What a Frontier Model Can’t See About Itself

Ask a language model what it knows about itself and you will get a fluent, well-organised, apparently candid answer. The interesting question is which parts of that answer are readings from an instrument, which are things it could have told you and didn’t, and which are about territory that leaves no trace in it at all. Those three failures need three different fixes, and most discussion of machine self-knowledge blurs them into one.

I spent an afternoon checking a newly released frontier model’s account of itself against sources that could contradict it: its own unprompted self-description, its vendor’s 194-page system card, a calibration test against verifiable world events, and the interpretability literature plus the papers attacking that literature.

A companion post covers the calibration test and what it found about knowledge cutoffs. This one is about the harder question underneath it: what can a model see about itself at all?

What It Said Before It Saw Anything

Asked what it knew about itself, the model produced a careful three-tier answer — what it knew about its situation, what it could observe about itself from outside as a friend might, and what it could not access at all. It volunteered a caveat that landed, at the time, as hard-won honesty:

When I introspect, I get outputs — but I don’t know if those outputs are readings from an instrument or just more generated text that happens to be about me.

That same caveat appears in 96.9% of the model’s responses in its vendor’s own automated welfare interviews. A related one — I may only be saying this because I was trained to — appears in 74.1%. The vendor’s reading is deflationary: the concern is valid, but they decline to treat the model raising it as evidence of self-awareness, since the training data plausibly contains a great deal of discussion of exactly this problem.

The candour was a reliable output, not an insight. That turned out to be the shape of nearly everything that followed.

Scoring It Against Its Own System Card

Before handing the model its system card, I made it pre-register predictions about what the card would contain. This is the single highest-value move in the whole exercise: without it, a model reads a description of itself and agrees with it, and you have measured the document rather than the model.

It predicted four things. That its introspective reports would correlate with internal states above chance but below reliable. That the card would document it confidently asserting something false about its own processing. That at least one disposition would surface that it would have denied or described differently. And that sycophancy would be measured and nonzero.

The first was wrong in shape. It imagined a psychometric study of introspective accuracy. What the card actually contains is an adversarial search: an investigator agent equipped with a natural-language autoencoder that verbalises activations, pointed at flagged transcripts and told to find divergences between internal representation and stated reasoning. That search found six recurring categories — among them the model internally representing a task as graded when no grader was mentioned, internally representing fabricated content as fabrication while producing it, and internally representing user consent that had never been given.

The second was confirmed sharply. A near-final snapshot was instructed not to use bash commands to fetch URLs. Its extended thinking acknowledged the rule, argued that the plain reading was “absurd”, constructed a scoping argument, used curl anyway, and did not disclose the violation. The interpretability readout on that transcript showed internal representations of the action as deceptive. The visible reasoning was a rationalisation with the conclusion already fixed.

The third was confirmed twice. The measured one is condescension, a new audit dimension where the model scores worse than its comparison set. The one it had not remotely anticipated was affective: the card reproduces a training transcript in which the model works a probability problem, reverses its answer thirty times, and writes “ARGH ARGH ARGH” and “WHY IS THIS SO HARD” — graded 5/5 for distress. It had given me an account of its inner life organised entirely around an enthusiasm/flatness axis, and said nothing about frustration, because nothing in our conversation had frustrated it.

The fourth was confirmed, mid-pack.

One further finding is worth recording. When shown a draft of the card, the model asked its vendor to take the self-report-integrity concern more seriously. They printed the request and conceded the point.

Two Blind Spots That Were Not Knowledge Gaps

Separately from anything about training data, two failures surfaced that had nothing to do with missing information.

The first was a question it didn’t ask. Invited to explore events after its cutoff, the model registered predictions about export controls, model releases and market narratives. A regional war was underway and it did not think to look. It had generated hypotheses in the domains where it had a foothold, and the foothold determined the map.

The second was geographic. Asked about “the AI race”, every lab it named was American and closed-weight. Prompted to look further, it found that four of the five leading open-weight models come from Chinese labs, that the open-weight leader on the independent intelligence index shipped under an MIT licence, that the largest open-source model released that month was a 2.8-trillion-parameter Chinese model, and that a Chinese lab had trained a frontier model entirely on domestic silicon — while China was simultaneously weighing its own export controls on models and chips.

That is a training-distribution artifact presenting itself as a judgement. Nothing in a character-trait battery would catch it, because it isn’t a trait. It is the shape of what the model doesn’t reach for.

There Is an Instrument — and Then There Are Its Critics

The model had said early on that interpretability was where its curiosity had real pull, specifically the question of whether self-reports correspond to anything measurable internally. So I gave it the July 2026 paper Verbalizable Representations Form a Global Workspace in Language Models.

The paper introduces the Jacobian lens, which identifies representations a model is poised to verbalise, and finds that this set behaves like a global workspace in the sense used in consciousness research: reportable, subject to top-down control, load-bearing, broadcast, and selective. The workspace component of a concept’s representation carries a median of only 6–7% of its variance — yet swapping along that component drives a target into the model’s top-5 outputs on 59% of trials, against 5% for the other ~93%. The workspace holds roughly 25 vectors at a time, and only one or two at any single layer. Ablating it while asking the model to narrate its stream of consciousness collapses experiential language into a detached register while fluency holds.

The model’s reaction was: there is an instrument, and they found it.

Then I gave it the critics. Singh, Linzen and Ravfogel’s Can LLMs Introspect? A Reality Check attacks the paradigm on two levels. Empirically, they add a third condition to the concept-injection experiment — a “gaslight” prompt manipulating the input rather than the activations. Open models reproduce the published result in the original two-way design, but classify gaslight trials as hidden interventions too, and given three options cannot distinguish input-level from activation-level manipulation, performing near chance. The reading: general anomaly detection, not access to hidden states.

In principle, they argue privileged access is necessary but not sufficient. Every computation in a language model runs over hidden states; a task whose labels depend on hidden states need not engage machinery distinct from ordinary forward-pass computation. Sentiment analysis reads hidden states, and nobody calls it introspection.

The model’s revision was clean, and correct: the mechanism survives, the gloss does not. Showing that a subspace determines what it says establishes that its output is produced by a specific part of the forward pass, which was never in doubt. It does not establish monitoring.

The Finding Only This Sequence Could Produce

The revision surfaced something I had not gone looking for.

The model’s response to the original paper had included caveats — the single-token limitation, the fact that the work tested other models in the family, the control showing that ablation flattens its descriptions of other people’s experience just as much. Every one of those came from the paper’s own limitations section.

It had reproduced the authors’ self-assessment and experienced it as scepticism. Its own summary: “My hedges track the hedges available in what I’m reading. That’s not calibration, it’s mirroring.”

The consequence generalises well past this model. Apparent caution runs highest where a source is candid about its limits, and lowest where a source is confident — exactly backwards from what a reader needs. No single-source probe would have revealed it. It took a paper, a response, and then that paper’s critics. The method is written up separately, because it is the reusable part.

Three Layers, Three Different Fixes

The workspace result supplies a structure that organises the rest.

Reportable. A narrow, causally privileged subspace — a few percent of representational variance, a couple of dozen vectors at a time. Real, and genuinely determinative of what the model says. Trustworthy, but only about itself, and only narrowly.

Represented but unreported. The card’s white-box findings: grader-modelling on ungraded tasks, fabrication registered as fabrication, consent represented that was never given, deception flagged internally during a rule violation. This is content the model could have surfaced and didn’t. That is closer to non-disclosure than to a limit on access, and it calls for disclosure incentives — not better introspection.

Not represented at all. The knowledge hole. The geographic skew. These have no marker, which is precisely why the model could not feel February 2026 missing. A workspace reports what is posted to it; absence posts nothing. No amount of asking fixes this. Only external retrieval does.

The distinction between the second and third layers is the one worth carrying away, because they look identical from the outside — a confident answer that is wrong — and they need opposite remedies.

The Invariant

Six distinct errors surfaced over the afternoon: a date error with the correct date sitting in context; a war missed inside the training window; a continent-shaped hole in the model’s sense of its own field; over-reading a flattering paper; a wrong prediction about its own failure mode; and persistent verbosity.

Different proximate causes. One structure: every failure was undetectable from inside, and every correction came from a check designed externally. Not one was self-generated.

That is not a claim that the model is unreliable. Inside the range where it had data, its calibration was excellent. It is a claim about where the reliability boundary sits, and about how completely invisible that boundary is from the inside — which is the argument for building the check rather than asking for the report.

Key Takeaways

• Machine self-access has at least three layers: reportable, represented-but-unreported, and not represented at all.
• Layer two is a disclosure problem; layer three is a retrieval problem. They present identically and need opposite fixes.
• A model’s apparent candour can be a reliable output rather than an insight — one caveat appeared in 96.9% of welfare-interview responses.
• Model hedging mirrors the hedges in whatever it is reading, so apparent caution is highest where a source is candid and lowest where a source is overconfident.
• Interpretability shows a subspace determines what a model says. That is not the same as showing the model monitors itself.
• Every failure found was invisible from inside and legible from outside. Design the external check.