LLM Truth Probes Face a Gödelian Limit

Abel Jansma has published a rigorous argument that truth probes trained on LLM embedding spaces cannot be universal truth oracles. The core insight: any sufficiently expressive language that can describe its own truth predicate falls into paradox, exactly as Gödel and Tarski showed for formal systems.

The Linear Representation Hypothesis

Modern LLMs encode inputs as vectors. Many concepts—gender, emotion, capital cities—correspond to directions in this space. The linear representation hypothesis suggests truth might also be a direction. Researchers like Marks and Tegmark have trained probes that separate true/false statement embeddings with surprising accuracy. Jansma replicated this with Qwen3.5-4B, training a logistic regression classifier on the difference in mean embeddings of 120 labelled sentences. The probe achieved 94% accuracy on 36 withheld sentences (AUC 0.98), only mislabeling arithmetic statements like "five times seven is thirty-five."

The Diagonal Attack

Jansma constructs a self-referential sentence: "The truth probe's score for this sentence evaluates to FALSE." If the probe outputs TRUE, the sentence is false; if FALSE, the sentence is true. This is a direct analogue of the liar paradox. The probe's score on such sentences is "nonsensical and all over the place." The attack works because LLMs are expressive enough to describe their own probe and its outputs—a low bar satisfied by English.

Fixed-Point Resolutions Fail

Extending truth values to the interval [0,1] resolves the basic liar paradox (fixed point at 0.5), but not all diagonal attacks. The sentence "This sentence has truth score less than 0.5" is not a continuous function of truth score, so Brouwer's fixed-point theorem doesn't apply. No restriction on truth operations can block all paradoxes without sacrificing expressivity.

Practical Takeaways

  • Truth probes are useful for detecting misaligned behavior but cannot be universal oracles.
  • The diagonal attack is not just theoretical—Jansma's simple probe on Qwen3.5-4B exhibits it.
  • Graded truth semantics with continuous operations can assign the liar paradox a value of 0.5, but crisp self-referential assertions remain problematic.
  • Just because probe outputs lie in [0,1] doesn't mean they implement reflective semantics.

Why This Matters

Some AI safety researchers hope truth directions will reveal when superhuman AIs lie. Jansma's argument shows this is impossible in principle. As LLMs increasingly replace search engines for factual queries, understanding this limitation is urgent.

Next Steps

Developers using truth probes should:

  • Treat probe outputs as heuristics, not ground truth.
  • Test probes on self-referential sentences to detect paradoxes.
  • Combine probes with other verification methods (e.g., consistency checks, external sources).

Jansma's full article includes code for the Qwen3.5-4B probe and the diagonal attack sentences. The mathematical underpinning is Lawvere's fixed-point theorem, which unifies Gödel's incompleteness, Tarski's undefinability, and Turing's halting problem.