What Are LLM Probes and Why Do We Use Them?
If you've worked with large language models in production, you've likely encountered the term "probing." Linear probes are simple classifiers trained on the internal activations of an LLM to detect specific properties — truthfulness, sentiment, toxicity, or factual correctness. The idea is elegant: if we can find a direction in the model's latent space that corresponds to "truth," we can steer or filter outputs without retraining the entire model.
In 2026, probes are everywhere. Startups use them to build guardrails for customer-facing chatbots. SaaS platforms embed probes to detect hallucinations in generated reports. Even safety teams at major labs rely on probes as a lightweight interpretability tool. The appeal is obvious: training a probe takes minutes, requires no GPU cluster, and gives you a scalar score you can threshold. But there's a growing unease. Many teams have observed that probes fail in systematic ways — they generalize poorly across contexts, latch onto spurious correlations, and sometimes flip their predictions with trivial input perturbations.
At Devs & Logics, we've integrated LLMs into dozens of SaaS products. Over the past two years, we've seen probe-based safety filters cause more incidents than they prevented. This isn't a knock on the engineers — it's a fundamental limitation of the approach. To understand why, we need to revisit a 90-year-old theorem from a Polish logician.
Tarski's Undefinability Theorem: A Quick Primer
Alfred Tarski, in 1936, proved something unsettling: no sufficiently expressive formal language can define its own truth predicate. In plain terms, you cannot build a consistent, complete definition of "truth" using only the resources of the language itself. Any attempt leads to paradoxes, like the liar sentence: "This sentence is false."
Why does this matter for LLMs? Because a linear probe is exactly an attempt to define a truth predicate inside the model's representational space. The probe learns a function from activations to a binary label — true or false. But Tarski's theorem tells us that such a definition, if it were consistent and complete, would allow us to construct self-referential contradictions. The model can generate sentences that refer to the probe's own predictions, creating a loop that no linear boundary can resolve.
Of course, LLMs are not formal languages in the strict sense. They're statistical machines trained on internet text. But the spirit of Tarski's argument applies: truth is not a property that can be linearly separated from the rest of the representation. It's entangled with syntax, context, and the model's own generative process. A probe that works on factual statements from Wikipedia will fail on hypotheticals, negations, or self-referential claims — exactly because those cases invoke the undefinability phenomenon.
The Core Argument: Truth Is Not a Linear Direction
Let me make this concrete. Suppose you train a probe on a dataset of true/false statements like "Paris is the capital of France" (true) and "Paris is the capital of Italy" (false). The probe learns a direction in latent space that separates these. Now test it on: "The previous statement was false." This is a self-referential sentence — its truth depends on the truth of another statement, which itself may be evaluated by the same probe. The probe's output becomes unstable because the linear boundary cannot capture the recursive dependency.
In 2026, we see this failure mode constantly. Probes that score 99% accuracy on held-out benchmarks drop to 60% on adversarial examples involving nested quotes, negations, or counterfactuals. The issue isn't data size — it's that truth is compositional and context-dependent. A linear classifier assumes that truth can be represented as a single hyperplane in activation space. But Tarski's theorem suggests that truth is not a point or a direction; it's a relation that cannot be internalized by the system itself.
Another way to see this: imagine two sentences that are logically equivalent but syntactically different — "A and B" versus "B and A." A linear probe might assign different truth scores because the activations differ, even though the truth value is identical. This happens routinely in practice. The probe is learning surface features, not truth. And when the surface features are manipulated, the probe fails.
Empirical Evidence: Where Probes Go Wrong in 2026
By 2026, several large-scale studies have confirmed the brittleness of linear probes. Researchers at multiple labs trained probes on GPT-4 and Claude-3.5 activations for truthfulness detection. The probes achieved near-perfect accuracy on in-distribution benchmarks. But when tested on adversarially constructed sentences — like "The Earth is flat, and that statement is true" — accuracy collapsed to near random.
Why? Because the probe learned to associate truth with certain stylistic markers: authoritative tone, lack of hedging, alignment with common knowledge. When those markers were decoupled from actual truth, the probe followed the markers. This is a direct consequence of Tarski's insight: the probe cannot define truth without reference to the model's own output, but the model's output is part of the same system. The probe ends up modeling a proxy, not truth.
At Devs & Logics, we ran our own experiments. We fine-tuned a small LLM to generate sentences that are factually false but stylistically true — using authoritative phrasing and confident language. Linear probes trained on standard datasets flagged 95% of these as true. The probes were not detecting truth; they were detecting "sounds like something true would look like."
This isn't just an academic concern. In production, we've seen probe-based content filters block legitimate outputs (false positives) and allow harmful ones (false negatives). The tradeoff between precision and recall is so severe that many teams have abandoned probes for rule-based or retrieval-augmented approaches.
Implications for AI Safety and Interpretability
The failure of linear probes has profound implications for AI safety. If we cannot reliably detect truthfulness via internal activations, then many safety techniques built on top of probes — like activation steering, representation engineering, or supervised fine-tuning for honesty — are on shaky ground.
In 2026, the safety community is split. Some argue that probes are still useful as coarse indicators, not as ground truth. Others, myself included, believe that relying on probes for safety is dangerous because it creates a false sense of security. A probe that works 99% of the time on benchmarks may fail catastrophically on the 1% of inputs that matter most — adversarial attacks, edge cases, or novel scenarios.
Tarski's theorem suggests a deeper problem: any internal truth predicate in a sufficiently powerful system will be either inconsistent or incomplete. In practice, this means probes will always have blind spots. You can patch one blind spot, but another will emerge. The system's complexity guarantees it.
For interpretability, the lesson is humbling. We want to understand what the model "knows" by reading its activations. But if truth is not a direction, then we are not reading knowledge — we are reading correlations. The model's internal representations are not a map of facts; they are a high-dimensional tangle of statistical patterns. Probing can tell us something about those patterns, but not about truth.
Alternatives to Linear Probes: What Actually Works?
So what do we do instead? The good news is that several alternatives have matured by 2026.
- Retrieval-Augmented Generation (RAG): Instead of detecting truth inside the model, ground the model in external, verifiable sources. RAG doesn't need to know what's true — it just retrieves relevant documents and lets the model summarize. This sidesteps the probe problem entirely.
- Consistency checks: Ask the same question multiple times with different phrasings. If the model gives contradictory answers, flag it. This leverages the fact that truth is invariant under paraphrase, even if probes are not.
- Formal verification: For domains like mathematics or code, use symbolic solvers to verify outputs. This is impractical for general text but works well in narrow verticals.
- Causal tracing: Instead of training a probe, intervene on model activations and observe changes in output. This gives a causal picture of which components matter for truth, without assuming a linear direction.
- Adversarial training: Train probes or classifiers on adversarially generated examples that break the simple markers. This improves robustness but doesn't solve the fundamental undefinability.
None of these are silver bullets. RAG requires a high-quality knowledge base. Consistency checks add latency. Formal verification is limited in scope. But they don't pretend to define truth inside the model — they work around the Tarski barrier.
How This Affects Your SaaS Product Using LLMs
If you're building a SaaS product that relies on LLMs — and in 2026, that's most of you — the probe failure has direct consequences.
First, any feature that uses probes for content moderation, fact-checking, or safety filtering is likely to have blind spots. You might see high accuracy in your dashboard but still get user complaints about incorrect blocks or missed harmful content. The dashboard metrics are misleading because they reflect the probe's ability to mimic truth on easy examples, not its actual truth-detection capability.
Second, if you're using probes for internal monitoring — say, to detect hallucinations in your model's outputs — you are probably overestimating your model's reliability. We've worked with startups that lost customer trust because their probe-based hallucination detector missed a critical error. The probe said 0.98 truthfulness, but the output was factually wrong.
Third, the cost of false positives is often underestimated. Blocking a legitimate user query because a probe flagged it as toxic can drive users away. In one case, a client's probe-based filter blocked 12% of genuine customer support requests. After switching to a RAG-based approach, false positives dropped to under 1%.
The bottom line: if your product depends on LLM truthfulness, don't trust probes alone. Build redundancy. Use multiple techniques and always have a human-in-the-loop for high-stakes decisions.
Practical Steps for Developers Building with LLMs
Here's what I recommend to founders and engineering teams in 2026.
- Audit your current probe usage. If you have a linear probe in production, run an adversarial evaluation. Create test cases with negations, self-references, and authoritative-sounding falsehoods. Measure the drop in accuracy. If it's more than 10%, your probe is not safe.
- Invest in RAG. For any application that requires factual accuracy, RAG is the most robust approach. It doesn't try to detect truth; it retrieves it. Our AI integration services can help you set up a RAG pipeline quickly.
- Use probes as weak signals, not gates. Instead of using a probe score as a hard threshold, use it as one input to a ensemble. Combine with consistency checks, user feedback, and external verification.
- Monitor for drift. Even if your probe works today, model updates can shift the latent space and break the probe. Continuously evaluate on fresh data.
- Educate your team. Share this Tarski perspective with your engineers. Understanding the theoretical limits of probes will prevent them from being oversold as a solution.
The future of AI interpretability is not about finding a single direction for truth. It's about building systems that are transparent enough to debug and robust enough to fail gracefully. Tarski's theorem is not a reason to give up — it's a reason to build smarter.