A well-calibrated system is confident when it is right and uncertain when it might be wrong. AI language models are not calibrated. They sound equally sure whether they are stating a verified fact or generating a complete fabrication. This makes hallucinations invisible to the user.
In probability theory, a system is calibrated when its confidence matches its accuracy. If a system says it is 90% confident in something, it should be right about 90% of the time. If it says 50%, it should be right about half the time.
A well-calibrated doctor says "I'm quite sure this is X" when it is probably X, and says "I'm not sure - let me run some tests" when they genuinely are not sure. Their confidence tracks their accuracy.
AI language models fail this test comprehensively. Their expressed confidence - the tone, the hedging, the certainty of their phrasing - has weak correlation with their actual accuracy. A hallucinated fact and a correct fact are delivered in the same authoritative voice.
Token probabilities are not truth probabilities. The model does produce probability distributions over tokens. But these represent "what token is most likely to come next given this context" - not "how likely is this statement to be true." A confidently wrong statement can have very high token probabilities because the patterns in training data support that token sequence.
RLHF rewards confidence. During reinforcement learning from human feedback, humans consistently prefer confident responses. Hedged answers, caveats, and expressions of uncertainty are penalized in the preference data. The model learns that sounding sure gets higher reward, regardless of whether it is sure.
The model has no self-knowledge. A human who is uncertain often knows they are uncertain - they can feel the gap in their knowledge. The model has no such introspective capacity. It does not know what it does not know. It cannot assess its own knowledge boundaries because it has no mechanism to do so.
The AI never invents - it says what it thinks is right. The problem is it thinks everything it says is right, because it has no mechanism to think otherwise.
If the AI said "I'm making this up" every time it hallucinated, hallucination would be a minor nuisance. You would know which answers to trust and which to verify.
But because of miscalibration, every answer looks the same. The fabricated medication dosage sounds as authoritative as the correct one. The invented case citation is presented with the same certainty as a real one. The hallucinated earnings figure carries the same confident tone as accurate financial data.
The user cannot tell which is which. There is no signal in the AI's output that distinguishes truth from fabrication. This is not a failure of the user's judgment. It is a failure of the system's design. The system provides no basis for the user to distinguish accurate from hallucinated output.
Training-based approaches. Some researchers are working on training models to express appropriate uncertainty - to say "I'm not sure" when they should. This helps but is limited. The model still cannot verify its own outputs. It can learn when it is statistically likely to be wrong, but it cannot check whether it actually is wrong on any specific answer.
Logprob-based detection. Monitoring the model's internal token probabilities can flag some hallucinations - low-confidence token sequences may indicate fabrication. But this catches only some hallucinations. High-confidence hallucinations - where the model's training strongly supports a wrong answer - produce high logprobs and are invisible to this method.
External verification. The only reliable approach is to not rely on the model's self-assessment at all. Instead, verify the output against external reality. This does not require the model to know whether it is right. It checks whether it is right, from the outside. This is what Check does - verification that does not depend on the model's own confidence.
120 verifications a day free. No card, no signup.