The industry measures hallucination with benchmarks - standardised tests that score how often models fabricate. These benchmarks are useful for comparing models. They are not useful for predicting whether a model will hallucinate on your specific task, with your specific data, in your specific environment.
Tests whether models generate truthful answers to questions that humans commonly answer incorrectly. Contains 817 questions across 38 categories designed to trigger common misconceptions. Measures: whether the model avoids repeating popular falsehoods. Limitation: tests common misconceptions, not domain-specific fabrication or novel hallucination.
A large-scale benchmark specifically for hallucination evaluation. Contains 35,000 samples across QA, dialogue, and summarization tasks with both hallucinated and correct examples. Measures: whether the model can identify hallucinated content. Limitation: tests detection, not generation - the model identifies others' hallucinations, not its own.
Fine-grained Atomic fact scoring. Breaks model-generated biographies into individual atomic facts and verifies each against Wikipedia. Measures: factual precision at the claim level. Limitation: limited to Wikipedia-verifiable facts about people. Does not cover real-time data, proprietary information, or domain-specific knowledge.
Stanford's comprehensive evaluation framework that includes accuracy, calibration, robustness, and fairness metrics across many tasks. Measures: broad model capability including but not limited to hallucination. Limitation: general-purpose evaluation - hallucination is one metric among many, not deeply probed.
A model that scores 95% on TruthfulQA still hallucinates. The benchmark measures average performance across known questions. Your production queries are not benchmark questions.
Your specific domain. Benchmarks test general knowledge. Your AI operates in a specific domain - healthcare, legal, finance, code. A model that performs well on general benchmarks can hallucinate heavily in your domain if its training data was thin in that area.
Real-time information. Benchmarks test static knowledge. Production systems need current information. No benchmark tests whether the model hallucinates about events after its training cutoff - which is exactly where production hallucination is most dangerous.
Multi-turn degradation. Benchmarks are mostly single-turn. Production conversations are multi-turn. Sycophancy and cascading hallucination compound over turns. A model that is accurate on turn 1 may hallucinate by turn 10.
Action-level accuracy. Benchmarks measure text accuracy. AI agents need action accuracy - are the commands valid, are the parameters correct, is the target in the right state? No major benchmark tests this.
Edge cases. Benchmarks are, by definition, the cases researchers thought to test. Production hallucination happens on the cases nobody anticipated - unusual queries, edge-case inputs, combinations of conditions that were never in any benchmark.
Benchmarks are useful for model selection. A model that scores well on TruthfulQA and FActScore is likely to hallucinate less than one that scores poorly. Use benchmarks to narrow your choices.
But do not use benchmark scores as a substitute for production verification. A 95% accuracy score means 5% fabrication. At scale - thousands of queries per day - that 5% is hundreds of hallucinated responses reaching users.
Real-time verification provides the protection that benchmarks cannot. It does not measure average accuracy across a test set. It verifies each specific output against each specific reality, in real time. Check does not care about benchmark scores - it checks whether this specific output is correct for this specific environment right now.
120 verifications a day free. No card, no signup.