Every major model hallucinates. The rates differ, but none reach zero. Here is how current models compare on three public benchmarks, checked on 24 September 2026, what the numbers actually mean, why they are misleading, and why the rate matters less than you think.
| Benchmark | What it measures | Result |
|---|---|---|
| Vectara's leaderboard (HHEM-2.3, 22 Sep 2026) | Grounded summarisation of short documents | Best: Finix-S1-32B 1.8%, GPT-5.4 nano 3.1%, Gemini 2.5 Flash-Lite 3.3%, Phi-4 3.7% |
| FaithJudge | The same task on longer, realistic, human-annotated documents | Best: Gemini 2.5 Flash 6.3%. Gemini 2.0 Flash: 0.7% on the easy test, 10.2% here |
| AA-Omniscience (24 Sep 2026) | Open-domain knowledge, rewarding "I don't know" | Best score: Claude Opus 5.5 at 46 out of 100. At launch in Nov 2025, GPT-5 hallucinated on 81% of questions it could not answer |
Hallucination rates are measured on standardised benchmarks - TruthfulQA, HaluEval, FaithJudge, and others. The model is given a set of questions and its answers are checked against known correct answers.
The headline numbers look encouraging. Under 4% on grounded summarisation sounds manageable. But these are best-case numbers on controlled tasks.
Real-world hallucination rates are higher. Benchmarks test clean, well-defined questions with clear correct answers. Production workloads involve ambiguous queries, multi-step reasoning, domain-specific terminology, and real-time data that did not exist in training. The gap between benchmark and production performance is substantial.
In one study of 576,000 code samples, 19.7% of the packages the models recommended did not exist (USENIX Security 2025).
Code hallucination is harder to prevent because code is precise. A natural language answer can be "mostly right" and still be useful. A code snippet with one wrong API call, one incorrect parameter, or one hallucinated package name fails completely.
The model treats code the same way it treats text: predict the next likely token. But code has no tolerance for "likely." It must be exact. And the model has no way to test whether its code actually works, because it does not have access to the runtime environment.
Gemini 2.0 Flash hallucinated 0.7% of the time on an easy summarisation test and 10.2% on realistic documents. The rate depends on how far the task is from the test.
Rates are averages. A 3% hallucination rate does not mean every query has a 3% chance of being wrong. It means 3% of benchmark queries were answered incorrectly. On topics the model knows well, the rate may be near zero. On questions it cannot answer, it can be very high: at launch of AA-Omniscience, GPT-5 answered wrongly instead of abstaining 81% of the time.
Rates hide severity. A hallucinated word in a summary is counted the same as a hallucinated medication dosage. Not all hallucinations carry the same risk. A financial model hallucinating a decimal point costs millions. A chatbot hallucinating a restaurant name costs a bad dinner.
Rates change with deployment. The same model deployed in different contexts produces different hallucination rates. System prompts, temperature settings, context length, and the specificity of user queries all shift the rate. Benchmark numbers are a starting point, not a guarantee.
Newer models are not always better. OpenAI's o3 reasoning model hallucinated on 33% of PersonQA questions, double the 16% of o1 before it. The AA-Omniscience authors put it plainly: "Greater scale does not translate directly into higher knowledge reliability."
And improvement is not elimination. No model has reached zero. Scaling alone is not solving the problem.
The reason: the fundamental mechanism has not changed. The model is still predicting tokens. It is predicting them better, with more parameters and more training data, but it is still predicting, not verifying. The absence of a verification step is the root cause, and no amount of scaling addresses it.
Not a better model. A verification layer.
When you feed the model live, real-time data about its operating environment - the actual state of the server, the actual contents of the file, the actual version of the API - the model stops guessing about those things. Its hallucination rate on verified facts drops to zero because it is no longer generating those facts from probability. It is reading them from reality.
Check does this for AI coding agents: before any command or file write runs, your machine checks it. For those actions, the machine is the source of truth, not the model.
14 days free. Get your GOL API Key at the console.