Every major model hallucinates. The rates differ, but none reach zero. Here is a benchmark comparison across GPT-4o, Claude, Gemini, Llama, and Mistral - what the numbers actually mean, why they are misleading, and why the rate matters less than you think.
| Model | General hallucination rate | Factual QA rate | Code hallucination rate |
|---|---|---|---|
| GPT-4o | 2.5 – 5% | 3 – 8% | 15 – 30% |
| Claude 3.5 Sonnet | 2 – 4% | 2 – 6% | 12 – 25% |
| Gemini 1.5 Pro | 3 – 6% | 4 – 9% | 18 – 35% |
| Llama 3 70B | 5 – 10% | 6 – 14% | 20 – 40% |
| Mistral Large | 4 – 8% | 5 – 12% | 18 – 38% |
| GPT-3.5 Turbo | 8 – 15% | 10 – 20% | 25 – 50% |
Hallucination rates are measured on standardised benchmarks - TruthfulQA, HaluEval, FaithJudge, and others. The model is given a set of questions and its answers are checked against known correct answers.
The headline numbers look encouraging. GPT-4o at 2.5-5% hallucination sounds manageable. But these are best-case numbers on controlled tasks.
Real-world hallucination rates are higher. Benchmarks test clean, well-defined questions with clear correct answers. Production workloads involve ambiguous queries, multi-step reasoning, domain-specific terminology, and real-time data that did not exist in training. The gap between benchmark and production performance is substantial.
Notice the code column. Even GPT-4o hallucinates at 15-30% on code generation tasks. Claude at 12-25%. Open-source models at 20-50%.
Code hallucination is harder to prevent because code is precise. A natural language answer can be "mostly right" and still be useful. A code snippet with one wrong API call, one incorrect parameter, or one hallucinated package name fails completely.
The model treats code the same way it treats text: predict the next likely token. But code has no tolerance for "likely." It must be exact. And the model has no way to test whether its code actually works, because it does not have access to the runtime environment.
A 3% hallucination rate on a benchmark becomes a 30% hallucination rate in a domain the model was not trained on. The rate depends on the gap between training data and reality.
Rates are averages. A 3% hallucination rate does not mean every query has a 3% chance of being wrong. It means 3% of benchmark queries were answered incorrectly. On topics the model knows well, the rate may be near zero. On topics outside its training data, the rate may be 50% or higher.
Rates hide severity. A hallucinated word in a summary is counted the same as a hallucinated medication dosage. Not all hallucinations carry the same risk. A financial model hallucinating a decimal point costs millions. A chatbot hallucinating a restaurant name costs a bad dinner.
Rates change with deployment. The same model deployed in different contexts produces different hallucination rates. System prompts, temperature settings, context length, and the specificity of user queries all shift the rate. Benchmark numbers are a starting point, not a guarantee.
The trend is clear: larger, newer models hallucinate less. GPT-4 hallucinates less than GPT-3.5. Claude 3.5 hallucinates less than Claude 2. Each generation improves.
But improvement is not elimination. No model has reached zero. And the rate improvements are flattening. The jump from GPT-3.5 to GPT-4 was dramatic. The jump from GPT-4 to GPT-4o was incremental. Scaling alone is not solving the problem.
The reason: the fundamental mechanism has not changed. The model is still predicting tokens. It is predicting them better, with more parameters and more training data, but it is still predicting, not verifying. The absence of a verification step is the root cause, and no amount of scaling addresses it.
Not a better model. A verification layer.
When you feed the model live, real-time data about its operating environment - the actual state of the server, the actual contents of the file, the actual version of the API - the model stops guessing about those things. Its hallucination rate on verified facts drops to zero because it is no longer generating those facts from probability. It is reading them from reality.
Check does exactly this. It reads the machine state, injects it into the model's context, and validates every output. The model's benchmark rate becomes irrelevant because the model is no longer the source of truth. Reality is.
120 verifications a day free. No card, no signup.