The EU AI Act is the first comprehensive AI regulation. It requires high-risk AI systems to be accurate, robust, and transparent. AI hallucination - confidently generating false information - violates all three requirements. Breaching the high-risk obligations can be fined up to 15 million EUR or 3% of global revenue, whichever is higher. The 35 million EUR / 7% tier applies only to prohibited AI practices, and lower caps apply to SMEs and small mid-caps. The obligations for high-risk AI systems apply from 2 December 2027 for the uses listed in Annex III, and from 2 August 2028 for AI in products covered by Annex I (Article 113, as amended in 2026).
Accuracy (Article 15). High-risk AI systems must "achieve an appropriate level of accuracy", and the levels and metrics must be declared in the instructions for use. A system that hallucinates - generating false information and presenting it as true - fails this requirement by definition.
Robustness (Article 15). Systems must be resilient to errors and inconsistencies. Hallucination is the opposite of robustness - it is the system generating errors as a normal part of operation, not as an exceptional failure.
Transparency (Article 13). Deployers must be able "to interpret a system's output and use it appropriately". When a system generates hallucinated content with the same confidence as accurate content, deployers cannot distinguish reliable output from fabricated output. Transparency is undermined.
Human oversight (Article 14). The Act requires that humans can effectively oversee the system. But hallucinations are designed to look correct - the model generates them with full confidence and proper formatting. Human oversight of a system that produces indistinguishable true and false output is not meaningful oversight.
The EU AI Act does not mention "hallucination" by name. It does not need to. Every hallucination violates at least one of its core requirements.
The Act classifies AI systems by risk level. High-risk systems face the strictest requirements. These include AI used in:
Healthcare. AI that is, or is a safety component of, a medical device under the EU medical device rules (Annex I), AI that decides eligibility for public healthcare services, and emergency patient triage (Annex III). A hallucinated dosage or fabricated guideline in these contexts is both an accuracy failure and a safety risk.
Legal. AI used by or on behalf of a judicial authority to research and interpret facts and the law, or used the same way in alternative dispute resolution (Annex III, point 8). Legal research tools used by law firms are not on the list. Fabricated citations and misrepresented holdings are accuracy failures with professional consequences.
Financial services. Evaluating creditworthiness or setting credit scores for individuals, and risk assessment and pricing for life and health insurance (Annex III, point 5). AI used to detect financial fraud is explicitly excluded. Hallucinated financial data or fabricated regulatory requirements create compliance and market risk.
Employment. Recruitment tools, performance evaluation, workforce management AI. Hallucinated assessments or fabricated candidate information affect hiring decisions.
Critical infrastructure. AI used as a safety component in managing critical digital infrastructure, road traffic, or the supply of water, gas, heating or electricity (Annex III, point 2). Hallucinated commands or fabricated system states can cause outages or security breaches.
Accuracy metrics. Providers must declare their system's levels of accuracy and the metrics used (Article 15(3)). The Act does not name hallucination rates. For a system that can hallucinate, a measured rate for your use case, with your data, is a direct way to show accuracy. "We use GPT-4" is not a compliance statement.
Risk management. Hallucination risk must be identified, assessed, and mitigated as part of the required risk management system. This means acknowledging that the model can generate false output and implementing controls to catch it.
Verification mechanisms. The most direct path to compliance is implementing a verification layer that validates AI output before it reaches users. This addresses accuracy (verified output is accurate), robustness (verification catches errors), and transparency (users can distinguish verified from unverified claims).
Documentation. The technical documentation (Annex IV) must describe the risk management system, the expected level of accuracy, and known limitations in performance. For a system that can hallucinate, that risk belongs in it: what mitigation is in place, how effective it is, and what residual risk remains.
The Act does not prescribe specific technical solutions. It sets requirements and leaves implementation to providers. But the requirements point clearly toward verification.
A verification layer that checks AI output against reality before delivery directly addresses the Act's accuracy requirement. It provides measurable, documented accuracy improvement. It creates an audit trail showing which outputs were verified and which were not.
Check provides this for AI coding agents. It checks every command and file write against the machine before it runs. The same verification architecture applies to any high-risk domain - the principle is the same, only the data source changes.
14 days free. Get your GOL API Key at the console.