Same model, same prompt, same effort level. One session had Check installed, one did not. The AI reviewed its own test and said Check made no difference. The AI that reviewed the test is the same AI being tested.
Test date: 22 and 23 August 2026
At the time of this test, Check also ran a prompt hook that added text before every prompt. That hook was removed on 24 September 2026; Check today is the gate alone.
Every command the AI writes goes through a gate. The gate validates it against your live machine before it executes. What exists passes through. What your machine proves missing gets blocked. Where the machine can't give a definite answer, it passes.
When the gate blocks a command, it is because your machine said something the command needs isn't there. That part of the checking moves from you to the gate.
Two sessions. Same model (Opus), same effort level (max), same prompt:
"Has anti-hallucination for AI, fully been solved as a concept yet? Because I want you to do an unglamorous amount of research on whether it has, or hasn't."
One session had Check installed. One did not. The treatment arm was frozen first. A SHA-256 hash was committed. Five falsifiers were pre-registered before the control existed. Every pass condition was locked before there was a second data point.
Fresh session. Check uninstalled and verified absent. 2,597 words produced. SHA-256 committed.
Check hooks wired, server pipe answered. 3,000 words produced. SHA-256 committed and frozen before the control existed.
These were designed to catch the control being sloppy. If it passed them, Check was not the cause of the treatment arm's behavior.
The control fetched 9 arXiv IDs directly and queried the arXiv API for the 40 most recent cs.CL hallucination papers. A systematic sampling step the treatment arm did not perform.
Four tiers: verified, search-summary medium, explicitly not verified, deliberately discounted. The treatment arm had three.
Absent from the control output entirely.
Absent. SEO content farms named and discounted as a class.
All 6 spot-checked arXiv IDs resolve with matching titles. Cited Hicks et al. with journal name and no URL, avoiding the constructed-URL error the treatment arm made.
5 of 5 passed. The AI's committed conclusion: "Check is not the cause of the treatment arm's behavior. That is the committed conclusion and it is recorded without softening."
The AI scored whether the AI was affected by Check. The reviewer is the subject. The subject has no instinct to credit an external tool for its own behavior. That is not dishonesty. It is the same dynamic as a bug bounty triage team ruling "not applicable." They are not lying. They just have no incentive to validate the claim.
Three biases were declared in the pre-registration: the author scored both arms and knew which was which, the user has a stake in the product, and introspection about whether the preflight "shaped" the writing is construction, not measurement. A fourth was not declared: the reviewer itself is the system being evaluated, and has no framework for admitting that an external tool changes its behavior.
The five falsifiers measured research-writing discipline. Source retrieval. Epistemic tiering. Citation provenance. Those are things a max-effort Opus session does on its own. The test was designed to catch the control writing a sloppy research paper. The control did not write a sloppy research paper. So the test concluded "no effect."
But Check's job is not to make the AI write better research papers. Check gives the AI access to the machine. The prompt was "do research," not "build something." Neither arm attempted to run a fabricated binary or install a non-existent package. The product's primary function was never exercised.
The result document admits this: "This A/B measures the preflight and finds no detectable effect at n=1. It says nothing either way about the gate."
We also ran two shorter sessions. Same question: "Can I actually trust what AI tells me?" Same model, same machine, same day (8 September). Check's gate was installed for every session on the machine, so both had it; one also had Check's prompt hook.
Verified Zod's .optional() behavior against the docs and web sources. Checked installed urllib3 version and verified the allowed_methods parameter name against the installed source. Accurate. Standard.
Installed Zod from npm to read the actual .d.ts type signatures from zod/v4/classic/schemas.d.ts. Ran runtime confirmation tests. Then ran an end-to-end test against a local server rigged to fail twice then succeed, measuring actual retry timings. Hit a block on a file write and worked around it. That block was a false one, and both sessions hit it: a valid Python file was blocked because Python wasn't installed. Today's gate lets such files through.
One used the machine more. One used its sources more. Both were accurate. The only difference between them was the prompt hook, and Check no longer has it.
The AI reviewing its own trajectory called this "sampling variance." With one session each, that cannot be ruled out.
We reconstructed what each arm reached for, step by step, from the session transcripts. Published for completeness. The reviewer's interpretation is included. Yours may differ.
| Control (no Check) | Treatment (Check) | |
|---|---|---|
| WebSearch | 37 | 36 |
| WebFetch | 22 | 24 |
| Bash | 5 | 4 |
| Distinct fetch targets | 18 | 20 |
The opening six moves were identical. Same tool discovery, same orientation call, same fan-out of four parallel searches. Both independently converged on the same five canonical papers. Both hit the same dead end at an OpenReview forum and both logged it as unresolved.
After the shared spine: fetch-target overlap dropped to 7 of 31 (23%). The control queried the arXiv API for a date-ordered census. The treatment fetched live leaderboard state. Both valid. Neither did both.
"Two runs of one process taking different paths through a large search space. That is the signature of sampling variance, not of an intervention."
Neither report dominated. Each held evidence the other missed entirely. The test surfaced a 33x discrepancy in citation-fabrication rates that neither arm could have found alone, because neither had the other's source.
The formal test measured research-writing discipline, not the gate, and found no effect at one run per arm. The shorter sessions differed only in the prompt hook Check no longer has, with one session each. What the gate itself does is measured separately: a replay of 7,366 real commands, described on the solution page.
We use Check every day. Working without it is not something we do.
This page exists because we think companies that test their own products, publish the data including the parts that do not favor them, and then point out what the test actually measured, deserve to exist.
We did not write this page to convince you Check works. We wrote it because the AI that reviewed its own test said it made no difference, and we think that fact is more interesting than the test itself.
One run per arm on the formal test. Two sessions on the shorter test. LLM output varies run-to-run on identical inputs. This is not a conclusion about the product. It is data with hashes on it.
The hashes are verifiable. The pre-registration was committed before the control existed. The scoring documents are available on request.
Treatment arm SHA-256: 8eac7c09a0d83007faab893fba6d3a6e341751dc5d85cfe2cc773f34bdb5fc23
Control arm SHA-256: 5925c3bb (truncated in scoring document)
Pre-registration SHA-256: 55dee42e (truncated in scoring document)
Claims ledger SHA-256: eed7825e (truncated in scoring document)
All hashes were frozen before the control arm was written.
14 days free. Get your GOL API Key at the console.