AI hallucination experiment — Context Gate
Tests whether forcing the source of truth into an AI's context reduces hallucination — interactive demo + report.
AI agents often hallucinate before they answer — they skip the file that holds the rule, then fill the blank with something plausible. This experiment tested a fix: a Context Gate that forces the source-of-truth file into the model's context before it acts.
Across three studies and two models, with no gate the model drifted on 90–100% of scenes — and usually never opened the rule file at all. Forcing the rule in cut that to near zero on DeepSeek, and sharply reduced it on Opus (which has a separate malformed-output ceiling the gate can't fix).
How it was tested
The model writes an 18-scene story that must cast only an approved 5-character roster — with the roster buried among 18 plausible rule files. Three arms: No gate, Rule pasted in, Context Gate. 20 runs per arm, scored deterministically. Run under three pressure conditions: unlimited file access, one file only, and a rushed deadline.
What it means
Context Gates don't make a model hallucination-proof. They remove one specific, common cause: answering before the source of truth is in context. The one-use Context Receipt is for enforcement and audit — proof the gate ran — not added intelligence.
Read / try it
Interactive demo · Full report · Harness + raw runs on GitHub
stack
← select a file to view