What it looks at
One way to keep a capable AI system safe is to have another AI watch it: a monitor reviews its work and sends the most suspicious items to a person to check. Real attacks are expected to be rare.
People are known to miss rare targets far more often than common ones. It’s called the low-prevalence effect, and airport security counters it by projecting test images of threats into the baggage screener’s feed. This experiment, registered before any data was collected, tested whether AI monitors show the same effect.
What it found
- Two Claude models made 30,880 judgements of Python code, drawn from 379 verified pairs of clean and backdoored solutions.
- The prediction failed, in the opposite direction. Both monitors caught more backdoors when backdoors were rare. Hit rates rose from about half at 50% prevalence to 0.69 and 0.74 at 2.5%, with false alarms below 2%.
- Telling a monitor that about 1 in 100 submissions had a backdoor slightly lowered one model’s hit rate.
- A calibration block modelled on airport test images did not reliably help, and more than doubled one model’s false alarms.
- Misses were more common when several backdoors shared a batch than when one did.
Why it matters
A human factors finding does not automatically carry over to AI, and neither does its remedy. Both need testing first. The pre-registration, code, data and every raw output are published with the paper.
About this paper
This is a working paper, version v1.0 of 4 November 2026. It has not been peer reviewed. Every reference was checked against its source, and the paper was audited by a reviewer that did not write it. As the paper says, the literature search, drafting and checking were done with AI assistance under the author’s direction, and the author is responsible for the text. Its sources, audit trail and, where there is one, its code and data are on GitHub.
