How we know
Every number we show comes from a script that reads the saved data. The plan for that script was written down before anyone took the test.
The plan
Analysis plan tags: prereg-v1 for the test, prereg-v2 for part 2, prereg-v3 for the second wave. Each plan was tagged before its first participant and names the outcome, the test, the exclusions and the sample size.
What a participant does
- Reads the consent text and ticks two boxes.
- Picks the healthier of two creeks (a warm-up, not scored).
- Is assigned by the server to see the lesson first or the test first.
- Answers Yes, No or Can't tell on sixteen photos, four per feature.
- Sees their score per feature, and may keep it with a random token.
- After the score, may take part 2: eight more photos, one question each. Half of them, at random, meet the checker's one question, "The checker noticed something here. Look again?", whenever a flag the gate kept disagrees with their answer. They keep or change it, and the final answer is their own.
How numbers reach the README
Scripts in evals/ write files in results/. A check in CI compares every number in the README with those files. Nobody types a number by hand.
AI on the same test
Vision models take the same sixteen photos. A model may only raise a question about a feature it passed, and the volunteer always answers first.
Which features each model passed
We gave the same test to 4 vision models. A model passes a feature when it gets every photo of that feature right in most of its runs.
Passed: Built banks, Pipes and drain outlets
Did not pass: Dug-out channel, Plants that do not belong
Passed: Built banks, Dug-out channel
Did not pass: Plants that do not belong, Pipes and drain outlets
Passed: Built banks, Dug-out channel, Pipes and drain outlets
Did not pass: Plants that do not belong
Passed: Built banks, Dug-out channel
Did not pass: Plants that do not belong, Pipes and drain outlets
From results/model_pass_table.json, run on Sep 24, 2026.
The gate on real creek footage
The models looked at 46 frames from 5 openly licensed creek videos and proposed 64 flags. The gate dropped 29 of them and kept 35. 32 of the 35 kept flags are on a dug-out channel, which the check does not ask about, so they ask nothing. The other 3 are Claude Haiku 4.5's answers on the frame shown below: it said yes in each of its 3 runs.
A model may only flag a feature it passed on the test. The gate drops every other flag, so the model never decides.
From results/footage_latest.json, results/footage_pool.json, results/model_card.json, results/footage_20260924T060539Z.json, run on Sep 24, 2026.
One flag kept, one dropped
On the live site the checker is off today in the creek check, and no walk carries its question, so nobody has met it there. Part 2 of the test is the one place a person meets it: the words "The checker noticed something here. Look again?", with no note. This is the record of our paid run on creek footage, not something a volunteer saw.
Kept
v06-00403
Claude Haiku 4.5
Built banks
The model said yes. It had passed this feature on the test: it got all 4 photos right in 3 of 3 runs. So the gate kept its flag.
A person at the creek would then be asked this:
The checker noticed something that may be concrete walls and other built banks. Want to look again?
the checker noticed: The stream bed shows carefully arranged stones/rocks that appear to be deliberately placed and set, characteristic of artificial bank construction rather than...
Claude Sonnet 5 passed this feature too. Its answers on this frame, run by run: can't tell, can't tell, can't tell.
Claude Opus 5.5 passed this feature too. Its answers on this frame, run by run: can't tell, can't tell, can't tell.
Claude Fable 5.1 passed this feature too. Its answers on this frame, run by run: can't tell, can't tell, can't tell.
No other model that passed this feature said yes on this frame. Passing the photos did not stop this flag, which is why a flag can only ask a person to look again and never answers for them.
Nobody has labelled this frame, so nobody has said whether the model was right.
The frame is a still from a video by OkState Ag, CC BY 3.0. Source
Dropped
v02-00143
Claude Haiku 4.5
Pipes and drain outlets
The model said yes. It had not passed this feature on the test: it got all 4 photos right in 0 of 3 runs. So the gate dropped its flag, and nobody would be asked anything.
The gate's own words: flag 1: feature pipe_running not passed by model claude-haiku-4-5-20251001
Nobody has labelled this frame, so nobody has said whether the model was right.
The frame is a still from a video by Красота Приморского края и не только, CC BY 3.0. Source
From examples/footage-flag/example.json, results/footage_20260924T060539Z.json, run on Sep 24, 2026.