41 of 41 tests passed. Three of them were failing.

The question I was not asking

Before you report something as verified, ask one thing: what output would have made this check report a problem?

If you cannot name that output, the check proved nothing. It is not weak evidence. It is not evidence at all.

This sounds too obvious to be worth writing down. I violated it four times in two days.

The exit code that was never mine

The worst one is also the smallest. A test runner loop, roughly this:

timeout 240 python "$t" 2>&1 | tail -2

Then it read $? to decide whether the suite passed.

That is not the exit code of the Python process. It is the exit code of tail, which succeeds essentially always. The check reported success unconditionally. It had no path to any other answer.

It told me 41 of 41 suites passed. Three of them were failing.

Three more of the same shape

Once I went looking, the same failure was everywhere in that week's work.

• A test that could not see its subject. It scanned page source for the literal string /topics/x. The page built that href from a template literal, assembling the path at runtime, so the literal never appeared in the source at all. The test passed because it was looking for something that structurally could not be there. A redirect bug shipped. A production link audit caught it days later.
• A grep against a data literal. I searched a TypeScript map for "grace" and got nothing, so I concluded a review finding was wrong and overruled it. In that file only hyphenated keys are quoted. Bare keys like grace: never match a quoted search. The grep was blind to 22 of the 25 keys in the file, and the finding I overruled was correct.
• A root cause nothing could refute. A browser session stopped authenticating after a reboot. Clean story: the reboot dropped a session-only cookie. Every piece fit, so I stopped. The cookie jar actually showed that cookie present and still unauthenticated, plus the persistent cookies wiped. The real cause was a change of cookie encryption backend, not an expired session. I had never asked what would disprove the tidy version.

One habit wearing four costumes

These are not four unrelated mistakes. They are one mistake: writing the check after forming the belief, and then shaping the check to fit the belief.

Every one of them felt rigorous at the time. That is the part worth sitting with. I was not being sloppy. Each check was written deliberately, and each produced a result that fit neatly with everything else I thought was true.

Coherence felt like proof. It is not. A story where all the pieces fit is just a story where all the pieces fit.

What I do now

Seven rules came out of this, and they have held up since.

Break it on purpose. After writing a guard, put the original bug back and confirm the guard fails. If you cannot make it fail, you did not write a guard. You wrote a comment that runs.

Never read $? through a pipe. Use set -o pipefail, or capture the exit code before piping anything.

Parse data, do not grep it. If the claim is about a config map, a JSON structure, or any data literal, load it and query it properly. Quoting and formatting vary in ways your search string does not know about.

Green on your laptop is not verified. If the repo has CI, local green tells you about your laptop. Go look at the pipeline.

Check the check for races. A CI job triggered on push that crawls the live site is auditing the previous deploy, not yours. Re-trigger it after the deploy actually lands, or the result means nothing.

Say hypothesis until something could have killed it. A root cause stays a hypothesis until a check with the power to refute it has run and failed to.

Know where your tools are blind. When the state you care about lives in gitignored directories, git status cannot report a dirty tree. A clean result there is not clean. It is silent.

Why this gets worse with agents

All four of these came out of sessions where I was working fast with an agent, and that is not incidental.

A model will happily write the check that confirms the belief it just formed. It is good at producing a coherent account, and coherence is exactly the signal that fails you here. The more fluent the explanation, the more it feels verified, and the less likely anyone is to go looking for the output that would have contradicted it.

What caught all four was an adversarial pass: a separate reviewer whose only job was to refute the finding, running with no stake in the original belief. Not the thing that formed the belief in the first place.

I have written before about adversarial verification as a quality filter. This is the stronger argument for it. In a loop where everything else is trying to agree with you, it is the only component with a reason to make the check fail.

The bottom line

One question, asked before you call anything done: what result would have made this fail?

If you cannot answer it, you have not verified anything. You have agreed with yourself, in a formal-looking way.

More field notes as the systems keep teaching me things. Follow @HeVeMi99 for updates.