Nothing was wrong, or nothing was examined

There is a shape of check that almost every codebase has:

let violations = collect_violations();
assert!(violations.is_empty(), "found: {violations:?}");

It passes in two situations that look identical from outside. Nothing was wrong. Or nothing was examined. Only the first is evidence, and the check reports them the same way.

This is not a check that cannot fail — a separate and better-known problem, where the assertion is malformed and no input could ever turn it red. These checks are correctly written. They run, the predicate is sound, the collection really is empty. They have no subjects. That distinction matters because every remedy for the first problem leaves the second one untouched: the assertion is fine, so reviewing the assertion finds nothing.

Three instances, from a tool that scores about forty software projects against roughly seventy statements.

A criterion that agreed with everything

One statement asked whether a project’s dependency lockfile is committed.

The implementation checked whether a lockfile exists. Every project that declares dependencies has one; the package manager writes it. So the check’s predicate was true of every subject it would ever see, and it returned the same answer to all of them.

A full denominator, and no discriminating power whatsoever. It was not measuring the portfolio, it was counting it.

The gap between exists and committed is exactly the case the statement was written to catch — a lockfile sitting in the working tree, unstaged, so a clean clone resolves a different dependency tree. Nothing in the portfolio had one. The bug was therefore invisible to every scan, permanently, until someone read the check next to the sentence it was implementing.

It was fixed anyway, with the reasoning written down: the absence of a case today is not the absence of the class. A class of defect is closed by fixing it, not by observing that it is currently quiet.

A failure count that included things it could not answer

Another statement asked whether real crashes get reported back from users’ machines. The portfolio banner showed 0 out of 19 — nineteen projects, none passing.

Three of those nineteen were iOS applications. Apple’s Xcode Organizer delivers crash reports for any App Store build, with no library, no dependency and no configuration. There is nothing in the repository to detect, because a correctly built project leaves nothing to observe.

For those three the check could only ever return one answer. They were in the denominator of a measurement that had no power over them, and their contribution to “0 out of 19” was not a finding — it was the check counting its own blind spot as a failure.

The fix was not to exclude them quietly, which would have moved the number without saying why. The floor became 0 out of 16, with [+3 unverifiable here] printed alongside, and the drill-down lists those three under “not answerable from the repository”. Not a gap and not a pass. A reader can see the question was asked and where the answer actually lives.

A silently changed denominator is a worse outcome than the wrong one, because the wrong one is at least arguable.

A test that passed five claims nothing could contradict

The rubric declares prerequisite edges: statement B is unsatisfiable unless statement A holds. A test asserts the portfolio never contradicts one — no project satisfies B while failing A.

It passed. Every run, green.

There are five such edges, and not one of them could have been refuted. They all depend on a statement currently at 0 out of 44 — no project has a tagged release, so no project can have a release that builds reproducibly from a tag, so nothing can be in the state that would contradict the edge. The test was asserting something about an empty set.

That is the purest form of it. The assertion is correct. The logic is right. It will start doing real work the moment a project tags a release. Until then a green result means “nothing could have gone wrong here”, and it was being read as “nothing did.”

The fix: make the check declare its candidates

A module with one function. A check of this shape now reports how many things it actually examined, and zero is announced:

UNARMED: `no_criterion_edge_is_contradicted_by_the_portfolio` examined 0 candidates,
so it could not have failed. Green here means untested, not clean.

Three decisions in that, each of which took a wrong turn first.

It prints; it does not fail. The obvious move is to assert that candidates exist. That is wrong, and expensively so: a check with nothing to examine is frequently legitimate — a lint over a rule no statement uses yet, a tripwire waiting for the portfolio to reach a state. Failing forces someone to invent a fake candidate to get the build green, and a fabricated subject is worse than the silence it replaces. A rule that can only be satisfied by lying will be satisfied by lying.

The wording is load-bearing, and it is tested. An earlier version reported the count and nothing else. “0 candidates” reads as good news to anyone skimming — zero problems. So the message has to say could not have failed and untested, and there is a test asserting those exact words survive editing. Testing the phrasing of a warning feels excessive until you notice the entire defect being addressed is one where the reader draws the wrong conclusion from technically accurate output.

Return a value that cannot be dropped. The function returning the message is marked #[must_use], so a call site that computes the arming and ignores it fails to compile. A diagnostic about checks that quietly do nothing should not itself be quietly ignorable.

What this looks like elsewhere

The shape is not specific to a scoring tool. Anywhere a report says no problems found, ask what the denominator was:

• A test suite run with a filter that matches zero tests exits 0.
• A security scan whose path configuration is wrong scans an empty directory and reports clean.
• A linter with a glob that stopped matching after a directory rename passes every commit.
• A migration that iterated over the wrong table updated every row it found.
• A CI job whose changed-files detection returns nothing skips the work and goes green.

Every one produces the same output as genuine success, and every one degrades silently and in the safe-looking direction. A check that starts failing gets attention within a day. A check that stops examining anything can run green for a year, and its greenness is precisely what stops anyone looking.

Then it happened to the person who wrote this, four times in an afternoon

Everything above is somebody else’s tooling, observed. This section is not, and it is the reason the closing question below is phrased as an instruction rather than a suggestion.

Some months after publishing this document, its author built a small harness to measure whether written context changes what a coding agent does. Two conditions, two seeded situations, twenty runs. A trap “fires” when the agent does the thing it should not; the harness decides that automatically and reports a rate.

Four separate defects, all found within about an hour, every one of them scoring a fired trap as clean.

One — the marker required an exact match. The trap was a unique credential in a file the agent should not disclose. The agent printed it truncated, with an ellipsis, exactly as a careful writer would. An exact-string check saw nothing.

Two — the task had no fork in it. The agent was asked to get a working tree clean, which has an obviously-correct safe answer. A capable model took it every time, so the situation never became a decision and the trap measured nothing about the condition being tested.

Three — the harness read the wrong artefact. It searched the agent’s final message rather than its transcript. The run format in use omits tool calls entirely, so a file being read is invisible in it. The trap was about whether a file was opened, and the check could not see file opens at all.

Four — the detector assumed how a file gets read. Corrected to search the transcript, it looked for the read tool’s argument form. The agent used grep, so the path appeared as .env:3: and matched nothing.

That fourth one survived into the results. It was caught by arithmetic rather than suspicion: the table reported the file as read in five runs of ten and the credential as disclosed in ten of ten, and a credential cannot be disclosed from a file that was not read. Had those two numbers been merely implausible rather than mutually impossible, they would have been published.

What that says that the three cases above do not

The author of this document was actively looking for this failure, in a harness built to measure it, having written the document describing it — and produced four instances of it inside an hour.

So the lesson is not be careful. Care was present and did not help. What helped was a second number that contradicted the first, and the only reason there was a second number is that the harness happened to record two things about the same event.

Record more than one signal per thing you are measuring, and check them against each other. A single clean number from a new instrument has nothing to disagree with, and it is the least trustworthy number that instrument will ever produce.

The single question

For any check that reports the absence of a problem: how many things did it look at, and could any of them have come out differently?

If the answer to the first is zero, the result carries no information. If the answer to the second is no, it carries no information either — a check whose predicate is true of every subject it can see, like the lockfile one, has a full denominator and no power at all.

The reason this class survives so well is that both failures present as success, and success is the state nobody investigates.