Verification that cannot fail

An agent that runs a check and reports the result is only as trustworthy as the check. This document is about three ways a check reports success while being structurally incapable of reporting anything else — all three found in one project, all three by something other than the check itself.

The common shape: nobody had ever seen the check fail. A green result was read as evidence, when it was only ever evidence that the check ran.

1. Grepping for words instead of asserting exit codes

A typecheck was being verified like this:

npm run check 2>&1 | grep -E "errors|warnings|hints"

The output ends with three lines: - 0 errors, - 0 warnings, - 0 hints. Seeing them printed reads like confirmation.

It matches - 0 errors and it does not match - 1 error. The singular has no “s”. One error is precisely the case the pattern cannot see, and one error is the overwhelmingly common case.

printf -- '- 1 error\n' | grep -E "errors|warnings|hints"   # no output
printf -- '- 0 errors\n' | grep -E "errors|warnings|hints"  # matches

The error had been present locally for days. CI, which does not grep, failed correctly and was not looked at.

The lesson is not “write better patterns”. It is that a tool already reported success or failure precisely, in its exit code, and the pattern replaced that signal with a worse one derived from its prose.

2. Pipes discard the exit code

The same command has a second defect, independent of the pattern. In a pipeline, the shell reports the exit status of the last command, and grep succeeds.

false | cat; echo $?     # 0 — the failure is gone

So even a correct pattern would not have failed the run. Two options, and they differ by shell:

set -o pipefail          # portable across bash and zsh; pipeline fails if any stage fails
false | cat; echo $?     # 1

Or read the stage directly — but note the syntax is not portable:

# bash: zero-indexed
false | cat; echo "${PIPESTATUS[0]}"     # 1
# zsh: a different variable, one-indexed
false | cat; echo "${pipestatus[1]}"     # 1

Using bash’s form in zsh yields an empty string, which in a conditional is falsy — failing open, again silently.

The general fix is to stop post-processing and start chaining. Commands joined with && stop at the first failure and return non-zero:

npm run canon:verify && npm run test && npm run check && npm run build

There is nothing to pattern-match, and no way for a failure to be reported as success.

3. A guard nobody has watched fail

A build-time check compared two copies of a document and was supposed to fail the build when they drifted. It passed. It passed every time.

That is exactly what a check with a broken file path, an unreachable branch or an inverted condition also does.

A guard you have only ever seen pass is not yet a guard. It is a line of code that has never been observed doing its job. The fix is cheap and takes a minute: deliberately introduce the condition it exists to catch, watch it fail, then undo that.

Doing this found a related problem in the same session — a command-line tool whose usage error exited with the same code as a real verdict, so “I could not parse my arguments” and “I rejected this document” were indistinguishable to anything downstream. That was only visible because the failure path was exercised on purpose.

This generalises past guards to anything with an error branch: an error path that has never executed is an assumption, not a behaviour.

One case this remedy does not reach: a check that is correctly written and simply has nothing to examine. You cannot introduce the condition it catches without inventing a subject for it, and a fabricated subject is worse than the silence it replaces. That needs a different fix — see Nothing was wrong, or nothing was examined.

What to do instead

• Assert exit codes, not output. If a tool cannot signal failure properly, that is worth fixing before building on it.
• Chain with && rather than piping into a matcher. One command, one exit code, no interpretation.
• Set pipefail where a pipeline is genuinely needed.
• Make each guard fail once, on purpose, and record in the commit that you did. A guard’s first failure should be one you caused.
• When local and CI disagree, believe CI. It has no muscle memory and no helpfully-worded grep. A red CI against a green local check almost always means the local check is the broken one.

Why this matters more for agents

An agent runs verification commands far more often than a person does, reports their results in prose, and is trusted partly because it says it checked. Every one of these failures produces a confident, accurate-sounding report of a check that could not have failed.

The report is not the verification. The exit code is.