Turn review comments into build failures

You have written this comment before. Not word for word, but close enough to paste: we don’t hardcode user-facing strings here, or route handlers go through the repository, not the client, or that needs a migration, not a schema edit.

You wrote it in March. You are writing it again now, on a pull request from someone who never saw the March one, because a comment on a diff is written on water. The diff merges, the comment scrolls away, and the convention survives only in the heads of whoever happened to read it.

That was tolerable when the people opening pull requests were five engineers who sat near each other. It stops being tolerable when the authors are a product manager, a designer, someone in support, and a fleet of agents that have never read a single one of your comments.

The instinct, and why it is not enough

The modern instinct is to write the convention into the instruction file: CLAUDE.md, AGENTS.md, a rules file. Better than a comment on a dead diff, certainly. But somebody finally measured how well it holds, and the number is worth knowing before you rely on it.

In a preprint from May 2026, Damon McMillan ran a factorial study on coding-agent configuration files. The instruction under test was about as easy as an instruction gets: begin every function body with a // @tracked comment. In the reference configuration, one 250-line CLAUDE.md with the rule on line two, the agent applied it to 67.7% of the functions it wrote. With no configuration file at all, it applied it to zero of 524.

Read those two numbers together, because they say something more interesting than “instruction files don’t work”. The file plainly causes the behaviour. It just does not compel it. Compliance also decayed as a session went on, at roughly 5.6% lower odds with each further function generated, and 65.7% of runs broke the rule at least once.

Two thirds is a good hit rate for advice. It is a terrible hit rate for a constraint.

The move

If you have written the same review comment twice, it is not feedback. It is a missing rule.

Mitchell Hashimoto named the underlying practice in February 2026, and his definition is the whole discipline in one sentence: “anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.”

Birgitta Böckeler’s taxonomy gives you the reason it works. She splits the checks in a harness into inferential ones, which need a model to interpret them, and computational ones, which are deterministic and therefore give guarantees. An instruction file is inferential. A failing build is computational. Prefer computational, every time you can get it.

Most of review is already rule-shaped

The objection here is that this only covers trivia, while real review is about judgment. The evidence says the ratio runs the other way.

Bacchelli and Bird hand-classified 570 comments from 200 Microsoft code review threads. Their most frequent category was code improvements, at 165 comments or 29%: readability, consistency, dead code, better practices, explicitly not correctness. Defects came fourth out of nine categories, at 78 comments or 14%.

So roughly twice as much of your review attention goes to convention as goes to finding bugs. That is the half a machine can hold for you, and holding it is the only way the judgment half gets any of your time back.

What it looks like

The comment, written for the fourth time:

"Route handlers shouldn't touch the db client directly. Go through
 the repository, or you bypass row-level auth."

The same thing, written once:

# .semgrep/rules/no-direct-db-in-handlers.yml
rules:
  - id: no-direct-db-in-handlers
    message: >-
      Route handlers must go through src/server/repository.
      Calling the db client here bypasses row-level auth.
    severity: HIGH
    languages: [typescript]
    paths:
      include:
        - "src/routes/**"
    pattern: db.$METHOD(...)

Notice what happened to the sentence you kept typing. It did not get summarised into a policy document. It became the error message, which means it now arrives at the exact moment somebody needs it, attached to the exact line that caused it, for every author forever, including the ones who cannot read code.

If your convention is expressible in your linter instead, prefer that, and check your config format while you are there. ESLint’s current major is 10, and as of v10 the old .eslintrc format is not read at all, so a rule sitting in one has quietly been doing nothing.

The false positive budget

Here is the discipline that makes this survivable, and it comes from the largest working example of it. Google’s Tricorder programme was analysing roughly 50,000 code review changes a day by January 2018, and it runs two different bars for two different jobs.

For a check that only comments in review: “analysis results shown during code review are allowed to include up to 10% effective false positives.” An effective false positive is defined by behaviour, not by correctness. If a developer takes no action after seeing the issue, it counts against you even when the finding was technically right.

For a check that breaks the build, the bar is zero. It should “produce no effective false positives”, and report “issues affecting only correctness rather than style or best practices.”

That is the honest cost of this move. A rule that fires on correct code will be routed around, disabled, or ignored, and it takes the credibility of your other rules with it. Google’s answer is brutal and correct: they track the ratio of “please fix” to “not useful” clicks, and disable any analyser that goes above 10% until its owner improves it.

So write the rule narrow. A rule that catches four of the five cases and never cries wolf beats one that catches all five and annoys everybody, because the second one will not be there in six months.

Try this week

Open the last thirty pull requests in your busiest repository and read only your own comments.

You are looking for repeats: the same correction, made to different people, about different code. Most people find between three and six. Write them down as sentences.

Then take the single most repeated one and give it a machine. If it is a naming or import convention, it is probably a lint rule you can write in ten minutes. If it is structural, it is a Semgrep rule, and Semgrep is free for up to ten contributors and ten repositories, so the first one costs you nothing but the afternoon.

Start it as a warning, not a build failure. Watch it for a week, count how often it is right, and promote it to blocking only when it has earned that. You are not automating your judgment. You are retiring the part of it you have already spent four times.