Notes on Building a Code Reviewer

It probably won’t surprise readers that software engineers at a voice AI evals startup produce a lot of generated code. In August, people at Coval opened 2,578 pull requests. Even that undercounts the reading: those pull requests picked up roughly 3,400 more commits after they were first marked ready, so a typical change asks to be read more than once before it merges. Most of the cost of producing this much code lands on the people who review it.

Much of what a reviewer here actually does on a first pass is run a frontier model over the change and exercise judgement on what comes back. So the question was never whether a model belongs in review—it was how much of that judgement can be automated, and whether we could decide, rather than discover, where it can’t. Whether a change should exist, or suits the system it lands in, stays with people. Reading the diff against the surrounding code and checking that what a change says matches what it does looked like the automatable share, provided the checking was cheap enough to run on everything. The serious reference point is the commercial review bots, which land around eighty cents a pull request—noticeable at our volume, but a price a working product can hold. At the other end, one frontier lab’s managed review offering publishes an official average of fifteen to twenty-five dollars per review, which we mention without judgement, as people who have watched an agent reverify a two-line change. Whatever we built needed to come in well under the eighty cents, since the point was to run it on every push.

The reviewer runs inside our internal agent service, and it shipped one day after that service existed at all. Over the following ten weeks it was turned off three times, each time for a different reason, before a fourth version took. The measurement mistakes we made along the way are included below, since they transfer better than the results do.

June

The service came up on June 4th, and the reviewer went in on June 5th: triggered by @sofia review on a pull request, running on qwen3-max, with an isolated checkout and changed-line validation from the first commit. Automatic reviews followed a day later, and lasted about an hour before we turned them back off. The reviews read well and were wrong at a rate that mattered—the model treated facts newer than its training data as fabrications, rejecting a model name as nonexistent because it postdated the cutoff, and claimed that files plainly present in the repository were missing. It also posted all of this as failing checks, so the errors were not so much noise as interruptions to merging. The days that followed went to fixes, with the reviewer switched back on to test each one; swapping the underlying model to claude-sonnet-4.5 on the morning of June 12th was the last of these, and that evening we disabled the reviewer in production, with a note that it would stay off “until the review quality issues are sorted.”

One layer from that week outlived every rewrite that followed, and a reader building something similar can take it as a starting point: the delivery mechanics. A GitHub webhook wants an answer in seconds, so the receiving lambda does no work—it verifies the payload’s HMAC signature, drops what should never become a review (bot senders, drafts, repositories off the allowlist), enqueues a job, and answers 202 in milliseconds. Everything slow happens behind the queue, where the worker’s first act is to ask GitHub whether the head it was enqueued for is still the head, since the branch may have moved while the job waited. Two details took longer to get right than the rest combined: the job’s identity is the head SHA, so GitHub’s redeliveries and duplicate events collapse into one review instead of two; and the automatic trigger fires only for pull requests targeting the default branch, because a stacked PR is otherwise reviewed twice—once on its own branch and again when it retargets.

What June established was narrower than “the model isn’t good enough.” It wasn’t the model; two different models produced the same failure. Nothing in that design required a claim to touch the code it was a claim about, and a model that may assert without citing will, at some rate, assert fiction with the same fluency as fact. Most of what the current design does about honesty can be traced to that week.

July

The reviewer stayed dark for six weeks while we built out the product side of the agent, and when it returned in late July it had been rebuilt as an overcorrection: a five-stage verification pipeline—candidate generation, changed-line validation, an independent verifier, blind evidence collection, a candidate-blind skeptic, and a final adjudicator—running on GPT-5-mini and failing closed on anything it could not prove, aimed at a named catalogue of false-positive families: partial-diff absence claims, unseen secrets and action contracts, future data-flow theories, speculative defensive handling.

It mostly told the truth, and it lasted one working day of real traffic anyway. We enabled it on a Friday afternoon; by Monday morning it was clear that reviews—which arrive in bursts and hold a worker for minutes each—were crowding the agent’s other work off the queue they all shared, and we stopped it. There was no kill switch at this point, so the fastest way to stop it was to point the production repository allowlist at a repository that cannot exist:

-  "code_review_allowed_repos": [],
+  "code_review_allowed_repos": ["__code_review_disabled__"],

An empty allowlist meant no restriction, so turning the reviewer off required inventing a repository for it to be restricted to. The proper controls arrived in August.

August 5th

The third attempt went live at 19:11 UTC on August 5th and was disabled at 22:32 the same evening, this time with nothing wrong in the findings themselves. The shutdown notice recorded a review that “completed silently and correctly, but used 37 GPT-5.4 discovery calls, took 10m44s, and cost about $2.07,” two repositories whose pull requests deterministically exceeded the 90k-token prompt limit and left stuck checks behind, and the observation that disabling publication alone would not stop the spending, since the models ran whether or not anyone saw the output. The verification pipeline had bought its honesty by doing each review many times over, and at $2.07 against the roughly eighty cents we were paying a vendor, the arithmetic ran the wrong way. Three attempts in, the record was easy to state: the first reviewer lied cheaply, and its replacement told the truth at a price nobody would pay.

The rebuild

What happened over the following week was mostly subtraction. The reviewer was rebuilt around a single bounded agent working in an exact checkout, now on gpt-5.6-terra, and the verifier, skeptic, and adjudicator hierarchy were all removed: 287 files changed, 6,666 lines added, 76,277 deleted, under a principle we wrote down at the time and have held to since—keep only complexity that wins paired evaluation.

A few things survived the deletion, each because it kept winning evals, not because it sounded plausible.

The checkout is exact and read-only. The reviewer gets read, grep, glob, and list, and nothing else—no shell, no LSP, no network, no edits—not primarily for review quality, but because it reads arbitrary pull request code while holding worker credentials, and those tools are the paths by which untrusted code could reach them. The restriction produced a discipline we’ve kept: the prompt forbids predicting that a check will fail, since a reviewer with no shell claiming “CI will break” is guessing, and shipping confident guesses as red marks was exactly June’s failure.

Every claim must quote a line the diff actually changed. This one constraint does more of the work than the five-stage pipeline did, and its effect is measurable. Asked to name sensitive risk domains in prose, the model claimed authorization in 38% of reviews, while about 18% of the changes touched anything authorization-shaped; required to quote the specific changed line instead, it produced 570 claims, of which 98% resolved to real code—the same model over the same pull requests, with only the form of the question changed. The constraint doesn’t filter over-claiming after the fact so much as make most of it impossible to produce.

Reviews are capped at three findings, a partly arbitrary number kept because it measurably changes whether people read the output. And silence is the usual outcome: across 1,732 recorded runs of this design, 922 published nothing, a mean of 0.70 findings per review. Most pull requests are fine, and the characteristic failure of review bots is less about missed bugs than about the few real findings drowning in observations about naming, so a reviewer that mostly says nothing is, on most pull requests, simply right.

The shape that remains is two phases: a discovery pass proposes candidates, and an adjudication pass, resuming the same session, must return a verdict on each—publish, or discard with a reason—with discovery also required to supply the strongest benign explanation for every candidate, which the judge then has to test. The reviewer went back on for five repositories on August 17th and hasn’t been off since. The two weeks that followed were hardening, very little of it model work: one visible comment per defect rather than one per push, thread reconciliation so a re-review answers its earlier findings rather than restating them, and the risk rating printed together with its rubric, since a reader handed a single word can’t tell whether it sits at the top of a three-step scale or the middle of a ten-step one. There is also a register system keyed to the pull request’s author, so the prose varies from PR to PR. One colleague receives his reviews in the voice of a pig, at his own request. Observed in the wild, on one of his pull requests:

Medium · correctness OINK, author, hooves in the trough: this new direction conflicts with the simultaneously mounted propose_org_action and get_agent contracts…

The severity calibration is unaffected. It says “author” because the prompt asked for the author’s first name and the context didn’t carry one, so it printed the word—which points at what building the register taught us about prompts generally: rules produce formulas, examples produce voice. Every attempt to specify the register as instructions collapsed into a template, and one good example fixed it.

What it runs on

The system around the model matters as much as the model, and most of it exists as the answer to one recurring question: what happens when reviews contend for something. July’s answer was that they shouldn’t contend with the rest of the agent. The agent already ran two lanes—one queue for internal automation, one for customer-facing product work—so reviews became a third: their own encrypted queue with a dead-letter queue behind it, worked by their own fleet. GitHub webhooks are only the reviewer’s front door; the other lanes have ingresses of their own.

The fleet’s scaling signal is the part worth copying: it scales on the age of the oldest queued message, not on queue depth. A slow review holds the queue while depth stays at one or two, so depth is the wrong signal—and once you scale on age, every threshold has to be positioned relative to how long a review is allowed to hold its message, because the age metric counts messages still being worked. The budget helps here by being a clock rather than a meter: members of a concurrent batch are charged once, at the longest of them, so a review can spend more model time than the budget names while its wall time—and with it the age of its message—stays under the number. Get that wrong and the system pages a human about a review that is behaving exactly as configured. We know because ours did.

At our volume this means three to eight two-vCPU workers, each clearing roughly six reviews an hour at current settings, against a peak of about fifty review jobs an hour when an afternoon’s worth of pull requests gets marked ready. Around two-thirds of what the reviewer spends goes to re-reviews of pull requests that kept moving after the first pass—the same follow-up pushes from the opening paragraph, seen from the other side.

The numbers a single review runs under:

budget per review	$1.50 · 960s on the review’s clock, shared by everything it runs
findings cap	3 per review
discovery ceiling	240s per pass, settable per model
tools	read, grep, glob, list—nothing else
roster	3× gpt-5.6-luna + 1× glm-5.3-flash, two at a time
median cost	about eleven cents

The roster line is the second half of this story.

Four runs of the same reviewer

With the reviewer stable, we could measure it carefully, and the first solid result changed what we thought the problem was. We ran the identical terra engine four times over one evaluation panel—same model, same prompt, same code. Individual recall came back between 37% and 56%, averaging 47%. The union of the four runs found 81% of the panel. Of the sixteen defects in it, three were found in every run, ten came and went between runs, and three never appeared. The note we wrote at the time was that recall was failing on consistency rather than capability: a single review is one draw from a fairly wide distribution, and where a better model improves that distribution somewhat, more draws walk toward the union. The practical reading was to stop buying capability and start buying repetition—which is only affordable if each draw is cheap, and cheap draws are the one thing the low end of the model market reliably supplies.

The roster

To test that reading, we replayed a panel of historical review rounds—real pull requests carrying 55 defects that human reviewers had flagged at the time—against different reviewer configurations.

Scoring these replays honestly took more work than running them, and the first method was wrong. The cheap way to score a replayed review is anchor matching—did the bot comment near the line the human commented on—and when we hand-audited 70 such hits against the original human comments, only 70% described the defect the human had described; the rest were nearby comments about something else. Chattier configurations get more chances to land near a label, so the metric was quietly paying for verbosity: one arm we had measured at 14% below baseline sat at 37% below once a blind judge compared every hit against the original comment, and an apparent improvement we were preparing to act on was an artifact of comment count. Every number below is blind-judged for that reason. The same campaign caught two more instrument failures on the way: a corpus flagged as a sealed holdout turned out to share every one of its pull requests with the panel we tuned against—case IDs that didn’t collide had hidden the overlap—and a results recorder trusted a sweep summary that never echoed the variant settings back, mislabeling twenty-five runs as baselines. The holdout was renamed to what it is, holdout labels now stay out of the repository entirely, and one number I had already circulated did not survive the rebuild from per-case fingerprints and was withdrawn.

With the scoring settled, the result:

configuration	verified defects (of 55)	cost for the sweep
gpt-5.6-terra, one run	16	$14.76
gpt-5.6-luna × 3	17	$3.76
gpt-5.6-luna × 3 + glm-5.3-flash × 1	19	$5.65

A caveat before anyone quotes this: Coval runs an actual benchmarking service, and this is not it. These are ad hoc measurements on a panel of our own pull requests, made to choose a configuration, with instruments we had to repair twice along the way—a grain of salt is appropriate. They still added up to a reviewer that works, so they aren’t worth nothing either.

A single run of luna is substantially worse than a single run of terra, so the result is not that the cheap model is secretly good. Its runs are decorrelated—they overlap little enough that each contributes findings the others missed—where terra’s repeat runs mostly re-find the same things. In production, somewhere between 72% and 82% of the roster’s findings come from exactly one member, which is to say that most of the value sits in the disagreement between runs. A different model family disagrees more thoroughly than a different run of the same family—glm is there because it is cheap and foreign—and that, more than pricing, is the argument for mixing vendors. One figure from these panels never appears next to another figure from the earlier one: the 81% union above and the 19-of-55 here come from corpora labeled by different processes, and recall numbers from differently sourced corpora turn out not to be comparable at all. They answer different questions in the same units.

Two designs lost along the way. Pooling—several discovery passes feeding one shared adjudication—reached the same union of findings at three times the discovery cost and with worse completion, since a single adjudicator owes a verdict on every pooled candidate and degrades as the list grows; independent whole reviews share nothing, so nothing contends. And agreement as a filter, which looks like an obvious way to cut noise given how few findings are corroborated, kept 23% of the comment volume but only 38% of the verified hits when we measured it, which is worse than running one model once. Agreement does carry information—a finding raised by one member corresponds to a real defect about 18% of the time, by three members about 67%—but it earns its keep as ordering, not filtering. Corroborated findings are listed first, and nothing is deleted for having been seen once, since being seen once is most of what the roster is for.

In its first week in production the roster brought the median cost per review from about thirty cents to eleven while publishing at roughly the same rate. It also re-asked the contention question one level down. We first ran the members sequentially against the review’s shared clock, so the member listed last received whatever time the others left over, and completed 6 of its first 34 reviews—nothing wrong with the model, which was being handed the tail of a sixteen-minute budget. The members now run two at a time, each seeing the full clock:

That exposed the next layer, because glm still failed most reviews with the full clock available. It was hitting the 240-second discovery ceiling, a limit tuned for terra’s latency profile—68% of glm’s production calls ended at exactly that wall, against zero of luna’s 1,553. The ceilings are settable per model now. Running an open-weight model in the roster carries one more operational cost worth naming: the same model is not the same model. Open weights are served by many upstreams at differing quantisations, so an unpinned model can change quality with nothing about your own service changing—and it surfaces looking like a model regression rather than a routing change. We pin the serving order per model, with fallbacks off, and treat a routing change as a deployment.

What it catches

On defects that a human reviewer went on to block a merge over, the reviewer’s recall is about one in nine—far below what benchmark-style reporting tends to suggest, and the figure we consider the real one. Of the findings it does publish, roughly 85% are engaged by a human and nearly all are resolved, which is the trade the design makes deliberately: it speaks rarely and is usually worth reading when it does.

It has also been reviewing its own construction, which produced the clearest single demonstration we have. The pull request that made roster members run concurrently—written carefully, tested, self-reviewed—went up while this post was being drafted, and within minutes the reviewer posted this, verbatim:

High · Reliability

When concurrency is greater than one, this creates each member checkout from Git’s committed tree, but the generated .sofia-review-context-<head>.patch is an untracked file created only in the primary checkout. OpenCodeReviewHarness.run requires exactly that file before starting a model, so every concurrent member fails before execution and an all-member flock cannot complete. Copy or regenerate the exact-diff context in each member worktree before invoking the harness.

(evidence: four lines cited across four files)

It was right. Every concurrent member would have failed a safety check before any model started—complete failure of the new path, caught before merge, by the system the pull request was extending. It went on to find four more real defects across the fix-up commits, including one where my fix for a budget-accounting bug introduced the same bug’s mirror image one retry attempt later. Whether one-in-nine recall is enough is an abstract question until the bug is one you wrote that morning and your tests missed.

Where it goes

The roster mixes two model families today, and since families decorrelate harder than sizes within a family, it will get more diverse before it gets bigger. The apparatus around it—the replay panel, the blind judge, the audited labels—is standing infrastructure now, so a new model can be evaluated for a roster seat within a day of its release; local models are moving toward the price region where the roster’s cheap seats live, and when one arrives the evaluation will be waiting.

The configuration that catches the most of what our human reviewers catch is also the cheapest one we have run, and most of its budget goes to models that would be dismissed on sight as too weak for code review. Individually, they are too weak. After watching four identical runs of a strong model disagree with one another, that stopped seeming like the relevant question; the useful unit of review turned out to be the ensemble rather than the model.