Why our referee never uses AI

People are sometimes surprised when we say this, given what we're building: the component that confirms a bug is real doesn't use AI. Not a small model, not a cheap one, not one "just for parsing." None.

We call it the referee. An explorer agent wanders through the staging site, notices something, and hands over a finding. The referee's only job is to answer one question: does this happen again when nobody is improvising?

What the referee actually gets

A finding isn't a paragraph. It's a small bundle:

• the steps that led there, as structured actions (goto /category/garden, click [data-testid=sort], select "Lowest price")
• the browser setup the explorer used: viewport, network profile, user agent, account
• a condition that should hold, written so code can evaluate it, for example prices_on_page are non-decreasing

The referee opens a fresh browser session with the same setup, runs the steps, and evaluates the condition. Then it does it again, in another clean session. And a third time.

If the condition fails in at least two of three replays, the finding moves forward. For findings that came only from agent exploration, with no rule behind them, we want three out of three. Anything less stays behind as a note, not a ticket.

Why not let a model help?

The tempting version goes like this: the steps are a bit fragile, the page changed slightly, so let a model "adapt" the replay. Click the button that looks like the sort button. Read the prices off the screenshot.

We've thought about it a lot, and every version of that idea breaks the thing the referee exists for.

It stops being a replay. If a model adapts the steps, the referee is now exploring too. You've got two improvisers agreeing with each other, which is a lot less convincing than one improviser and one strict witness.

It stops being repeatable. Ask a model the same question twice and you can get two answers. A referee that confirms a bug on Tuesday and rejects the same bug on Wednesday is worse than no referee, because now every result needs a second opinion.

It costs money on every check. Replays happen a lot. Three per finding, plus every later re-check. If the referee is plain browser automation, that's close to free. If each replay calls a model, the cheapest part of the pipeline becomes one of the most expensive.

It hides failures. When a deterministic replay can't find [data-testid=sort], it says so. That's useful. Maybe the selector changed, maybe the button is gone. A model that helpfully clicks "something nearby" papers over exactly the kind of change we'd want to know about.

The tradeoff we accept

A strict referee will reject some real bugs. If an explorer reached a broken state through a path that depends on timing, or on a random banner, the replay might not hit it. Those findings fall back to "suspect" and wait for a person.

We think that's the right way round. A missed finding can be caught next deploy, or by a human. A false alarm in Jira costs trust, and trust is slow to rebuild. Anyone who has worked with a noisy alerting system knows what happens: people stop reading the alerts.

There's a nice side effect too. Because the referee only understands structured steps and code conditions, the explorer has to produce them. That pressure makes findings clearer. "The sort seems off" can't be refereed. prices_on_page are non-decreasing after selecting "Lowest price" can, and it's also halfway to being a regression test.

We're still building the referee, and the hard parts so far are boring ones: waiting for the page to settle, matching the explorer's browser setup exactly, making sure each session really starts clean. None of those need a model. They need patience.