Oracles, ranked: how we decide what counts as a bug

Every test needs an oracle: some way of knowing what the right answer is. A checkout test that clicks "Pay" and sees a page load hasn't proven anything until something says whether that page was the right page.

The awkward part is that oracles aren't equally reliable. "The spec says the button is blue" and "the model thinks this looks off" are both reasons to raise a hand, but you'd never block a release on them with the same confidence. So we ranked them. The rank decides how much weight a finding gets, and in one case, whether it can become a bug at all.

Here's the list, top to bottom, with one online store example for each.

The ones we trust most

1. Specified expectations. Someone wrote it down. A ticket says "free shipping applies to orders over $50." Add $49.99 to the cart and shipping should be charged; add one more cent and it shouldn't. If the cart disagrees, there's not much to argue about.

2. Universal invariants. Things that are wrong on any site, for any user, on any day. A 5xx response. An uncaught JavaScript error in the console. An image that returns 404. A product link that quietly 302s to the homepage. A product name rendered as Café instead of Café. Nobody needs to write a ticket for these. They're bugs everywhere.

3. Business invariants. Rules that hold for a whole category of site. In a store, an in-stock product shouldn't have a price of 0.00. The same product shouldn't appear twice on one listing page. If you choose "lowest price first," the prices on the page should go up, not wander. These need a little knowledge of the domain, but not of your particular codebase.

The ones that compare things

4. Consistency. The same fact shown in two places should match. The card on the category page says $24.00. The product page should say $24.00. The structured data in the page source should say 24.00. The cart, after adding it, should say $24.00. Four sources, one truth. When one of them drifts, something upstream is computing price twice.

5. Metamorphic relations. You don't know the right answer, but you know how answers should relate. Filter a category by "in stock" and the result count can't go up. Sort by price ascending, then descending, and you should get the same set of products in reverse order. We wrote about this one with a cart example back in March, in Metamorphic testing, explained with a shopping cart.

6. Differential. Compare against another version of the same thing. Yesterday's staging build showed 312 results for "garden hose." Today's shows 4. Maybe someone cleaned up the catalog. Maybe search broke. The difference alone doesn't say which, which is why this sits lower on the list.

7. Statistical. Things that are unusual rather than wrong. The product page usually loads in 1.2 seconds and today it takes 6. Out of 400 products, one has a price twelve times the category median. Often a real problem, sometimes a real luxury item.

And last, judgment

8. Agent judgment. An explorer agent looks at a page and says "this layout seems broken on mobile" or "this discount label is confusing." Sometimes that's the most interesting finding of the day. It's also the one we trust least, because free-text bug verdicts from models are right roughly a quarter of the time in published agentic testing benchmarks.

So here's the rule we settled on: agent judgment alone only produces "suspect." It never goes straight to the issue tracker. A suspect finding can climb, though. If the agent can turn "the discount label is confusing" into something checkable, like strikethrough_price > sale_price, it's now a business invariant and gets evaluated by code. If it can't, it waits for a human to look.

This isn't us being dismissive of models. Agents are great at deciding where to look. We just don't want them deciding what counts as broken, for the same reasons we laid out in Why the LLM should suggest, and code should decide.

Why the order matters in practice

The rank feeds directly into how confident we are about a finding. A broken image confirmed on replay is about as solid as it gets. A statistical outlier confirmed on replay is still just an outlier. Treating them the same would mean either blocking releases on noise or ignoring real breakage.

It also tells us where to spend effort. Every time a finding comes from the bottom of the list, the useful question is: could we have caught it higher up? An agent noticing duplicate products is nice. A business invariant that checks for duplicates on every listing page, every deploy, for free, is better.

We expect the list to stay roughly this shape. What will grow is the middle: more business rules, more consistency checks, more relations that hold across a store. That's where most of the real bugs we've read about seem to live.