Pi Day math: why we replay three times and accept two

Happy Pi Day. March 14, or 3.14 if you write dates the American way, is the one day a year when it's socially acceptable to bring pie to a meeting and call it math.

Our contribution is smaller than pi, but it also starts with a 3. It's the rule our referee will use to decide whether a finding is real: replay it three times, accept it if it reproduces at least twice.

The problem the rule solves

When an agent reports a finding, the referee replays the exact steps in a clean browser session with no model involved, as we described in how the LLM suggests and code decides. If the condition fails again, the finding is real. If it doesn't, it was probably noise.

But "replay it once" is fragile. Real bugs are sometimes intermittent: a race between two requests, a cache that's warm on one try and cold on the next. And noise sometimes looks real: a slow image that happened to time out once. A single replay throws away too many real bugs and lets in too many flukes.

So we replay three times and count.

The math, in plain words

Say a finding reproduces with some probability on each attempt. Call it r. A solid bug might reproduce 90% of the time. A coin-flip intermittent bug, 50%. A fluke, maybe 30%.

"At least 2 out of 3" can happen two ways: all three replays reproduce, or exactly two do.

• All three: r × r × r, which is r³.
• Exactly two: there are three ways to pick which replay misses, and each has probability r × r × (1 − r). That's 3r²(1 − r).

Add them up and simplify, and the chance of acceptance is:

P(accept) = 3r² − 2r³

Now plug in numbers.

• r = 0.9, a solid bug: 3(0.81) − 2(0.729) = 0.972. About 97% of the time, it's accepted.
• r = 0.5, a true coin flip: 3(0.25) − 2(0.125) = 0.5. Exactly 50%, the same as a single replay.
• r = 0.3, mostly noise: 3(0.09) − 2(0.027) = 0.216. About 22%.

That shape is the whole point. The rule pushes things apart. Findings that usually reproduce get accepted more reliably than with a single replay (97% instead of 90%). Findings that usually don't get rejected more reliably (22% instead of 30%). The midpoint stays where it is. It's a small amplifier for the difference between signal and noise, for the price of two extra replays.

Why agent-only findings need 3 out of 3

There's one category we treat more strictly: findings that come only from an agent's judgment, with no specified expectation or rule behind them. These are the ones most likely to be wrong, because, as published benchmarks show, a model's free-text bug verdict is right only about one time in four.

For those, the bar is 3 out of 3. The chance of acceptance is just r³.

• r = 0.3: 0.3 × 0.3 × 0.3 = 0.027. About 2.7%.

Compared to 22% under the 2-of-3 rule, that's roughly eight times fewer flukes getting through. The cost is real too. A solid bug with r = 0.9 now passes only about 73% of the time (0.9³ = 0.729) instead of 97%. We think that trade is right for this category: an agent-only finding that misses once goes back for another look rather than straight to a developer's queue.

What the math doesn't do

It doesn't make a bad oracle good. If the condition being checked is wrong, three perfect reproductions just mean you're reliably wrong. Replays measure whether something happens again, not whether it should have happened at all. That's a different problem, and a different post.

It also assumes the replays are independent, which is why each one runs in a fresh browser session with no shared state. Three replays in the same warm session would be closer to one replay counted three times.

For now, we're going to go find some pie. Ideally a slice that reproduces at least two out of three times.