Evidence on every run, even the green ones

Picture a bug report that says "checkout button didn't work, couldn't reproduce." Everyone has seen one. Usually it gets a shrug, a "works on my machine," and a slow slide to the bottom of the backlog.

Now picture the same report with a screenshot of the moment it broke, a full browser trace you can scrub through, the console error that fired two seconds earlier, and the network request that came back with a 502. That report gets fixed.

The difference isn't the bug. It's the evidence.

So one of the pieces we've been building this summer is an evidence store, and the rule we set for it is simple: every attempt produces evidence, whether it passed or failed.

What goes in the manifest

Each attempt, meaning a single run of a single test or a single replay by the referee, writes a manifest. It's a small file that lists what was captured and where it lives:

• a screenshot at the end of the attempt, plus at any step that fired a rule
• the Playwright trace for the whole attempt, with DOM snapshots and actions
• the browser console log
• the network log, with status codes and timings
• the rule results: which universal and business rules ran on each step, and what they returned

That last one is the part people tend to skip, and it's the part we care about most.

An empty list is evidence too

When a rule finds nothing, it's tempting to record nothing. Why store "no broken images" for every page?

Because "we checked and found nothing" and "we didn't check" look identical if you don't write it down. If a sort-order bug shows up in production next month, the first question will be "did the gate check sort order on that page?" A manifest that says sort_order_ascending: [] for that step answers it. A manifest that's silent leaves you guessing, and guessing in a postmortem is miserable.

An empty result list is a positive statement: these rules ran here, on this build, and they were satisfied. That's what lets a green run mean something.

How long we keep it

Evidence costs disk space, and traces in particular aren't tiny. Keeping everything forever would be expensive and mostly pointless. Keeping nothing would defeat the purpose. The retention idea we've settled on for now:

1. Passing runs are kept for a week. Long enough to compare against "the last green one" when something breaks.
2. Failing runs are kept for a month, even if nobody files anything.
3. Anything linked to a Jira ticket is kept until the ticket closes, plus 90 days. Regressions have a way of coming back right after someone says "that's fixed."

These numbers aren't sacred. They're a starting point that we expect to tune once real runs start piling up, which should be soon since we're getting ready to point the system at a real staging site.

Replay it later, step by step

The most useful thing a trace gives you is time travel. A developer looking at a finding three days later can open the trace, step to action 14, see exactly what the page looked like, which element was clicked, what the network was doing, and what the console said right then.

That matters a lot for agent findings. An explorer might take a weird path to a bug: open a category, sort by price, scroll, go back, scroll again. Written steps describe that path. The trace shows it. And because our referee replays without any AI, each replay produces its own trace too, so you can put the original attempt next to the clean replay and see whether they really hit the same thing.

There's a quieter benefit. When a test fails because the VPN dropped or staging returned a wall of 503s, the evidence makes that obvious in about five seconds. Network log full of connection errors, screenshot of a gateway page. That's an infrastructure failure, not a test failure, and the manifest is what lets the system tell them apart without a human squinting at logs.

We sometimes joke that the green runs are the boring ones. They are. That's exactly why we want proof they happened.