What we learned reading 40 papers on AI testing

We spent the first two weeks of the year reading. Papers on agentic UI testing, LLM-driven test generation, flaky tests, oracles, static analysis, sequential statistics. Some were brilliant. Some were a demo with a methods section attached. By the end we had a shared document of notes that's longer than anything we've written so far, and a much clearer idea of what we should and shouldn't build.

This isn't a literature review. It's the handful of lessons that actually changed our design.

1. The hard part isn't clicking, it's knowing what's wrong

We expected the difficult problem to be navigation: getting an agent through a login, a multi-step form, a checkout. Modern models are decent at that. They get lost sometimes, but they recover.

The difficult problem is the oracle. Once the agent is on a page, how does it know whether what it sees is correct? Across the published benchmarks for agentic testing, a model's free-text verdict that something is a bug is right roughly a quarter of the time. About one in four.

That number reframed the whole project for us. If you let a model decide what's broken, three of every four alarms are noise. So the model can't be the judge. It can only point.

2. Old ideas are doing a lot of the heavy lifting

Some of the most useful reading was decades old. Metamorphic testing, introduced by T. Y. Chen and colleagues in 1998, deals with exactly the situation where you don't know the right answer but you do know how answers should relate to each other.

A shopping example: you don't know what the correct search results for "boots" are. But you do know that if you add a filter for size 42, every result should be one that was already in the unfiltered list. And if you sort by price ascending, each price should be no higher than the next. Those relations can be checked by code, every time, with no judgment involved.

We came away convinced that a good part of "AI testing" should actually be checking relations like these, with the AI's job being to find new places to apply them.

3. A false positive is whatever people ignore

Sadowski and colleagues wrote about building static analysis tools at Google (CACM, 2018), and one idea from that paper has stuck with us: the "effective false positive." If a developer looks at a report and doesn't act on it, it's a false positive for practical purposes, whether or not it was technically correct.

That's a harsh standard, and the right one. A tool that reports a real issue in a way nobody trusts has failed just as much as a tool that reported a fake one. For us it means every finding has to arrive with reproduction steps and evidence, and the system should track how often each kind of finding turns out to be real.

4. Flaky tests are a tax everyone pays

Google's testing blog wrote about flaky tests back in 2016, and the problem hasn't gone anywhere. A test that sometimes fails for no reason teaches people to rerun until green, and that habit eventually hides a real failure.

Adding AI to testing makes this worse, not better, if you're careless. A model that behaves slightly differently each run is a flake generator. So we want the parts of the system that can block a release to be fully deterministic, and the parts that use a model to be clearly separated from them. More on the math of flakiness next week.

5. Statistics already solved "how much evidence is enough"

When should a new test be trusted enough to block a deploy? After one green run? Ten? We were about to invent a rule of thumb when we found Abraham Wald's sequential probability ratio test, which answers exactly this kind of question: keep observing until the evidence is strong enough either way, then stop.

It's not glamorous, and that's the appeal. It gives you a principled stopping point instead of a number someone picked in a meeting.

6. Watch the evaluation setup

This one is more of an impression than a finding. Many of the setups we read about evaluate on small demo apps or on sites the authors controlled, with tasks written by the authors. That's reasonable for research, but it means the results tell you less about a real store with a few thousand products, a legacy admin panel and a cookie banner that loads late.

We don't hold that against anyone. It's a reminder that whatever we build, we'll have to prove on a real site eventually, not on a benchmark.

What we're taking forward

If we had to compress two weeks into one line, it would be this: let the model explore, and let code decide. Agents are good at curiosity and bad at verdicts. Deterministic checks are bad at curiosity and good at verdicts. The interesting design work is in the handoff between the two.

That handoff is what we'll spend the next couple of months arguing about. If you've read something that should be on our list, send it our way. We're not done reading.