Checking vs. testing: the distinction that shaped our design

There's a distinction in software testing that we keep coming back to, and it comes from James Bach and Michael Bolton as part of Rapid Software Testing: the difference between checking and testing.

Paraphrasing their idea: checking is the part a machine can do. You make an observation, apply a decision rule, and get a pass or a fail. "The cart total equals the sum of the line items." "The page returns 200." "The price is greater than zero." Testing is the bigger human activity around it: exploring the product, questioning it, learning how it behaves, noticing things nobody wrote a rule for.

Checking lives inside testing. Testing is what decides which checks are worth having.

We'd read about this distinction years ago. It took designing an AI testing system to realize how much it matters.

Where AI tools blur the line

A lot of AI testing tools pitch themselves as testing: the agent explores, notices, judges. And the exploring part is real. A model can wander through a store, try a weird coupon code, resize the window, and spot that a label is cut off.

The problem is that the same tools then treat the agent's opinion as a check. "The agent thinks the price is wrong" becomes a red mark in a report. But an opinion isn't a decision rule. Ask again and you may get a different answer. As we noted after reading a stack of papers, free-text bug verdicts from models are right only about one time in four.

So you end up with something that has the flexibility of testing and the authority of checking, without the reliability of either.

Three kinds of executors

Bach and Bolton's split gave us a cleaner way to divide the work. We ended up with three kinds of executors, each with a clear job.

Code and tools. These do checking. Smoke tests, regression tests, accessibility scans, rules that run on every page. Same input, same verdict, every time. Because they're deterministic, they're the only thing allowed to block a release.

Agents. These do something close to testing, in a limited way. They explore, form hypotheses, and propose findings. They're allowed to be curious. They're not allowed to block anything on their own, and their findings have to be confirmed by code before they count.

Humans. These do the parts of testing that need judgment we can't automate: is this design confusing, does this flow make sense for our customers, should we ship this anyway. Humans also have final sign-off.

The loop that connects them

The useful part isn't the split. It's what moves between the three.

When an agent finds something suspicious, it doesn't hand over a paragraph. It hands over a condition that code can evaluate, plus the steps to get there. Say it notices that a product card shows $24.99 while the product page shows $29.99. The finding becomes something like card_price == product_page_price, which a replay can check without any model involved.

If the check confirms it, the finding becomes a ticket, and then a regression test. The agent's moment of curiosity turns into a check that runs on every deploy from then on.

Same with humans. When a tester notices that the shipping estimate disappears on narrow screens, the goal is to turn that observation into a check where possible, so a person doesn't have to notice it again next month.

So the flow is always the same direction: testing discovers, checking keeps. Whatever humans or agents learn should end up as code if it can be written down as a rule.

What stays human

Not everything can be turned into a check, and we don't want to pretend otherwise. "This checkout feels slow" can partly become a performance budget. "This error message is rude" mostly can't. A good tester's sense that something is off, before they can say why, is still the most valuable thing in the room.

We see our job as taking the repetitive checking off people's plates so they have more time for that kind of testing. Not replacing it.

Which raises the next question we're working on: if the agent isn't the judge, who is, and how do they decide?