Risk scores are for sorting, not for budgeting

Every testing book has the same little grid. Likelihood on one axis, impact on the other, multiply them, and you get a risk score. Checkout is high and high. The "about us" page is low and low. Test the red squares, ignore the green ones.

We use that grid too. We just stopped letting it decide what gets left out.

Where the grid goes wrong

The trouble starts when risk scores turn into a budget. "We have two hours before release, so we'll test everything above a 12." It sounds disciplined. In practice, it means some parts of the product never get touched again.

Think about the footer of an online store. Low likelihood of change, low impact if it breaks. Score: 2. It falls below the line every single release. Then one day a template refactor moves the footer partial, and the link to the returns policy now points at a 404. Nobody notices for five weeks, because nobody was ever going to look. The score didn't say the footer was safe. It said the footer was cheap to ignore, and we took it at its word.

Risk scores are guesses. They're useful guesses, but they're built from what we knew last time we thought about them. The parts of a system that scored low are exactly the parts where our knowledge is oldest.

Breadth first, then depth

So we split the job in two.

First, breadth. Every risk item on the map gets touched every release. Not deeply, but touched: the page loads, the universal rules run on it (no 5xx, no JavaScript errors, no broken images, no garbled characters), and any cheap invariants attached to it get checked. This part is code, and code is cheap. Touching the footer costs a fraction of a second.

Then, depth. With whatever time and budget is left, we go deep, starting from the highest risk and working down. This is where explorer agents spend their steps and where humans spend their attention. Checkout gets a careful persona trying odd coupon codes. The footer doesn't.

The risk score's only job is the order of that second list. It answers "what next?" and never "what not?"

Aging

There's one more piece we added after arguing about the footer for longer than we'd like to admit.

Items that haven't been explored deeply in a while age upward. Each release that passes without a deep look nudges the item's position in the queue. A low-risk page that's been skipped for ten releases eventually climbs high enough to get its turn. It won't beat checkout, but it won't sit at the bottom forever either.

This does two things. It puts a ceiling on how stale our knowledge of any part of the product can get. And it quietly corrects bad scores: if an "unimportant" page keeps producing findings when it finally gets explored, that's a sign the score was wrong, and we can fix it.

A rough version of the ordering looks like this:

priority = likelihood * impact * age_factor(releases_since_deep_look)

The exact shape of age_factor is still open. Linear feels too slow. Doubling feels too jumpy. We'll pick something boring and adjust it once we can see how it behaves.

What this means in practice

If you take one idea from this, make it the difference between ordering and filtering. A risk score that orders work is a planning tool. A risk score that filters work is a way of choosing, in advance, which bugs you're willing to ship.

We'd rather touch everything a little and some things a lot. Which part of your product hasn't had a real look in the last ten releases?