Metamorphic testing, explained with a shopping cart

Here's a question that sounds easy: if a shopper searches a store for "lamp", what is the correct list of results?

We don't know. Neither do you. Neither, most of the time, does the team that built the store. The answer depends on the catalog, the search index, today's stock and a ranking function someone tuned last quarter. Writing a test that says "the result must be exactly these 38 products" is a test that breaks every time someone adds a lamp.

This is the oracle problem, and it shows up everywhere in e-commerce. There are lots of outputs where nobody can tell you the right answer in advance.

Metamorphic testing gets around it with a small change of question. Instead of asking "is this output correct?", you ask "how must these two outputs relate to each other?" T. Y. Chen and colleagues described the idea in 1998, and it has been quietly useful ever since in places where exact expectations are hard to write: compilers, search engines, scientific software. Online stores turn out to be a very good fit.

Five relations for a product listing

Take a category page with a few hundred products, a sort dropdown, some filters, and pagination. We don't know what it should show. We do know a lot about how its answers relate.

1. Sorting doesn't change the set. Sort by "price, low to high", then by "newest", then by "name". The order should change. The set of products should not. If sorting by price makes a product disappear, or a new one appear, something is wrong, and we never needed to know which products belonged there to see it.

2. Adding a filter never grows the results. Start with "Lamps", 212 results. Add "Brass". The count can stay at 212 or go down. It can't go to 215. A filter that adds products is either a bug in the filter or a bug in the count, and both are worth a ticket.

3. Removing the filter takes you back. Add "Brass", then remove it. You should be looking at the same 212 products you started with. This one catches a surprising amount: stale caches, filter state that leaks into the URL and never leaves, a "clear" button that clears the chip but not the query.

4. Pages are disjoint, and together they make the total. Page 1 and page 2 shouldn't share a product. Walk every page, collect the product IDs, and the union should match the count in the header. No repeats, nothing missing. Off-by-one errors in offsets love to hide here, and they're almost invisible to a person scrolling quickly.

5. Case shouldn't matter, and neither should the locale's idea of case. "LAMP", "lamp" and "Lamp" should return the same set. Then try the harder version. In Turkish, the uppercase of "i" is "İ" (dotted), and the lowercase of "I" is "ı" (dotless). A search box that lowercases with the wrong locale can turn "İSTANBUL" into something that matches nothing. We want "İstanbul" and "istanbul" to return the same products, and that's a relation we can check without knowing a single product name.

None of these need a spec. None need a model to decide anything. Each one is a pair of requests and a comparison.

Why we like them so much

Metamorphic relations sit in a useful spot in our oracle hierarchy. They're weaker than an explicit expectation ("the coupon SAVE10 takes 10% off"), because they only tell you that two answers disagree, not which one is wrong. But they're much stronger than a model's opinion, because the verdict is a comparison in code. Two sets either match or they don't.

They also travel well. "Filtering never grows results" is true for almost every store we can imagine. That means we can write it once and point it at any listing page, which is exactly the kind of check we want running on every deploy.

And they fit the split we wrote about in Why the LLM should suggest, and code should decide. An explorer agent is good at spotting that a page has filters, a sort control and pagination, and proposing "check that these relations hold here." Code then runs the requests and compares the sets. The agent suggests where to look. The relation decides.

A few practical notes from designing these:

• Compare sets of IDs, not rendered text. Prices get formatted, names get truncated, and you want the comparison to fail for real reasons.
• Pin what you can. If the catalog changes between request A and request B, a perfectly good relation will look broken. Run the pair close together, and when you can't, treat a single failure as a reason to replay, not a verdict.
• Ranking is not a set. "Sorting doesn't change the set" is safe. "Sorting by relevance is stable" is not, and we don't pretend it is.

We're still building the list. Right now it has the five above plus a handful for carts: adding the same item twice should increase the quantity rather than create a second line, removing every item should return the cart to empty with a total of zero, and changing the currency and changing it back should give you the price you started with.

If you've got a relation that's saved you on a listing page, we'd honestly like to hear it.