Staging is not fake production

There's a comfortable lie a lot of teams tell themselves: staging is production, just quieter. Same code, same data, fewer people. If it works there, it'll work for real.

Half of that is true, and the other half is where the trouble starts.

We're getting ready to point Testomatica at a real staging site soon, so we've spent the last few weeks writing down what staging actually is and what it isn't. These rules aren't glamorous. They're the difference between findings you can trust and findings that send a developer chasing ghosts on a Friday afternoon.

What staging gets right

A good staging environment usually has production-like data. Real product names, real category trees, prices that look like prices, maybe a nightly or weekly copy of the catalog. That's valuable. Business rules like "an in-stock product must have a price above zero" or "the cheapest item comes first when you sort by lowest price" can be checked against data that has the same shape and the same weird corners as the real thing.

So staging is a great place to ask "is the logic right?"

What staging gets wrong, on purpose

It doesn't have production traffic. Nobody is buying anything, caches are cold, search indexes may be rebuilt on a different schedule. It often shares infrastructure with other environments, so the database you're hitting might also be serving three other test setups and a developer's migration script. And in many setups, including the one we're preparing for, you reach it over a VPN from somewhere else, which adds its own latency.

That has a blunt consequence for us: no load tests and no performance comparisons on staging.

If the product page took 4.2 seconds yesterday and 6.8 seconds today, that number tells you almost nothing. Maybe the deploy made it slower. Maybe someone kicked off a reindex. Maybe the VPN was having a day. A test system that reports "performance regression, 62% slower" from that data is producing noise with a decimal point on it. We still record timings as evidence, and a page that times out completely is still a real problem. We just don't turn staging timings into verdicts.

Read-only, enforced in the browser

Staging data is shared and usually somebody's carefully prepared fixture. Our agents don't get to change it. Requests that could write data are blocked in the browser before they leave, so an explorer persona can fill in a checkout form, but it can't place the order. It can open the review form, but it can't post the review.

This rules out some tests, and we're fine with that. "Can a user place an order" is a question for a test environment you're allowed to dirty. On someone else's staging, the polite and safe answer is to look, not touch. We wrote more about the people who keep these environments alive in our SysAdmin Day post, and not breaking their data is the least we can do.

Find the data, don't hardcode it

Here's a mistake we made in our own early sketches. A test said, more or less, "open product 48213 and check that the price is shown." It worked for a week. Then the catalog was refreshed, 48213 became a discontinued item with no price, and the test started failing for a reason that had nothing to do with any deploy.

Fixed IDs rot. On staging they rot faster, because the data gets refreshed by people who have no idea your test exists.

So we use data finders instead. A finder describes what you need, not which row you want:

findProduct({ inStock: true, minVariants: 2, hasDiscount: true })

At run time, the finder searches the site (or a listing page) for something that matches and hands back whatever it found. If nothing matches, that's reported as "precondition not met," which is an infrastructure-ish outcome, not a test failure. A missing discounted product on staging is not a bug in the deploy.

Metamorphic pairs run back to back

Metamorphic testing checks relations between two queries instead of exact answers. Search "laptop," then search "laptop" with an in-stock filter, and the second result set should be a subset of the first. Sort by lowest price, and the first item shouldn't cost more than the last.

On staging, the source query and the follow-up query run immediately one after the other, in the same session, with nothing scheduled between them. If you run the source at 10:00 and the follow-up at 10:40, a price import in between can break the relation and you'll file a bug that's really just "the data changed." Back to back doesn't make that impossible, but it makes it rare enough that the referee's replays can sort it out.

None of this makes staging less useful. It makes it honest. Staging is very good at telling you whether the logic is right, and pretty bad at telling you whether it's fast. Treating it as fake production gets both of those backwards.

The question we keep asking ourselves as the first real run gets closer: what else are we assuming about staging that we haven't written down yet?