The release gate: when a test system is allowed to say stop

Letting a tool block your production deploy is a big ask. Most teams have been burned by a red pipeline that turned out to be a timeout, and after the third time, someone adds || true to the step and nobody talks about it again.

So we've been careful with the release gate. It's the part of Testomatica with the most authority, and authority without good reasons is just friction.

One question, three answers

When a production deploy starts, it asks Testomatica a single question about the build it's about to ship. There are three possible answers:

• Go. Nothing we know about should stop this.
• Stop. One or more stop conditions are true, and here they are.
• Human decision. We can't give a clean go or stop, so a person needs to look.

That third answer is the one most tools leave out, and it's the one we care about most. More on that below.

When the answer is stop

We kept the list of stop conditions short and specific. Each one is something code can check, and each one comes with the evidence that triggered it.

1. An open critical bug against this build or the code it touches. If checkout returns a 500 on staging and the ticket is still open, shipping anyway is a choice someone should make on purpose.
2. A failing gate test, unless that failure is already known and accepted. Gate tests are the ones that earned their place through a long stable streak in quarantine, so a failure there is a strong signal. A failure that's been triaged, linked to a ticket and explicitly accepted doesn't block again on every deploy.
3. Low change coverage. If the diff touches pricing logic and almost none of the risk items connected to pricing were exercised on this build, we don't know enough to say go.
4. Heavy-risk items not checked. Our risk score is for ordering work, not for skipping it. If the highest-risk items for this release weren't touched at all, that's a stop.

What's not on the list matters too. An agent-only suspect doesn't stop a release. An approval-queue finding with a middling confidence score doesn't stop a release. Agents explore and propose. They never block alone.

Flakes would make this unbearable

A quick bit of arithmetic shaped how gate tests run. Say the gate has 150 tests and each one has a 1% chance of failing for no real reason. Run them with no retry and the chance that at least one flakes is about 78%. Most of your deploys would be blocked by noise.

Give each failing test one retry and the chance of a false block drops to about 1.5%. That's the difference between a gate people trust and a gate people bypass. A retry that passes is still recorded as evidence, so a test that keeps needing it gets noticed and sent back to quarantine.

Override, with a reason

Sometimes the right call is to ship anyway. A critical bug might be in a feature that's behind a flag nobody can see yet. A heavy-risk item might have been checked by hand at 11 p.m. and the person who did it is confident.

So an operator can override a stop. The only requirement is a written reason, and that reason goes to the audit log along with who overrode, when, and which conditions were true at the time. Not a checkbox. A sentence.

We don't think this is bureaucracy. It's the opposite of || true. The override exists so that nobody needs to sneak around the gate, and the written reason is what makes the next postmortem short.

When the system can't answer

Here's the case we spent the most time on. What happens if Testomatica is down, the staging run timed out, the evidence store is unreachable, or the gate tests didn't finish?

The default is a human decision. Never a silent pass.

A test system that returns "go" when it doesn't know is worse than having no test system, because it tells everyone the build was checked when it wasn't. If we can't answer, the deploy pauses and asks a person, with a plain explanation of what's missing: "gate run incomplete, 41 of 150 tests did not execute due to staging timeouts."

We're getting ready to point the whole flow at a real staging site soon, and the gate will be the last piece to get real authority. It'll start by recommending, not blocking. Trust, like a gate test, should have to earn its place.