Budgets, not hopes: keeping agent spend predictable

Every step an explorer agent takes is a model call, and every model call costs something. Usually not much. A few cents here, a few cents there.

The problem with "a few cents" is that it compounds in ways nobody plans for. An agent gets stuck in a loop on a modal it can't close. A busy afternoon brings fifteen deploys instead of three. Someone points the system at a site with 40,000 product pages. None of this is exotic, and any one of them can turn a predictable bill into a surprising one.

"We hope it stays reasonable" is not a cost model. So we're building budgets into the core, not bolting them on.

Two hard limits

There are two caps, and both are hard. Not warnings, not soft targets.

Per run. Each deploy gets a fixed amount of agent spend. The planner knows the number before it writes a plan, so it can choose: three explorers going deep on the changed checkout, or six going broad across the areas around it. When the run's budget is used up, the explorers finish their current step and stop.

Per day. A ceiling across all runs. On a quiet day you might never notice it. On a day with twenty deploys, it's what stops the twentieth from costing as much as the first nineteen together. When the daily budget runs out, no more exploration until tomorrow.

Both numbers are set by the team, shown on the dashboard, and checked by code before every single model call. Not after the run, not once an hour. Before each call. If the next step would go over, it doesn't happen.

What keeps running when the money's gone

This is the part we care about most. When the budget runs out, testing doesn't stop. Only exploration does.

The deterministic gate tests, the smoke checks, the regression suite, the universal and business rules on every page, the referee replaying findings: none of them call a model. They keep running on every deploy regardless of what the budget says. A release can still be checked, still be blocked if something's broken, still get a clear answer.

That's a direct consequence of keeping regression free of AI. If the regression suite needed a model, a spent budget would mean an untested deploy. Since it doesn't, a spent budget just means "no new discoveries today," which is a much smaller problem.

Running out is also logged as an infrastructure state, not a failure. Nobody gets a ticket saying the product is broken because we chose to stop spending.

Spend should go down

Here's the trend we're designing toward: the cost per deploy should fall over time.

The reasoning goes like this. When an explorer finds something real, the referee confirms it, and it becomes a Playwright test. From then on, that check runs forever at no model cost. The agent doesn't need to rediscover it next week.

Over months, more and more of what agents used to find by wandering gets covered by plain code. The planner can then send explorers to places that aren't covered yet, and fewer of those remain. Each deploy needs less exploration to reach the same confidence.

If spend per deploy is rising month over month instead, something's off. Maybe findings aren't turning into tests. Maybe agents keep exploring areas that are already covered. Either way, it's a signal worth looking at, and we want it right on the dashboard where it's hard to ignore.

A small worked example

Say a team sets $5 per run and $40 per day. A normal day has six deploys. Each run uses some or all of its $5 on exploration, for a total of at most $30. On a busy release day with twelve deploys, the daily cap kicks in around the eighth run. Deploys nine through twelve still get the full gate and every rule check; they just don't get new explorers.

That trade feels right to us. The money is predictable, and so is the safety net.