Data Privacy Day: what our test agents are never allowed to see

January 28 is Data Privacy Day, picked because it's the anniversary of the Council of Europe opening its data protection convention for signature back in 1981. Most people mark it by finally reading one cookie banner all the way through. We marked it by writing down the rules our test agents will have to live by.

It felt like the right week. We're still designing Testomatica, which means this is the cheapest moment to decide what the system is not allowed to do. Retrofitting privacy into a tool that already sends screenshots to a model is a lot harder than never sending them in the first place.

The uncomfortable starting point

An exploratory agent works by looking at pages and deciding what to do next. "Looking" means sending page content, and often a screenshot, to a language model. If that page is a customer's order history, you've just shipped someone's name, address and purchase list to a third party, for the sake of a test.

That's not acceptable, and "the staging data is probably fake" isn't a policy. Plenty of staging databases are last month's production copy with the emails left in.

So here are the rules, as they stand today.

Only public pages go to the model

By default, the only pages an agent can show a model are the ones anyone on the internet could see without logging in: the home page, category pages, product pages, search results, the cart before checkout.

Signed-in personas are a different category. A returning customer with saved addresses, a vendor in their dashboard, an admin user: we want to test those flows, but nothing from those sessions goes to a model until the content has been masked. Names, emails, phone numbers, addresses and order numbers get replaced before the model ever sees the page. If masking isn't configured for a site, signed-in exploration simply doesn't run. The deterministic tests still cover those flows, because they don't need a model at all.

Staging is read-only, enforced in the browser

Our agents don't get to change data. Not "they're told not to," but blocked at the browser level: requests that could write data are intercepted and stopped before they leave.

That covers the obvious cases, like placing an order or editing a profile, and some less obvious ones. An agent that's curious about the "delete account" button should be able to find it and look at it. It should never be able to press it and have it work. If a test genuinely needs to write, that's a scripted test with its own fixtures, not an agent improvising.

Secrets stay encrypted, and nobody reads them back

Testing a staging site usually means credentials: basic auth on the staging domain, test account passwords, sometimes an API key. Those are stored encrypted with AES-256-GCM, decrypted only at the moment a browser needs them, and never shown back in the dashboard. You can replace a secret. You can't view it.

They also never go into a prompt. The model doesn't need to know the password to decide that the login page is worth exploring. Code fills in the form.

Everything leaves a trail

Every action the system takes is written to an audit log: which persona visited which page, what was sent to a model, what was blocked, who changed a setting. If someone asks "did your tool ever send our checkout page to an LLM?", the answer should be a query, not a guess.

What this costs us

These rules make the product a little less impressive in a demo. An agent that can wander anywhere, log in as anyone and click every button looks great on a slide. It's also the kind of thing a security reviewer should reject on sight.

We'd rather build the version that passes that review. It's narrower, and it's honest about where the model is allowed to look.

If your team has a rule we haven't thought of, we'd like to hear it while it's still cheap for us to add.