First sketches: a browser, a model and a loop

For months most of our work lived in documents. Coverage maps, oracle lists, rules about who's allowed to judge what. This month we finally started writing the thing that moves: an agent that opens a browser and looks around.

It's rough. It runs on a laptop against a few local test pages we built ourselves. But the shape is starting to feel right, so here's where it stands.

The loop

The core is almost embarrassingly small:

1. Give the model a goal and a short description of the current page.
2. The model picks one tool and its arguments.
3. Our code runs that tool in the browser and records what happened.
4. Feed the result back. Repeat until the goal is done, the step limit is hit, or the budget runs out.

The browser side is a standard automation library. We don't expose the library itself to the model. We wrap it as a few tools, each with a narrow job:

• navigate(url)
• click(target)
• type(target, text)
• read_page() which returns a trimmed, structured view of what's visible
• note_finding(condition, description)

That's about it for now. Scrolling and waiting are on the list.

What the model doesn't get

The model can't run arbitrary code. No "evaluate this JavaScript," no shell, no writing a quick script to scrape the page. It's a conscious choice and probably the most important one in the sketch.

Free-form code execution would make the agent more capable on paper. It would also make every session unpredictable, much harder to replay, and much harder to keep safe. If the agent can only click, type, navigate and read, then everything it did can be described as a list of those actions. That list is the whole point.

Logs a referee can replay

Every step gets written as a structured record, roughly:

{ "step": 7, "tool": "click", "target": "[data-testid=sort-select]",
  "url": "/category/garden", "result": "ok", "duration_ms": 412 }

No prose, no "the agent decided to." Just the action, where it happened, and what came back. The model's reasoning is stored separately, for us to read when debugging, but nothing downstream depends on it.

The test for this format is simple: can a script with no model in it read the log and do the exact same thing in a fresh browser? If yes, the referee can confirm or reject a finding without asking anyone's opinion. We explained why that matters in Why our referee never uses AI.

What we expect to go wrong

We haven't pointed this at a real site yet, and we're not going to pretend the local pages tell us much. But some problems are easy to predict, because every browser automation project runs into them.

Pages that aren't done loading. The model reads the page, sees no products, and concludes the category is empty, when really the product grid arrives 800 ms later via a background request. We'll need a sensible definition of "settled" before read_page() returns.

Selectors that don't exist. The model will ask to click "the sort dropdown" with a target that's close but wrong, or that only exists on desktop. The tool has to fail clearly and say why, not guess.

Timeouts. Something will hang. A slow image, a request that never resolves, a modal waiting for input. Each tool needs its own time limit, and a timeout has to be recorded as a timeout, not as a bug on the site.

Getting lost. Left alone, the model may wander into the footer and read the privacy policy for twenty steps. Step limits help. A planner that says "focus on what changed in this deploy" will help more.

None of these are exciting. They're the kind of thing that eats a week each. We'd rather find them on our own test pages than on someone's staging site.

The next step is making read_page() better. What the model sees decides almost everything about what it does, and right now it sees too much of some things and too little of others.