Interactions

When a task requires acting on a page — not just reading it — browser-backed runners run an act-observe loop driven by the selected LLM backend. The same loop powers the vendor browser-agents (ChatGPT Agent, Claude Computer Use, Gemini Agent).

Act vs extract

The instruction is classified first. Read-only instructions run the extraction path (one model call over the fetched page). Instructions with an action verb — log in, click, type, submit, download, buy, book — run the act loop.

The loop

Observe: collect the URL, title, trimmed page text, and an indexed list of up to 150 visible interactive elements. Each element gets a stable data-dr-act selector.
Decide: the model returns exactly one action as strict JSON, using only selectors from the element list.
Act:the action runs against the engine's real API; the page settles (network idle), and the action is appended to the run's step trace.

{ "action": "click",  "selector": "[data-dr-act=\"3\"]" }
{ "action": "type",   "selector": "[data-dr-act=\"1\"]", "text": "..." }
{ "action": "press",  "key": "Enter" }
{ "action": "goto",   "url": "https://..." }
{ "action": "done",   "answer": { ... } }
{ "action": "fail",   "reason": "...", "answer": { ... } }

Human input vs bot challenges

Full contract: MFA / OTP. When a target site asks for MFA, 2FA, or an OTP, the run pauses immediately and asks the user for the code. It does not keep trying other runners.

ask_user / requires_action fires only for a value the account owner holds — MFA, OTP, TOTP, SMS or email codes, a security answer, or an explicit confirm. CAPTCHA and other bot-detection gates (reCAPTCHA, hCaptcha, Turnstile, DataDome, Cloudflare challenge, “prove you are human”) are solved, re-routed, or end blocked with reason bot_challenge. They never become a human text prompt. It never emits kind: "text" for a visual bot challenge.

Termination

The loop ends on done (the model proves every required step succeeded), fail (a step was rejected or impossible), a 100-step cap, or a 400-second limit. Anything short of done is reported as a failedrun — the model is instructed never to claim success it can't prove.

Interactions

Act vs extract

The loop

Human input vs bot challenges

Termination

See also

Command Palette