Web Agent Bench

🧭 web-agent-bench

A fair, reproducible benchmark for the different ways to give an LLM a web browser. Measured on token usage, speed, and reliability, with one identical Claude brain driving every approach.

[](./LICENSE) [](https://nodejs.org) [](https://www.python.org)

---

The contenders

Five locally-runnable drivers across four tool methodologies. agent-browser is run in two harnesses (a minimal API loop and the Claude Code Agent SDK) to isolate what the harness itself costs:

| Tool | Methodology | Agent loop | What it feeds the model | |---|---|---|---| | 🎭 Playwright MCP | accessibility tree | Claude Agent SDK (MCP) | a11y snapshot + refs | | ⚡ agent-browser (thin) | accessibility refs | minimal API loop (Bash + CLI) | snapshot → @e1 refs | | ⚡ agent-browser (Claude Code) | accessibility refs | Claude Agent SDK (Bash + CLI) | snapshot → @e1 refs | | 🐍 browser-use | hybrid DOM + vision | its own Python loop | DOM elements + a screenshot | | 🎯 Stagehand | NL primitives | its own loop (act/observe/extract) | structured, cache-able actions |

They live at different layers, which is exactly what makes a naïve comparison misleading. This repo fixes the model and varies the browser tooling, and for agent-browser the harness around the same tool.

Fairness model

The diagram above is the whole idea: one Claude model, several sets of "hands," one local site.

• Playwright MCP runs on the Claude Agent SDK (the MCP server is its tool layer).
• agent-browser (thin) runs on a minimal API loop (src/drivers/thinLoop.ts) that drives the CLI via a single Bash tool. It carries only the task prompt, a short CLI guide, and the tool schema (no Claude Code preset), so its token counts reflect the browser work rather than a large built-in system prompt re-read every turn.
• agent-browser-cc runs the same agent-browser CLI on the Claude Code Agent SDK, with the same guide. It's the A/B partner to the thin driver: identical tool layer, only the harness differs, so it isolates what the heavy harness costs (and whether it buys anything).
• browser-use and Stagehand bring their own loop.
• All drivers share the same model and account: the API-loop and own-loop drivers (agent-browser thin, browser-use, Stagehand) point their Anthropic client at a tiny local proxy (src/proxy/oauthProxy.ts); the Agent SDK drivers (Playwright MCP, agent-browser-cc) use the logged-in claude CLI.
• Every cost is re-priced uniformly from raw token counts (each tool reports cost differently, or not at all). See src/core/pricing.ts.
• Every driver gets the same hands within the browser: UI actions only (snapshot/click/type/scroll/screenshot). Shell, in-page JS eval, and direct network/file access are blocked, so no driver can fetch an answer instead of doing the task.

Quickstart

Prerequisites: Node ≥ 22, Python ≥ 3.13 + uv, Claude Code logged in (no ANTHROPIC_API_KEY needed), macOS or Linux.

npm run setup          # Node + Python deps, browsers, agent-browser CLI
claude setup-token     # one-time, opens browser login, prints sk-ant-oat01-...
cp .env.example .env   # paste that token into .env
npm run list           # see every suite, scenario, driver, model
npm run bench          # run the default suite

What npm run setup does

npm install
npx playwright install chromium
uv venv --python 3.13 .venv && source .venv/bin/activate
uv pip install -r requirements.txt
python -m playwright install chromium
npm install -g agent-browser && agent-browser install

Authentication (all on your Claude subscription)

• Playwright MCP and agent-browser-cc authenticate via the logged-in claude CLI the Agent SDK spawns.
• agent-browser (thin) / browser-use / Stagehand use CLAUDE_CODE_OAUTH_TOKEN (from claude setup-token, in .env); the local proxy injects the Claude Code identity + beta headers so the subscription accepts the request. Verify any time with npm run proxy:check.

Usage

npm run bench                                      # all drivers, all scenarios, 5 reps, opus
npm run bench -- --suite=scroll --model=sonnet     # one suite, on Sonnet
npm run bench -- --suite=hard --drivers=agent-browser,stagehand --reps=5

| Flag | Default | Values | |---|---|---| | --suite | controlled | controlled, scroll, dynamic, hard, auth, vision | | --drivers | all five | playwright-mcp, agent-browser, agent-browser-cc, browser-use, stagehand | | --scenarios | all in suite | scenario ids (npm run list) | | --model | opus | opus, sonnet, haiku, or a full model id | | --reps | 5 | integer |

> For a full run, use scripts/clean-run.sh: it sweeps all suites and schedules the two agent-browser drivers in separate waves (they share one machine-wide daemon and can't run concurrently).

Each run lands in results/runs/// (raw per-rep JSON), where ` defaults to _`. Cost is normalized at the chosen model's pricing.

Test suites

Each suite isolates one place where the approaches diverge. Success is checked deterministically: a scenario only reveals its gated code after the correct interaction happens, and every gated answer is generated at runtime (never a literal in the source), so an agent can't pass by guessing or by reading the benchmark's own source.

| Suite | What it tests | The question it answers | |---|---|---| | controlled | login, filter, gated todos | baseline: token/speed floor on easy tasks | | scroll | below-the-fold + load-more | viewport-cap (must scroll) vs one-shot full-tree snapshot | | dynamic | value / job that finishes after a delay | waiting / polling / re-checking behavior | | hard | validated checkout wizard, paginated search with decoys, decoy-modal trap (all behind a cookie-consent overlay) | reliability under genuinely tricky, failure-prone flows | | auth | login works, but a 2FA wall blocks the vault | every driver fails: fresh login can't satisfy MFA → why you need pre-baked sessions | | vision | a code and a colour-marker rendered as server-side images (pixels only) | isolates vision-capable drivers from DOM-only ones |

Every fixture is a small React app served from memory (esbuild-bundled, fully offline). Gated answer codes are generated at runtime and released only after the genuine interaction, behind session-gated endpoints (cookies, per-session refs, sequential paging) or, for vision, rendered into pixels, so they can't be scraped from the DOM, the JS bundle, or a direct API call.

Results

Every run streams per-rep progress to the console and writes raw per-rep JSON to results/runs///: one file per rep, plus a raw.json with all results and per-driver aggregates.

Regenerate the comparison plots for any run with:

python py/plots.py                  # latest run (PNGs land in /analysis/plots/)
python py/plots.py --run      # a specific run

The numbers below are one full run on claude-sonnet-4-6 (5 reps per scenario, ~300 tasks). Pass rates are a near tie (95 to 100 percent for every driver), so the interesting differences are in how much context each approach carries.

Tokens per browser step (median + IQR). The same agent-browser CLI costs 2.7k tokens per step on a minimal API loop and 35k inside Claude Code; Playwright MCP carries the full accessibility snapshot plus tool schemas on every call:

Same CLI, two harnesses (tokens per task, median + IQR). Identical tool layer and commands; only the harness differs:

Per suite (median tokens per task, log scale). The gap is task-shaped: huge on extraction-style suites (controlled, dynamic, vision), narrower where page content dominates everyone's context (scroll, hard):

Cost note: tokens, not dollars. Most of the heavy drivers' volume is cache reads, which are nearly free on a subscription, so the billed gap is much smaller than the token gap.

Metrics

| Metric | Meaning | |---|---| | Reliability | success rate over N reps (the least-confounded signal) | | Tool calls / turns | browser actions / LLM round-trips; fewer = more batching | | Duration | wall-clock per task (median & p95) | | Tokens | fresh input + cache reads + output the model processed | | Cost | normalized from tokens at the run's model pricing (a uniform cross-tool comparison, not your subscription bill) |

Extending

Add a suite: create src/suites//{site.ts, scenarios.ts, index.ts} and register it in src/suites/registry.ts.

Add a driver: implement the Driver interface (src/core/types.ts) and register it in src/drivers/index.ts. Tool layers can reuse makeAgentSdkDriver (Agent SDK) or runThinLoop (minimal API loop).

Acknowledgements

This benchmark measures these projects; all credit for the tools themselves goes to their authors:

• Playwright MCP by Microsoft
• agent-browser by Vercel Labs
• browser-use by Browser Use
• Stagehand by Browserbase
• Claude Agent SDK by Anthropic

License

[MIT](./LICENSE) © 2026 Nicola Sosio

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: nsosio
• Source: nsosio/web-agent-bench
• License: MIT

Install and usage instructions live in the source repository linked above.