Browsercontrol

BrowserControl

Give your AI agent real browser superpowers. Vision-first browser automation for any MCP-compatible AI agent — local, private, and zero-cost.

What it is • How it looks • Quick start • Connect your AI • Tools • Examples • Why • Contributing

---

✨ What it is

BrowserControl is an MCP server that gives your AI agent a real browser it can see, click, type, and debug — using a vision-first approach based on the Set of Marks (SoM) pattern.

Instead of fragile CSS selectors, XPath, or DOM trees, the agent sees an annotated screenshot with numbered red boxes over every interactive element, then just calls:

click(5)        →  clicks the 5th element
type_text(3, "hello world")  →  types into the 3rd element
upload_file(7, "/path/to/file.pdf")  →  uploads a file via the native browser input

No selectors to guess. No selectors to break when the page changes. Just point at numbers.

---

🖼️ How it looks

Every tool that interacts with the page returns an annotated screenshot plus a textual element map:

┌─────────────────────────────────────────────────┐
│  GitHub                          [Sign in]      │
├─────────────────────────────────────────────────┤
│                                                 │
│   ┌──────────────────────┐    ┌──────────────┐  │
│   │ 1  🔍 Search…        │    │ 2  Pulls     │  │
│   └──────────────────────┘    │ 3  Issues    │  │
│                               │ 4  Codespace │  │
│   ┌───────────────────────┐   └──────────────┘  │
│   │ 5  Sign in            │                     │
│   └───────────────────────┘                     │
└─────────────────────────────────────────────────┘

Found 5 interactive elements:
  [1] input - Search or jump to...
  [2] a - Pulls
  [3] a - Issues
  [4] a - Codespaces
  [5] button - Sign in

The model receives the image and the list, so it can either reason over the pixels or read the text — whichever is cheaper.

---

🚀 Quick start

1 · Install

# pip
pip install browsercontrol

# uv (recommended — much faster)
uv add browsercontrol

> No extra setup needed. Chromium auto-installs on first run. If the auto-install fails for any reason, run python -m playwright install chromium once and you're set.

2 · Run the server

# As a CLI
browsercontrol

# As a module
python -m browsercontrol

# With FastMCP (for `fastmcp dev` / inspector UI)
fastmcp run browsercontrol.server:mcp

3 · Connect it to your AI

Pick your client — full configs in [Connect your AI](#connect-your-ai) below.

// Claude Desktop — claude_desktop_config.json
{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol"
    }
  }
}

Restart Claude Desktop, then try:

> "Open Hacker News and tell me the top story."

That's it — your agent now has a browser.

---

🧠 Why BrowserControl?

A dozen MCP servers can open a web page. Here's what BrowserControl does that most don't:

| | | |---|---| | 🟢 Set of Marks (SoM) — Numbered red boxes on every interactive element. The agent picks a number. No selectors, ever. | 🟢 Shadow DOM + iframe aware — Recursively descends into open shadow roots and same-origin iframes (with proper coordinate offsets). Modern web apps just work. | | 🟢 True persistent sessions — Uses launch_persistent_context. Cookies, localStorage, login state and history all survive restarts. Log in once. | 🟢 Built-in devtools — Console logs, network requests (with timing), JS errors, page performance, element inspection, computed styles. No second tool needed. | | 🟢 Native file uploads — Uses Playwright's set_input_files. The most reliable path through ` — works even when click-and-pick dialogs fail. | 🟢 **Multi-tab orchestration** — Open, switch, close, list tabs. Clicking a link that opens a new tab doesn't lose the agent. | | 🟢 **Smart localhost routing** — Auto-falls-back localhost → 127.0.0.1` when a dev server is up but proxy-resolved DNS fails. | 🟢 100% local & private — No LLM API key, no cloud, no telemetry, no usage cap. Your browsing stays on your machine. | | 🟢 Zero marginal cost — Runs on your hardware, no per-action fees. | 🟢 Cross-platform — CI tests on Ubuntu, Windows, and macOS. |

---

🧩 Connect your AI

BrowserControl speaks MCP stdio, so it works with anything that speaks MCP.

🟣 Claude Desktop

Add to claude_desktop_config.json:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol"
    }
  }
}

✨ Gemini CLI / Google AI Studio

Configure MCP servers in Gemini's settings, or via environment:

export MCP_SERVERS='{"browsercontrol": {"command": "browsercontrol"}}'

For Google AI Studio, configure in the MCP settings panel.

🛠️ Cline (VS Code)

1. Install Cline
2. Open Cline settings → MCP Servers → Add:

{
  "browsercontrol": {
    "command": "browsercontrol"
  }
}

🤖 Continue.dev (VS Code / JetBrains)

Add to ~/.continue/config.json:

{
  "mcpServers": [
    { "name": "browsercontrol", "command": "browsercontrol" }
  ]
}

🎯 Cursor IDE

Cursor Settings → Features → Model Context Protocol → Add:

{ "browsercontrol": { "command": "browsercontrol" } }

⚡ Zed Editor

Add to ~/.config/zed/settings.json:

{
  "context_servers": {
    "browsercontrol": {
      "command": { "path": "browsercontrol" }
    }
  }
}

🐍 Python (programmatic)

import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client

async def main() -> None:
    params = StdioServerParameters(command="browsercontrol", args=[])
    async with stdio_client(params) as (read, write):
        async with ClientSession(read, write) as session:
            await session.initialize()
            tools = await session.list_tools()
            result = await session.call_tool(
                "navigate_to", {"url": "https://news.ycombinator.com"}
            )

asyncio.run(main())

📦 uvx / pipx

{
  "mcpServers": {
    "browsercontrol": { "command": "uvx",  "args": ["browsercontrol"] }
  }
}

{
  "mcpServers": {
    "browsercontrol": { "command": "pipx", "args": ["run", "browsercontrol"] }
  }
}

⚙️ Advanced config (env vars)

Pass environment variables to tune the browser:

{
  "mcpServers": {
    "browsercontrol": {
      "command": "browsercontrol",
      "env": {
        "BROWSER_HEADLESS": "false",
        "BROWSER_VIEWPORT_WIDTH": "1920",
        "BROWSER_VIEWPORT_HEIGHT": "1080",
        "LOG_LEVEL": "DEBUG"
      }
    }
  }
}

See [Configuration](#configuration) for the full list.

---

🛠️ Tools

Every action returns an annotated screenshot + element map (when relevant), so the model always has the latest visual context.

Navigation

| Tool | Description | |---|---| | navigate_to(url) | Open a URL. Auto-falls back to 127.0.0.1 for localhost if the proxy blocks it. | | go_back() / go_forward() / refresh_page() | Standard history controls. | | scroll(direction, amount) | direction ∈ {up, down, left, right}. amount ∈ {small, medium, large, page, top, bottom} or raw pixels (e.g. "500"). |

Interaction

| Tool | Description | |---|---| | click(element_id) | Click by SoM number. Resolves the actual DOM element at the coordinates first, so overlays don't fool it. | | click_at(x, y) | Click raw coordinates. | | type_text(element_id, text) | Uses element.fill() for reliable form input — not char-by-char typing. | | press_key(key) | Any keyboard key: Enter, Tab, Escape, ArrowDown, Backspace, etc. | | hover(element_id) | Hover for tooltips/menus. | | scroll_to_element(element_id) | Scroll the element into view. | | wait(seconds) | Sleep — for animations or lazy-loaded content. |

Tabs

| Tool | Description | |---|---| | create_tab(url=None) | Open a new tab, optionally navigated to a URL. | | switch_tab(index) | Switch by 0-based index. | | close_tab(index) | Close a tab. Auto-falls-back to another open page. | | list_tabs() | List all tabs with title, URL, active marker. |

Forms

| Tool | Description | |---|---| | select_option(element_id, option) | Pick from a ` dropdown by visible text or value. | | checkcheckbox(elementid, check=True) | Toggle a checkbox. | | uploadfile(elementid, filepath) | **Native** file upload via setinput_files — works on every site that has an `. |

Content

| Tool | Description | |---|---| | get_page_content() | Whole page as Markdown (script/style stripped, 30 KB cap). | | get_text(element_id) | Read text from a specific element. | | get_page_info() | Current URL + title. | | run_javascript(script) | Run JS, return serialized result + screenshot. | | screenshot(annotate=True, full_page=False) | Annotated viewport, clean viewport, or clean full-page. |

DevTools

| Tool | Description | |---|---| | get_console_logs(clear=False) | Last 50 captured console messages, with level + source location. | | get_network_requests(num_requests=30, clear=False) | Method, URL, status, duration (ms). Requests paired by identity (no URL-collision bugs). | | get_page_errors() | Uncaught JS exceptions with stack traces. | | run_in_console(code) | Eval JS with structured error handling; returns JSON-stringified result + screenshot. | | inspect_element(element_id) | Computed styles, dimensions, attributes, classes, href, src, value. | | get_page_performance() | TTFB, FCP, DOMContentLoaded, load time, resource count, JS heap. | | get_cookies() / set_cookie(name, value, domain=None, path="/") | Full cookie control. domain is auto-inferred from current URL when omitted. | | delete_cookie(name) / clear_cookies() | Targeted or total cookie wipe. | | set_viewport(width, height) | Change viewport on the fly (great for responsive testing). |

Recording

| Tool | Description | |---|---| | start_recording(name="") | Begin a Playwright trace (screenshots + DOM snapshots + sources). | | stop_recording() | Save the trace to ~/.browsercontrol/recordings/. | | take_snapshot(name="") | Save PNG + HTML + URL triplet to ~/.browsercontrol/snapshots/. | | list_recordings() | List recent recordings and snapshots. |

> View recordings with: npx playwright show-trace ~/.browsercontrol/recordings/.zip

---

⚙️ Configuration

All settings are environment variables. Sensible defaults out of the box.

| Variable | Default | Description | |---|---|---| | BROWSER_HEADLESS | true | Run without a visible window. Set false to watch the browser. | | BROWSER_VIEWPORT_WIDTH | 1280 | Viewport width in pixels. | | BROWSER_VIEWPORT_HEIGHT | 720 | Viewport height in pixels. | | BROWSER_TIMEOUT | 30000 | Navigation timeout (ms). | | BROWSER_USER_DATA_DIR | ~/.browsercontrol/user_data | Browser profile dir (cookies, history, extensions persist here). | | BROWSER_EXTENSION_PATH | — | Path to a .crx/unpacked extension to load at startup. | | LOG_LEVEL | INFO | DEBUG / INFO / WARNING / ERROR. |

# Headful + phone-sized viewport for mobile testing
BROWSER_HEADLESS=false BROWSER_VIEWPORT_WIDTH=375 BROWSER_VIEWPORT_HEIGHT=812 browsercontrol

# Verbose logs for debugging the MCP server itself
LOG_LEVEL=DEBUG browsercontrol

---

📚 Examples

🔍 Web research

> "Find the Wikipedia article on Python and tell me who created it."

→ navigate_to("https://wikipedia.org")
→ type_text(2, "Python programming language")
→ press_key("Enter")
→ get_page_content()
→ "Python was created by Guido van Rossum, first released in 1991."

🐛 Debug a web app

> "Check localhost:3000 for any errors and tell me what's broken."

→ navigate_to("http://localhost:3000")    # auto-fallback to 127.0.0.1 if needed
→ get_console_logs()
→ "2 errors:
   [ERROR] Uncaught TypeError: Cannot read property 'map' of undefined (app.js:42)
   [ERROR] Failed to load resource: 404 /api/users"
→ get_network_requests()
→ "GET /api/users -> 404 (12ms)  ← check your API route"

🧪 Record a test run

> "Walk through the login flow on staging.example.com while recording."

→ start_recording("login_flow")
→ navigate_to("https://staging.example.com/login")
→ type_text(3, "user@example.com")
→ type_text(4, "hunter2")
→ click(5)             # Sign In button
→ take_snapshot("after_login")
→ stop_recording()
→ "Saved ~/.browsercontrol/recordings/login_flow.zip"

📝 Fill out a form

> "Fill out the contact form at example.com/contact with a polite message."

→ navigate_to("https://example.com/contact")
→ type_text(2, "Alex Doe")
→ type_text(3, "alex@example.com")
→ type_text(4, "Hi! Just saying hello.")
→ click(5)             # Submit

🗂️ Multi-tab research

> "Open GitHub in one tab and Hacker News in another, then tell me the top repo and the top story."

→ create_tab("https://github.com/trending")
→ create_tab("https://news.ycombinator.com")
→ list_tabs()          # → returns both tabs with indices
→ switch_tab(0)        # focus GitHub
→ get_page_content()
→ switch_tab(1)        # focus HN
→ get_page_content()

📂 Upload a file

> "Upload my resume to the job application form."

→ navigate_to("https://jobs.example.com/apply")
→ upload_file(8, "/Users/me/Documents/resume.pdf")

📱 Mobile responsive check

> "Switch to iPhone 14 viewport and screenshot the homepage."

→ set_viewport(390, 844)
→ navigate_to("https://my-site.com")
→ screenshot(annotate=False, full_page=True)

---

🏗️ Architecture

┌─────────────────┐     ┌────────────────────┐     ┌─────────────┐
│   AI Agent      │────▶│  BrowserControl    │────▶│  Chromium   │
│ (Claude, Gemini,│◀────│   MCP Server       │◀────│ (Playwright)│
│  GPT, local…)   │     │                    │     │             │
└─────────────────┘     └────────────────────┘     └─────────────┘
        │                        │                       │
        │  call_tool("click",5)  │  page.click(handle)    │
        │  ◀─────────────────────│  ◀─────────────────────│
        │  annotated PNG +       │  screenshot.png        │
        │  element map           │  + SoM overlay         │

Action loop

1. Agent calls a tool — e.g. click(5).
2. Server resolves the element — Looks up element 5 from the last SoM pass, then calls document.elementFromPoint(x, y) to get the actual DOM element (handles overlays, sticky headers, and shadow boundaries).
3. Browser executes — Playwright drives the real Chromium.
4. Server re-screenshots — New screenshot, fresh element detection (with shadow DOM + iframe recursion), numbered boxes drawn with Pillow.
5. Annotated result returns — Both image and textual map are sent back so the model can pick its preferred representation.

Why SoM over selectors?

• No DOM drift — If the site gets redesigned, the numbered elements still point at the right pixels.
• No hallucinated selectors — The model never invents div.flex-container > button.btn-primary:nth-child(3).
• Tiny tokens — A list of {"id": 7, "tag": "button", "text": "Submit"} is far cheaper than a full DOM tree.
• Vision + text — The model can use either or both. Vision for visual reasoning, text for fast selection.

---

🆚 How it compares

A few notes, kept honest:

| Capability | BrowserControl | Playwright MCP | Stagehand / Browser-Use | AgentQL | |---|:-:|:-:|:-:|:-:| | Set of Marks (numbered boxes) | ✅ | ❌ text-tree onl

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: adityasasidhar
• Source: adityasasidhar/browsercontrol
• License: MIT

Install and usage instructions live in the source repository linked above.