Clawdcursor

Clawd Cursor

Safe desktop control for any AI agent. Compiles the screen into a UI map and acts on elements by stable id (screenshot/vision only as a last resort), verifies its own actions, and gates everything through one safety checkpoint. Local · cross-OS · any model.

Quickstart · Why it's different · The engine · How it works · Tools · Platforms · Changelog

---

What it is

Clawd Cursor is a local MCP server that gives any tool-calling agent — Claude Code, Cursor, Windsurf, OpenClaw, the Claude Agent SDK, or your own loop — safe control of the real desktop. It clicks, types, reads the screen, opens apps, and drives any GUI the way a human would: native apps, the browser, even a canvas.

Most "let an agent use the computer" tools take a screenshot and feed it to a vision model — slow, expensive, and brittle. Clawd Cursor compiles the screen into one UI map: it fuses the accessibility tree and OCR into a confidence-scored set of elements, each tagged with a stable el_NN id, and acts on elements by id — not pixel coordinates. Coordinates appear only in the last-resort screenshot/vision tier (live pixels off the current frame), for canvas-only apps or tasks that genuinely need spatial reasoning. The result is cheaper, faster, private, and — uniquely — it checks that each action actually did what it claimed.

> If a human can do it on a screen, your agent can too. No API, no integration, no problem — only the right sequence of reads, clicks, keys, and waits. Use it as the last-mile fallback: native API exists? Use it. CLI? Use it. Clawd Cursor is for the click, the legacy app, the GUI with no public surface.

---

Why it's different

The desktop-agent space is crowded. The closest install-and-go peers are Windows-MCP and Terminator (desktop MCP servers); browser-only tools (browser-use, Playwright MCP) are adjacent; and OmniParser / UI-TARS are vision-centric parsing approaches you'd build an agent around, not products you install. Here's the honest comparison across those approaches — what Clawd Cursor does that the popular options don't:

| | Clawd Cursor | browser-use | Playwright MCP | OmniParser / UI-TARS | computer-use | |------------------------------------------|:-----------------------:|:-----------:|:--------------:|:--------------------:|:------------:| | Any desktop app, not just the web | ✅ | web only | web only | ✅ | ✅ | | Cross-OS (Windows + macOS + Linux) | ✅ | — | — | varies | sandbox | | Perception without a vision model | ✅ compiled a11y + OCR map | DOM | a11y tree | ❌ vision-centric | ❌ vision | | Verifies its own actions (deviation) | ✅ | — | — | — | — | | Single safety chokepoint (allow/confirm/block) | ✅ | — | — | — | — | | Any model / vendor | ✅ | ✅ | not an agent | model-specific | Claude only | | MCP-native (one config, any host) | ✅ | library | test framework | — | tool-use API | | Local-only, no cloud required | ✅ | ✅ | ✅ | needs a model | screens → cloud |

Three things here are genuinely rare:

1. Cheapest-tier-first perception, fully local. Accessibility tree (free) → OCR (cheap) → screenshot (expensive — the only tier that puts pixels in the model's context; "screenshot" and "vision" are the same step). The agent climbs only when it must, so token cost tracks task difficulty — and with a local model, nothing leaves the machine. Vision-centric agents (OmniParser, UI-TARS) need a screenshot in the model for every observation.
2. It verifies. Pass expect on a consequential action and Clawd Cursor re-checks the live screen (with a short settle window for async UIs) and reports a DEVIATION instead of a hollow "success." A completed task can't be marked done on evidence that was already true before it acted.
3. One safety gate. Every call — from an editor over stdio, an external agent over HTTP, or the built-in loop — routes through a single safety.evaluate() chokepoint (allow / confirm / block) before it touches the desktop. The agent cannot bypass it.

Plus: an on-screen "desktop control in progress" banner with a blinking red dot whenever an agent is driving — double-click it to stop. A human at the machine always knows, and always has a kill switch.

---

Install

clawdcursor is an MCP server published to npm — install it into any MCP-capable agent (Claude Code, Claude Desktop, Cursor, Windsurf, Zed, OpenAI Codex, or your own loop) the same way you install any other MCP server.

1 — Install the engine + grant consent (once)

npm i -g clawdcursor
clawdcursor consent --accept    # one-time desktop-control consent (required)
clawdcursor grant               # macOS only — approve Accessibility + Screen Recording

> Zero-install also works — swap clawdcursor for npx -y clawdcursor in any > snippet below and npx fetches it on demand. A global install is recommended > anyway: it's pinnable and inspectable on disk (safer for a tool with full desktop > control than auto-fetching latest every run), and it's the path on which the macOS > native helper builds at install time. Requires Node.js 20+.

> Per-OS prerequisites. Windows installs clean — sharp and @nut-tree-fork/nut-js > ship prebuilt binaries, so no C++/Python build tools are needed. macOS needs Xcode > Command Line Tools (xcode-select --install) for screenshots / vision; core > accessibility-driven control still works without them. Linux needs a few system > packages npm can't install: tesseract-ocr (OCR), python3-gi + gir1.2-atspi-2.0 > (accessibility tree), and — on Wayland — ydotool (synthetic input).

2 — Add it to your agent (pick your host)

Claude Code

claude mcp add clawdcursor -s user -- clawdcursor mcp --compact

OpenAI Codex — add to ~/.codex/config.toml:

[mcp_servers.clawdcursor]
command = "clawdcursor"
args = ["mcp", "--compact"]

Cursor / Windsurf / Zed / Claude Desktop — add to the host's MCP config:

{
  "mcpServers": {
    "clawdcursor": { "command": "clawdcursor", "args": ["mcp", "--compact"] }
  }
}

That's the whole setup. Ask your agent: "open Outlook and reply to the latest email from Sarah."

Or: one-command plugin (Claude Code)

Skip the manual config — this repo ships a plugin that registers the tools and bundles the usage skill in one step. It resolves the package's bin (never a hard-coded dist/ path), so an upgrade can't break it:

claude plugin marketplace add AmrDab/clawdcursor
claude plugin install clawdcursor@clawdcursor

One-line installers (clone + build; handles the macOS native build)

# Windows (PowerShell)
powershell -c "irm https://clawdcursor.com/install.ps1 | iex"

# macOS / Linux
curl -fsSL https://clawdcursor.com/install.sh | bash

> Notes. You never run clawdcursor mcp yourself — the host spawns it over stdio > on demand. clawdcursor doctor is not part of MCP setup; it only configures the > built-in LLM for the autonomous agent daemon. On macOS, Accessibility is > required (primary control path); Screen Recording is optional (vision fallback > only). For editor permission allowlists, use the server-level wildcard > mcp__clawdcursor rather than per-tool entries — it survives tool renames.

---

The engine

The perception + verification core (the UI State Compiler, since v1.5.0):

• compile_ui fuses the accessibility tree and OCR into one confidence-scored map of the screen, every element tagged with a stable el_NN id. Act on an element by {element_id, snapshot_id} instead of pixels — near-free in tokens, and it survives DPI, resize, and layout shifts. find_button / find_field locate a target by meaning and hand you the id.
• Reactive verification. expect on an action → Clawd Cursor confirms the outcome on the live screen and returns a DEVIATION when the UI didn't obey.
• Cross-platform parity. The compiler, secure-field redaction, and coordinate handling run on Windows, macOS, and Linux; the external-agent (MCP) surface resolves el_NN refs through the safety gate and discloses when it attached to your existing browser.

> Set-of-Mark-style element IDs and a11y/OCR fusion aren't new ideas on their own — what's rare is doing them locally, a11y-first (no vision model required), with a built-in verification gate and one safety chokepoint, across three operating systems, behind a single MCP config.

See the [changelog](CHANGELOG.md) for the full release history, or the latest release.

---

How it works

Where the brain lives decides how you run it. Both modes can run side-by-side.

| Brain lives… | Mode | Command | What you call | |---|---|---|---| | In your editor (Claude Code, Cursor, Windsurf, Zed) | Direct tools | clawdcursor mcp | Each tool, via stdio MCP | | In a headless agent with its own LLM (OpenClaw, Agent SDK, your loop) | Direct tools | clawdcursor agent --no-llm | Same, over HTTP MCP | | Inside Clawd Cursor itself (scheduled / "submit and walk away") | Thin agent loop | clawdcursor agent + doctor-configured LLM | task / submit_task | | External brain that delegates sub-tasks to the built-in loop | Direct + delegation | clawdcursor agent + your client | task({instruction:…}) to hand off |

The loop

Read the a11y tree (cheap) → act on named targets → verify from fresh observations → escalate perception only when needed (OCR → screenshot, the one tier that sends pixels to the model). Sparse a11y tree? system.detect_webview switches Electron/WebView2 apps to browser.* over CDP. Canvas-only (Paint, Figma, games)? Screenshot + coordinate click.

flowchart TB
    task["User task"] --> loop["Agent LLM loopplans · chooses tools · verifies"]
    loop --> observe{"Cheapest observationthat answers the question"}

    observe -- "obs·a11y — free" --> a11y["A11y tree(structured text + el_NN handles)"]
    observe -- "obs·ocr — cheap" --> ocr["OCR (OS-level, no vision LLM)"]
    observe -- "obs·dom — medium" --> dom["Browser DOM (CDP)"]
    observe -- "obs·vision — expensive" --> vision["Screenshot (image into context)"]

    a11y --> act
    ocr --> act
    dom --> act
    vision --> act

    act["Actclick/type/key/drag · invoke/set_value · open_app · batch"] --> safety
    safety["Single safety gatesafety.evaluate() → allow / confirm / block"] -- allowed --> tools["Tool registry98 granular + 7 compound"]
    safety -- needs user --> confirm["Human confirmation"] --> tools
    safety -- denied --> blocked["blocked"]

    tools --> desktop["Real desktop"]
    desktop --> verify{"expect → does state match?"}
    verify -- pass --> done["done"]
    verify -- "DEVIATION" --> loop

    classDef agentNode fill:#dbeafe,stroke:#2563eb,color:#0f172a;
    classDef gate fill:#ede9fe,stroke:#7c3aed,color:#0f172a;
    classDef obsNode fill:#fef9c3,stroke:#ca8a04,color:#0f172a;
    classDef actNode fill:#ffedd5,stroke:#ea580c,color:#0f172a;
    classDef stop fill:#fee2e2,stroke:#dc2626,color:#0f172a;
    class loop,verify agentNode;
    class safety,confirm,tools gate;
    class observe,a11y,ocr,dom,vision obsNode;
    class act actNode;
    class blocked stop;

batch for deterministic stretches. When the next N steps are known, collapse them into one call — each step still routes through the safety gate; on any guard miss or error the batch halts with a per-step trace.

Task delegation. With an LLM configured on the daemon, an external agent can hand off at any point: task({"instruction":"…"}). The built-in loop takes the wheel and reports back — offload grunt work to a cheaper model without burning your own context.

---

The toolbox

Two catalogs ship side-by-side. The toolbox is 7 compound tools, each with an action enum covering ~10–20 verbs (~1,500 tokens total — about 12× smaller than granular, the computer_20250124 shape editor hosts already know). The granular surface is the 98 underlying primitives, one schema per verb (for runtimes that need top-level tools, or for debugging). Both run through the same safety.evaluate() chokepoint; the full catalog is always visible via MCP tools/list.

| Toolbox | Actions | |---|---| | computer | screenshot, click, double_click, right_click, triple_click, hover, move, scroll, scroll_horizontal, drag, drag_path, type, key, wait | | accessibility | read_tree, find, get_element, focused, invoke, focus, set_value, get_value, expand, collapse, toggle, select, state, list_children, wait_for, compile_ui, find_button, find_field, smart_click, smart_type, smart_read | | window | list, active, focus, maximize, minimize, restore, close, resize, list_displays, screen_size, open_app, open_file, open_url, switch_tab, navigate | | system | clipboard_read, clipboard_write, system_time, ocr, undo, shortcuts_list, shortcuts_run, delegate, detect_webview, relaunch_with_cdp, system_prompt, build_uri, open_uri, open_app, open_file, open_url, detect_app, app_guide, learn_app | | browser | connect, page_context, read_text, click, type, select_option, evaluate, wait_for, list_tabs, switch_tab, scroll | | task | run (default; bounded-sync — waits up to timeouts, returns {status:"running"} + progress if longer, re-call to keep waiting), status, abort. Delegates to the built-in loop. Requires clawdcursor agent with an LLM. | | batch | {steps:[…]} — collapse N calls into one round-trip; each step {name, arguments, expect?}, re-perceived and safety-gated, halts with a trace on any miss. |

computer({ action: "key", combo: "mod+s" })          // Cmd+S / Ctrl+S, resolved per-OS
accessibility({ action: "invoke", name: "Send" })    // click by name, not pixels
window({ action: "open_app", name: "Outlook" })
task({ instruction: "open Notepad and type hello" }) // hand off to the thin loop

---

Cheapest-tier-first perception

Every observation has a cost. Start at the cheapest rung that works; climb only when it fails. The live log (CLAWD_LOG=pretty, default on a TTY) shows the ladder in real time via per-call badges.

| Tier | Badge | Cost | Source | When | |---|---|---|---|---| | T1 structured | obs·a11y | ~free | accessibility.*, window.*, browser.read_text, clipboard | Default. Text + bounds, no image, no vision LLM. | | T2 OCR | obs·ocr | cheap | system.ocr, smart_read / smart_click / smart_type | A11y tree empty/sparse. OS-level text out, no image bytes. | | T3 DOM | obs·dom | medium | browser.read_text / page_context (CDP) | WebView / Electron / Chrome content. | | T4 screenshot (vision) | obs·vision | expensive | computer.screenshot | The only tier that puts pixels in the model's context. Canvas-only apps or spatial reasoning. Last resort. |

Acting tools log act. Watching `obs·a11y → act → obs·a11y

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: AmrDab
• Source: AmrDab/clawdcursor
• License: MIT
• Homepage: https://clawdcursor.com

Install and usage instructions live in the source repository linked above.