Computer Use Mcp

computer-use-mcp

> Computer Use MCP is an open source high performance MCP server + client for controlling your desktop computer with AI Agents. Tools include screenshot, mouse, keyboard, clipboard, app management, and window-level targeting — all in-process via Rust NAPI. Easy to install via npm and works with Claude Code, Claude DEsktop, Codex CLi, Codex Desktop, Gemnini Cli, Kiro CLi, Kiro VS Code Extension, Github Copilot, Cursor, OpenCode, OpenClaw, Hermes Agent and any other provider that supports Model Context Protocol.

macOS + Windows + Linux · Node.js 18+ · MIT License

---

Table of Contents

1. [What is this?](#what-is-this)
2. [Tool priority guidance](#tool-priority-guidance)
3. [How it works](#how-it-works)
4. [Architecture](#architecture)
5. [Installation](#installation)
6. [Permissions setup](#permissions-setup)
7. [Quick start](#quick-start)
8. [Using with MCP clients](#using-with-mcp-clients)

• [Codex](#codex)
• [Claude Desktop](#claude-desktop)
• [Cursor](#cursor)
• [Windsurf](#windsurf)
• [Any MCP-compatible client](#any-mcp-compatible-client)

1. [Using as a library](#using-as-a-library)
2. [All tools](#all-tools)
3. [Window targeting and focus strategies](#window-targeting-and-focus-strategies)
4. [API reference](#api-reference)
5. [Building from source](#building-from-source)
6. [Security](#security)
7. [Limitations](#limitations)
8. [Troubleshooting](#troubleshooting)
9. [License](#license)

---

What is this?

computer-use-mcp lets an AI model (or any program) control your computer — take screenshots, move the mouse, type text, press keys, read/write the clipboard, open and manage apps, target specific windows, and query display information.

It works on macOS, Windows, and Linux, with platform-specific native implementations backed by a Rust NAPI module. On macOS it uses CoreGraphics/AppKit/AXUIElement; on Windows it uses SendInput/EnumWindows/IUIAutomation/DXGI — all direct Win32 API calls through Rust, no Python dependencies. On Linux it uses X11/XTest for input synthesis, xdotool/wmctrl for window management, and scrot for screenshots.

It implements the Model Context Protocol (MCP), which means any MCP-compatible AI client (Claude Desktop, Cursor, Windsurf, etc.) can use it as a tool server with zero extra code.

It also ships a typed TypeScript client so you can drive your Mac programmatically from your own scripts.

---

Tool priority guidance

Desktop computer use is powerful but should be your last resort. Prefer more precise tools when they exist:

| Priority | Approach | When to use | |---|---|---| | 1. Connector / integration | Structured APIs, MCP connectors, app-specific plugins | Always prefer if available | | 2. Shell / filesystem | Terminal commands, file I/O, execFile | File operations, CLI tools, system tasks | | 3. Browser automation | Playwright, Puppeteer, Browser Use | Web pages, web apps | | 4. Desktop computer use | This package | Native desktop apps, simulators, installers, modal dialogs, UI-only workflows |

Desktop control is the broadest and slowest fallback. It works for anything on screen, but structured tools are faster, more reliable, and easier to recover from.

---

How it works

Most computer-use tools work by spawning shell commands (xdotool, osascript, cliclick) for every action. This is slow, unreliable, and causes focus-stealing — your terminal window keeps jumping to the front.

This package takes a different approach: a Rust native module (.node addon) that calls OS APIs directly in-process:

macOS:

| What | How | |---|---| | Mouse & keyboard | CGEvent (CoreGraphics) — same API the OS uses | | App management | NSWorkspace (AppKit) | | Window enumeration | CGWindowListCopyWindowInfo (CoreGraphics) — direct FFI, no subprocess | | Window activation | AXUIElement (Accessibility API) — window-level raise | | Display info | CoreGraphics display APIs | | Screenshots | screencapture CLI (fastest reliable method on macOS) | | Clipboard | pbcopy / pbpaste |

Windows:

| What | How | |---|---| | Mouse & keyboard | SendInput (Win32) — direct input synthesis | | App management | EnumWindows + CreateToolhelp32Snapshot | | Window enumeration | EnumWindows + GetWindowText + GetWindowRect | | Window activation | AttachThreadInput + SetForegroundWindow | | Display info | EnumDisplayMonitors + GetDpiForMonitor | | Screenshots | DXGI Desktop Duplication (GPU framebuffer) with GDI BitBlt fallback | | Clipboard | OpenClipboard / SetClipboardData (native Win32) | | UI Automation | IUIAutomation COM (direct vtable calls from Rust) |

Linux:

| What | How | |---|---| | Mouse & keyboard | X11/XTest (X11) or ydotool (Wayland) — direct synthetic events | | Text input | xdotool type (X11) or ydotool type (Wayland) — Unicode support | | App management | /proc filesystem + wmctrl + GNOME Shell D-Bus | | Window enumeration | wmctrl (X11) or GNOME Shell D-Bus Eval (Wayland) | | Window activation | xdotool windowactivate (X11) or GNOME D-Bus (Wayland) | | Display info | X11 XDisplayWidth/XDisplayHeight | | Screenshots | XDG Desktop Portal (GNOME Wayland), grim (wlroots), scrot (X11) | | Clipboard | wl-copy/wl-paste (Wayland) or xclip/xsel (X11) | | Workspaces | wmctrl -d for listing, wmctrl -t for moving windows |

Mouse, keyboard, focus, window enumeration, and display operations run in-process via the native module. Screenshots and clipboard access still rely on the system utilities that are most reliable on macOS, but the control path avoids per-action shell hops.

---

Comparison with alternatives

Feature matrix

| Feature | computer-use-mcp (ours) | CursorTouch/Windows-MCP | sinmb79/windows-computer-mcp | Claude Computer Use | OpenAI CUA | |---|---|---|---|---|---| | Platform | macOS + Windows + Linux | Windows only | Windows only | Linux (Docker) | Linux (Docker) | | Language | Rust NAPI + TypeScript | Python | Python | Python (reference) | Python (reference) | | Protocol | MCP (stdio + in-process) | MCP (stdio) | MCP (stdio) | Claude API built-in | OpenAI API built-in | | Tools | 58 | 14 | 10 | 3 (computer, bash, editor) | 1 (computer) | | Screenshot | DXGI + GDI + PNG/JPEG | dxcam + PIL | mss + PIL | Xvfb screenshot | Xvfb screenshot | | Mouse input | SendInput (Rust) | ctypes→SendInput | pyautogui | xdotool | xdotool | | Keyboard | SendInput Unicode (Rust) | SendKeys wrapper | pyautogui.write | xdotool | xdotool | | UI Automation | IUIAutomation COM (Rust) | comtypes COM | ❌ | ❌ | ❌ | | Clipboard | Native Win32 (Rust) | pywin32 | ❌ | ❌ | ❌ | | Window targeting | HWND + focus strategies | ❌ | title match | ❌ | ❌ | | Virtual desktops | Registry + keyboard | comtypes COM (fragile) | ❌ | ❌ | ❌ | | File system tool | ✅ (Node.js fs) | ✅ (Python os) | ❌ | ✅ (bash) | ❌ | | Registry | ✅ (PowerShell) | ✅ (PowerShell) | ❌ | ❌ | ❌ | | Scripting | PowerShell + AppleScript | PowerShell | ❌ | bash | ❌ | | Zoom (region inspect) | ✅ (full-res crop) | ❌ | ❌ | ✅ (20251124) | ❌ | | Annotation overlay | ✅ (Rust pixel drawing) | ✅ (PIL drawing) | ❌ | ❌ | ❌ | | Grid reference lines | ✅ | ✅ | ❌ | ❌ | ❌ | | Snapshot (combined) | ✅ (screenshot+UI tree+windows) | ✅ (screenshot+elements) | ❌ | ❌ | ❌ | | Multi-select/edit | ✅ | ✅ | ❌ | ❌ | ❌ | | Process management | ✅ | ✅ | ❌ | ✅ (bash) | ❌ | | Notifications | ✅ (toast) | ✅ (toast) | ❌ | ❌ | ❌ | | Scrape (web fetch) | ✅ | ✅ | ❌ | ❌ | ❌ | | Tool guide/discovery | ✅ | ❌ | ❌ | ❌ | ❌ | | Focus strategies | strict/besteffort/none/preparedisplay | ❌ | ❌ | ❌ | ❌ | | Coordinate validation | ✅ | ❌ | ✅ | ❌ | ❌ | | PNG format option | ✅ (quality=0) | ❌ (JPEG only) | ✅ (PNG only) | ❌ | ❌ | | No Python required | ✅ | ❌ (Python 3.13) | ❌ (Python 3.11) | ❌ | ❌ | | Memory footprint | ~120MB | ~180MB | ~150MB | ~200MB+ | ~200MB+ |

Performance benchmark

Measured on Windows Server 2022 (2560×1080, single monitor). All times are median of 15 runs.

| Operation | computer-use-mcp | CursorTouch/Windows-MCP | sinmb79/windows-computer-mcp | Speedup vs CT | |---|---|---|---|---| | Screenshot (800px JPEG) | 17ms | 31ms | - | 1.8× | | Screenshot (800px PNG) | 10ms | - | - | - | | Screenshot (full-res PNG) | 12ms | - | 94ms | 7.6× vs sinmb79 | | Clipboard round-trip | 0.7ms | 22ms | - | 31× | | List windows | 1.3ms | 127ms | 0.6ms | 98× | | Get frontmost app | 0.1ms | 129ms | - | 939× | | List running apps | 0.4ms | 13ms | - | 35× | | Scrape (example.com) | 7ms | 179ms | - | 25× | | PowerShell execution | 213ms | 171ms | - | 0.8× | | Registry list | 244ms | 212ms | - | 0.9× | | Virtual desktops list | 0.2ms | 0.0ms | - | ~same | | Cursor position | 0.1ms | 0.0ms | 0.0ms | ~same |

> PowerShell and registry operations are bottlenecked by subprocess spawn time (~200ms), not our code. Both implementations pay the same cost. All other operations are 2–939× faster due to direct Rust→Win32 API calls vs Python→ctypes/comtypes marshaling.

---

Architecture

Your AI client (Claude, Cursor, etc.)
        │
        │  MCP protocol (JSON-RPC over stdio or in-memory)
        ▼
  MCP Server  (src/server.ts)
  ├── Registers 46 tools with Zod schemas
  ├── Validates all inputs at the boundary
  └── Delegates to Session
        │
        ▼
  Session  (src/session.ts)
  ├── Manages TargetState (bundleId + windowId + provenance)
  ├── Resolves target: target_window_id → target_app → session state
  ├── Applies focus strategy (strict / best_effort / none)
  ├── Returns structured FocusFailure diagnostics on failure
  └── Calls native module
        │
        ▼
  NAPI Native Module  (computer-use-napi.node)
  ├── mouse.rs         — CGEvent mouse events
  ├── keyboard.rs      — CGEvent keyboard events (static keycode map)
  ├── apps.rs          — NSWorkspace app management
  ├── windows.rs       — CGWindowListCopyWindowInfo + AXUIElement window raise
  ├── display.rs       — CoreGraphics display queries
  ├── accessibility.rs — AXUIElement tree walk / find / perform / set / menu (v5)
  ├── spaces.rs        — CGS Space enumeration (read-only, v5)
  └── screenshot.rs    — screencapture + JPEG parsing

Data flow for a window-targeted tool call:

AI sends: { tool: "key", args: { text: "command+v", target_window_id: 12345, focus_strategy: "strict" } }
  → Zod validates parameters
  → Session resolves window 12345 → bundleId "com.apple.iWork.Numbers"
  → Session checks focus_strategy: strict
  → Session confirms frontmost app == "com.apple.iWork.Numbers"
  → Session confirms window 12345 is on-screen
  → If not confirmed: attempt recovery (unhide → activate → raise → poll)
  → If still not confirmed: return FocusFailure with suggestedRecovery
  → If confirmed: Rust keyPress("command+v")
  → Session updates TargetState { bundleId, windowId: 12345, establishedBy: 'keyboard' }
  → Returns: { content: [{ type: "text", text: "Pressed command+v" }] }

---

Installation

Option 1: npx (no install required — recommended)

Run the server directly without installing anything:

npx --yes --prefer-offline @zavora-ai/computer-use-mcp

That's it. Use this path in your MCP client config:

command: npx
args: ["--yes", "--prefer-offline", "@zavora-ai/computer-use-mcp"]

Option 2: npm install

npm install @zavora-ai/computer-use-mcp

Option 3: Build from source

Requires Rust and Cargo.

git clone https://github.com/zavora-ai/computer-use-mcp
cd computer-use-mcp
npm install
npm run build

---

Permissions setup

macOS

macOS requires explicit permission for apps that control the computer. You need to grant Accessibility access to your terminal.

Step-by-step

1. Open System Settings (Apple menu → System Settings)
2. Go to Privacy & Security → Accessibility
3. Click the + button
4. Navigate to your terminal app and add it:

• Terminal.app: /Applications/Utilities/Terminal.app
• iTerm2: /Applications/iTerm.app
• VS Code terminal: /Applications/Visual Studio Code.app
• Cursor: /Applications/Cursor.app

1. Make sure the toggle next to your terminal is on (blue)
2. You may need to restart your terminal after granting permission

> Why is this needed? macOS sandboxes apps from controlling other apps by default. The Accessibility permission grants the ability to send synthetic mouse and keyboard events via CGEvent.

Verifying permissions work (macOS)

Run the built-in demo:

npx @zavora-ai/computer-use-mcp demo

If permissions are correct, you'll see Calculator open, compute 42+58, and close. If you see an error about permissions, revisit the steps above.

Windows

No special permissions are required for most operations on Windows. The native module uses standard Win32 APIs (SendInput, EnumWindows, DXGI) that work without elevation.

Notes:

• UI Automation access may be blocked by UIPI (User Interface Privilege Isolation) when targeting elevated processes. Run your terminal as Administrator if you need to automate elevated apps.
• No Python, pywin32, or any Python dependencies are required — everything is implemented in Rust.

Linux

No special permissions are required beyond having an X11 display available. Ensure the following tools are installed:

sudo apt-get install -y xdotool wmctrl xclip scrot

Notes:

• XTest extension must be enabled (it is by default on most X11 setups).
• For Wayland sessions, you may need to run under XWayland or set GDK_BACKEND=x11.
• No Python dependencies required — the native module uses Rust + X11 directly.

---

Quick start

Run the MCP server

npx --yes --prefer-offline @zavora-ai/computer-use-mcp

The server speaks MCP over stdio and is ready to connect to any MCP client.

Test it works

# Takes a screenshot and saves it
node -e "
import('@zavora-ai/computer-use-mcp').then(async ({ createComputerUseServer }) => {
  const { connectInProcess } = await import('@zavora-ai/computer-use-mcp/client')
  const s = createComputerUseServer()
  const c = await connectInProcess(s)
  const shot = await c.screenshot({ width: 1024 })
  const img = shot.content.find(x => x.type === 'image')
  if (img?.type === 'image') {
    const { writeFileSync } = await import('fs')
    writeFileSync('/tmp/test.jpg', Buffer.from(img.data, 'base64'))
    console.log('Screenshot saved to /tmp/test.jpg')
  }
  await c.close()
})
"

---

Using with MCP clients

Codex

Codex reads MCP server definitions from ~/.codex/config.toml. Add this block:

[mcp_servers.computer-use]
command = "npx"
args = ["--yes", "--prefer-offline", "@zavora-ai/computer-use-mcp"]

If you already use the Codex CLI, you can verify the server is visible with:

codex mcp list

Claude Desktop

1. Find your Claude Desktop config file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

1. Add the server:

{
  "mcpServers": {
    "computer-use": {
      "command": "npx",
      "args": ["--yes", "--prefer-offline", "@zavora-ai/computer-use-mcp"]
    }
  }
}

1. Restart Claude Desktop.

1. You should see a hammer icon (🔨) in the chat input — click it to see the available tools.

1. Try asking Claude: "Take a screenshot and describe what you see" or "Open Safari and navigate to example.com"

---

Cursor

1. Open Cursor settings: Cmd+, → search for "MCP"

1. Click Add MCP Server and fill in:

• Name: computer-use
• Command: npx
• Args: ["--yes", "--prefer-offline", "@zavora-ai/computer-use-mcp"]

1. Save and reload the window (Cmd+Shift+P → "Reload Window").

1. In the AI chat, the comp

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: zavora-ai
• Source: zavora-ai/computer-use-mcp
• License: MIT

Install and usage instructions live in the source repository linked above.