# Desktop Pilot Mcp > Universal macOS app automation MCP server โ€” 30-100x faster than screenshot-based computer-use. Uses Accessibility API, AppleScript, and CGEvent. - **Type:** MCP server - **Install:** `agentstack add mcp-versoxbt-desktop-pilot-mcp` - **Verified:** Yes โ€” security-reviewed for prompt injection and unsafe behavior - **Seller:** [VersoXBT](https://agentstack.voostack.com/s/versoxbt) - **Installs:** 0 - **Category:** [Integrations](https://agentstack.voostack.com/c/integrations) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [VersoXBT](https://github.com/VersoXBT) - **Source:** https://github.com/VersoXBT/desktop-pilot-mcp - **Website:** https://github.com/VersoXBT/desktop-pilot-mcp#readme ## Install ```sh agentstack add mcp-versoxbt-desktop-pilot-mcp ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About Desktop Pilot MCP Native macOS automation for Claude. 30-100x faster than screenshots. --- Desktop Pilot is an MCP server that gives Claude direct access to any macOS application through the Accessibility API, AppleScript, and CGEvent -- no screenshots, no pixel coordinates, no vision model overhead. It reads the actual UI tree and acts on semantic element references, the same way Playwright works for browsers. **One snapshot of Telegram takes 20ms and returns structured data. The same operation with screenshot-based computer-use takes ~3 seconds and returns pixels.** ``` pilot_snapshot { "app": "Telegram" } [e1] Window "Saved Messages" [e2] MenuButton "Main menu" [e3] Button "All chats (111 unread chats)" [e7] Button "Code (4 unread chats)" [e18] TextField "Write a message..." [e20] Button "Record Voice Message" pilot_click { "ref": "e18" } // focus the text field pilot_type { "ref": "e18", "text": "Hello from Claude" } pilot_click { "ref": "e20" } // send ``` No coordinates. No screenshots. No guessing. Just refs. --- ## Quick Start (2 minutes) ```bash npx desktop-pilot-mcp ``` **Step 1.** Add to your Claude config and restart Claude: For **Claude Code**, add to `~/.claude.json` under your project's `mcpServers`: ```json { "desktop-pilot": { "command": "npx", "args": ["-y", "desktop-pilot-mcp"] } } ``` For **Claude Desktop**, add to `~/Library/Application Support/Claude/claude_desktop_config.json`: ```json { "mcpServers": { "desktop-pilot": { "command": "npx", "args": ["-y", "desktop-pilot-mcp"] } } } ``` **Step 2.** Grant Accessibility permission when macOS prompts you (one-time). If the prompt doesn't appear: **System Settings > Privacy & Security > Accessibility** -- add your terminal app or Claude Desktop. **Step 3.** Ask Claude to interact with any app: > "Take a snapshot of Telegram and show me what's on screen" That's it. No API keys, no accounts, no configuration files. Alternative: build from source Requires Swift 6.0+ (included with Xcode 16+). ```bash git clone https://github.com/VersoXBT/desktop-pilot-mcp.git cd desktop-pilot-mcp swift build -c release ``` Then use the binary path directly in your Claude config: ```json { "desktop-pilot": { "command": "/absolute/path/to/desktop-pilot-mcp/.build/release/desktop-pilot-mcp", "args": [] } } ``` --- ## Benchmarks Real measurements from testing against Telegram, Finder, and other macOS apps: | Operation | computer-use (screenshots) | Desktop Pilot | Speedup | |-----------|---------------------------|---------------|---------| | Snapshot (read full UI tree) | ~3000ms | 20ms | **150x** | | Snapshot (Finder, 45 elements) | ~3000ms | 78ms | **38x** | | Click element | ~3000ms | ~50ms | **60x** | | Read element value | ~3000ms | Save As...", "app": "TextEdit" } ``` | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `path` | string | Yes | Menu path with ` > ` separator. | | `app` | string | No | App name or bundle ID. Omit for frontmost. | --- ### `pilot_script` Run AppleScript or JXA (JavaScript for Automation) code targeting a specific app. ```json { "app": "Finder", "code": "tell application \"Finder\" to get name of every window", "language": "applescript" } ``` | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `app` | string | Yes | Target app name. | | `code` | string | Yes | AppleScript or JXA code to execute. | | `language` | string | No | `applescript` (default) or `jxa`. | --- ### `pilot_screenshot` Capture a screenshot of a specific element or the full screen. Returns base64 PNG. Use sparingly -- `pilot_snapshot` is usually better for understanding UI state. ```json { "ref": "e1" } ``` | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `ref` | string | No | Element ref to screenshot. Omit for full screen. | --- ### `pilot_batch` Execute multiple tool calls in sequence within a single MCP round-trip. Use to reduce latency when performing multi-step actions. ```json { "actions": [ { "tool": "pilot_click", "params": { "ref": "e18" } }, { "tool": "pilot_type", "params": { "ref": "e18", "text": "Hello" } }, { "tool": "pilot_click", "params": { "ref": "e20" } } ] } ``` | Parameter | Type | Required | Description | |-----------|------|----------|-------------| | `actions` | array | Yes | Array of `{ tool, params }` objects to execute in order. | --- ### `pilot_list_apps` List all running macOS applications with their names, bundle IDs, PIDs, and window counts. Use to discover available apps before taking a snapshot. ```json {} ``` No parameters required. --- ## Architecture ``` Sources/ DesktopPilot/ Core/ AppRegistry.swift # App discovery via NSWorkspace ElementStore.swift # Actor-based ref-to-element mapping Router.swift # Smart per-app, per-action routing Snapshot.swift # Batch AX tree traversal Layers/ LayerProtocol.swift # InteractionLayer protocol AccessibilityLayer.swift # AXUIElement tree reading + actions AppleScriptLayer.swift # System Events + sdef scripting CGEventLayer.swift # Raw keyboard/mouse injection ScreenshotLayer.swift # Screen capture fallback MCP/ Server.swift # JSON-RPC 2.0 with Content-Length framing Tools.swift # 10 tool definitions + dispatch Types.swift # PilotElement, AppSnapshot, AppInfo Platform/ PlatformProtocol.swift # Cross-platform bridge interface macOS/ AXBridge.swift # Low-level AXUIElement C API wrapper Permissions.swift # Accessibility permission management SystemEvents.swift # AppleScript/JXA execution helper DesktopPilotCLI/ main.swift # Entry point: permission check + server start Tests/ DesktopPilotTests/ DesktopPilotTests.swift # 22 tests: types, router, registry, MCP, tools ``` **Key design decisions:** - **Zero dependencies.** The entire server is built on Apple frameworks only (ApplicationServices, AppKit, CoreGraphics). No SwiftNIO, no Vapor, no third-party JSON library. This keeps the binary at 427KB. - **Actor-based element store.** Refs are ephemeral -- they reset on each snapshot. The `ElementStore` actor guarantees thread-safe access to the AXUIElement-to-ref mapping across concurrent tool calls. - **Content-Length framing.** The MCP server uses the standard JSON-RPC 2.0 protocol with `Content-Length` header framing over stdin/stdout, matching the MCP specification exactly. - **Batch attribute reading.** Instead of N individual AXUIElementCopyAttributeValue calls per element, the snapshot builder uses `AXUIElementCopyMultipleAttributeValues` to read 6 attributes in a single call. This is why snapshots are fast. --- ## Supported Apps Desktop Pilot works with any macOS application that exposes an accessibility tree (which is virtually all of them): | Category | Examples | Primary Layer | |----------|----------|---------------| | Apple native | Finder, Safari, Mail, Notes, Calendar, Music | AppleScript + Accessibility | | Productivity | Microsoft Office, Google Chrome, Firefox | Accessibility | | Electron | VS Code, Discord, Slack, Spotify, Signal | Accessibility | | Creative | Final Cut Pro, Logic Pro, Xcode | AppleScript + Accessibility | | Communication | Telegram, iMessage, WhatsApp | Accessibility | | System | System Settings, Activity Monitor, Terminal | Accessibility | --- ## Use Cases - **Automate any macOS workflow** -- file management, app configuration, data entry across apps - **Build AI agents** that operate native desktop applications Claude can't reach through web APIs - **Test macOS apps** by driving the UI through structured element refs instead of fragile pixel coordinates - **Cross-app orchestration** -- copy data from one app, process it, paste into another, all in a single Claude session - **Accessibility auditing** -- inspect the full UI tree of any app to verify accessibility compliance --- ## Troubleshooting **"Accessibility permission not granted"** Open System Settings > Privacy & Security > Accessibility and add the binary or your terminal app. Restart the MCP server after granting. **"Failed to capture screenshot"** Grant Screen Recording permission in System Settings > Privacy & Security > Screen Recording. Required only for `pilot_screenshot`. **Stale refs (`Unknown ref 'e5'`)** Refs reset on every `pilot_snapshot` call. Always take a fresh snapshot before interacting with elements. If an app's UI has changed since the last snapshot, the old refs are invalid. **Electron apps not responding to `pilot_type`** Some Electron apps (VS Code, Discord) swallow raw key events. The router handles this by using Accessibility (AXSetValue) instead of CGEvent for Electron apps. If typing still fails, try `pilot_script` with a System Events keystroke. **Empty snapshots** The app may not have any open windows, or it may use a non-standard UI framework (games, OpenGL/Metal renderers). Use `pilot_screenshot` as a fallback for custom-rendered content. --- ## Development ```bash # Build debug swift build # Build release swift build -c release # Run tests swift test # Run the server directly swift run desktop-pilot-mcp ``` The project is split into a library target (`DesktopPilot`) and an executable target (`DesktopPilotCLI`) for testability. All core logic lives in the library; the CLI is a thin entry point. --- ## License MIT ## Contributors VersoXBT๐Ÿ’ป ๐Ÿ“– Claude๐Ÿค– ๐Ÿ’ก ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source โ€” we do not rehost the code. - **Author:** [VersoXBT](https://github.com/VersoXBT) - **Source:** [VersoXBT/desktop-pilot-mcp](https://github.com/VersoXBT/desktop-pilot-mcp) - **License:** MIT - **Homepage:** https://github.com/VersoXBT/desktop-pilot-mcp#readme Install and usage instructions live in the source repository linked above. ## Pricing - **Free** โ€” Free ## Security capabilities Automated source analysis of v0.1.0 โ€” what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source โ€” more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** โ€” security scan: passed โ€” Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-versoxbt-desktop-pilot-mcp - Seller: https://agentstack.voostack.com/s/versoxbt - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack โ€” the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.