Desktop Pilot Mcp

Desktop Pilot MCP Native macOS automation for Claude. 30-100x faster than screenshots.

---

Desktop Pilot is an MCP server that gives Claude direct access to any macOS application through the Accessibility API, AppleScript, and CGEvent -- no screenshots, no pixel coordinates, no vision model overhead. It reads the actual UI tree and acts on semantic element references, the same way Playwright works for browsers.

One snapshot of Telegram takes 20ms and returns structured data. The same operation with screenshot-based computer-use takes ~3 seconds and returns pixels.

pilot_snapshot { "app": "Telegram" }

[e1] Window "Saved Messages"
  [e2] MenuButton "Main menu"
  [e3] Button "All chats (111 unread chats)"
  [e7] Button "Code (4 unread chats)"
  [e18] TextField "Write a message..."
  [e20] Button "Record Voice Message"

pilot_click { "ref": "e18" }       // focus the text field
pilot_type  { "ref": "e18", "text": "Hello from Claude" }
pilot_click { "ref": "e20" }       // send

No coordinates. No screenshots. No guessing. Just refs.

---

Quick Start (2 minutes)

npx desktop-pilot-mcp

Step 1. Add to your Claude config and restart Claude:

For Claude Code, add to ~/.claude.json under your project's mcpServers:

{
  "desktop-pilot": {
    "command": "npx",
    "args": ["-y", "desktop-pilot-mcp"]
  }
}

For Claude Desktop, add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "desktop-pilot": {
      "command": "npx",
      "args": ["-y", "desktop-pilot-mcp"]
    }
  }
}

Step 2. Grant Accessibility permission when macOS prompts you (one-time).

If the prompt doesn't appear: System Settings > Privacy & Security > Accessibility -- add your terminal app or Claude Desktop.

Step 3. Ask Claude to interact with any app:

> "Take a snapshot of Telegram and show me what's on screen"

That's it. No API keys, no accounts, no configuration files.

Alternative: build from source

Requires Swift 6.0+ (included with Xcode 16+).

git clone https://github.com/VersoXBT/desktop-pilot-mcp.git
cd desktop-pilot-mcp
swift build -c release

Then use the binary path directly in your Claude config:

{
  "desktop-pilot": {
    "command": "/absolute/path/to/desktop-pilot-mcp/.build/release/desktop-pilot-mcp",
    "args": []
  }
}

---

Benchmarks

Real measurements from testing against Telegram, Finder, and other macOS apps:

| Operation | computer-use (screenshots) | Desktop Pilot | Speedup | |-----------|---------------------------|---------------|---------| | Snapshot (read full UI tree) | ~3000ms | 20ms | 150x | | Snapshot (Finder, 45 elements) | ~3000ms | 78ms | 38x | | Click element | ~3000ms | ~50ms | 60x | | Read element value | ~3000ms | Save As...", "app": "TextEdit" }

| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `path` | string | Yes | Menu path with ` > ` separator. |
| `app` | string | No | App name or bundle ID. Omit for frontmost. |

---

### `pilot_script`

Run AppleScript or JXA (JavaScript for Automation) code targeting a specific app.

```json
{
  "app": "Finder",
  "code": "tell application \"Finder\" to get name of every window",
  "language": "applescript"
}

| Parameter | Type | Required | Description | |-----------|------|----------|-------------| | app | string | Yes | Target app name. | | code | string | Yes | AppleScript or JXA code to execute. | | language | string | No | applescript (default) or jxa. |

---

pilot_screenshot

Capture a screenshot of a specific element or the full screen. Returns base64 PNG. Use sparingly -- pilot_snapshot is usually better for understanding UI state.

{ "ref": "e1" }

| Parameter | Type | Required | Description | |-----------|------|----------|-------------| | ref | string | No | Element ref to screenshot. Omit for full screen. |

---

pilot_batch

Execute multiple tool calls in sequence within a single MCP round-trip. Use to reduce latency when performing multi-step actions.

{
  "actions": [
    { "tool": "pilot_click", "params": { "ref": "e18" } },
    { "tool": "pilot_type",  "params": { "ref": "e18", "text": "Hello" } },
    { "tool": "pilot_click", "params": { "ref": "e20" } }
  ]
}

| Parameter | Type | Required | Description | |-----------|------|----------|-------------| | actions | array | Yes | Array of { tool, params } objects to execute in order. |

---

pilot_list_apps

List all running macOS applications with their names, bundle IDs, PIDs, and window counts. Use to discover available apps before taking a snapshot.

{}

No parameters required.

---

Architecture

Sources/
  DesktopPilot/
    Core/
      AppRegistry.swift     # App discovery via NSWorkspace
      ElementStore.swift    # Actor-based ref-to-element mapping
      Router.swift          # Smart per-app, per-action routing
      Snapshot.swift        # Batch AX tree traversal
    Layers/
      LayerProtocol.swift   # InteractionLayer protocol
      AccessibilityLayer.swift  # AXUIElement tree reading + actions
      AppleScriptLayer.swift    # System Events + sdef scripting
      CGEventLayer.swift        # Raw keyboard/mouse injection
      ScreenshotLayer.swift     # Screen capture fallback
    MCP/
      Server.swift          # JSON-RPC 2.0 with Content-Length framing
      Tools.swift           # 10 tool definitions + dispatch
      Types.swift           # PilotElement, AppSnapshot, AppInfo
    Platform/
      PlatformProtocol.swift    # Cross-platform bridge interface
      macOS/
        AXBridge.swift          # Low-level AXUIElement C API wrapper
        Permissions.swift       # Accessibility permission management
        SystemEvents.swift      # AppleScript/JXA execution helper
  DesktopPilotCLI/
    main.swift              # Entry point: permission check + server start
Tests/
  DesktopPilotTests/
    DesktopPilotTests.swift # 22 tests: types, router, registry, MCP, tools

Key design decisions:

• Zero dependencies. The entire server is built on Apple frameworks only (ApplicationServices, AppKit, CoreGraphics). No SwiftNIO, no Vapor, no third-party JSON library. This keeps the binary at 427KB.
• Actor-based element store. Refs are ephemeral -- they reset on each snapshot. The ElementStore actor guarantees thread-safe access to the AXUIElement-to-ref mapping across concurrent tool calls.
• Content-Length framing. The MCP server uses the standard JSON-RPC 2.0 protocol with Content-Length header framing over stdin/stdout, matching the MCP specification exactly.
• Batch attribute reading. Instead of N individual AXUIElementCopyAttributeValue calls per element, the snapshot builder uses AXUIElementCopyMultipleAttributeValues to read 6 attributes in a single call. This is why snapshots are fast.

---

Supported Apps

Desktop Pilot works with any macOS application that exposes an accessibility tree (which is virtually all of them):

| Category | Examples | Primary Layer | |----------|----------|---------------| | Apple native | Finder, Safari, Mail, Notes, Calendar, Music | AppleScript + Accessibility | | Productivity | Microsoft Office, Google Chrome, Firefox | Accessibility | | Electron | VS Code, Discord, Slack, Spotify, Signal | Accessibility | | Creative | Final Cut Pro, Logic Pro, Xcode | AppleScript + Accessibility | | Communication | Telegram, iMessage, WhatsApp | Accessibility | | System | System Settings, Activity Monitor, Terminal | Accessibility |

---

Use Cases

• Automate any macOS workflow -- file management, app configuration, data entry across apps
• Build AI agents that operate native desktop applications Claude can't reach through web APIs
• Test macOS apps by driving the UI through structured element refs instead of fragile pixel coordinates
• Cross-app orchestration -- copy data from one app, process it, paste into another, all in a single Claude session
• Accessibility auditing -- inspect the full UI tree of any app to verify accessibility compliance

---

Troubleshooting

"Accessibility permission not granted" Open System Settings > Privacy & Security > Accessibility and add the binary or your terminal app. Restart the MCP server after granting.

"Failed to capture screenshot" Grant Screen Recording permission in System Settings > Privacy & Security > Screen Recording. Required only for pilot_screenshot.

Stale refs (Unknown ref 'e5') Refs reset on every pilot_snapshot call. Always take a fresh snapshot before interacting with elements. If an app's UI has changed since the last snapshot, the old refs are invalid.

Electron apps not responding to pilot_type Some Electron apps (VS Code, Discord) swallow raw key events. The router handles this by using Accessibility (AXSetValue) instead of CGEvent for Electron apps. If typing still fails, try pilot_script with a System Events keystroke.

Empty snapshots The app may not have any open windows, or it may use a non-standard UI framework (games, OpenGL/Metal renderers). Use pilot_screenshot as a fallback for custom-rendered content.

---

Development

# Build debug
swift build

# Build release
swift build -c release

# Run tests
swift test

# Run the server directly
swift run desktop-pilot-mcp

The project is split into a library target (DesktopPilot) and an executable target (DesktopPilotCLI) for testability. All core logic lives in the library; the CLI is a thin entry point.

---

License

MIT

Contributors

VersoXBT💻 📖 Claude🤖 💡

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: VersoXBT
• Source: VersoXBT/desktop-pilot-mcp
• License: MIT
• Homepage: https://github.com/VersoXBT/desktop-pilot-mcp#readme

Install and usage instructions live in the source repository linked above.