Opendesk

opendesk

Give any AI agent eyes and hands on your desktop.

Opendesk is a computer use framework that lets AI agents navigate your computer just like a human would — screenshots, mouse, keyboard, UI interaction, OCR, workflow recording, scheduling, and remote machine control.

macOS · Linux · Windows

[](https://pypi.org/project/opendesk/) [](https://www.npmjs.com/package/@vitalops/opendesk-sdk) [](LICENSE) [](https://vitalops.github.io/opendesk/docs/)

---

https://github.com/user-attachments/assets/5a6fab31-9f53-4ddb-9efb-17f0afe97844

Single Machine DemoScreenshot, click, type, navigate — all from natural language.

https://github.com/user-attachments/assets/629cf31b-12ab-4913-bc03-963f1cfbd682

Control Multiple MachinesDrive remote desktops over encrypted WebSocket — same tools, same agent.

https://github.com/user-attachments/assets/659c9e30-e8f6-4a5a-ab81-0fa7ccaf8fb8

UI TestingTest a full web app — navigate, create, verify — with zero selectors.

---

SDKs

| Language | Location | Package | Install | |----------|----------|---------|---------| | Python | [python/](python/) | opendesk (PyPI) | pip install 'opendesk[core,mcp]' | | JavaScript / TypeScript | [js/](js/) | @vitalops/opendesk-sdk (npm) | npm install @vitalops/opendesk-sdk |

More SDKs can be added to this repo following the same pattern.

---

MCP install

opendesk works as an MCP server with any MCP-compatible client — Claude Code, Claude Desktop, Cursor, Windsurf, Continue, or any custom tool.

Python

pip install 'opendesk[core,mcp]'
opendesk install        # shortcut for Claude Code

> Requires Python 3.10+

JavaScript / TypeScript

npm install @vitalops/opendesk-sdk
npx opendesk-js install        # shortcut for Claude Code

Other MCP clients (Cursor, Windsurf, Continue, custom)

Point your client at the opendesk-mcp binary:

{
  "mcpServers": {
    "opendesk": { "command": "opendesk-mcp" }
  }
}

For JS:

{
  "mcpServers": {
    "opendesk": {
      "command": "node",
      "args": ["/path/to/node_modules/@vitalops/opendesk-sdk/bin/opendesk-mcp.js"]
    }
  }
}

Once connected, try:

Take a screenshot of my screen
Click the Chrome icon
Open Spotify and play lo-fi beats
Show me the audit log
Replay everything from this session

---

SDK usage

Use opendesk programmatically in your own agent or app.

Python

from opendesk import create_registry, allow_all_context

registry = create_registry()
ctx = allow_all_context()

result = await registry.get("screenshot").execute(ctx, ...)

JavaScript / TypeScript

import { OpenDeskClient } from "@vitalops/opendesk-sdk";

const client = new OpenDeskClient();
await client.screenshot({ marks: true });
await client.ui({ action: "click", app: "Safari", title: "Go" });

---

Architecture

opendesk is built in independently-importable layers:

┌──────────────────────────────────────────────────────────────┐
│  Integrations   MCP  ·  Claude Code  ·  OpenAI  ·  LangChain │
├──────────────────────────────────────────────────────────────┤
│  Tools          screenshot · mouse · keyboard · ui ·         │
│                 clipboard · ocr · learn · schedule · audit   │
├──────────────────────────────────────────────────────────────┤
│  Computer       LocalComputer  ·  RemoteComputer  (ABC)      │
├──────────────────────────────────────────────────────────────┤
│  Remote         server · client · discovery (mDNS)           │
├──────────────────────────────────────────────────────────────┤
│  Protocol       frames · codec (msgpack) · peer · transports │
│                 auth (X25519 + AEAD, pairing)                │
└──────────────────────────────────────────────────────────────┘

| Layer | What it does | |-------|-------------| | Computer | The capability surface of a computer (observe / act / subscribe). LocalComputer drives the local machine; RemoteComputer forwards every call over the wire to a paired peer. Tools and integrations target this ABC — they never know whether the machine is local or remote. | | Tools | One class per capability, agent-friendly Pydantic schemas. Calls into the active Computer on the ToolContext. | | Integrations | Thin adapters for MCP, Anthropic, OpenAI, LangChain — add one tool, get all four. | | Remote | opendesk serve / opendesk pair, mDNS discovery, client helper. | | Protocol | Five-frame wire protocol (msgpack binary, no base64 ever), WebSocket transport, mutual X25519 + ChaCha20-Poly1305 auth and encryption. | | Automation | learn + schedule backed by pynput recording, JSON storage, APScheduler daemon. |

Full details → [docs/architecture.md](docs/architecture.md)

---

Tools

| Tool | What it does | |------|-------------| | screenshot | Capture the screen with numbered boxes on every clickable element (Set-of-Marks) | | ui | Click and type by element name — no coordinates needed | | mouse | Pixel-level mouse control for anything ui can't reach | | keyboard | Type text, press keys, send hotkeys | | app | Open, close, and focus applications | | clipboard | Read and write the system clipboard | | ocr | Extract text from any region of the screen | | learn | Record a workflow once, replay it anytime | | schedule | Run any task or learned procedure on a timer |

Full reference → [docs/tools.md](docs/tools.md)

---

Automation

Record a task once, replay it forever, or put it on a schedule.

Record

"Start recording task expense-form"

Perform the workflow yourself. The agent captures every click, keystroke, and screenshot.

Replay

"Stop recording"
"Replay expense-form"

The agent re-executes using the current screen state — no hardcoded coordinates.

Schedule

"Every morning at 9am, open my email in Chrome, take a screenshot, and summarize what's there"
"Schedule expense-form every friday at 5pm"

opendesk scheduler start

Supported timing: every 30m · every 2h · every day at 09:00 · every friday at 17:00 · raw cron

Full guide → [docs/automation.md](docs/automation.md)

---

Remote computer use

Control another machine from your agent — same tools, same MCP server, the Computer abstraction just lives on the other end of an encrypted WebSocket.

On the machine being controlled (one time):

pip install 'opendesk[core,remote]'
opendesk pair        # prints a 6-digit code, listens

On the controller (one time):

pip install 'opendesk[remote]'
opendesk discover                          # list opendesk peers on the LAN
opendesk pair-with   --name mini

After pairing, the controlled machine runs the long-lived server:

opendesk serve            # accepts paired peers only

…and the controller drives it through the existing MCP server (Claude Code, Claude Desktop, Cursor — anything that speaks MCP). The agent gets new admin tools — opendesk_peers, opendesk_use, opendesk_status — and every existing tool accepts an optional peer: argument:

screenshot                       → controls the local machine
screenshot peer=mini             → controls the paired remote
opendesk_use mini                → make mini the default for this session
screenshot                       → [on mini] ...

With exactly one paired peer the agent doesn't have to specify anything — it becomes the implicit default. With multiple, the agent must pick explicitly (no silent fallback).

One controller at a time. Pair as many machines as you like, but only one drives the desktop at a time — a second peer trying to connect while one is active gets a clean BUSY error. Same peer reconnecting bumps the previous session (no waiting out a stale TCP). Two ways to free the slot from the controlled machine:

• opendesk disconnect — cooperative. Server asks the controller to

leave via a session.evicted PUSH; a cooperative client (the in-tree RemoteComputer) suppresses its auto-reconnect and raises SessionEvicted. Trust is preserved.

• opendesk unpair — enforced. Revokes trust + closes the

session; next reconnect fails authentication.

Security model: pairing exchanges long-lived X25519 keypairs via a 6-digit code-authenticated handshake (PBKDF2-stretched, ~CPU-month to brute force). Subsequent connections use mutual static-key authentication. Every frame is ChaCha20-Poly1305 AEAD-encrypted with per-direction counters. No CA-signed certificates required — the keys ARE the trust.

Full guide → [docs/remote.md](docs/remote.md)

---

Installation options

pip install opendesk                              # core framework only
pip install 'opendesk[core,mcp]'                  # + screen capture + MCP server (recommended)
pip install 'opendesk[core,mcp,remote]'           # + control another machine over LAN
pip install 'opendesk[core,mcp,learn]'            # + task recording and replay
pip install 'opendesk[core,mcp,learn,schedule]'   # + scheduled tasks
pip install 'opendesk[all]'                       # everything

---

Platform support

| Feature | macOS | Linux | Windows | |---------|:-----:|:-----:|:-------:| | Screenshot | ✓ | ✓ | ✓ | | Mouse & keyboard | ✓ | ✓ | ✓ | | UI element access | AppleScript | AT-SPI2 | UI Automation | | Clipboard | pbcopy/pbpaste | xclip/xsel | pyperclip | | OCR | Vision / tesseract | tesseract | WinRT / tesseract | | App control | open -a | xdg-open | start | | Task recording | ✓ | ✓ | ✓ | | Scheduled tasks | ✓ | ✓ | ✓ | | Remote control (LAN) | ✓ | ✓ | ✓ | | LAN discovery (mDNS) | ✓ | ✓ | ✓ |

---

System permissions

macOS

• System Settings → Privacy & Security → Screen Recording — enable for your terminal
• System Settings → Privacy & Security → Accessibility — enable for mouse/keyboard control

Linux

sudo apt install xclip xdotool python3-atspi

Windows

No extra permissions needed — opendesk uses Win32 APIs by default.

See [docs/permissions.md](docs/permissions.md) for full setup guide.

---

Integrations

Claude Code

opendesk install        # registers opendesk-mcp globally
opendesk uninstall      # removes the registration

Claude Desktop

Add to your config file:

• macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json
• Linux: ~/.config/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "opendesk": { "command": "opendesk-mcp" }
  }
}

Python API

import asyncio
from opendesk import create_registry, allow_all_context

async def main():
    registry = create_registry()
    ctx = allow_all_context()

    result = await registry.get("screenshot").execute(
        ctx, registry.get("screenshot").Params(marks=True)
    )
    print(result.output)

asyncio.run(main())

Works with Anthropic SDK, OpenAI, and LangChain — see [docs/integrations.md](docs/integrations.md)

On-device models (Ollama, LM Studio, vLLM, llama.cpp)

Any OpenAI-compatible local server works out of the box:

from openai import OpenAI
from opendesk.integrations.openai_compat import OpenAIAdapter

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
adapter = OpenAIAdapter()
result = await adapter.run_loop(client, model="qwen2.5:72b", messages=messages)

---

Citation

If you use opendesk in your research or project, please cite it:

@software{opendesk,
  author  = {Abraham, Abhigith Neil and Rahman, Fariz and Rahman, Fadil},
  title   = {opendesk: Open Desktop Automation Framework},
  year    = {2026},
  url     = {https://github.com/vitalops/opendesk},
  version = {0.2.0},
  license = {MIT}
}

A CITATION.cff is included — GitHub's "Cite this repository" button will pick it up automatically.

---

License

MIT

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: vitalops
• Source: vitalops/opendesk
• License: MIT
• Homepage: https://vitalops.ai/opendesk/docs/

Install and usage instructions live in the source repository linked above.