Open Compute

open-compute

EN | [DE](README_de.md)

[](CHANGELOG.md) [](pyproject.toml) [](https://github.com/ellmos-ai/open-compute/actions/workflows/tests.yml) [](tests) [](llms.txt) [](https://github.com/ellmos-ai) [](CHANGELOG.md) [](LICENSE)

A model-agnostic computer-use core: one agent loop, any reasoning model behind a single interface.

open-compute is a small, dependency-light Python core for building computer-use agents (LLM-driven GUI / desktop / browser automation). It implements the perception → model-tool-call → action → feedback loop and keeps the reasoning model swappable behind a single ComputerBackend interface. No provider is privileged: Anthropic Claude and OpenAI CUA are two equally-ranked API backends, and the offline mock backend is the default. A keyless path also exists today via Mode A, where the host model itself reasons — and it can run that loop either inline or in a self-spawned subagent for context economy (see [usage pattern](#usage-pattern--inline-a-vs-self-subagent-b)). The core has zero runtime dependencies; vendor SDKs (anthropic, openai) are optional, lazily imported extras — import open_compute works with none of them installed, and the default mock wiring runs fully offline.

> [!NOTE] > AI / LLM Integration Notice: open-compute includes a machine-readable [llms.txt](llms.txt) file designed for AI agents, RAG crawlers, and LLM-assisted workflows.

---

Why

Every computer-use model — Anthropic's Claude computer tool and OpenAI's computer-use tool — shares the same agent-loop shape but differs in transport, coordinate frame, and action names. open-compute factors out the common parts so you write the loop once and swap the reasoning model freely behind one ComputerBackend interface:

• A canonical action schema with one mapper per backend.
• Normalized (0..1) coordinates internally, denormalized per backend /

resolution / DPI in one tested utility — the DPI problem solved centrally.

• A central safety gate ("confirm before risky actions") evaluated before

every action.

• A hybrid perception interface (screenshot + Set-of-Marks / accessibility /

DOM), so you can move from pure pixel-vision to semantic targeting later.

---

Architecture

                        +-----------------------------------------+
                        |        AGENT LOOP / ORCHESTRATOR        |
                        |  goal -> perceive -> backend -> safety  |
                        |        -> execute -> re-perceive        |
                        +-------------------+---------------------+
                                            |
        +-----------------------------------+-----------------------------------+
        |                                   |                                   |
+-------v---------+              +----------v-----------+            +----------v----------+
| PERCEPTION      |              | CANONICAL ACTIONS    |            | SAFETY / POLICY     |
| - screenshot    |              | click/type/key/      |            | - confirm-at-action |
| - set-of-marks  |              | scroll/drag/wait/    |            | - allow / deny list |
|   (OmniParser)* |              | screenshot + OS ext  |            | - read-only mode    |
| - accessibility*|              | (launch/activate)    |            | - audit log         |
+-------+---------+              +----------+-----------+            +----------+----------+
        |                                   |                                   |
        +-----------------+-----------------+----------------------------------+
                          |
              +-----------v------------+   COORDINATE / DPI NORMALIZATION
              | BACKEND ABSTRACTION    |   - internal: normalized (0..1)
              | (ComputerBackend)      |   - denormalize per backend:
              +-----+--------+---------+     * Claude: global px (display_w x display_h)
                    |        |    |          * OpenAI: px (computer_call)
        +-----------+        |    +-----------+   * Mock: synthetic
        |                    |                |
+-------v-------+   +--------v-------+  +-----v---------+
| Claude        |   | OpenAI CUA     |  | Mock backend  |
| computer_2025 |   | computer-use-  |  | (no SDK,      |
| 1124 + beta   |   | preview [?]    |  |  offline)     |
| (host runs)   |   | (host runs)    |  |               |
+---------------+   +----------------+  +---------------+

  * = stub / interface in this release (see Status)

---

Install

> [!IMPORTANT] > Not on PyPI — install from Git. This project has no PyPI release yet. The > name open-compute on PyPI is taken by an unrelated project ("multi-agent > systems for healthtech"), so a plain pip install open-compute installs > someone else's package. Always install from this repository:

pip install "git+https://github.com/ellmos-ai/open-compute.git"                        # core only, zero runtime deps
pip install "open-compute[claude] @ git+https://github.com/ellmos-ai/open-compute.git" # + anthropic SDK

The same extra @ git+… form works for every extra below:

| Extra | Adds | |---|---| | claude | anthropic SDK | | openai | openai SDK | | local | mss — real Windows screenshots + input | | wgc | WGC fallback for DirectX surfaces (pulls numpy/OpenCV) | | compose | Pillow — Before\|After composite + annotated shots | | watch | watchdog — native FS events for the directory-watch feed | | clirec | external clirec package for oc rec workflows | | record | clirec[record] capture backend compatibility | | mcp | mcp SDK — MCP server (console script: open-compute-mcp) | | dev | pytest | | all | anthropic, openai, playwright, mss, WGC, Pillow, watchdog, clirec, mcp |

Extras combine as usual, e.g. open-compute[local,wgc,claude]. Working from a clone instead? pip install -e ".[local,claude]" from the repository root.

Until clirec has a package release, install it directly when using oc rec:

pip install git+https://github.com/ellmos-ai/clirec.git

Python 3.10+.

---

Quick start

Mode A — No API key: session-agent as reasoner (chat skill)

Run oc capture / oc do manually from a Claude Code session. The session model sees the PNG via the Read tool and decides the next action:

# 1. Install the local extra (Windows only; provides real screenshots + input)
pip install "open-compute[local] @ git+https://github.com/ellmos-ai/open-compute.git"

# 2. Capture a screenshot — saved automatically to _session/ (never loose on Desktop)
oc capture
# -> {"path": ".../_session/0001_20260620_143200.png", "width": 1920, "height": 1080}
# Then: read the PNG with your Read tool to see the screen.

# 3a. Execute one canonical action (single, backwards-compatible)
oc do '{"type":"mouse_move","x":0.5,"y":0.5}' --mode allow_all
oc do '{"type":"left_click","x":0.25,"y":0.1}' --yes   # --yes = agent pre-approved

# 3b. Execute with Before|After composite (Pillow optional)
oc do '{"type":"left_click","x":0.5,"y":0.3}' --label "click_ok" --yes
# -> {"result":"executed","action":"left_click","composite":"_session/0002_click_ok.png"}

# 3c. Execute a batch/macro (JSON array, one call = multiple actions)
oc do '[{"type":"mouse_move","x":0.5,"y":0.5},{"type":"left_click","x":0.5,"y":0.3}]' --yes
# -> {"result":"batch","count":2,"width":1920,"height":1080}

# 3d. Ensure the target window is in the foreground before acting
oc do '{"type":"left_click","x":0.5,"y":0.3}' --ensure-foreground "Word" --yes

# 3e. Save a full-res after-shot + annotated click marker (v0.5, Pillow optional)
oc do '{"type":"left_click","x":0.5,"y":0.3}' --yes --fullres
# -> {"result":"executed",...,"fullres_annotated":"_session/...fullres.png"}

# 3f. Capture only the active window's bounding rect (v0.5, Windows)
oc capture --window "Word"
# -> {"path":"...","width":800,"height":600,"window":"Word","region":{...}}

# 3g. Watch a directory for changes (v0.5)
oc watch-dir ~/Downloads --for 5       # collect 5 s, print JSON events
oc watch-dir ~/Downloads --once        # one-time snapshot diff

# 3h. Explicit companion handoff (mutations need a granted, scoped lease)
oc session companion --owner local-user
oc session request-control --owner agent-a --scope window:42 --ttl 60
oc session grant --lease-id 
oc window minimize --hwnd 42 --yes

# 3i. Bounded, deduplicated window capture (full screen needs explicit opt-in)
oc capture-series --window "Word" --max-frames 8 --stable-frames 2

# 4. Recapture and repeat until done (or read the "composite" After-shot directly).

See SKILL.md for the full loop protocol, action schema, coordinate guide, and environment variable reference.

Mode B — Autonomous loop with an API backend

The backend is selected by name; claude and openai are equally supported (each needs its own key + extra). For a keyless path, use Mode A above — the host model reasons itself, optionally in a self-spawned subagent (see [usage pattern](#usage-pattern--inline-a-vs-self-subagent-b)).

# Claude (needs ANTHROPIC_API_KEY + open-compute[local,claude]):
oc run "Find the latest invoice in the Downloads folder" --backend claude --max-steps 15

# OpenAI (needs OPENAI_API_KEY + open-compute[local,openai]):
oc run "Find the latest invoice in the Downloads folder" --backend openai --max-steps 15

Or in Python — get_backend(name, ...) builds whichever you name; inject your own executor or use LocalExecutor:

from open_compute import AgentLoop, Config, get_backend
from open_compute.drivers.local import LocalExecutor   # Windows; needs mss
from open_compute.safety import SafetyPolicy

executor = LocalExecutor()   # real display + input
config = Config(backend="claude", scope="os",
                display_width=executor.width, display_height=executor.height)
backend = get_backend("claude", executor.width, executor.height, model="claude-opus-4-8")

loop = AgentLoop(
    config,
    backend=backend,
    executor=executor,
    policy=SafetyPolicy(mode="confirm",
                        confirm_callback=lambda a: input(f"run {a.type.value}? [y/N] ") == "y"),
)
loop.run("Find the latest invoice in the Downloads folder")

Offline dry-run (no API key, no display, mock only)

from open_compute import AgentLoop, Config

loop = AgentLoop(Config(backend="mock", safety_mode="allow_all"))
result = loop.run("Open the settings page and enable dark mode")
print(result.done, result.steps)
for trace in result.traces:
    print(trace.index, trace.backend_message, [a.type.value for a in trace.executed])

---

MCP server (native tool-calls, keyless)

Expose the keyless Mode A loop to any MCP client as native tools — the client is the reasoner (no API key, model-agnostic). Versus driving oc by hand, a long-lived server keeps one warm LocalExecutor resident (no Python restart per action) and returns screenshots as MCP image blocks. Windows-only for real capture/input.

pip install "open-compute[mcp,local,uia,wgc] @ git+https://github.com/ellmos-ai/open-compute.git"
open-compute-mcp          # stdio server (console script)

Tools: capture · do (single or batch canonical actions) · tree · click_name · invoke (UIA semantic targeting) · list_windows · get_screen_size · watch_dir · push_status · rec_replay · signal_show / signal_hide / signal_status / signal_abort (human-in-the-loop screen signal) · chat · talk (push-to-talk). Coordinates are normalized 0..1; list_windows and get_screen_size describe that frame, so the client can name a window exactly instead of guessing a title substring.

Hardware-composited windows (wgc extra). A GDI grab of a DirectX window — Roblox Studio, Blender, a GPU-accelerated browser — does not fail; it quietly returns an all-black rectangle. capture(window=...) therefore checks the frame and, when it comes back blank, re-grabs it through Windows.Graphics.Capture. Install open-compute[wgc] for that fallback; without it a black frame is still returned rather than failing the call. OC_WGC_WINDOWS (comma-separated title substrings) skips the GDI attempt outright for windows known to need WGC. Note that WGC only produces a frame when the window redraws: an idle or non-capturable window fails fast (bounded, a few seconds) instead of hanging.

Capture budget (token cost). A vision model is billed per pixel, so a full-HD capture is by far the most expensive thing this server returns — and every frame stays in the conversation, so the cost is paid again on each following request. Because all coordinates here are normalized 0..1, shrinking the image costs nothing in control accuracy; only legibility drops. Three knobs:

| Variable | Effect | Cost of a 1920×1080 grab | |---|---|---| | (unset) | full resolution | ~1600 tokens | | OC_CAPTURE_SCALE=0.5 | halve both edges | ~690 tokens | | OC_CAPTURE_MAX_DIM=768 | cap the longest edge | ~440 tokens | | OC_CAPTURE_GRAYSCALE=1 | drop colour | payload only — not tokens, which follow pixel count alone |

OC_CAPTURE_SCALE=0.5 is the sweet spot for GUI work: buttons and field borders stay clearly identifiable, only small body text gets hard to read. Both size knobs compose (scale first, then the cap), and a failure to shrink never fails the capture — the original frame is returned instead.

Safety. OC_SAFETY_MODE is an operator ceiling (confirm default · read_only · allow_all); a per-call mode can only tighten it, never loosen it, so a prompt-injected agent cannot escape a read_only/confirm server via mode="allow_all". Because stdio MCP has no server→client confirm callback, confirm/read_only return a needs_confirmation/deny result without acting. For interactive use, run the server with OC_SAFETY_MODE=allow_all in an isolated VM and let the client's tool-permission dialog be the human-in-the-loop. Optional OC_DENY (comma-separated action types) is a hard deny list.

Auto-signal (OC_SIGNAL_AUTO). Set it to a SessionMode name (e.g. control) to auto-show the screen-usage overlay the first time a state-changing tool (do / click_name / invoke / rec_replay) actually passes the safety gate — no separate signal_show call to remember before the model starts steering. It never overrides an already-visible signal (manual or auto, any mode) and never fires from a gate-blocked call or a read-only tool. Unset or off (the default) disables it; an invalid mode name surfaces as auto_signal_error in the tool result instead of failing the call. See signal_show/signal_hide/signal_status below for the manual controls and OC_SIGNAL_CONFIG for per-mode colors.

Auto-hide (OC_SIGNAL_IDLE_HIDE). An auto-shown overlay takes itself down once the steering stops: every state-changing tool call re-arms an idle countdown, and when it expires with no further action the overlay is hidden. The value is seconds, default 60; 0, an empty value, or off disables the auto-hide and keeps the overlay up until signal_hide (the pre-0.7 behavior). Only an overlay that OC_SIGNAL_AUTO put up is ever swept away — one you asked for with signal_show stays until you hide it, and a manual signal_show over an auto-shown overlay takes ownership and cancels the countdown. signal_status reports both (auto_shown, idle_hide_armed); an unusable value surfaces as signal_idle_hide_error in the tool result instead of failing the action.

Troubleshooting: do/click_name only ever return needs_confirmation and never act. That is the confirm ceiling working as designed under stdio MCP — there is no confirm callback, so the server reports instead of acting. Fix for interactive use: set "env": {"OC_SAFETY_MODE": "allow_all"} in the server registration and let the client's tool-approval

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ellmos-ai
• Source: ellmos-ai/open-compute
• License: MIT

Install and usage instructions live in the source repository linked above.