# Open Compute

> Model-agnostic computer-use core: one agent loop for Claude, OpenAI CUA, and a mock backend — normalized coordinates, canonical action schema, central safety gate.

- **Type:** MCP server
- **Install:** `agentstack add mcp-ellmos-ai-open-compute`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [ellmos-ai](https://agentstack.voostack.com/s/ellmos-ai)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [ellmos-ai](https://github.com/ellmos-ai)
- **Source:** https://github.com/ellmos-ai/open-compute

## Install

```sh
agentstack add mcp-ellmos-ai-open-compute
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# open-compute

**EN** | [DE](README_de.md)

[](CHANGELOG.md)
[](pyproject.toml)
[](https://github.com/ellmos-ai/open-compute/actions/workflows/tests.yml)
[](tests)
[](llms.txt)
[](https://github.com/ellmos-ai)
[](CHANGELOG.md)
[](LICENSE)

**A model-agnostic computer-use core: one agent loop, any reasoning model behind a single interface.**

open-compute is a small, dependency-light Python core for building computer-use
agents (LLM-driven GUI / desktop / browser automation). It implements the
**perception → model-tool-call → action → feedback** loop and keeps the
reasoning model swappable behind a single `ComputerBackend` interface. **No
provider is privileged**: Anthropic Claude and OpenAI CUA are two equally-ranked
API backends, and the offline `mock` backend is the default. A **keyless** path
also exists today via Mode A, where the host model itself reasons — and it can
run that loop either inline or in a self-spawned subagent for context economy
(see [usage pattern](#usage-pattern--inline-a-vs-self-subagent-b)). The core has
**zero runtime dependencies**; vendor SDKs (`anthropic`, `openai`) are
**optional, lazily imported** extras — `import open_compute` works with none of
them installed, and the default mock wiring runs fully offline.

> [!NOTE]
> **AI / LLM Integration Notice**: `open-compute` includes a machine-readable [`llms.txt`](llms.txt) file designed for AI agents, RAG crawlers, and LLM-assisted workflows.

---

## Why

Every computer-use model — Anthropic's Claude `computer` tool and OpenAI's
computer-use tool — shares the same agent-loop *shape* but differs in transport,
coordinate frame, and action names. open-compute factors out the common parts so
you write the loop once and swap the reasoning model freely behind one
`ComputerBackend` interface:

- A **canonical action schema** with one mapper per backend.
- **Normalized (0..1) coordinates** internally, denormalized per backend /
  resolution / DPI in one tested utility — the DPI problem solved centrally.
- A **central safety gate** ("confirm before risky actions") evaluated before
  every action.
- A **hybrid perception** interface (screenshot + Set-of-Marks / accessibility /
  DOM), so you can move from pure pixel-vision to semantic targeting later.

---

## Architecture

```
                        +-----------------------------------------+
                        |        AGENT LOOP / ORCHESTRATOR        |
                        |  goal -> perceive -> backend -> safety  |
                        |        -> execute -> re-perceive        |
                        +-------------------+---------------------+
                                            |
        +-----------------------------------+-----------------------------------+
        |                                   |                                   |
+-------v---------+              +----------v-----------+            +----------v----------+
| PERCEPTION      |              | CANONICAL ACTIONS    |            | SAFETY / POLICY     |
| - screenshot    |              | click/type/key/      |            | - confirm-at-action |
| - set-of-marks  |              | scroll/drag/wait/    |            | - allow / deny list |
|   (OmniParser)* |              | screenshot + OS ext  |            | - read-only mode    |
| - accessibility*|              | (launch/activate)    |            | - audit log         |
+-------+---------+              +----------+-----------+            +----------+----------+
        |                                   |                                   |
        +-----------------+-----------------+----------------------------------+
                          |
              +-----------v------------+   COORDINATE / DPI NORMALIZATION
              | BACKEND ABSTRACTION    |   - internal: normalized (0..1)
              | (ComputerBackend)      |   - denormalize per backend:
              +-----+--------+---------+     * Claude: global px (display_w x display_h)
                    |        |    |          * OpenAI: px (computer_call)
        +-----------+        |    +-----------+   * Mock: synthetic
        |                    |                |
+-------v-------+   +--------v-------+  +-----v---------+
| Claude        |   | OpenAI CUA     |  | Mock backend  |
| computer_2025 |   | computer-use-  |  | (no SDK,      |
| 1124 + beta   |   | preview [?]    |  |  offline)     |
| (host runs)   |   | (host runs)    |  |               |
+---------------+   +----------------+  +---------------+

  * = stub / interface in this release (see Status)
```

---

## Install

> [!IMPORTANT]
> **Not on PyPI — install from Git.** This project has no PyPI release yet. The
> name `open-compute` on PyPI is taken by an **unrelated** project ("multi-agent
> systems for healthtech"), so a plain `pip install open-compute` installs
> someone else's package. Always install from this repository:

```bash
pip install "git+https://github.com/ellmos-ai/open-compute.git"                        # core only, zero runtime deps
pip install "open-compute[claude] @ git+https://github.com/ellmos-ai/open-compute.git" # + anthropic SDK
```

The same `extra @ git+…` form works for every extra below:

| Extra | Adds |
|---|---|
| `claude` | anthropic SDK |
| `openai` | openai SDK |
| `local` | mss — real Windows screenshots + input |
| `wgc` | WGC fallback for DirectX surfaces (pulls numpy/OpenCV) |
| `compose` | Pillow — Before\|After composite + annotated shots |
| `watch` | watchdog — native FS events for the directory-watch feed |
| `clirec` | external clirec package for `oc rec` workflows |
| `record` | clirec[record] capture backend compatibility |
| `mcp` | mcp SDK — MCP server (console script: `open-compute-mcp`) |
| `dev` | pytest |
| `all` | anthropic, openai, playwright, mss, WGC, Pillow, watchdog, clirec, mcp |

Extras combine as usual, e.g. `open-compute[local,wgc,claude]`. Working from a
clone instead? `pip install -e ".[local,claude]"` from the repository root.

Until `clirec` has a package release, install it directly when using `oc rec`:

```bash
pip install git+https://github.com/ellmos-ai/clirec.git
```

Python 3.10+.

---

## Quick start

### Mode A — No API key: session-agent as reasoner (chat skill)

Run `oc capture` / `oc do` manually from a Claude Code session. The session
model sees the PNG via the Read tool and decides the next action:

```bash
# 1. Install the local extra (Windows only; provides real screenshots + input)
pip install "open-compute[local] @ git+https://github.com/ellmos-ai/open-compute.git"

# 2. Capture a screenshot — saved automatically to _session/ (never loose on Desktop)
oc capture
# -> {"path": ".../_session/0001_20260620_143200.png", "width": 1920, "height": 1080}
# Then: read the PNG with your Read tool to see the screen.

# 3a. Execute one canonical action (single, backwards-compatible)
oc do '{"type":"mouse_move","x":0.5,"y":0.5}' --mode allow_all
oc do '{"type":"left_click","x":0.25,"y":0.1}' --yes   # --yes = agent pre-approved

# 3b. Execute with Before|After composite (Pillow optional)
oc do '{"type":"left_click","x":0.5,"y":0.3}' --label "click_ok" --yes
# -> {"result":"executed","action":"left_click","composite":"_session/0002_click_ok.png"}

# 3c. Execute a batch/macro (JSON array, one call = multiple actions)
oc do '[{"type":"mouse_move","x":0.5,"y":0.5},{"type":"left_click","x":0.5,"y":0.3}]' --yes
# -> {"result":"batch","count":2,"width":1920,"height":1080}

# 3d. Ensure the target window is in the foreground before acting
oc do '{"type":"left_click","x":0.5,"y":0.3}' --ensure-foreground "Word" --yes

# 3e. Save a full-res after-shot + annotated click marker (v0.5, Pillow optional)
oc do '{"type":"left_click","x":0.5,"y":0.3}' --yes --fullres
# -> {"result":"executed",...,"fullres_annotated":"_session/...fullres.png"}

# 3f. Capture only the active window's bounding rect (v0.5, Windows)
oc capture --window "Word"
# -> {"path":"...","width":800,"height":600,"window":"Word","region":{...}}

# 3g. Watch a directory for changes (v0.5)
oc watch-dir ~/Downloads --for 5       # collect 5 s, print JSON events
oc watch-dir ~/Downloads --once        # one-time snapshot diff

# 3h. Explicit companion handoff (mutations need a granted, scoped lease)
oc session companion --owner local-user
oc session request-control --owner agent-a --scope window:42 --ttl 60
oc session grant --lease-id 
oc window minimize --hwnd 42 --yes

# 3i. Bounded, deduplicated window capture (full screen needs explicit opt-in)
oc capture-series --window "Word" --max-frames 8 --stable-frames 2

# 4. Recapture and repeat until done (or read the "composite" After-shot directly).
```

See `SKILL.md` for the full loop protocol, action schema, coordinate guide, and
environment variable reference.

### Mode B — Autonomous loop with an API backend

The backend is selected by name; `claude` and `openai` are equally supported
(each needs its own key + extra). For a **keyless** path, use Mode A above — the
host model reasons itself, optionally in a self-spawned subagent (see
[usage pattern](#usage-pattern--inline-a-vs-self-subagent-b)).

```bash
# Claude (needs ANTHROPIC_API_KEY + open-compute[local,claude]):
oc run "Find the latest invoice in the Downloads folder" --backend claude --max-steps 15

# OpenAI (needs OPENAI_API_KEY + open-compute[local,openai]):
oc run "Find the latest invoice in the Downloads folder" --backend openai --max-steps 15
```

Or in Python — `get_backend(name, ...)` builds whichever you name; inject your
own executor or use `LocalExecutor`:

```python
from open_compute import AgentLoop, Config, get_backend
from open_compute.drivers.local import LocalExecutor   # Windows; needs mss
from open_compute.safety import SafetyPolicy

executor = LocalExecutor()   # real display + input
config = Config(backend="claude", scope="os",
                display_width=executor.width, display_height=executor.height)
backend = get_backend("claude", executor.width, executor.height, model="claude-opus-4-8")

loop = AgentLoop(
    config,
    backend=backend,
    executor=executor,
    policy=SafetyPolicy(mode="confirm",
                        confirm_callback=lambda a: input(f"run {a.type.value}? [y/N] ") == "y"),
)
loop.run("Find the latest invoice in the Downloads folder")
```

### Offline dry-run (no API key, no display, mock only)

```python
from open_compute import AgentLoop, Config

loop = AgentLoop(Config(backend="mock", safety_mode="allow_all"))
result = loop.run("Open the settings page and enable dark mode")
print(result.done, result.steps)
for trace in result.traces:
    print(trace.index, trace.backend_message, [a.type.value for a in trace.executed])
```

---

## MCP server (native tool-calls, keyless)

Expose the keyless **Mode A** loop to any MCP client as **native tools** — the
client is the reasoner (no API key, model-agnostic). Versus driving `oc` by hand,
a long-lived server keeps **one warm `LocalExecutor`** resident (no Python restart
per action) and returns screenshots as MCP **image** blocks. Windows-only for real
capture/input.

```bash
pip install "open-compute[mcp,local,uia,wgc] @ git+https://github.com/ellmos-ai/open-compute.git"
open-compute-mcp          # stdio server (console script)
```

**Tools:** `capture` · `do` (single or batch canonical actions) · `tree` ·
`click_name` · `invoke` (UIA semantic targeting) · `list_windows` ·
`get_screen_size` · `watch_dir` · `push_status` · `rec_replay` · `signal_show` /
`signal_hide` / `signal_status` / `signal_abort` (human-in-the-loop screen
signal) · `chat` · `talk` (push-to-talk). Coordinates are
normalized 0..1; `list_windows` and `get_screen_size` describe that frame, so the
client can name a window exactly instead of guessing a title substring.

**Hardware-composited windows (`wgc` extra).** A GDI grab of a DirectX window —
Roblox Studio, Blender, a GPU-accelerated browser — does not fail; it quietly
returns an **all-black** rectangle. `capture(window=...)` therefore checks the
frame and, when it comes back blank, re-grabs it through Windows.Graphics.Capture.
Install `open-compute[wgc]` for that fallback; without it a black frame is still
returned rather than failing the call. `OC_WGC_WINDOWS` (comma-separated title
substrings) skips the GDI attempt outright for windows known to need WGC.
Note that WGC only produces a frame when the window *redraws*: an idle or
non-capturable window fails fast (bounded, a few seconds) instead of hanging.

**Capture budget (token cost).** A vision model is billed per pixel, so a full-HD
`capture` is by far the most expensive thing this server returns — and every frame
stays in the conversation, so the cost is paid again on each following request.
Because all coordinates here are normalized 0..1, shrinking the image costs
**nothing in control accuracy**; only legibility drops. Three knobs:

| Variable | Effect | Cost of a 1920×1080 grab |
|---|---|---|
| *(unset)* | full resolution | ~1600 tokens |
| `OC_CAPTURE_SCALE=0.5` | halve both edges | ~690 tokens |
| `OC_CAPTURE_MAX_DIM=768` | cap the longest edge | ~440 tokens |
| `OC_CAPTURE_GRAYSCALE=1` | drop colour | payload only — **not** tokens, which follow pixel count alone |

`OC_CAPTURE_SCALE=0.5` is the sweet spot for GUI work: buttons and field borders
stay clearly identifiable, only small body text gets hard to read. Both size knobs
compose (scale first, then the cap), and a failure to shrink never fails the
capture — the original frame is returned instead.

**Safety.** `OC_SAFETY_MODE` is an operator **ceiling** (`confirm` default ·
`read_only` · `allow_all`); a per-call `mode` can only *tighten* it, never loosen it,
so a prompt-injected agent cannot escape a `read_only`/`confirm` server via
`mode="allow_all"`. Because stdio MCP has no server→client confirm callback,
`confirm`/`read_only` return a `needs_confirmation`/`deny` result **without acting**.
For interactive use, run the server with `OC_SAFETY_MODE=allow_all` **in an isolated
VM** and let the client's tool-permission dialog be the human-in-the-loop. Optional
`OC_DENY` (comma-separated action types) is a hard deny list.

**Auto-signal (`OC_SIGNAL_AUTO`).** Set it to a `SessionMode` name (e.g.
`control`) to auto-show the screen-usage overlay the first time a
state-changing tool (`do` / `click_name` / `invoke` / `rec_replay`) actually
passes the safety gate — no separate `signal_show` call to remember before
the model starts steering. It never overrides an already-visible signal
(manual or auto, any mode) and never fires from a gate-blocked call or a
read-only tool. Unset or `off` (the default) disables it; an invalid mode
name surfaces as `auto_signal_error` in the tool result instead of failing
the call. See `signal_show`/`signal_hide`/`signal_status` below for the
manual controls and `OC_SIGNAL_CONFIG` for per-mode colors.

**Auto-hide (`OC_SIGNAL_IDLE_HIDE`).** An auto-shown overlay takes itself down
once the steering stops: every state-changing tool call re-arms an idle
countdown, and when it expires with no further action the overlay is hidden.
The value is **seconds, default 60**; `0`, an empty value, or `off` disables the
auto-hide and keeps the overlay up until `signal_hide` (the pre-0.7 behavior).
Only an overlay that `OC_SIGNAL_AUTO` put up is ever swept away — one you asked
for with `signal_show` stays until you hide it, and a manual `signal_show` over
an auto-shown overlay takes ownership and cancels the countdown. `signal_status`
reports both (`auto_shown`, `idle_hide_armed`); an unusable value surfaces as
`signal_idle_hide_error` in the tool result instead of failing the action.

**Troubleshooting: `do`/`click_name` only ever return `needs_confirmation` and never
act.** That is the `confirm` ceiling working as designed under stdio MCP — there is
no confirm callback, so the server reports instead of acting. Fix for interactive
use: set `"env": {"OC_SAFETY_MODE": "allow_all"}` in the server registration and let
the client's tool-approval

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ellmos-ai](https://github.com/ellmos-ai)
- **Source:** [ellmos-ai/open-compute](https://github.com/ellmos-ai/open-compute)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-ellmos-ai-open-compute
- Seller: https://agentstack.voostack.com/s/ellmos-ai
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.