Hypruse

hypruse

Computer use for Hyprland. An MCP server that gives AI agents native hands on your Wayland desktop: workspaces, windows, mouse, keyboard, screenshots.

No ydotool daemon. No root. No portals. No X11.

Why

Computer use exists on macOS and Windows. On Linux there is effectively nothing: the Claude Desktop Linux beta explicitly ships without screen control, Anthropic's reference implementation is an X11 container, and the existing Wayland attempts lean on setuid uinput hacks or GNOME-only portals.

Meanwhile Hyprland already exposes everything an agent needs, better than any accessibility bridge: a complete IPC surface for state and window management, and first-class Wayland protocols for input. hypruse just wires them to MCP:

• Semantic first. desktop returns the real window/workspace tree (addresses, classes, titles, geometry) in one call. The agent switches workspaces and focuses windows the way you do (instantly, over IPC), not by squinting at pixels.
• Vision when it matters. Screenshots of a monitor, an exact window crop, or a zoomed region, with the geometry/scale metadata to map any pixel back to a clickable coordinate: a coarse-to-fine loop [grounded in the GUI-agents research](#research).
• Native input. Clicks and scrolls are spoken directly over the Wayland wire (zwlr_virtual_pointer_v1); typing goes through wtype's virtual keyboard with a proper XKB keymap, unicode-safe on any layout.

How it works

agent (Claude Code, or any MCP client)
   │ stdio
   ▼
hypruse
   ├── hyprctl -j ········▶ desktop state: monitors, workspaces, windows, layers
   ├── hyprctl dispatch ··▶ focus / move / close / launch / movecursor
   ├── grim ··············▶ screenshots: monitor, window crop, region
   ├── busctl (AT-SPI) ···▶ accessibility tree: named controls, current
   │                        values, exact coords (ui / marks / click_ui)
   ├── wtype ·············▶ keyboard (zwp_virtual_keyboard_v1, real XKB keymap)
   └── raw Wayland wire ··▶ click & scroll (zwlr_virtual_pointer_v1)

Optional binaries gate two more tools: imagemagick draws the numbered overlay for marks, and wl-clipboard backs the opt-in clipboard tool.

Design decisions:

• No ydotool / uinput. That path needs a daemon, udev rules or root, and types US scancodes that break on other layouts. hypruse is just another Wayland client of your compositor, same standing as wlrctl.
• No portals. xdg-desktop-portal-hyprland does not implement the RemoteDesktop portal (InputCapture is capture, not injection), so anything built on libei/portals silently degrades on Hyprland. hypruse doesn't try.
• Cursor positioning via hyprctl dispatch movecursor (global logical coordinates, exact on any monitor layout), with only button/axis events on the virtual pointer, sidestepping the known multi-monitor bugs of absolute virtual-pointer motion (hyprwm/Hyprland#6749).

Tools

| tool | what it does | |---|---| | desktop | One-call semantic snapshot: monitors, workspaces, windows (address/class/title/geometry), active window, cursor, and layer surfaces (launchers, bars, notification popups) with a best-effort kind and geometry | | screenshot | Focused monitor, exact window crop by address, or x,y,WxH region; returns image + coordinate-mapping metadata; fast JPEG by default (lossless=true for PNG); stable=true waits for the frame to settle | | zoom | Native-resolution re-capture around an estimated point (optionally clamped to a window): the precision step before clicking small controls, same metadata contract | | ui | Read a window's accessibility tree (AT-SPI, GTK/Qt apps that expose one) and return clickable elements by name with exact global coordinates, no screenshot; reports current values too (typed text, slider position, checkbox state); falls back to vision when an app exposes nothing | | marks | Set-of-Marks capture: the window screenshot with every accessible control drawn as a numbered mark, plus a JSON legend (role, name, current value, exact click point per number); needs ImageMagick for the drawing, degrades to the legend alone without it | | click_ui | Click a control by accessible NAME or by a marks number in one call: the coordinate comes from the tree, the click goes through the real pointer (visible, same safety guarantees); an ambiguous name returns the candidates instead of guessing | | pointer | move / click / drag / scroll (discrete wheel notches) in global coordinates | | keyboard | Type literal text (unicode-safe) or press app-level combos (ctrl+shift+t, esc, F5); optional window address focuses the target first so keystrokes land in the right app. Compositor binds (super+...) go through use_bind, not here | | hypr | Switch workspace, focus/move/close windows, fullscreen, floating (pure IPC, milliseconds; close_window also waits up to 1s for the destroy event, so its result and then= observation reflect the close instead of racing it) | | launch | Start an app (optionally silent on another workspace), block on its actual openwindow event, return its address; detects single-instance apps (browsers) whose window ignores exec rules and moves it to the requested workspace | | binds | The user's own keybinds, decoded (SUPER+Q, action, description); the agent runs one with use_bind | | use_bind | Execute a keybind by combo (SUPER+F), running its bound action, so the agent drives the owner's own launchers and shortcuts | | sequence | Run an ordered list of actions (pointer/keyboard/clickui/hypr/waitfor) in one call; stops the moment the desktop changes in a way the current step did not expect, so a click/type/enter micro-sequence costs one round-trip instead of several | | wait_for | Block on real compositor events (window open/close, workspace change, title change, layer surfaces appearing/closing, urgency, screen sharing on/off) with a match filter and timeout; a filtered wait whose condition already holds (window already gone, workspace already active, layer already mapped) answers instantly with already: true instead of missing an event that fired before it subscribed. Replaces sleep-and-hope in multi-step automations | | clipboard | Read or write the text clipboard via wl-clipboard; opt-in, exists only with HYPRUSE_CLIPBOARD=1 in the server env |

The acting tools (pointer, keyboard, click_ui, hypr, use_bind, sequence) take an optional then argument that appends the result to the same call, so the agent sees the effect without a second round-trip: then='desktop' adds a fresh semantic snapshot (~20 ms, cheap, best for window/focus changes), then='screenshot' a stable capture (best for visual changes), then='ui' the acted-on window's controls with their current values (a few hundred exact tokens, best after typing or toggling; click_ui reads the window it clicked even if the click handed focus to a dialog, the others read the focused window), then='none' nothing (the default everywhere except sequence, which defaults to 'desktop').

Features

The tools group into five capabilities, ordered most-reliable-and-cheapest first. An agent that reaches for them in this order is both faster and more accurate, and many tasks never need a screenshot at all.

1. Semantic desktop control (start here)

desktop returns the entire window and workspace tree in one call: every window's address, class, title, and geometry, the active window, the cursor, and any layer surfaces on screen (launchers, bars, notification popups). hypr and launch then act on it over IPC in milliseconds: switch workspace, focus/move/close/fullscreen/float a window by address, or start an app.

Use it well: never take a screenshot to find or arrange windows. Read desktop, act on the address you want. launch blocks on the real openwindow event and hands back the new window's address, so there is nothing to poll or guess; it also relocates single-instance apps (browsers) that ignore workspace rules.

2. Click controls by name, no pixels

When an app exposes an accessibility tree (most GTK and Qt apps), you can target controls by name instead of by pixel. ui lists every control with its exact global coordinate, and reports the current value of the controls that carry one: the text in a field, a slider's percentage, a checkbox's state. click_ui resolves a name and clicks it in one call, through the real cursor (so the beacon and every safety guarantee still apply). marks draws numbered marks over a screenshot with a legend, for when you want to see the options first and then click_ui(mark=N).

Use it well: reach for click_ui name="Save" before estimating any pixel, since it is exact and spends no image. Read a form's state with ui (did the box actually tick?) instead of screenshotting it. An ambiguous name returns the candidates rather than guessing. When an app exposes no tree (terminals, canvas apps, Electron/Chrome without --force-renderer-accessibility) the tool says so, and you fall back to vision.

3. Vision when it matters: the zoom loop

For everything the accessibility tree cannot name, screenshot (monitor, window crop, or region) and zoom (a native-resolution re-capture around a point) carry a strict coordinate contract, global = geometry + pixel / scale, that stays exact on every monitor and fractional scale.

Use it well: don't guess a small control from a full-screen image. Work coarse-to-fine: screenshot the window, estimate the target, zoom there, re-estimate on the sharp crop, then click. This two-step loop is the [research-backed](#research) way to hit small targets.

4. Fewer round-trips: the latency lever

For an agent the model calls dominate task latency, not the desktop, so the real speedups are structural. sequence runs an ordered micro-plan (click, type, press enter, wait) in a single call, stopping the moment the desktop changes structurally in a way a step did not intend (a window opening, closing, or moving, an unexpected workspace switch, or a seat-taking launcher or on-screen keyboard; it deliberately ignores bare focus changes and notification popups). then='desktop' | 'screenshot' | 'ui' fuses a fresh view of the result into the acting call itself. wait_for blocks on real compositor events (a window or launcher opening, a title changing, a workspace switch, an urgency hint, screen-sharing starting) instead of sleeping and hoping.

Use it well: collapse a known click/type/enter flow into one sequence. After typing into a form, add then='ui' to read the effect back in a few hundred exact tokens instead of a screenshot. After a launch or a shortcut that opens something, wait_for the event rather than sleeping.

5. Safe delegation: trust layers

hypruse hands an agent your real seat, so it ships the controls to bound what that agent can do. Beyond the always-on approval prompts and the Waybar activity beacon, opt-in env flags narrow what an agent can touch: HYPRUSE_READONLY exposes only the observation tools; HYPRUSE_CONFINE restricts input to the windows the agent launched, or a class/workspace allowlist; HYPRUSE_AUTH_GUARD (on by default) refuses to drive authentication dialogs; HYPRUSE_STRICT refuses to act if you took the seat back; HYPRUSE_MARK tags agent-owned windows and announces when the agent opens a window or captures the screen.

Use it well: run read-only for the first week. When you trust a workflow, allowlist its tools and, if you want to walk away, confine the agent to a scope so your password manager on another workspace stays untouchable. Keep a panic bind handy (hypruse stop, or pkill -f hypruse). The [Security model](#security-model) has the full story.

Install

Requirements: Hyprland, grim, wtype (most Hyprland setups already have both), and uv. The accessibility tools (ui/marks/click_ui) use busctl, which ships with systemd. Optional: wl-clipboard for the opt-in clipboard tool, imagemagick for numbered marks captures.

Arch Linux, from the AUR:

yay -S hypruse        # or hypruse-git for main

Then let it set itself up and verify the environment:

hypruse init     # detects your MCP clients, registers (asks first), runs doctor
hypruse doctor   # just the diagnostics

Manual registration, Claude Code:

claude mcp add -s user hypruse -- uvx hypruse

From a source checkout:

claude mcp add -s user hypruse -- uv run --directory /path/to/hypruse hypruse

Any other MCP client: run uvx hypruse as a stdio server. hypruse is also in the official MCP registry as io.github.IlyasKhallouki/hypruse, so clients that browse the registry can install it from there.

Read-only mode: set HYPRUSE_READONLY=1 in the server config to expose only the observation tools (desktop, screenshot, zoom, ui, marks, binds, wait_for). The agent can see and narrate but cannot click, type, or launch. A good first week.

Claude Desktop (Linux beta)

The Linux beta ships without Anthropic's first-party computer use, but stdio MCP servers work in chat, which makes hypruse the workaround. In ~/.config/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "hypruse": {
      "command": "uvx",
      "args": ["hypruse"],
      "env": { "HYPRUSE_SCREENSHOT_MODE": "image" }
    }
  }
}

Two Desktop-specific notes: use image mode (Desktop renders inline MCP images and has no file-read tool), and the app must run natively inside your Hyprland session so the server inherits WAYLAND_DISPLAY/HYPRLAND_INSTANCE_SIGNATURE; from a VM or container it cannot reach your compositor. If your Desktop install bypasses tool-approval prompts, treat the [Waybar indicator + panic keybind](waybar/) as mandatory, not optional.

Security model

Read this section before installing. hypruse hands an agent your mouse, your keyboard, your screen contents, and an app launcher. The layers that keep that sane:

1. Approval: MCP clients gate tool calls. In Claude Code, allowlist the read-only tools (desktop, screenshot) and leave pointer/keyboard/hypr/launch on ask-first until you trust a workflow.
2. Visibility: the server maintains an activity beacon ($XDG_RUNTIME_DIR/hypruse/state.json); the shipped [Waybar module](waybar/) is invisible when idle and shows a robot indicator while an agent has hands on your desktop.
3. Interruption: click the indicator, or bind a panic key. The portable form works for every install: bind = SUPER SHIFT, BackSpace, exec, pkill -f hypruse. If hypruse is on your PATH (the AUR or a pipx install), bind = SUPER SHIFT, BackSpace, exec, hypruse stop is nicer: it signals the server to shut down gracefully, releasing any held pointer button and clearing the beacon. For a uvx install use exec, uvx hypruse stop; for a source checkout, exec, uv run --directory /path/to/hypruse hypruse stop. Killing it mid-action is safe either way: button press/release pairs never span tool calls, and even a long drag's held button is released on the way out.
4. The seat is shared. There is one cursor and one keyboard focus, and Hyprland's focus-follows-mouse means a cursor move alone can retarget keystrokes. Don't type while an agent is driving; watch the indicator.
5. Scope: stdio only (no network listener), nothing persisted except the beacon and the capped screenshot cache in $XDG_RUNTIME_DIR (tmpfs, newest 20), and no clipboard access unless you opt in: HYPRUSE_CLIPBOARD=1 registers a clipboard tool (never in read-only mode); clipboards hold passwords, so leave it off unless a workflow needs it. A screenshot sees everything

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: IlyasKhallouki
• Source: IlyasKhallouki/hypruse
• License: MIT

Install and usage instructions live in the source repository linked above.