Gui Tool

gui-tool

Desktop GUI automation CLI. All commands return JSON to stdout (errors to stderr).

gui-tool screenshot --output /tmp/screen.png                              # full screen
gui-tool screenshot --window-id 123 --output /tmp/app.png                 # cropped to window
gui-tool screenshot --window-id 123 --grid --output /tmp/grid.png         # grid overlay
gui-tool screenshot --window-id 123 --grid --cell B2 --output /tmp/z.png  # zoom into cell

gui-tool windows list                                # JSON list of all windows
gui-tool windows raise 123                           # bring window to front

gui-tool mouse click                                 # click at current position
gui-tool mouse click --button right                  # right-click
gui-tool mouse click --cell B2.C1 --window-id 123   # click cell (recursive zoom OK)

gui-tool key type "hello" --window-id 123            # type text
gui-tool key press "ctrl+a" --window-id 123          # key combo (ctrl/shift/alt/super + key)

---

Grid Targeting

No pixel coordinates exist. The grid is the only way to click. Each cell has a red crosshair (+) at its center — a click on that cell lands exactly on that crosshair, nowhere else. If the crosshair isn't on your target, the click will miss. Zoom until a crosshair sits directly on the target.

Orient → Zoom → Click → Verify

Orient. Take a grid screenshot. Identify which cell contains your target and note where within that cell it sits (e.g. "the button is in D1, near the bottom-left").

Zoom. Crop into that cell to get a sub-grid. Pick the sub-cell matching your position note — don't re-read text or hunt for icons in zoomed views, they'll be blurry. Translate spatially: "bottom-left of D1" → sub-cell B5 or C5. Keep zooming (append with dots: D1.C5.F3) until a crosshair is on the target. 2–3 levels is typical.

Click. gui-tool mouse click --cell D1.C5 --window-id 123

Verify. Zoom into the area you just clicked to check the result — a full-page screenshot is too small to see subtle state changes (button color, focus ring, selection highlight). If the click missed, re-orient from scratch — the screen state may have changed.

Navigating Without Zooming Out

Zoomed views show dimmed context from adjacent parent cells with their labels visible. If your target is in a neighboring cell, reference it directly — do not zoom all the way out:

# Zoomed into G1, but target is actually in H1 (visible in dimmed context)
# WRONG: take a fresh full-grid screenshot and start over
# RIGHT: just use H1 directly
gui-tool screenshot --window-id 123 --grid --cell H1 --output /tmp/zoom.png

Only take a fresh full-grid screenshot (no --cell) when the screen has changed (after clicking, typing, or switching windows) or when you're genuinely lost.

Between-Cell Targeting

Target straddles two cells? Use + to center on the boundary:

gui-tool mouse click --cell D3+E3 --window-id 123   # horizontal
gui-tool mouse click --cell D3+D4 --window-id 123   # vertical
gui-tool mouse click --cell D3+E4 --window-id 123   # diagonal

Small Icons and Buttons

Tiny targets (like/dislike buttons, close icons, checkboxes) need 3+ zoom levels. On the first zoom, note the icon's position carefully — at deeper zooms it becomes a few blurry pixels. Trust your spatial note, not what the zoomed crop looks like.

Dot Notation

B2.C1 = sub-cell C1 within parent B2. Change the last segment to try a neighbor at the same depth (B2.D2). Append to go deeper (B2.C1.F3).

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ZachRouan
• Source: ZachRouan/agent-desktop-interface
• License: MIT
• Homepage: https://crates.io/crates/gui-tool

Install and usage instructions live in the source repository linked above.