# Macos Control

> >

- **Type:** Skill
- **Install:** `agentstack add skill-nakaokarei-swiftautogui-macos-control`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [NakaokaRei](https://agentstack.voostack.com/s/nakaokarei)
- **Installs:** 0
- **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [NakaokaRei](https://github.com/NakaokaRei)
- **Source:** https://github.com/NakaokaRei/SwiftAutoGUI/tree/master/plugins/swift-auto-gui/skills/macos-control
- **Website:** https://nakaokarei.github.io/SwiftAutoGUI/documentation

## Install

```sh
agentstack add skill-nakaokarei-swiftautogui-macos-control
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# macos-control

The `sagui` CLI provides mouse control, keyboard input, screenshot capture, image recognition, and screen queries for macOS automation.

## Preflight: ensure `sagui` is installed

Before issuing any `sagui` command, run:

```bash
command -v sagui
```

If the command prints a path, you're ready — continue to the sections below.

If it prints nothing (exit code 1), `sagui` is not installed. **Stop and ask the user for permission to install it** before doing anything else. Show them these options and wait for confirmation:

**Recommended — Homebrew:**

```bash
brew install NakaokaRei/tap/sagui
```

**Fallback — build from source** (requires Xcode / Swift 6.2+ toolchain; verify with `swift --version`):

```bash
git clone https://github.com/NakaokaRei/SwiftAutoGUI.git /tmp/SwiftAutoGUI
cd /tmp/SwiftAutoGUI
swift build -c release
# Copy the binary somewhere on $PATH. /usr/local/bin requires sudo:
sudo cp .build/release/sagui /usr/local/bin/sagui
# Or, no sudo, into a user dir already on PATH:
# cp .build/release/sagui ~/.local/bin/sagui
```

After install, re-run `command -v sagui` to confirm the binary is reachable, then continue.

## Permissions

The first time `sagui` issues an event or screenshot, macOS will prompt for these permissions. Without them, commands silently fail or return blank screenshots.

- **Accessibility** — required for `sagui key` and `sagui mouse`. Enable in System Settings → Privacy & Security → Accessibility for the app running Claude Code (Terminal.app, iTerm, etc.).
- **Screen Recording** — required for `sagui screen screenshot`, `sagui screen pixel`, and `sagui screen locate*`. Enable in System Settings → Privacy & Security → Screen Recording for the same app.

If a command unexpectedly fails or returns empty output, ask the user to verify both permissions are granted to their terminal app.

This skill is **macOS-only** — `sagui` builds on CoreGraphics. On other platforms, tell the user the skill cannot run and stop.

## Coordinates

All coordinates use **CGWindow coordinate system** — logical points with origin at the **top-left of the primary display**.

- x increases to the right
- y increases downward
- Values are in logical points (not pixels)

Use `sagui screen size` to get screen dimensions.

## Commands

### key — Simulate keyboard input

```bash
sagui key shortcut command c             # Keyboard shortcut (Cmd+C)
sagui key shortcut command shift a       # Multi-modifier shortcut
sagui key down shift                     # Press key without releasing
sagui key up shift                       # Release a held key
sagui key type "Hello, World!"           # Type text character by character
sagui key type "slow" --interval 0.1     # Type with delay between keystrokes
sagui key shortcut return                # Return key (alias for returnKey)
sagui key shortcut backspace             # Backspace (alias for delete)
sagui key list                           # List every valid key name
```

| Subcommand | Arguments | Optional |
|---|---|---|
| `shortcut` | `` (key names) | |
| `down` | `` | |
| `up` | `` | |
| `type` | `` | `--interval ` (default: 0) |
| `list` | | |

#### Supported key names

**Modifiers**: `command`, `shift`, `rightShift`, `control`, `rightControl`, `option`, `rightOption`, `capsLock`, `function`

**Letters**: `a`-`z`

**Numbers**: `zero`-`nine`

**Function keys**: `f1`-`f20`

**Arrow keys**: `upArrow`, `downArrow`, `leftArrow`, `rightArrow`

**Navigation**: `home`, `end`, `pageUp`, `pageDown`, `help`

**Special**: `return`, `returnKey`, `enter`, `tab`, `space`, `escape`, `backspace`, `delete`, `forwardDelete`

**Keypad**: `keypad0`-`keypad9`, `keypadDecimal`, `keypadMultiply`, `keypadPlus`, `keypadMinus`, `keypadDivide`, `keypadEnter`, `keypadClear`, `keypadEquals`

**Media**: `volumeUp`, `volumeDown`, `mute`, `brightnessUp`, `brightnessDown`

**JIS**: `jisYen`, `jisUnderscore`, `jisEisu`, `jisKana`

### mouse — Control mouse cursor

Coordinates are **screen points** (origin top-left).

```bash
sagui mouse position                                                  # Print current position
sagui mouse move --x 500 --y 300                                      # Move to absolute position
sagui mouse move-relative --dx 50 --dy -30                            # Move relative to current
sagui mouse click                                                     # Left click at current position
sagui mouse click --right                                             # Right click
sagui mouse click --double                                            # Double-click
sagui mouse click --triple                                            # Triple-click
sagui mouse click --x 500 --y 300                                    # Click an exact position
sagui mouse click --right --x 500 --y 300                            # Right-click an exact position
sagui mouse drag --from-x 100 --from-y 100 --to-x 400 --to-y 400    # Drag
sagui mouse scroll --vertical 5                                       # Scroll up
sagui mouse scroll --vertical -3                                      # Scroll down
sagui mouse scroll --horizontal 2                                     # Scroll left
```

| Subcommand | Required | Optional |
|---|---|---|
| `position` | | |
| `move` | `--x`, `--y` | |
| `move-relative` | `--dx`, `--dy` | |
| `click` | | `--x ` and `--y `, `--right`, `--double`, `--triple` |
| `drag` | `--from-x`, `--from-y`, `--to-x`, `--to-y` | |
| `scroll` | | `--vertical `, `--horizontal ` |

### screen — Screenshots and screen queries

```bash
sagui screen size                                          # Print screen dimensions
sagui screen screenshot                                    # Save screenshot to screenshot.png
sagui screen screenshot --output capture.png               # Save to specific path
sagui screen screenshot --region 0,0,500,500               # Capture specific region
sagui screen pixel --x 100 --y 200                         # Get pixel color (R G B A)
sagui screen locate button.png                             # Find image on screen (X Y W H)
sagui screen locate button.png --confidence 0.8            # Find with lower threshold
sagui screen locate-center button.png                      # Find image center (X Y)
```

| Subcommand | Required | Optional |
|---|---|---|
| `size` | | |
| `screenshot` | | `--output ` (default: `screenshot.png`), `--region ` |
| `pixel` | `--x`, `--y` | |
| `locate` | `` | `--confidence ` |
| `locate-center` | `` | `--confidence ` |

**Image recognition** uses Metal-accelerated normalized cross-correlation. Default confidence threshold is 0.95. Lower it for fuzzy matching.

### agent — AI-powered automation

```bash
sagui agent "Open Safari and search for Swift"                    # Uses OPENAI_API_KEY and default model
sagui agent "Click the submit button" --api-key sk-...            # Run with OpenAI
sagui agent "Fill in the form" --model gpt-5.6-sol --reasoning-effort low --max-iterations 10
sagui agent "Press the Save button" --vision-mode automatic       # Prefer semantic AX elements when available
```

| Required | Optional |
|---|---|
| `` (string) | `--api-key ` (or `OPENAI_API_KEY` env), `--model ` (default: `gpt-5.6-sol`), `--reasoning-effort ` (default: `low` for GPT-5.6), `--vision-mode ` (default: `always`), `--max-iterations ` (default: 20), `--delay ` (default: 1.0), `--no-screen-context` |

The command prints the effective reasoning effort at startup and a concise
`Reasoning:` summary for every step. This summary explains the selected actions;
it is not the model's hidden chain of thought.

## Workflow

### Screenshot + mouse/keyboard interaction

1. `sagui screen size` — get screen dimensions.
2. `sagui screen screenshot --output /tmp/screen.png` — capture current state.
3. Read the screenshot image to identify target coordinates.
4. `sagui mouse click --x  --y ` — interact at the identified position.
5. `sagui key type "text"` or `sagui key shortcut command a` — keyboard input.
6. `sagui screen screenshot --output /tmp/verify.png` — verify result.

### Image-based interaction

1. `sagui screen locate-center target.png` — find element position.
2. `sagui mouse click --x  --y ` — click the found coordinates atomically.

## Notes

- All mouse and keyboard commands require Accessibility permissions.
- Use `sagui --version` when reporting issues or checking a CI environment.
- Screenshot and image recognition commands require Screen Recording permissions.
- `key type` supports any Unicode character. Use `key shortcut` with modifier keys for keyboard shortcuts.
- Scroll: positive vertical = up, negative = down; positive horizontal = left, negative = right.
- Image recognition confidence defaults to 0.95; lower for fuzzy matching (e.g., 0.8).
- Supported screenshot formats: PNG, JPG/JPEG, GIF, BMP, TIFF (detected from file extension).

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [NakaokaRei](https://github.com/NakaokaRei)
- **Source:** [NakaokaRei/SwiftAutoGUI](https://github.com/NakaokaRei/SwiftAutoGUI)
- **License:** MIT
- **Homepage:** https://nakaokarei.github.io/SwiftAutoGUI/documentation

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-nakaokarei-swiftautogui-macos-control
- Seller: https://agentstack.voostack.com/s/nakaokarei
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.