Peekaboo

Peekaboo

Native macOS GUI automation via peekaboo, using an AX-first hybrid approach that prioritizes accessibility-tree interaction when reliable and switches to screenshot, coordinate, JavaScript, or vision-based fallbacks when AX coverage breaks down in real apps.

Terminology used in this file:

• AX (Accessibility tree): The macOS accessibility hierarchy used for deterministic element discovery and interaction (B1, T2, etc.).
• peekaboo: The macOS automation CLI used for discovery (see, image), interaction (click, type, hotkey), and app/window/dialog control.
• UI-TARS: Vision model server used when AX discovery fails or times out on complex UIs.
• FSM: Finite State Machine controlling retries, transitions, and recovery paths.
• Retina scale: Display pixel ratio (typically 2.0) used to map screenshot coordinates correctly on Retina displays.
• snapshot ID: The snapshot_id returned by see; required for stable element resolution and invalidated by UI changes.

Setup

brew install jq
brew install --cask peekaboo
mkdir -p ./scripts
chmod +x ./scripts/peekaboo-safe.sh ./scripts/health-check.sh
./scripts/health-check.sh

• Grant Accessibility and Screen Recording permissions for your terminal agent host (for example, Terminal, iTerm2, Claude Code).
• Use ./scripts/peekaboo-safe.sh as the only invocation entrypoint for peekaboo commands.
• For full setup details and troubleshooting, see references/installation-guide.md if present in your local skill copy.

Staying Updated

This skill ships with an UPDATES.md changelog and UPDATE-GUIDE.md for your AI agent.

After installing, tell your agent: "Check UPDATES.md in the peekaboo skill for any new features or changes."

When updating, tell your agent: "Read UPDATE-GUIDE.md and apply the latest changes from UPDATES.md."

Follow UPDATE-GUIDE.md so customized local files are diffed before any overwrite.

---

Quick Start

Run a 3-step baseline flow: health check, discover, interact.

# 1) Health check
./scripts/health-check.sh

# 2) Discover UI state
./scripts/peekaboo-safe.sh see --app Safari --json --path /tmp/peekaboo/

# 3) Interact
./scripts/peekaboo-safe.sh click --on B1 --app Safari

Hard Rules (MUST FOLLOW)

1. All peekaboo calls MUST use ./scripts/peekaboo-safe.sh wrapper - never direct peekaboo calls
2. Sequential execution only - no parallel GUI operations (causes race conditions)
3. NEVER run tccutil reset - resets ALL app permissions, not just one
4. Always --mode screen - window mode returns 698B stubs (beta3 bug)
5. Health check required - run ./scripts/health-check.sh before automation
6. Fresh snapshots mandatory - re-run see before each click sequence (IDs expire)
7. Use see --analyze for Q&A - 50x more token-efficient than full screenshots

Navigation Strategy: Navigate Programmatically, Interact Visually

Core principle: Never waste turns clicking through UI to reach a target. Get there programmatically, then use peekaboo for the last-mile interaction.

| Method | When | Example | |--------|------|---------| | URL scheme | System Settings, Safari pages | open "x-apple.systempreferences:com.apple.Desktop-Settings.extension" | | AppleScript | Apps with scripting dictionary | osascript -e 'tell app "Finder" to open folder "Desktop"' | | CLI/flags | Apps with CLI args | open -a TextEdit /path/to/file | | Menu command | Navigate within already-open app | ./scripts/peekaboo-safe.sh menu click --path "View > Show Sidebar" | | AX sidebar click | Last resort - only if above fail | Unreliable for tightly-packed items (System Settings proven broken) |

Hostile AX apps (use URL scheme / AppleScript only):

• System Settings - SwiftUI toggles invisible to AX roles, sidebar clicks misfire

Routing Algorithm (Interaction Tier)

if target_is_native_macos_ui():
    use_ax_interaction()  # click --on B1, type, hotkey
elif target_is_swiftui_toggle():
    use_screenshot_coordinate_click()  # screenshot -> identify -> click --coords
elif target_is_web_form():
    use_safari_js_injection()  # osascript JavaScript
elif target_is_web_button():
    use_coordinate_click()  # click --coords X,Y --no-auto-focus
elif see_timeout_complex_app():
    start_ui_tars_server()  # vision provider required
elif dialog_or_sheet():
    add_no_auto_focus_flag()  # prevent focus conflicts
else:
    escalate_to_human()

FSM State Machine

| State | Entry Condition | Actions | Exit Criteria | Retry Budget | Next State | |-------|----------------|---------|---------------|--------------|------------| | PREFLIGHT | Start | health-check.sh, mkdir /tmp/peekaboo | allcheckspass | 1 | CLASSIFYTARGET | | CLASSIFYTARGET | Health pass | Determine interaction tier | tieridentified | 1 | ACQUIREWINDOW | | ACQUIREWINDOW | Target known | app switch, window focus | appfrontmost | 3 | WAITFORUISTABLE | | WAITFORUISTABLE | Window focused | sleep, check spinner state | uielementsstable | 2 | DISCOVER | | DISCOVER | UI stable | see --app --json --path | elementsfound | 3 | INTERACT | | INTERACT | Elements found | click/type/menu based on tier | actioncompleted | 5 | VERIFY | | VERIFY | Action attempted | see/image for state change | successconfirmed | 2 | EXTRACT or CLEANUP | | EXTRACT | Data needed | see --analyze or JS injection | dataextracted | 2 | CLEANUP | | CLEANUP | Task complete | clean snapshots, restore clipboard | cleanupdone | 1 | DONE | | RECOVER | Error detected | Retry from previous state | retrylimithit | 3 | HUMANHANDOFF or ABORT | | HUMANHANDOFF | Auth/CAPTCHA/TCC | Log issue, preserve evidence | humanresolved | inf | CLASSIFY_TARGET | | ABORT | Fatal error | Log failure, preserve evidence | - | - | - | | DONE | Success | Return results | - | - | - |

Bug-Specific Transitions

• see timeout -> start UI-TARS -> retry DISCOVER
• Safari AX click misfires -> force coordinate click path
• type targets URL bar -> force JS injection path
• daemon file-not-found -> daemon run --mode manual &
• Chrome window capture fail -> switch to Safari
• System Settings AX sidebar misfire -> use URL scheme: open "x-apple.systempreferences:com.apple.PANE_ID"
• SwiftUI toggle invisible to AX -> screenshot + coordinate click (not AX click)

Core Commands (Canonical Forms Only)

Discovery

• ./scripts/peekaboo-safe.sh see --app AppName --json --path /tmp/peekaboo/
• ./scripts/peekaboo-safe.sh see --analyze "What is the current state?" --app AppName
• ./scripts/peekaboo-safe.sh image --mode screen --path /tmp/peekaboo/screenshot.png

Interaction

• ./scripts/peekaboo-safe.sh click --on B1 --snapshot "$sid" --app AppName
• ./scripts/peekaboo-safe.sh click --coords X,Y --no-auto-focus --app AppName
• ./scripts/peekaboo-safe.sh type "text" --return --profile human --app AppName
• ./scripts/peekaboo-safe.sh hotkey --keys "cmd,w" --app AppName

App Control

• ./scripts/peekaboo-safe.sh app switch --to AppName
• ./scripts/peekaboo-safe.sh window focus --window-title "Title" --app AppName
• ./scripts/peekaboo-safe.sh menu click --path "File > Save As..." --app AppName

Safari Workarounds

• Navigate: ./scripts/peekaboo-safe.sh open "https://url" --app Safari
• Form fill: osascript -e 'tell application "Safari" to do JavaScript "document.querySelector(\"input[name=field]\").value = \"value\"" in document 1'
• Button click: ./scripts/peekaboo-safe.sh click --coords X,Y --no-auto-focus --app Safari

Agent Mode

• ./scripts/peekaboo-safe.sh agent "Natural language task" --model gpt-4o --max-steps 10
• ./scripts/peekaboo-safe.sh agent --resume --max-steps 5

Token Budget

see --analyze ~100 tokens | see --json ~600 | image --mode screen ~1400. Soft limit 50 turns, hard limit 100. Prefer see --analyze over full AX trees. Details in references/token-budget.md.

Capability Matrix

| Feature Area | Status | Last Validated | Notes | |-------------|--------|----------------|-------| | Native macOS UI | validated | 2026-02-22 | 20 tests passed, core capability | | Agent mode | validated | 2026-02-22 | Multi-step automation works | | Safari forms | partial | 2026-02-22 | JS injection required, see playbooks | | System dialogs | validated | 2026-02-22 | Use --no-auto-focus | | File pickers | partial | 2026-02-22 | Compact mode issues | | Menu navigation | validated | 2026-02-22 | App menus work, Apple menu limited | | Spaces management | validated | 2026-02-22 | Full virtual desktop support | | Live capture | validated | 2026-02-22 | Recording and diff detection | | Chrome automation | unsupported | 2026-02-22 | Issue #67, use Safari | | Social media ops | validated | 2026-02-22 | Reddit shadow DOM login, Booking URL-first search, X signup, cookie extraction, CAPTCHA detection -- see playbooks |

Playbook Index

Each playbook is a self-contained field-tested recipe. Status: validated = tested in production, experimental = untested or partially tested.

| Playbook | Use Case | Status | |----------|----------|--------| | native-app-automation.md | Any macOS app workflow | validated | | safari-login.md | Generic web authentication | validated | | dialog-and-file-picker.md | System dialogs, save/open | validated | | cross-app-data-transfer.md | Clipboard transfer between apps | validated | | reddit-login.md | Reddit auth (shadow DOM + React events) | validated | | reddit-data-extraction.md | Post Insights extraction (author-only analytics) | validated | | booking-search.md | Hotel search (URL-first, VLM extraction) | validated | | x-signup.md | X/Twitter account registration + email verification | validated | | file-upload.md | macOS file dialog from web apps (Cmd+Shift+G) | validated | | captcha-solver.md | CAPTCHA detection and handling | experimental | | cookie-extractor.md | Browser session cookie extraction | experimental | | instagram-monitor.md | Instagram content monitoring | experimental |

Reference Index

• references/commands.md - Complete command reference with all flags
• references/safari-workarounds.md - Web automation canonical patterns
• references/patterns.md - Common automation workflows and recipes
• references/failures.md - Error recovery procedures and symptom table
• references/validated-tests.md - Field validation results and capabilities
• references/token-budget.md - Token costs and optimization strategies
• references/hard-tests.md - Edge-case and stress test catalog
• references/installation-guide.md - Full setup from zero to working

Pre-flight Checklist

./scripts/health-check.sh
# Must return: cli_exists=true, accessibility_granted=true, ax_test_success=true
mkdir -p /tmp/peekaboo

Error Recovery

After 3 retries on same state: log state + evidence screenshot + escalate to HUMAN_HANDOFF. Quick recovery: app switch + fresh see + health-check.sh.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: buildoak
• Source: buildoak/fieldwork-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.