Macos Agent Ops

macos-agent ops

Contract

Prereqs:

• macOS host with Accessibility and Automation permissions granted for Terminal and target apps.
• Homebrew-installed macos-agent available on PATH.
• agent-out available on PATH for canonical project-scoped artifact directories.
• cliclick, osascript, and im-select available on PATH.
• ABC/US input source enabled in macOS Input Sources.
• For full AX surface (ax attr/action/session/watch), Hammerspoon runtime should be healthy (hs + hs.ipc).

Inputs:

• Command entrypoint: $AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh.
• Optional env vars:
• MACOS_AGENT_OPS_INPUT_SOURCE_ID (preferred; example: abc) for deterministic input source target.
• MACOS_AGENT_REAL_E2E_INPUT_SOURCE (legacy fallback) if MACOS_AGENT_OPS_INPUT_SOURCE_ID is unset.
• MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1 to bypass automatic input source switch.
• CODEX_MACOS_AGENT_AX_BACKEND=auto|hammerspoon|applescript to control AX backend preference.

Outputs:

• macos-agent JSON/text command output.
• Artifacts generated by macos-agent under a canonical agent-out project run directory.

Exit codes:

• 0: success
• 1: runtime failure or missing runtime dependency
• 2: usage error

Failure modes:

• wait app-active timeouts because focus was stolen (Control Center/Spotlight/notifications).
• window.activate failure when app relaunch/update state is stuck; wrapper now uses --reopen-on-fail.
• missing im-select causes input-source failure before keyboard-input actions.
• not authorized / Apple Events denial from TCC permissions.
• AX target mismatch (selector too broad / stale node id) during ax operations.
• AX gate failures (--gate-*) when app/window/selector readiness conditions are not met.
• AX postcondition failures (--postcondition-*) when state does not change as expected after mutation.
• observe screenshot --if-changed-baseline path missing or unreadable.
• trace directory validation failures when using --trace/--trace-dir and path is not writable.

Scripts (only entrypoints)

• $AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh

Workflow

1. Resolve Homebrew macos-agent binary path:

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh where

1. Ensure deterministic input source (default target: abc):

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh input-source
# or explicit source id
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh input-source --id com.apple.keylayout.ABC

1. Run readiness checks (preflight + AX list smoke check):

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh doctor
# optional target for AX smoke
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-app Arc
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh doctor --ax-bundle-id com.google.Chrome

1. Inspect unified permission state directly (screen_recording / accessibility / automation / ready):

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json preflight --include-probes

1. Run quick app foreground check with reopen recovery:

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Finder
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh app-check --app Arc --timeout-ms 15000
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh app-check --bundle-id com.google.Chrome

1. Run AX-only health check (AX tree probe):

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXWindow
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh ax-check --app Arc --role AXTextField --title-contains Search

1. Use AX wait primitives before mutation to stabilize dynamic UI:

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-present --app Arc --role AXButton --title-contains "Compose" \
  --match-strategy contains --timeout-ms 2500 --poll-ms 80

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json wait ax-unique --app Arc --role AXTextField --title-contains "Search" \
  --selector-explain --timeout-ms 2500 --poll-ms 80

1. Run robust mutating AX actions with gating + postconditions:

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax click --app Arc --role AXButton --title-contains "Play" \
  --match-strategy contains --selector-explain --reselect-before-click \
  --fallback-order ax-press,ax-confirm,frame-center,coordinate \
  --gate-app-active --gate-window-present --gate-ax-unique \
  --postcondition-focused true --postcondition-timeout-ms 2000 \
  --allow-coordinate-fallback

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json ax type --app Arc --role AXTextField --title-contains "Search" \
  --text "lofi" --clear-first --submit --paste \
  --gate-app-active --gate-window-present --gate-ax-present \
  --postcondition-attribute AXValue --postcondition-attribute-value "lofi" \
  --postcondition-timeout-ms 2000 --allow-keyboard-fallback

1. Use selector-frame screenshots for visual proof of target alignment:

ARTIFACT_DIR="$(agent-out project --topic macos-agent-ops --mkdir)"
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --role AXButton --title-contains "Play" --selector-padding 12 \
  --path "$ARTIFACT_DIR/macos-agent-selector.png"

1. Use diff-aware screenshots to avoid noisy artifacts when nothing changed:

ARTIFACT_DIR="$(agent-out project --topic macos-agent-ops --mkdir)"
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$ARTIFACT_DIR/macos-agent-state.png" \
  --if-changed --if-changed-threshold 2

1. Use one-shot debug bundle for triage artifacts:

ARTIFACT_DIR="$(agent-out project --topic macos-agent-ops --mkdir)"
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json debug bundle --active-window \
  --output-dir "$ARTIFACT_DIR/macos-agent-debug"

1. Run routine scripted scenarios:

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh scenario --file /path/to/scenario.json

1. Pass through raw macos-agent commands with JSON errors + trace when needed:

$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json --error-format json --trace \
  input click --x 200 --y 160

Screenshot-Based Triage Rules

• During runtime failure triage, the agent SHOULD capture an active-window screenshot before retry/remediation.
• For wait app-active and window.activate failures, screenshot capture SHOULD be first-line diagnostics.
• Screenshot artifacts MUST be written under a canonical agent-out run directory.
• The agent SHOULD include screenshot path + failing command + error message in the final diagnostic summary.

Recommended command (via the skill entrypoint):

ARTIFACT_DIR="$(agent-out project --topic macos-agent-failure --mkdir)"
$AGENT_HOME/skills/tools/app-runtime/macos-agent-ops/scripts/macos-agent-ops.sh run -- \
  --format json observe screenshot --active-window \
  --path "$ARTIFACT_DIR/active-window.png"

Common Errors And Prevention

• error: input source mismatch after switch attempt ...
• Run .../macos-agent-ops.sh input-source --id abc and verify current is com.apple.keylayout.ABC or ...US.
• Ensure im-select is installed: brew install im-select.
• If you intentionally need non-ABC IME, set MACOS_AGENT_OPS_SKIP_INPUT_SOURCE_SWITCH=1.

• error: timed out waiting for app-active ...
• Keep hands off keyboard/mouse while checks are running.
• Close Control Center/Spotlight overlays and reduce notification interruptions.
• Increase timeout (--timeout-ms 12000 or higher) for slow app transitions.

• error: ax.list ... timed out during wait ax-present / wait ax-unique
• Run app-check first, then retry AX waits after app focus is stable.
• Increase AX wait timeout/poll (--timeout-ms, --poll-ms) and narrow selectors.
• Keep backend on auto unless you need to force Hammerspoon behavior (CODEX_MACOS_AGENT_AX_BACKEND).

• window activate failed ...
• Wrapper already uses --reopen-on-fail; rerun once to allow quit/relaunch recovery.
• For flaky apps (e.g., Spotify updater), kill stale relauncher and retry.

• not authorized / Apple Events failures
• Re-run doctor and fix Accessibility/Automation in System Settings.

• selector returned zero AX matches or ambiguous matches
• Narrow selectors and use --match-strategy exact|prefix|suffix|regex when contains is too broad.
• Use --selector-explain and wait ax-unique before ax click/ax type.
• Prefer --reselect-before-click in dynamic pages where node ids churn.

• gate failure (--gate-*) before mutation
• Add/adjust readiness gates based on context (--gate-app-active, --gate-window-present, --gate-ax-present, --gate-ax-unique).
• Increase gate timeout/poll for slower app transitions (--gate-timeout-ms, --gate-poll-ms).

• postcondition failure (--postcondition-*) after mutation
• Verify you are asserting the correct target/attribute (AXValue, focus state, etc.).
• Increase postcondition timeout on animation-heavy apps (--postcondition-timeout-ms).

• error: --if-changed-baseline path does not exist ...
• Verify baseline path exists before running capture.
• If baseline is optional, remove --if-changed-baseline and let current output file be baseline.

• trace directory is not writable (trace.write errors)
• Use a writable custom path with --trace-dir.
• Prefer agent-out project --topic macos-agent-ops --mkdir for reproducible artifacts.

• Homebrew macos-agent missing
• Install and link with Homebrew: brew install macos-agent.

Practical Notes

• Prefer canonical flags in examples:
• --window-title-contains (instead of legacy --window-name)
• input type --submit (instead of legacy --enter)
• Keep --error-format json enabled in automation loops when you need machine-parseable failures.
• Use debug bundle first when a run fails and root cause is unclear.
• Parse preflight JSON permissions (screen_recording / accessibility / automation / ready) before mutating runs.

References

• skills/tools/app-runtime/macos-agent-ops/references/e2e-ref-01-preflight-focus.md
• skills/tools/app-runtime/macos-agent-ops/references/e2e-ref-02-finder-routine.md
• skills/tools/app-runtime/macos-agent-ops/references/e2e-ref-03-matrix-routine.md

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: graysurf
• Source: graysurf/agent-kit
• License: MIT

Install and usage instructions live in the source repository linked above.