Computer Use

Computer Use

Use this skill for desktop UI through Orca's computer-use surface.

Routing:

• Use Orca built-in browser commands (orca snapshot, orca click, orca fill, etc.) only for the browser page embedded inside the Orca app.
• Use orca computer for native desktop apps, external browser windows, app/webview chrome, Orca settings/app UI, and any desktop UI outside Orca's embedded browser.

Preconditions

• Prefer orca computer ...; on Linux, use orca-ide computer ... if orca is unavailable. In this Orca worktree, use ./config/scripts/orca-dev computer ... only when testing the local dev runtime.
• Prefer --json. Screenshot bytes are omitted from JSON and written to screenshot.path.
• Do not push, submit forms, send messages, buy items, delete data, change account settings, or expose secrets unless the user explicitly asked for that action.
• If an app contains sensitive content, read only what the user requested.

orca status --json
orca computer capabilities --json

Core Loop

orca computer list-apps --json
orca computer get-app-state --app com.spotify.client --json
orca computer click --app com.spotify.client --element-index 42 --json

Use the fresh state returned by each action for the next element index. Element indexes go stale after navigation, focus changes, scrolling, window changes, or app re-rendering.

App Selectors

Prefer bundle IDs from list-apps; names are acceptable when unambiguous. Use pid: only when bundle ID or name matching is ambiguous.

orca computer get-app-state --app com.microsoft.edgemac --json
orca computer get-app-state --app Spotify --json
orca computer get-app-state --app pid:12345 --json

For apps with multiple windows or ambiguous titles, run list-windows first. Once you choose a window, pass the same --window-id or --window-index to get-app-state and later actions until the target window changes.

Commands

orca computer permissions --json
orca computer capabilities --json
orca computer list-apps --json
orca computer list-windows --app  --json
orca computer get-app-state --app  --json
orca computer get-app-state --app  --restore-window --json
orca computer click --app  --element-index  --json
orca computer click --app  --x 100 --y 100 --json
orca computer perform-secondary-action --app  --element-index  --action  --json
orca computer set-value --app  --element-index  --value "text" --json
orca computer type-text --app  --text "text" --json
orca computer press-key --app  --key Return --json
orca computer hotkey --app  --key CmdOrCtrl+A --json
orca computer paste-text --app  --text "text" --json
orca computer scroll --app  (--element-index  | --x  --y ) --direction down --json
orca computer drag --app  --from-element-index  --to-element-index  --json
orca computer drag --app  --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json

Use --no-screenshot only when pixels are not needed. Use --text-stdin or --value-stdin for sensitive text so payloads do not land in shell history. On Linux and Windows, action payloads still pass through a short-lived local operation file, so avoid sending secrets unless the user explicitly asked for them:

printf '%s' "$TEXT" | orca computer set-value --app  --element-index  --value-stdin --json

Action Rules

• Prefer semantic actions: set-value for editable fields, click for controls, perform-secondary-action only for listed action names.
• Use type-text only after focusing a field and confirming the app has a focused text receiver.
• Use press-key for single/navigation keys such as Return, Escape, Tab, and arrows. Use hotkey for shortcuts; prefer CmdOrCtrl+... for cross-platform combos.
• Some actions work in background apps, but this is app-dependent. If success does not change the UI, refresh state and choose a more semantic action or restore/focus the window.
• Coordinates are window-local; use coordinates from the latest screenshot/state for the same target window.

Screenshots

get-app-state returns tree+screenshot. Use the tree for indexes/actions and the screenshot for visual confirmation; failed capture usually means hidden, minimized, off-screen, or permission-blocked.

App Notes

Browsers: for Edge, Chrome, and similar browser windows, set the address/search field directly, then press Return. Do not assume raw typing went to the address bar.

orca computer get-app-state --app com.microsoft.edgemac --json
orca computer set-value --app com.microsoft.edgemac --element-index  --value "test123" --json
orca computer press-key --app com.microsoft.edgemac --key Return --json

Spotify: refresh after playback clicks; the UI often changes asynchronously.

Slack: the accessibility tree may be shallow while the screenshot contains useful information. Reading visible Slack UI is fine when requested; sending messages or triggering workflows still needs explicit permission.

Errors

• app_not_found: run list-apps and retry with the bundle ID.
• element_not_found: index is stale; run get-app-state again.
• action_failed: inspect the element role/actions and try a more semantic action.
• Empty tree or no screenshot: app may have no visible window, be minimized, or need permissions.
• Permission errors: run orca computer permissions --json, use the setup UI, then retry.

Next Action

Confirm Orca status unless already checked, run orca computer capabilities --json, then get the target app state with orca computer get-app-state --app --json.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ccppvv
• Source: ccppvv/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.