MCP MacOSControl

MCP macOS Control

A native macOS MCP (Model Context Protocol) server built in Swift that provides 74 tools for comprehensive computer control — mouse, keyboard, screen capture, OCR, window management, Vision analysis, CoreML intelligence, accessibility tree reading, event-driven UI waiting via AXObserver, application lifecycle event waiting via NSWorkspace, and iPhone Mirroring automation via macOS Sequoia.

[](https://github.com/aerocristobal/MCP-MacOSControl/actions/workflows/ci.yml)

Overview

MCP macOS Control enables AI agents to perceive and interact with macOS applications and, through iPhone Mirroring, iOS apps on a connected iPhone. All processing happens on-device using native Apple frameworks.

Key capabilities:

• Full mouse and keyboard automation via CoreGraphics events
• Screen capture with OCR text extraction (Vision framework)
• iPhone Mirroring control with normalized coordinates (macOS 15+)
• Accessibility tree reading via AXUIElement API
• On-device CoreML LLM inference (zero cloud tokens)
• Real-time screen monitoring with continuous capture
• Multi-display support
• Rate limiting, structured logging, and CI/CD

Requirements

• macOS 13.0 (Ventura) or later (macOS 15 for iPhone Mirroring)
• Apple Silicon or Intel Mac (Apple Silicon recommended)
• Swift 5.9+ / Xcode 16+
• MCP SDK 0.12.0+

Installation

git clone https://github.com/aerocristobal/MCP-MacOSControl.git
cd MCP-MacOSControl
swift build -c release

The executable is at .build/release/mcp-macos-control.

Claude Desktop Configuration

Add to ~/Library/Application Support/Claude/claude_desktop_config.json:

{
  "mcpServers": {
    "macos-control": {
      "command": "/path/to/MCP-MacOSControl/.build/release/mcp-macos-control"
    }
  }
}

Permissions

macOS requires explicit permissions:

1. Accessibility (System Settings > Privacy & Security > Accessibility) -- required for mouse, keyboard, window activation, and accessibility tree
2. Screen Recording (System Settings > Privacy & Security > Screen Recording) -- required for screenshots, OCR, and continuous capture

See [docs/PERMISSIONS.md](docs/PERMISSIONS.md) for detailed setup.

Security

The threat model, AppleScript denylist, audit-record schema, and NIST SP 800-53 control mapping live in [docs/SECURITY.md](docs/SECURITY.md). Vulnerabilities should be reported via a private GitHub Security Advisory — see §6 of that document.

Cooperative tool-call cancellation (STORY-027). The server honours the MCP notifications/cancelled message. Five long-running tools propagate cancellation to their resource teardown path — wait_for_ui_event (AXObserver unregister), wait_for_element_state (polling loop exits within one cycle), wait_for_app_event (NSWorkspace observer removal), smart_interact (between-layer abort, layer-skip), and run_applescript (osascript SIGTERM, SIGKILL escalation after 1000ms). SIGTERM to the server drains every in-flight call via the same pathway. See [docs/SECURITY.md §5](docs/SECURITY.md) for the full table.

Supply-chain controls (STORY-021, see §8). Every push and pull request:

• Generates a CycloneDX 1.5+ SBOM (via Syft) covering every transitive Swift package.
• Runs Grype SBOM scanning + GitHub's native Dependency Review against the SBOM.
• Fails the build on CVSS ≥ 9.0 (Critical) findings; CVSS 7.0–8.9 (High) warns via a PR comment without blocking.
• Enforces a license allow-list: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC.

Tagged releases attach the SBOM (and a VEX document, when statements exist) to the GitHub Release as evidence artifacts. Dependabot opens monthly Swift-package and weekly GitHub-Actions upgrade PRs against the same gate. Policy is declared in [.github/sbom-policy.yml](.github/sbom-policy.yml).

Compliance Artifacts

Machine-readable compliance posture for downstream SSP authors, assessors, and continuous-monitoring consumers. All three artifacts are versioned in oscal/ and validated against the NIST OSCAL schemas on every push.

| Artifact | OSCAL model | Purpose | |----------|-------------|---------| | [oscal/component-definition.json](oscal/component-definition.json) | OSCAL 1.1.2 Component Definition | Mirror of [docs/SECURITY.md](docs/SECURITY.md) §7/§8 control implementation, with rel:implementation and rel:verification links into source and tests. Maintained per STORY-022. | | [oscal/plan-of-action-and-milestones.json](oscal/plan-of-action-and-milestones.json) | OSCAL 1.1.2 POA&M | One open item per accepted-risk statement in docs/SECURITY.md §4, plus historical closed items. Each item carries status, owner, related-controls, and milestones. UUID registry: [oscal/poam-id-allocations.md](oscal/poam-id-allocations.md). Maintained per STORY-037. | | [oscal/assessment-results.json](oscal/assessment-results.json) | OSCAL 1.1.2 Assessment Results | Continuous-monitoring observations derived from the STORY-024 AuditRecord stream. One observation per AuditRecord. Append-only; nightly emit job amends new observations without rewriting prior ones. AuditRecord ↔ Observation mapping: [oscal/assessment-results-mapping.md](oscal/assessment-results-mapping.md). Maintained per STORY-037. |

Drift between docs/SECURITY.md and the OSCAL artifacts is enforced bidirectionally by [swift run oscal-drift check](Sources/OscalDrift/main.swift) — CI fails on any of: a §7/§8 control claim without an OSCAL implemented-requirement, an OSCAL implemented-requirement not referenced by §7/§8, a §4 accepted-risk statement without a matching open POA&M item, or a POA&M item whose security-md-section no longer exists.

Tools (75 total, 16 modules)

Mouse Control (MouseModule -- 9 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | click_screen | Click at coordinates (left/right/middle) | x, y | | double_click | Double-click at coordinates | x, y | | move_mouse | Move cursor | x, y | | mouse_down | Hold mouse button | -- | | mouse_up | Release mouse button | -- | | drag_mouse | Drag between two points | fromx, fromy, tox, toy | | scroll | Scroll in a direction | direction | | get_screen_size | Get screen resolution | -- | | list_displays | List all connected displays | -- |

Keyboard Control (KeyboardModule -- 4 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | type_text | Type text at cursor | text | | press_keys | Key combos (e.g., [["cmd", "c"]]) | keys | | key_down | Hold a key | key | | key_up | Release a key | key |

Screen Capture and OCR (ScreenCaptureModule -- 2 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | take_screenshot | Capture screen or window as PNG | -- | | take_screenshot_with_ocr | Screenshot + text extraction | -- |

Window Management (WindowModule -- 2 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | list_windows | List all open windows | -- | | activate_window | Bring window to foreground | title_pattern |

Continuous Capture (ContinuousCaptureModule -- 6 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | start_continuous_capture | Start real-time capture | capture_type | | stop_continuous_capture | Stop capture session | -- | | get_capture_frame | Get latest frame | -- | | list_capturable_displays | List available displays | -- | | list_capturable_windows | List capturable windows | -- | | list_capturable_applications | List running apps | -- |

Vision Analysis (VisionModule -- 5 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | classify_image | Scene/object classification | imagedata | | detect_objects | Object detection with bounding boxes | imagedata | | detect_rectangles | Rectangle/UI element detection | imagedata | | detect_saliency | Attention region detection | imagedata | | detect_faces | Face detection | image_data |

CoreML Intelligence (CoreMLModule -- 8 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | list_coreml_models | List available models | -- | | load_coreml_model | Load model for inference | name, path | | unload_coreml_model | Unload model | name | | get_model_info | Model metadata | name | | generate_text_llm | On-device text generation | modelname, prompt | | analyze_screen_with_llm | Screen + Vision + LLM | modelname, instruction | | intelligent_screen_summary | NLP-based screen summary | -- | | extract_key_info | Extract entities from OCR | ocr_results |

Real-Time Analysis (RealtimeModule -- 4 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | analyze_screen_now | Quick capture + Vision analysis | -- | | start_screen_monitoring | Continuous analysis | -- | | get_monitoring_results | Latest analysis results | -- | | stop_screen_monitoring | Stop monitoring | -- |

System (SystemModule -- 3 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | check_permissions | Check Accessibility and Screen Recording | -- | | wait_milliseconds | Pause execution | milliseconds | | wait_for_text | Poll OCR until text appears | text |

Accessibility (AccessibilityModule -- 5 tools)

| Tool | Description | Required Params | |------|-------------|----------------| | accessibility_tree | Read AXUIElement tree of macOS app | -- | | click_element | Click an element by semantic AX attributes | -- | | perform_ax_action | Dispatch any standard AX action (AXPress, AXIncrement, AXShowMenu, etc.) | -- | | find_elements | Find elements matching a semantic predicate | -- | | element_at_position | Resolve the AX element under a screen coordinate | x, y |

Returns structured JSON with role, title, value, position, size, and children for each UI element. Configurable max_depth (default 6). Note: works for macOS apps only -- iPhone Mirroring content requires iphone_screenshot_with_ocr.

Event-Driven Waiting (WaitForUIEventModule -- 1 tool)

| Tool | Description | Required Params | |------|-------------|----------------| | wait_for_ui_event | Subscribe to an AXObserver notification and block until it fires | notification, application |

Replaces fixed sleeps and screenshot-polling loops with precise event-driven synchronization. Supported notifications: AXWindowCreated, AXUIElementDestroyed, AXFocusedUIElementChanged, AXValueChanged, AXSelectedTextChanged, AXTitleChanged, AXMainWindowChanged, AXFocusedWindowChanged. Multiple concurrent calls on the same (application, notification) pair share a single underlying AXObserver. Default timeout 30s, hard cap 300s. See the ax_observer_notifications MCP prompt for routing guidance.

Element-State Polling (WaitForElementStateModule -- 1 tool)

| Tool | Description | Required Params | |------|-------------|----------------| | wait_for_element_state | Poll the AX tree until an element matches a state condition | condition, application |

The documented fallback to wait_for_ui_event for transitions that have no AXObserver notification (a button becoming enabled, a spinner disappearing, a row becoming selected). Condition syntax is " = ". Boolean fields: enabled, exists, focused, selected, expanded, visible_in_viewport, is_main, is_minimized, is_frontmost (e.g. enabled = true, exists = false). String field: value (exact, case-sensitive, quoted — e.g. value = 'Connected'). Fixed 100ms poll cadence (override with MCP_MACOS_CONTROL_POLL_INTERVAL_MS). Default timeout 30s, hard cap 120s — for long event-driven waits prefer wait_for_ui_event.

Application Lifecycle Events (WaitForAppEventModule -- 1 tool)

| Tool | Description | Required Params | |------|-------------|----------------| | wait_for_app_event | Subscribe to an NSWorkspace application-lifecycle notification and block until it fires | event |

The event-driven way to sequence "open app → app is ready → interact" workflows. Unlike wait_for_ui_event (AXObserver, requires an already-running AX-queryable target), this is pre-launch capable and needs no accessibility permission. Supported events: launched, activated, terminated, deactivated, hidden, unhidden. Optional bundle_identifier (reverse-DNS, e.g. com.apple.TextEdit) filters to one app; omit it to resolve on the next matching event from any application. Resolves only on the next transition, never the current state. Multiple concurrent calls on the same (event, bundle_identifier) share one underlying observer. Default timeout 30s, hard cap 300s. See the interaction_hierarchy MCP prompt for when to prefer it over wait_for_ui_event.

Interaction Router (SmartInteractModule -- 1 tool) — recommended for AI agents

| Tool | Description | Required Params | |------|-------------|----------------| | smart_interact | State an intent (click/type) and a target; the router auto-selects and falls through AX semantic → AppleScript → hit-test → coordinate layers | intent |

The recommended entry point for UI interaction. Instead of choosing between click_element, run_applescript, element_at_position, and click_screen yourself, state your intent and let the router pick the most reliable available layer. It consults the per-app capability registry to skip layers known to fail for the target app, and returns a decision_log documenting every layer attempted, skipped, or failed (with reasons), plus interaction_method and confidence. On exhaustion it returns the structured error all_layers_failed with retry suggestions. The router only knows the action dispatched, not its effect — verify after acting with wait_for_ui_event / wait_for_element_state. See the interaction_hierarchy MCP prompt.

iPhone Mirroring (IPhoneMirroringModule -- 21 tools)

Requires macOS 15 (Sequoia) with iPhone Mirroring configured. All coordinates use normalized 0.0-1.0 range relative to the iPhone screen content area.

| Tool | Description | Required Params | |------|-------------|----------------| | iphone_status | Check mirroring status and connection | -- | | iphone_launch | Launch/activate iPhone Mirroring | -- | | iphone_calibrate | Force content rect re-calibration | -- | | iphone_tap | Tap at normalized coordinates | x, y | | iphone_double_tap | Double-tap | x, y | | iphone_long_press | Long-press with duration | x, y | | iphone_swipe | Swipe with ease-in-out curve | startx, starty, endx, endy | | iphone_scroll | Scroll wheel events | -- | | iphone_type_text | Paste text via clipboard | text | | iphone_clear_text | Select all + delete | -- | | iphone_press_key | Send key event | key | | iphone_home | Go to home screen (Cmd+1) | -- | | iphone_app_switcher | Open App Switcher (Cmd+2) | -- | | iphone_spotlight | Open Spotlight (Cmd+3) | -- | | iphone_screenshot | Capture iPhone screen content | -- | | iphone_screenshot_with_ocr | Screenshot + OCR (normalized coords) | -- | | iphone_analyze_screen_now | Vision analysis on iPhone screen | -- | | iphone_analyze_with_llm | iPhone screen + CoreML LLM | modelname, instruction | | iphone_open_app | Open iOS app by name via Spotlight | appname | | iphone_wait_for_text | Poll iPhone OCR for text | text | | iphone_reconnect | Wait for mirroring reconnection | -- |

How iPhone Mirroring works: The server detects the iPhone Mirroring window, calibrates the iPhone screen content area via Vision rectangle detection, and translates normalized coordinates to absolute screen coordinates. Swipe gestures use ease-in-out easing with an initial nudge to ensure iOS gesture recognition. Text input uses clipboard paste for universal language support.

Environment Vari

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aerocristobal
• Source: aerocristobal/MCP-MacOSControl
• License: MIT

Install and usage instructions live in the source repository linked above.