Naturo

Naturo — Desktop Automation Engine (Eyes + Hands for AI Agents)

> See, click, type, capture. Desktop automation core only.

[](https://github.com/AcePeak/naturo/actions/workflows/build.yml) [](https://codecov.io/gh/AcePeak/naturo) [](https://pypi.org/project/naturo/) [](https://pypi.org/project/naturo/) [](https://pypi.org/project/naturo/) [](https://github.com/AcePeak/naturo#platform-support) [](https://opensource.org/licenses/MIT)

> Why naturo? It is the only open-source Windows automation engine with commercial-RPA-grade multi-framework recognition — UIA + MSAA/IA2 + Java Access Bridge + Electron/CDP + vision fusion. UIA-only rivals (UFO², Windows-MCP, Terminator) are blind to Electron and Java app content. See the reproducible proof: [Recognition coverage benchmark → docs/RECOGNITION.md](docs/RECOGNITION.md).

> Automating Chinese desktop apps? 钉钉 / 飞书 / 企业微信 / 同花顺 / WPS are built on the exact frameworks (CEF, 自绘/Duilib, hybrid, COM) that UIA-only engines can't see — and naturo can: [CJK coverage & China-market wedge → docs/CJKCOVERAGE.md](docs/CJKCOVERAGE.md).

> How it works (deep dive): the architecture, cascade fusion, correctness tagging, and the native C++ core, with diagrams — [How Naturo Works → docs/blog/how-naturo-works.md](docs/blog/how-naturo-works.md).

> New here? Get Claude to open, type into, and save a Notepad file in under five minutes: [5-minute quickstart → docs/QUICKSTART.md](docs/QUICKSTART.md).

Tutorials

Guided, end-to-end walkthroughs — each with complete, runnable examples (CLI + the Python SDK), and every command checked against the code:

1. [Automate Notepad in 5 minutes](docs/tutorials/01-automate-notepad.md) — see → click → type → save. The four verbs every naturo automation is built from.
2. [Automate Excel with naturo](docs/tutorials/02-automate-excel.md) — open a workbook, read/write cells, inspect ranges, and build a chart over Excel's COM interface.
3. [Build an AI agent that uses naturo](docs/tutorials/03-ai-agent-with-naturo.md) — run naturo as an MCP server for Claude Desktop/Code, plus a minimal Python tool-use loop that drives naturo.

What You Get

• 🖥️ Screen Capture — Screenshot any window or monitor
• 🌳 UI Tree Inspection — Walk the accessibility tree (UIA / MSAA / IAccessible2 / Java Access Bridge)
• 🔍 Element Finding — CSS-like selectors + fuzzy search for UI elements
• 🖱️ Click & Type — Hardware-level input simulation
• ⌨️ Key Combos — Send any keystroke or shortcut
• 🎮 Hardware Keyboard — Scan-code input bypasses virtual-key detection (games, anti-cheat)
• 📸 Annotated Screenshots — AI-ready screenshots with numbered bounding boxes
• 📋 Menu Traversal — Extract app menu structures with shortcuts
• 🪟 Window Management — Focus, close, minimize, maximize, move, resize windows
• 📦 App Control — Launch, quit, switch, hide/unhide applications
• 💬 Dialog Handling — Detect and interact with system dialogs (message boxes, file pickers)
• 📌 Taskbar & Tray — List and click taskbar items and system tray icons
• 🖥️ Multi-Monitor — Enumerate monitors, capture specific screens, DPI-aware coordinates
• 🗂️ Virtual Desktops — List, switch, create, close desktops and move windows between them
• 🍎 macOS Support — Coming soon (native implementation in development)
• 🎬 Recording & Playback — Record user actions, replay them, export to Python/Bash scripts
• 🏷️ Selector Management — Save, share, and reuse UI element selectors across sessions
• 🌐 Browser Automation — Full Chrome DevTools Protocol support: navigate, click, type, screenshot, wait, intercept network, stealth mode
• 🔬 Cascade Recognition — UIA + CDP + AI Vision multi-source fusion for Electron/CEF apps where single-source fails
• 👁️ Visual Regression Testing — Compare screenshots across runs, generate HTML reports, detect unintended UI changes
• 🤖 AI-Ready — JSON output, agent-friendly CLI, MCP server

Platform Support

| Platform | Status | Notes | |----------|--------|-------| | Windows 10/11 | ✅ Full support | Primary platform. All features available. | | Windows 7 SP1+ | ⚠️ Best-effort | Basic features only, no UIAutomation v3. | | macOS 13+ | 🚧 Coming soon | Native support is under active development. | | Linux | 🚧 Coming soon | Backend is a placeholder. Not usable yet. | | Python | 3.9+ | Required for all platforms. |

> Why Windows 10+? UIAutomation v2/v3 APIs (caching, virtualized controls) require Windows 8+. Windows 7 has been out of support since January 2020. Most enterprise customers have migrated to Windows 10/11.

Install

pip install naturo

MCP Server Setup

Naturo includes a built-in MCP server with 60+ tools for AI agent integration.

Quick install (one line)

After pip install naturo, connect naturo to your agent with a single copy-paste command:

Claude Code

claude mcp add naturo -- naturo mcp start

VS Code (GitHub Copilot / MCP)

code --add-mcp '{"name":"naturo","command":"naturo","args":["mcp","start"]}'

Cursor — add to ~/.cursor/mcp.json (global) or .cursor/mcp.json (per project):

{ "mcpServers": { "naturo": { "command": "naturo", "args": ["mcp", "start"] } } }

Windsurf — add to ~/.codeium/windsurf/mcp_config.json:

{ "mcpServers": { "naturo": { "command": "naturo", "args": ["mcp", "start"] } } }

> naturo must be on your PATH — it is after pip install naturo. Restart the client after editing a JSON config so it picks up the new server.

Claude Desktop (manual config)

Add to your Claude configuration file (claude_desktop_config.json):

{
  "mcpServers": {
    "naturo": {
      "command": "naturo",
      "args": ["mcp", "start"]
    }
  }
}

> Config file location: > - Windows: %APPDATA%\Claude\claude_desktop_config.json > - macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

Other AI Agents (SSE / HTTP)

For agents that connect over HTTP instead of stdio:

# SSE transport (Server-Sent Events)
naturo mcp start --transport sse --port 3100

# Streamable HTTP transport
naturo mcp start --transport streamable-http --port 3100

Verify Setup

# List all 60+ MCP tools
naturo mcp tools

# Install MCP dependencies if needed
naturo mcp install

Quick Start

# Check version
naturo --version

# Capture a screenshot
naturo capture --path screen.png

# List open windows
naturo list windows

# Inspect UI tree
naturo see --window "Notepad" --depth 5

# Click an element
naturo click "Button:Save"

# Type text
naturo type "Hello, World!"

# Type with hardware scan codes (bypass anti-cheat detection)
naturo type "Hello" --input-mode hardware

# Press key combo
naturo press ctrl+s

# Find element — the universal locator auto-detects the strategy from the query
naturo find "Edit:filename"                # UIA tree search (name / role:name)
naturo find button.png                     # image template match (.png/.jpg/…)
naturo find 'app://notepad.exe/Edit'       # resolve a selector path
naturo find @notepad/save-btn              # resolve a saved selector (@app/name)

# App management
naturo app launch "notepad"
naturo app focus "notepad"
naturo app quit "chrome" --force
naturo app minimize "notepad"
naturo app restore "notepad"
naturo app inspect "notepad"             # Probe frameworks (UIA, CDP, MSAA...)
naturo app relaunch "notepad"

# Dialog handling
naturo dialog detect                       # Detect active dialogs
naturo dialog accept                       # Click OK/Yes
naturo dialog dismiss                      # Click Cancel/No
naturo dialog type "hello.txt" --accept    # Type filename then OK

# Taskbar & tray
naturo taskbar list                        # List taskbar items
naturo taskbar click "Chrome"              # Click taskbar button
naturo tray list                           # List tray icons
naturo tray click "Volume"                 # Left-click tray icon
naturo tray click "Wi-Fi" --right          # Right-click for menu

# Virtual desktops (Windows 10/11)
naturo desktop list                        # List virtual desktops
naturo desktop switch 1                    # Switch to desktop 1
naturo desktop create --name "Work"        # Create named desktop
naturo desktop close                       # Close current desktop
naturo desktop move-window 1 --app "Notepad"  # Move window to desktop 1

# Type Windows paths literally (--raw disables escape interpretation)
naturo type "C:\Users\test\report.txt" --raw --app notepad

# Paste text via clipboard (fast for large content)
naturo type "large content" --paste        # Set clipboard → Ctrl+V → restore
naturo type --paste --file data.txt        # Read file → paste

# Read element values
naturo get e47                             # Read text/value by element ref
naturo get --aid txtSearch                 # Read by AutomationId
naturo get --role Edit --name Search       # Read by role + name
naturo get --role Button --app notepad --all -j  # All buttons (JSON)

# Write element values
naturo set e47 "hello world"               # Set text field value
naturo set --aid txtSearch "query"          # Set by AutomationId
naturo set e12 --toggle                    # Toggle a checkbox
naturo set e8 --select                     # Select a list/radio item
naturo set e5 --expand                     # Expand a combo box

# Highlight UI elements
naturo highlight --app notepad             # Show actionable elements
naturo highlight --app notepad --all       # Show all elements
naturo highlight e11 --app notepad         # Highlight specific ref
naturo highlight --app notepad -A out.png  # Save annotated screenshot

Python SDK

Prefer to stay in Python? import naturo gives you an ergonomic, in-process API over the same engine the CLI and MCP server use — no subprocess, no output parsing. Import-and-go in under 10 lines:

import naturo

app = naturo.launch("notepad")           # App handle; waits until ready
naturo.type("hello", window="Notepad")   # IME-immune type ladder (ValuePattern→clipboard→keystroke)
tree = naturo.see(window="Notepad")      # root Element; walk .children / .descendants / .find
el = naturo.find("Button:Save", window="Notepad")
if el:
    el.click()                           # elements act on themselves
naturo.capture("shot.png", window="Notepad")
app.quit()

Core verbs are available both as module-level functions and as methods on a reusable Desktop / Session session (or a launched App): see, find, click, type, press, get_value, set_value, capture, launch, quit, wait, windows.

import naturo

# Context-managed app quits on exit; the fused cascade tree is one flag away.
with naturo.launch("calculator") as app:
    for key in ("4", "2", "multiply", "7", "enter"):
        app.press(key)
    tree = app.see()                      # or app.see(cascade=True) for UIA+CDP+JAB+COM
    for el in tree.descendants():
        if el.role == "Text" and el.name:
            print(el.name)

Runnable scripts live in [examples/](examples/) (notepad_hello.py, form_filler.py, ui_inspector.py, window_capture.py).

Cascade Recognition

Most desktop automation tools rely on a single accessibility API (UIA) — when it fails (Electron apps, custom-rendered UI), you're stuck. Naturo cascades through multiple recognition sources automatically:

UIA → CDP → AI Vision
 ↓      ↓       ↓
Win32  Chrome   Claude/GPT
native DevTools  screenshot

# Progressive multi-source recognition
naturo see --app feishu --cascade --fill-gaps --stats

# Result: UIA finds 700+ elements, AI Vision adds 130+ that UIA missed
#   uia      705 elements    6s   [ok]
#   cdp        0 elements   15s   [skipped]
#   vision   133 elements   72s   [ok]

# Click an AI-discovered element by ref
naturo click e805 --app feishu    # "视频会议" (Video Meeting) found by AI Vision

How it works:

• UIA/MSAA finds native Win32/WPF/UWP controls (fastest, most accurate)
• CDP reaches into Electron/Chrome web content via DevTools Protocol
• AI Vision screenshots the window and asks Claude/GPT to enumerate every visible element — catches anything the other sources miss
• IoU dedup prevents duplicates: if UIA already found an element, AI Vision skips it
• Tree merge attaches AI-discovered elements to the correct UIA parent container

Requires ANTHROPIC_API_KEY or OPENAI_API_KEY for AI Vision. Set NATURO_AI_MODEL to choose the model (default: claude-sonnet-4-20250514).

CLI Commands

See (observe the desktop)

| Command | Description | Since | |---------|-------------|-------| | capture | Screenshot screen/window | 0.1.0 | | see | Inspect UI element tree | 0.1.0 | | find | Search UI elements (fuzzy match) | 0.1.0 | | get | Read element properties (text, value, state) | 0.2.1 | | highlight | Visual overlay showing all actionable elements | 0.3.0 | | list windows | List open windows | 0.1.0 | | list apps | List running applications | 0.1.0 | | list screens | List monitors and resolutions | 0.1.0 | | diff | Compare two UI snapshots | 0.1.1 | | menu-inspect | List app menu structure with shortcuts | 0.1.0 |

Act (interact with the desktop)

| Command | Description | Since | |---------|-------------|-------| | click | Click element/coordinates (--paste, --copy, --cut modifiers) | 0.1.0 | | type | Type text (supports --paste for clipboard) | 0.1.0 | | set | Set element value/state (toggle, select, expand) | 0.3.0 | | press | Press key combination (e.g., ctrl+s) | 0.1.0 | | scroll | Scroll mouse wheel | 0.1.0 | | drag | Drag from/to coordinates | 0.1.0 | | move | Move mouse cursor | 0.1.0 | | wait | Wait for element/window to appear | 0.1.0 |

App management

| Command | Description | Since | |---------|-------------|-------| | app launch | Launch application by name or path | 0.1.0 | | app quit | Quit application (supports --force) | 0.1.0 | | app focus | Focus an application window (alias: app switch) | 0.1.0 | | app close | Close an application window (graceful or forced) | 0.1.0 | | app minimize | Minimize an application window (alias: app hide) | 0.1.0 | | app maximize | Maximize an application window | 0.1.0 | | app restore | Restore a minimized/maximized window (alias: app unhide) | 0.1.0 | | app move | Move and/or resize an application window | 0.1.0 | | app list | List running applications with visible windows | 0.1.0 | | app windows | List open windows (filter by app/PID) | 0.1.0 | | app find | Find application by name or PID | 0.1.0 | | app inspect | Probe app frameworks (UIA, CDP, MSAA...) | 0.3.0 | | app relaunch | Quit and relaunch an application | 0.3.0 |

System

| Command | Description | Since | |---------|-------------|-------| | clipboard get | Read clipboard text content | 0.3.1 | | clipboard set | Write text to clipboard | 0.3.1 | | clipboard clear | Clear clipboard contents | 0.3.1 | | clipboard info | Show clipboard format and size | 0.3.1 | | dialog detect | Detect active system dialogs | 0.1.0 | | dialog accept | Accept (OK/Yes) a dialog | 0.1.0 | | dialog dismiss | Dismiss (Cancel/No) a dialog | 0.1.0 | | dialog click-button | Click specific dialog button | 0.1.0 | | dialog type | Type in dialog input field | 0.1.0 | | taskbar list | List taskbar items | 0.1.0 | | taskbar click | Click taskbar item | 0.1.0 | | tray list | List system tray icons | 0.1.0 | | tray click | Click tray icon (left/right/double) | 0.1.0 | | desktop list | List virtual desktops | 0.1.0 | | desktop switch | Switch to a virtual desktop | 0.1.0 | | desktop create | Create a new virtual desktop | 0.1.0 | | desktop close | Close a virtual desktop | 0.1.0 | |

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: AcePeak
• Source: AcePeak/naturo
• License: MIT

Install and usage instructions live in the source repository linked above.