Vnc Mcp Server

VNC MCP Server

An advanced Model Context Protocol (MCP) server that empowers AI agents to see, reason, and control computers via the VNC (RFB) protocol. This server transforms a standard VNC connection into a high-level "Vision-Action" interface for AI Agents.

📊 System Architecture

graph TD
    subgraph "AI Environment"
        A[AI Agent / Claude Desktop] -->|MCP Protocol| B[VNC MCP Server]
    end

    subgraph "Bridge Layer"
        B -->|Image Processing| C[Sharp / Tesseract.js]
        B -->|RFB Protocol| D[VNC Connection]
    end

    subgraph "Target System"
        D -->|Input Simulation| E[Windows / macOS / Linux]
        E -->|Framebuffer| D
    end

    style B fill:#f00,stroke:#333,stroke-width:4px

🌟 Key Features

👁️ Advanced Vision

• Regional Screenshot: Precise screen capture of specific areas to save tokens.
• OCR (Optical Character Recognition): Extract text from any part of the screen using Tesseract.js.
• Visual Search: Find specific icons, buttons, or images on the screen using template matching.
• Pixel Analysis: Get exact Hex/RGB colors of any pixel for state verification.
• Change Detection: Wait for a specific region to update before proceeding.

🕹️ Human-Like Control

• Precise Input: Single clicks, double clicks, and smooth mouse movement.
• Drag & Drop: Simulate complex dragging operations between coordinates.
• Scrolling: Directional mouse wheel support (Up/Down/Left/Right).
• Keyboard Mastery: Type strings and perform complex key combinations (e.g., Ctrl+C, Cmd+Space).

⚡ Hybrid Control (VNC + SSH Shell)

• Background Execution: Run any shell command (bash/cmd/powershell) on the server.
• 🔗 Remote Execution via SSH: Run commands directly on the VNC host via integrated SSH client.
• 🛡️ Security Layer: Robust filtering system with platform-specific allowlists and denylists.
• Cross-Platform Support: Works on Windows, Linux, and macOS.
• Seamless Integration: Use terminal to install software then use VNC to control apps.

🧠 Intelligence & Health

• OS Detection & Platform Awareness: Intelligent OS detection based on screen resolution patterns (Windows/macOS/Linux).
• Connection Health: Real-time connectivity monitoring with latency measurement and heartbeat status.
• Clipboard Sync: Full bidirectional clipboard support.
• Auto-Connect: Configure server to connect immediately via environment variables.
• Advanced Sequencing: Execute multiple actions in a single call for faster automation.

🚀 Getting Started

Prerequisites

• Node.js v18+
• A running VNC server on the target system.
• (Optional) SSH service enabled on the target system for remote command execution.

Quick Start

1. Clone & Install:

``bash git clone https://github.com/volkan-m/vnc-mcp-server.git cd vnc-mcp-server npm install npm run build ``

1. Test Environment (Docker with Chrome & SSH):

Run a full Linux desktop with Google Chrome, VNC, and SSH enabled: ``bash docker run -d \ -p 5901:5900 \ -p 2222:22 \ -v /dev/shm:/dev/shm \ --name mcp-vnc-chrome \ -e VNC_PASSWORD=vncpass \ dorowu/ubuntu-desktop-lxde-vnc:latest `` Connect to VNC at localhost:5901 and SSH at localhost:2222 (port 22 inside).

1. MCP Client Configuration:

Google Antigravity / Claude Desktop

Add this to your claude_desktop_config.json:

{
  "mcpServers": {
    "vnc": {
      "command": "node",
      "args": ["/absolute/path/to/vnc-mcp-server/dist/index.js"],
      "env": {
        "VNC_HOST": "localhost",
        "VNC_PORT": "5901",
        "VNC_PASSWORD": "vncpass"
      }
    }
  }
}

🛡️ Security Configuration (Optional)

You can restrict which system commands the AI can execute:

| Variable | Description | Default Denied | |---|---|---| | LINUX_DENIED_COMMANDS | Blocked Linux commands | rm -rf,rm -f,shred,mkfs,:(){ :|:& };: | | WIN_DENIED_COMMANDS | Blocked Windows commands | del,format,rd,sfc,attrib | | MAC_DENIED_COMMANDS | Blocked macOS commands | rm -rf,rm -f | | [OS]_ALLOWED_COMMANDS | If set, ONLY these are allowed | (Empty by default) |

🔗 Remote Execution via SSH

The server now supports direct command execution on the target machine via SSH. When connecting through vnc_connect, you can provide an optional ssh object:

{
  "host": "192.168.1.50",
  "ssh": {
    "user": "root",
    "password": "your-password",
    "port": 22
  }
}

If SSH is connected, vnc_system_command automatically routes all commands to the target machine instead of the local server.

📊 Enhanced Status Monitoring

Screen Info

vnc_get_screen_info returns detailed information about the remote session:

• Resolution: Current display dimensions
• OS Hint: Intelligent OS detection (Windows/macOS/Linux based on resolution patterns)
• SSH Status: Whether SSH connection is active

Connection Health

vnc_get_connection_health provides comprehensive connectivity metrics:

• VNC Status: Connected/Disconnected state
• Latency: Real-time latency measurement from screen refresh operations
• Resolution: Current display dimensions
• SSH Status: SSH connection state

🎯 Advanced Vision Features

Screenshot with Coordinate Overlay

Use the overlay parameter in vnc_screenshot to add a grid with labeled coordinates (A1, B2, etc.) for easier navigation:

{
  "format": "jpeg",
  "quality": 85,
  "overlay": true
}

This helps AI agents understand and communicate screen positions more accurately.

OCR with Multiple Languages

The vnc_ocr_region tool supports multiple languages via Tesseract.js. Specify the language code:

{
  "x": 0,
  "y": 0,
  "width": 1920,
  "height": 1080,
  "lang": "eng+deu+fra"
}

Supported languages: eng (English), deu (German), fra (French), spa (Spanish), and many more.

Batch Actions with vnc_execute_sequence

Execute multiple input actions in a single call for faster automation:

{
  "actions": [
    {"type": "move", "x": 100, "y": 100},
    {"type": "click", "x": 100, "y": 100},
    {"type": "wait", "ms": 500},
    {"type": "type", "text": "Hello World"},
    {"type": "key", "key": "Enter"}
  ]
}

This approach reduces RPC overhead and improves responsiveness for complex automation sequences.

🧰 API Reference (Tools)

| Tool | Category | Description | |---|---|---| | vnc_connect | Setup | Connect to VNC (optional: include ssh configuration). | | vnc_disconnect | Setup | Disconnect from VNC server. | | vnc_get_screen_info | Status | Returns resolution, OS hint (Windows/macOS/Linux), and SSH status. | | vnc_get_connection_health| Status | Provides heartbeat status, latency measurement, resolution, and SSH connection state. | | vnc_screenshot | Vision | Captures PNG/JPEG of full screen or region with optional coordinate overlay. | | vnc_ocr_region | Vision | Extracts text from a specific region with OCR (supports multiple languages). | | vnc_find_image | Vision | Finds a template image on screen with configurable tolerance. | | vnc_wait_for_change | Vision | Blocks until a screen region updates within timeout. | | vnc_get_pixel_color | Vision | Returns Hex/RGB color value of a specific pixel. | | vnc_mouse_move | Input | Moves mouse to coordinates. | | vnc_mouse_click | Input | Performs mouse clicks (Left/Middle/Right buttons). | | vnc_mouse_double_click | Input | Performs a quick double click. | | vnc_mouse_drag | Input | Drags from point A to point B with button control. | | vnc_mouse_scroll | Input | Directional wheel scrolling (Up/Down/Left/Right). | | vnc_key_tap | Input | Taps keys with modifiers (e.g., C-v, M-Tab, S-Enter). | | vnc_type_string | Input | Types a sequence of characters. | | vnc_execute_sequence | Input | Executes multiple input actions in a single call for performance. | | vnc_system_command | System | Execute shell commands (Local or Remote via SSH). | | vnc_launch_gui_app | System | Launches GUI applications on the remote desktop. | | vnc_set_clipboard | System | Sets remote system clipboard text. | | vnc_get_clipboard | System | Retrieves last known remote clipboard content. |

🛡 License

MIT License. See [LICENSE](LICENSE) for details.

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: volkan-m
• Source: volkan-m/vnc-mcp-server
• License: MIT

Install and usage instructions live in the source repository linked above.