AnyTool

AnyTool: Universal Tool-Use Layer for AI Agents

✨ One Line of Code to Supercharge any Agent with Fast, Scalable and Powerful Tool Use ✨

[](https://github.com/HKUDS/AnyTool/) [](https://www.python.org/) [](https://opensource.org/licenses/MIT/) [](./COMMUNICATION.md) [](./COMMUNICATION.md)

| ⚡ Fast - Lightning Tool Retrieval  |  📈 Self-Evolving Tool Orchestration  |  ⚡ Universal Tool Automation |

🎯 What is AnyTool?

AnyTool is a Universal Tool-Use Layer that transforms how AI agents interact with tools. It solves three fundamental challenges that prevent reliable agent automation: overwhelming tool contexts, unreliable community tools, and limited capability coverage -- delivering the first truly intelligent tool orchestration system for production AI agents.

💡 Research Highlights

⚡ Fast - Lightning Tool Retrieval

• Smart Context Management: Progressive tool filtering delivers exact tools in milliseconds through multi-stage pipeline, eliminating context pollution while maintaining speed.

• Zero-Waste Processing: Pre-computed embeddings and lazy initialization eliminate redundant processing - tools are instantly ready across all executions.

📈 Scalable - Self-Evolving Tool Orchestration

• Adaptive MCP Tool Selection: Smart caching and selective re-indexing maintain constant performance from 10 to 10,000 tools with optimal resource usage.

• Self-Evolving Tool Optimization: System continuously improves through persistent memory, becoming more efficient as your tool ecosystem expands.

🌍 Powerful - Universal Tool Automation

• Quality-Aware Selection: Built-in reliability tracking and safety controls deliver production-ready automation through persistent learning and execution safeguards.

• Universal Tool-Use Capability: Multi-backend architecture seamlessly extends beyond web APIs to system operations, GUI automation, and deep research through unified interface.

⚡ Easy-to-Use and Effortless Integration

One line to get intelligent tool orchestration. Zero-config setup transforms complex multi-tool workflows into a single API call.

from anytool import AnyTool

# One line to get intelligent tool orchestration
async with AnyTool() as tool_layer:
    result = await tool_layer.execute(
        "Research trending AI coding tools from GitHub and tech news, "
        "collect their features and user feedback, analyze adoption patterns, "
        "then create a comparison report with insights"
    )

---

📋 Table of Contents

• [🎯 Quick Start](#-quick-start)
• [🚀 Technical Innovation & Implementation](#-technical-implementation)
• [🔧 Configuration Guide](#-configuration-guide)
• [📖 Code Structure](#-code-structure)
• [🔗 Related Projects](#-related-projects)

---

🎯 Quick Start

1. Environment Setup

# Clone repository
git clone https://github.com/HKUDS/AnyTool.git
cd AnyTool

# Create and activate conda environment (includes ffmpeg for video recording)
conda create -n anytool python=3.12 ffmpeg -c conda-forge -y
conda activate anytool

# Install dependencies
pip install -r requirements.txt

> [!NOTE] > Create a .env file and add your API keys (refer to anytool/.env.example).

2. Execution Mode: Local vs Server

AnyTool's Shell and GUI backends support two execution modes. You can configure the mode in anytool/config/config_grounding.json:

{
  "shell": { "mode": "local", ... },  // or "server"
  "gui":   { "mode": "local", ... }   // or "server"
}

Local Mode (Default — no server needed)

In local mode, Shell and GUI operations are executed directly in-process via subprocess / asyncio. This is the simplest setup — no local server required. Just use AnyTool as normal, see [Quick Integration](#3-quick-integration) for usage examples.

> [!TIP] > Use local mode when you are running AnyTool on the same machine you want to control (your own laptop / desktop). This is the recommended mode for most users.

Server Mode (for remote VMs / isolation)

In server mode, Shell and GUI operations are sent over HTTP to a running local_server Flask service. This is required when:

• Controlling a remote VM — the agent runs on your host, while the server runs inside the VM.
• Process isolation / sandboxing — you want script execution in a separate process for security or stability.
• Multi-machine deployments — the agent and the execution environment are on different machines.

To use server mode, set "mode": "server" in config_grounding.json, then install platform-specific dependencies and start the server:

> [!IMPORTANT] > Platform-specific setup required: Different operating systems need different dependencies for desktop control. Please install the required dependencies for your OS before starting the local server:

macOS Setup

# Install macOS-specific dependencies
pip install pyobjc-core pyobjc-framework-cocoa pyobjc-framework-quartz atomacos

Permissions Required: macOS will automatically prompt for permissions when you first run the local server. Grant the following:

• Accessibility (for GUI control)
• Screen Recording (for screenshots and video capture)

> If prompts don't appear, manually grant permissions in System Settings → Privacy & Security.

Linux Setup

# Install Linux-specific dependencies
pip install python-xlib pyatspi numpy

# Install system packages
sudo apt install at-spi2-core python3-tk scrot

> [!NOTE] > Optional dependencies: > - Accessibility: pyatspi + at-spi2-core > - Window management: wmctrl > - Cursor in screenshots: libx11-dev + libxfixes-dev

Windows Setup

# Install Windows-specific dependencies
pip install pywinauto pywin32 PyGetWindow

After installing the platform-specific dependencies, start the local server:

python -m anytool.local_server.main

> [!NOTE] > See [anytool/local_server/README.md](anytool/local_server/README.md) for complete API documentation and advanced configuration.

Mode Comparison

| | Local Mode ("local") | Server Mode ("server") | |---|---|---| | Setup | Zero — just run your agent | Start local_server first | | Use case | Same-machine development | Remote VMs, sandboxing, multi-machine | | Shell execution | asyncio.subprocess in-process | HTTP → Flask → subprocess | | GUI execution | Direct pyautogui / ScreenshotHelper | HTTP → Flask → pyautogui | | Dependencies | Only core AnyTool | Core + Flask + platform deps | | Network | None required | HTTP between agent ↔ server |

3. Quick Integration

AnyTool is a plug-and-play Universal Tool-Use Layer for any AI agent. The task passed to execute() can come from your agent's planning module, user input, or any workflow system.

import asyncio
from anytool import AnyTool
from anytool.tool_layer import AnyToolConfig

async def main():
    config = AnyToolConfig(
        enable_recording=True,
        recording_backends=["gui", "shell", "mcp", "web"],
        enable_screenshot=True,
        enable_video=True,
    )
    
    async with AnyTool(config=config) as tool_layer:
        result = await tool_layer.execute(
            "Research trending AI coding tools from GitHub and tech news, "
            "collect their features and user feedback, analyze adoption patterns, "
            "then create a comparison report with insights"
        )
        print(result["response"])

asyncio.run(main())

> [!TIP] > MCP Server Configuration: For tasks requiring specific tools, add relevant MCP servers to anytool/config/config_mcp.json. Unsure which servers to add? Simply add all potentially useful ones, AnyTool's Smart Tool RAG will automatically select the appropriate tools for your task. See [MCP Configuration](#mcp-configuration) for details.

---

Technical Innovation & Implementation

🧩 Challenge 1: MCP Tool Context Overload

The Problem. Current MCP agents suffer from a fundamental design flaw: they load ALL configured servers and tools at every execution step, creating an overwhelming action space, creates three critical issues:

• ⚡ Slow Performance with Massive Context Loading

Complete tool set from all pre-configured servers loaded simultaneously at every step, degrading execution speed

• 🎯 Poor Accuracy from Blind Tool Setup

Users cannot preview tools before connecting, leading to over-setup "just in case" and confusing tool selection

• 💸 Resource Waste with No Memory

Same tools reloaded at every execution step with no caching, causing redundant loading

✅ AnyTool's Solution: Tool Context Management Framework

Motivation: "Load Everything" → "Retrieve What's Needed" Improvement: Faster tool selection, cleaner context, and efficient resource usage through smart retrieval and memory.

Technical Innovation:

🎯 Multi-Stage Tool Retrieval Pipeline

• Progressive MCP Tool Filtering: server selection → tool name matching → tool semantic search → LLM ranking
• Reduces MCP Tool Search Space: Each stage narrows down candidate tools for optimizing precision and speed

💾 Long-Term Tool Memory

• Save Once, Use Forever: Pre-compute tool embeddings once and save them to disk for instant reuse
• Zero Waste Processing: No more redundant processing - tools are ready to use immediately across all execution steps

🧠 Adaptive Tool Selection

• Adaptive MCP Tool Ranking: LLM-based tool selection refinement triggered only when MCP tool results are large or ambiguous
• Tool Selection Efficiency: Balances MCP tool accuracy with computational efficiency

🚀 On-Demand Resource Management

• Lazy MCP Server Startup: MCP server initialization triggered only when specific tools are needed
• Selective Tool Updates: Incremental re-indexing of only changed MCP tools, not the entire tool set

---

🚨 Challenge 2: MCP Tool Quality Issues

The Problem. Current MCP servers suffer from community contribution challenges that create three critical issues:

• 🔍 Poor Tool Descriptions

Misleading claims, non-existent advertised tools, and vague capability specifications lead to wrong tool selection.

• 📊 No Reliability Signals

Cannot assess MCP tool quality before use, causing blind selection decisions.

• ⚠️ Security and Safety Gaps

Unvetted community tools may execute dangerous operations without proper safeguards.

✅ AnyTool Solution: Self-Contained Quality Management

Motivation: "Blind Tool Trust" → "Smart Quality Assessment" Improvement: Reliable tool selection, safe execution, and autonomous recovery through quality tracking and safety controls.

Technical Innovation:

🎯 Quality-Aware Tool Selection

• Description Quality Check: LLM-based evaluation of MCP tool description clarity and completeness.
• Performance-Based Ranking: Track call/success rates for each MCP tool in persistent memory to prioritize reliable options.

💾 Learning-Based Tool Memory

• Track Tool Performance: Remember which MCP tools work well and which fail over time.
• Smart Tool Prioritization: Automatically rank tools based on past success rates and description quality.

🛡️ Safety-First Execution

• Block Dangerous Operations: Prevent arbitrary code execution and require user approval for sensitive MCP tool operations.
• Execution Safeguards: Built-in safety controls for all MCP tool executions.

🚀 Self-Healing Tool Management

• Autonomous Tool Switching: Switch failed MCP tools locally without restarting expensive planning loops.
• Local Failure Recovery: Automatically switch to alternative MCP tools on failure without escalating to upper-level agents.

---

🔄 Challenge 3: Limited MCP Capability Scope

The Problem. Current MCP ecosystem focuses primarily on Web APIs and online services, creating significant automation gaps that prevent comprehensive task completion:

• 🖥️ Missing System Operations

No native support for file manipulation, process management, or command execution on local systems.

• 🖱️ No Desktop Automation

Cannot control GUI applications that lack APIs, limiting automation to web-only scenarios.

• 📊 Incomplete Tool Coverage

Limited server categories in community and incomplete tool sets within existing servers create workflow bottlenecks.

✅ AnyTool Solution: Universal Capability Extension(MCP + System Commands + GUI Control ≈ Universal Task Completion)

Motivation: "Web-Only MCP" → "Universal Task Completion" Improvement: Complete automation coverage through multi-backend architecture that seamlessly extends MCP capabilities beyond web APIs.

🏗️ Multi-Backend Architecture

• MCP Backend: Community servers for Web APIs and online services
• Shell Backend: Bash/Python execution for system-level operations and file management
• GUI Backend: Pixel-level automation for any visual application without API requirements
• Web Backend: Deep web research and data extraction capabilities

💡 Self-Evolving Capability Discovery

• Intelligent Gap Detection: Planning agent identifies when MCP tools are insufficient for task requirements
• Automatic Backend Selection: Shell/GUI backends automatically fill capability gaps without manual intervention
• Dynamic Capability Expansion: Previously impossible tasks become achievable through backend combination

🎭 Unified Tool Orchestration

• Uniform Tool Schema: All backends expose identical interface for seamless agent tool selection
• Transparent Backend Switching: Agents select optimal tools across backend types without knowing implementation details
• Intelligent Tool Routing: Automatic routing to the most appropriate backend based on task requirements

🚀 Seamless Integration Layer

• Single Tool Interface: Unified API that abstracts away backend complexity from AI agents.
• Cross-Backend Coordination: Enable complex workflows that span multiple backend capabilities.
• Consistent Safety Controls: Apply security and safety measures uniformly across all backend types.

---

🔧 Configuration Guide

Configuration Overview

AnyTool uses a layered configuration system:

• config_dev.json (highest priority): Local development overrides. Overrides all other configurations.
• config_agents.json: Agent definitions and backend access control
• config_mcp.json: MCP server registry
• config_grounding.json: Backend-specific settings and Smart Tool RAG configuration
• config_security.json: Security policies with runtime user confirmation for sensitive operations

---

Agent Configuration

Path: anytool/config/config_agents.json

Purpose: Define agent roles, control backend access scope, and set execution limits to prevent infinite loops.

Example configuration:

{
  "agents": [
    {
      "name": "GroundingAgent",
      "class_name": "GroundingAgent",
      "backend_scope": ["gui", "shell", "mcp", "system", "web"],
      "max_iterations": 20
    }
  ]
}

Key Fields:

| Field | Description | Options/Example | |-------|-------------|-----------------| | backend_scope | Accessible backends | [] or any combination of ["gui", "shell", "mcp", "system", "web"] | | max_iterations | Maximum execution cycles | Any integer (e.g., 15, 20, 50) or null (unlimited) |

---

MCP Configuration

Path: anytool/config/config_mcp.json (copy from config_mcp.json.example)

Purpose: Register MCP servers with connection details. AnyTool automatically discovers tools from all registered servers and makes them available through Smart Tool RAG.

Example configuration:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/se

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [HKUDS](https://github.com/HKUDS)
- **Source:** [HKUDS/AnyTool](https://github.com/HKUDS/AnyTool)
- **License:** MIT

Install and usage instructions live in the source repository linked above.