Any Docs Mcp

AnyDocsMCP

> Scrape any documentation site and inject it into your AI coding agent's context.

Works with Windsurf, Cursor, Claude Code, and Antigravity (Jules/Gemini).

Landing Page | [MCP Server Docs](mcp-server/README.md) | [Scraper Docs](scraper/README.md)

Why This Works

Inspired by @gaojude/next-agents-md and Vercel's research on retrieval-led reasoning:

> A compressed 8KB docs index embedded directly in AGENTS.md achieved a 100% pass rate, while skills maxed out at 79%. Passive context beats active retrieval because there is no decision point -- the agent doesn't need to decide "should I look this up?" The information is already there.

AnyDocsMCP applies this pattern to any documentation site, not just Next.js. The scraper produces a compact pipe-delimited index that fits in ~500 bytes, pointing the agent to individual doc files it can read on-demand. This shifts agents from pre-training-led reasoning to retrieval-led reasoning.

How It Works

  You                          AnyDocs MCP                    Your Project
   │                               │                              │
   │  "scrape react docs"          │                              │
   ├──────────────────────────────>│                              │
   │  CLI command to run           │                              │
   ││  stored in AppData           │
   │                               ├─────────────────>            │
   │  "add react-docs to project"  │                              │
   ├──────────────────────────────>│  copy docs + inject index    │
   │                               ├─────────────────────────────>│
   │                               │                              │
   │  AI agent now has persistent  │  .windsurf/rules/            │
   │  access to react docs         │  .windsurf/docs/react-docs/  │

1. Scrape -- The MCP server returns a CLI command. You run it to scrape a documentation site.
2. Store -- Scraped docs are stored globally in %APPDATA%/AnyDocsMCP/v2/ (shared across projects).
3. Inject -- Docs are copied into your project and a compact index (~500 bytes) is added as an always-on IDE rule.
4. Use -- The AI agent sees the index in every conversation and reads individual doc files on-demand.

Quick Start

Prerequisites

• Python 3.10+ (for scraper)
• Node.js 18+ (for MCP server)
• OpenRouter API Key (for LLM-powered scraping)
• Chrome/Chromium (optional, for SPA sites)

Installation

git clone https://github.com/jusedit/any-docs-mcp.git
cd any-docs-mcp

# Scraper
cd scraper && pip install -r requirements.txt

# MCP Server
cd ../mcp-server && npm install && npm run build

Configure Your IDE

Add to your IDE's MCP configuration:

{
  "mcpServers": {
    "anydocs": {
      "command": "node",
      "args": ["/path/to/any-docs-mcp/mcp-server/dist/index.js"],
      "env": {
        "OPENROUTER_API_KEY": "sk-or-...",
        "ANYDOCS_IDE": "windsurf"
      }
    }
  }
}

Set ANYDOCS_IDE to your IDE: windsurf (default), cursor, claude, or antigravity.

See [MCP Server README](mcp-server/README.md) for full IDE configuration examples.

Usage

Once configured, the AI agent has 6 tools available:

| Tool | Description | |------|-------------| | scrape_docs | Returns CLI command to scrape a documentation site | | list_docs | Lists all doc sets in global storage | | remove_docs | Deletes a doc set from global storage | | add_docs_to_project | Copies docs into project + injects index as IDE rule | | remove_docs_from_project | Removes docs + rule from project | | list_project_docs | Lists docs currently added to this project |

IDE Support

| IDE | Docs Path | Rules Path | Format | |-----|-----------|------------|--------| | Windsurf | .windsurf/docs// | .windsurf/rules/docs-.md | trigger: always_on | | Cursor | .cursor/docs// | .cursor/rules/docs-.mdc | alwaysApply: true | | Claude Code | .docs// | CLAUDE.md | ` markers | | **Antigravity** | .agent/docs// | .agent/rules/docs-.md` | Always On |

Scraper Pipeline

The scraper uses a 7-step LLM-assisted pipeline:

1. Engine Selection -- cURL vs Selenium (auto-detects SPAs via link-count heuristic)
2. URL Discovery -- Sitemap + BFS quick-sample (uses Selenium for SPAs)
3. LLM Analysis -- Scope rules + CSS selectors with retry loop
4. Sample Crawl -- 5 pages for selector refinement
5. LLM Refinement -- Improve prune selectors from actual output
6. Full Crawl -- All pages with quality check + cross-page dedup
7. Index Generation -- Compact AGENTS.md (~500 bytes, no LLM)

Project Structure

any-docs-mcp/
  scraper/                 # Python scraping engine
    cli.py                 # CLI entry point
    pipeline.py            # 7-step orchestrator
    engine_selector.py     # cURL vs Selenium decision
    discovery.py           # URL discovery (sitemap + BFS)
    llm_analyzer.py        # LLM-powered scope + selector analysis
    crawler.py             # Parallel page fetcher + HTML-to-MD
    content_cleaner.py     # UI artifact removal
    dedup.py               # Cross-page deduplication
    grouper.py             # AGENTS.md index generator
    tests/                 # Unit tests (53 tests)
    requirements.txt
  mcp-server/              # TypeScript MCP server
    src/
      index.ts             # MCP tools + handlers
      storage.ts           # AppData docs management
      ide-adapter.ts       # IDE-specific adapters
    package.json
  docs/                    # GitHub Pages landing page
  .env.example             # Environment variable template
  LICENSE                  # MIT

Environment Variables

| Variable | Required | Default | Description | |----------|----------|---------|-------------| | OPENROUTER_API_KEY | Yes* | -- | API key for LLM-powered scraping | | ANYDOCS_IDE | No | windsurf | Target IDE | | ANYDOCS_STORAGE_ROOT | No | %APPDATA%/AnyDocsMCP/v2/ | Override storage path | | ANYDOCS_SCRAPER_PATH | No | auto-detected | Override scraper path |

Security

Use at your own risk. Scraped content is injected into your AI agent's context. Only scrape documentation from trusted sources. Review content before use in production.

License

MIT -- see [LICENSE](LICENSE)

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: jusedit
• Source: jusedit/any-docs-mcp
• License: MIT
• Homepage: https://siliconforest.de

Install and usage instructions live in the source repository linked above.