Site2cli

Turn any website into a CLI/API for AI agents. Discover APIs automatically. Extract structured data like Firecrawl — but local, free, and open-source.

Free, open-source alternative to Firecrawl, Browserbase browser-to-api, and Browserbase browser-to-cli — runs 100% locally, no API keys, no per-page billing.

---

The Problem

AI agents interact with websites through browser automation, which is slow, expensive, and unreliable:

| | Without site2cli | With site2cli | |---|---|---| | Speed | 10-30s per action (browser) | 95% for discovered APIs | | Setup | Write custom Playwright scripts | site2cli discover | | Output | Screenshots, raw HTML | Structured JSON, typed clients |

CLI Overview

What's New in v0.7.0 — browser-to-api+ for everyone

• Offline trace replay — site2cli discover --replay trace.json rebuilds the spec without re-running the browser. Save real captures with --save-trace.
• Four artifacts every run — OpenAPI 3.1 (.yaml / .json), Python client, zero-dep JavaScript ES-module client (Node 18+ / Deno / Bun / browser), and a dark-theme HTML coverage report with gap candidates.
• chunk command — RAG-ready chunking (fixed / sentence / heading) for .md, .txt, and PDFs.
• search command — DuckDuckGo search piped into --scrape, --extract, or --chunk in one go.
• PDF parsing — pdf_to_text, pdf_to_markdown, pdf_page_count via the [rag] extra.
• 559 tests (up from 500), all passing in

# CLI
site2cli run kayak.com search_flights from=SFO to=JFK date=2025-04-01

# Or as MCP tools for AI agents
site2cli mcp generate kayak.com
site2cli mcp serve kayak.com

---

Extract & Scrape — Open-Source Firecrawl Alternative

site2cli includes a complete web extraction pipeline — no API keys for scraping, no pay-per-page pricing, runs 100% locally.

Comparison with Firecrawl

| Feature | Firecrawl | site2cli | |---|---|---| | Scrape to markdown | Yes (cloud) | Yes (local) | | Structured extraction | Yes ($) | Yes (local LLM) | | JSON Schema validation | Yes | Yes | | Batch extraction | Yes | Yes | | Main content extraction | Yes | Yes | | Pricing | $0.001-0.004/page | Free | | Runs locally | No (SaaS) | Yes | | API discovery | No | Yes | | MCP server generation | No | Yes | | Progressive optimization | No | Yes (3 tiers) | | Open source | Partial | Yes (MIT) |

Extract Structured Data

LLM-powered extraction with natural language prompts and JSON Schema validation:

# Extract data using natural language
site2cli extract https://example.com -p "Extract the page title and all links"

# Extract with JSON Schema validation
site2cli extract https://news.ycombinator.com \
  -s '{"type":"object","properties":{"stories":{"type":"array"}}}'

# Use a Pydantic model as schema
site2cli extract https://example.com -s "myapp.models.Article"

# Batch extraction from multiple URLs
site2cli extract https://example.com -u https://example.org -p "Get the page title"

# Save results to file
site2cli extract https://example.com -p "Extract all headings" -o results.json

Scrape Any Page to Markdown / Text / HTML

# Convert page to markdown (default)
site2cli scrape https://example.com

# Convert to plain text
site2cli scrape https://example.com --format text

# Extract just the main content (skip nav/footer/sidebar)
site2cli scrape https://example.com --main-content

# Save raw HTML of main content
site2cli scrape https://example.com --format html -o output.html

Crawl an Entire Website

# Crawl a docs site to markdown
site2cli crawl https://docs.example.com -d 3 -n 100

# Stream pages as JSONL (great for piping)
site2cli crawl https://example.com --format jsonl --stream

# Generate a sitemap (URLs only, no content)
site2cli crawl https://example.com --sitemap

# Save all pages to a directory
site2cli crawl https://example.com -o output/

# Resume a previous crawl
site2cli crawl https://example.com --resume 

Monitor Pages for Changes

# One-shot: compare against last snapshot
site2cli monitor https://example.com/pricing

# Continuous polling every 5 minutes
site2cli monitor https://example.com/api --interval 300

# Get notified via webhook on change
site2cli monitor https://example.com --webhook https://hooks.slack.com/xxx

# List all watches, show history
site2cli monitor --list
site2cli monitor --history 

RAG-Ready Chunking (chunk command)

# Chunk a markdown file by headings (best for docs)
site2cli chunk README.md --strategy heading --output chunks.jsonl

# Chunk a PDF — requires site2cli[rag]
site2cli chunk paper.pdf --strategy sentence --size 500

# Fixed-size chunks with overlap (general fallback)
site2cli chunk notes.txt --strategy fixed --size 1000 --overlap 200

# Crawl + chunk in one shot — JSONL ready for any embedding pipeline
site2cli crawl https://docs.example.com --rag -o chunks.jsonl

Each chunk emits { text, index, total, url, title, section } so you can pipe it straight into pgvector, chroma, qdrant, weaviate, or any other vector DB.

Web Search → Scrape → Extract (search command)

# Plain search (DuckDuckGo, no API key)
site2cli search "Python web scraping 2026"

# Search + scrape top N results to markdown
site2cli search "pricing plans" --scrape --format markdown

# Search + scrape + chunk for RAG
site2cli search "API docs" --scrape --chunk heading -o chunks.jsonl

# Search + LLM-powered structured extraction
site2cli search "restaurant reviews austin" \
  --extract -p "extract name, rating, price tier"

Capture Screenshots

# Full-page screenshot
site2cli screenshot https://example.com -o page.png

# Capture a specific element
site2cli screenshot https://example.com --selector ".pricing-table"

# Viewport-only JPEG with quality
site2cli screenshot https://example.com --viewport --format jpeg --quality 80

Use a Proxy

# Any command supports --proxy
site2cli discover example.com --proxy http://proxy:8080
site2cli extract https://example.com -p "titles" --proxy socks5://proxy:1080
site2cli scrape https://example.com --proxy http://user:pass@proxy:8080

---

Use with Claude Code / Claude Desktop

# Add site2cli as an MCP server for Claude Code
claude mcp add site2cli -- uvx --from 'site2cli[mcp]' site2cli --mcp

# Or add to Claude Desktop's config (~/.claude/claude_desktop_config.json):
# {
#   "mcpServers": {
#     "site2cli": {
#       "command": "uvx",
#       "args": ["--from", "site2cli[mcp]", "site2cli", "--mcp"]
#     }
#   }
# }

Once configured, Claude can call any discovered site's API as a tool: > "Use site2cli to get data about the Pokemon Ditto"

Note: You need to run site2cli discover first to populate the registry. The MCP server exposes all discovered sites as tools.

---

Manage Browser Auth & Sessions

# Import a Chrome profile for authenticated discovery
site2cli auth profile-import --browser chrome

# Manage cookies
site2cli cookies list example.com
site2cli cookies export example.com

# Reuse browser sessions across commands
site2cli discover example.com --session my-session
site2cli run example.com search --session my-session

# Background browser daemon (persistent browser across CLI calls)
site2cli daemon start
site2cli daemon status
site2cli daemon stop

# Unified MCP server for ALL discovered sites
site2cli --mcp
# or: site2cli mcp serve-all

Comparison

| Feature | browser-use 2.0 | Firecrawl | CLI-Anything | Stagehand v3 | site2cli | |---|---|---|---|---|---| | Works on any site | Yes | Yes | Yes | Yes | Yes | | Structured output | No | Yes | Yes | Yes | Yes | | Auto-discovery | No | No | No | No | Yes | | Structured extraction | No | Yes ($) | No | No | Yes (free) | | Scrape to markdown | No | Yes ($) | No | No | Yes (free) | | MCP server generation | Acts as MCP | No | No | Yes | Generates MCP | | Progressive optimization | No | No | No | Auto-cache | Yes (3 tiers) | | Runs locally | Yes | No (SaaS) | Yes | Yes | Yes | | Cookie banner handling | No | Yes | No | No | Yes | | Auth page detection | No | No | No | No | Yes | | Self-healing | No | No | No | Yes | Yes | | No browser needed (after discovery) | No | N/A | No | No | Yes | | Session persistence | Yes | No | No | No | Yes | | Daemon mode | Yes (~50ms) | No | No | No | Yes | | Full site crawling | No | Yes | No | No | Yes | | Change detection/monitoring | No | Yes ($) | No | No | Yes (free) | | Screenshot capture | No | Yes | No | Yes | Yes | | Community spec sharing | No | No | No | No | Yes |

How It Works

site2cli uses Progressive Formalization — a 3-tier system that automatically graduates interactions from slow-but-universal to fast-but-specific:

graph LR
    A["Tier 1: BrowserExploration"] -->|"Patterndetected"| B["Tier 2: CachedWorkflow"]
    B -->|"APIdiscovered"| C["Tier 3: DirectAPI Call"]
    style A fill:#ff6b6b,color:#fff
    style B fill:#ffd93d,color:#000
    style C fill:#6bcb77,color:#fff

The Discovery Pipeline captures browser traffic and converts it into structured interfaces:

graph TD
    A[Launch Browser + CDP] --> B[Capture Network Traffic]
    B --> C[Group by Endpoint Pattern]
    C --> D[LLM-Assisted Analysis]
    D --> E[OpenAPI 3.1 Spec]
    E --> F[Python Client]
    E --> G[CLI Commands]
    E --> H[MCP Server]

What Gets Generated

From a single discovery session, site2cli produces:

| Output | Description | |---|---| | OpenAPI 3.1 Spec | Full API specification with schemas, parameters, auth | | Python Client | Typed httpx client with methods for each endpoint | | CLI Commands | Typer commands you can run from terminal | | MCP Server | Tools that AI agents (Claude, etc.) can call directly |

As a Python Library

from site2cli.discovery.analyzer import TrafficAnalyzer
from site2cli.discovery.spec_generator import generate_openapi_spec
from site2cli.generators.mcp_gen import generate_mcp_server_code

# Analyze captured traffic
analyzer = TrafficAnalyzer(exchanges)
endpoints = analyzer.extract_endpoints()

# Generate OpenAPI spec
spec = generate_openapi_spec(api)

# Generate MCP server
mcp_code = generate_mcp_server_code(site, spec)

---

What's New in v0.7.0

• Full feature parity with Browserbase browser-to-api and browser-to-cli — plus generates more (Python client, MCP tools, Typer CLI) than either.
• --replay and --save-trace on discover — offline trace I/O via the portable Trace JSON format.
• --spec-format yaml|json on discover — emit OpenAPI 3.1 in either format.
• Auto-emit four artifacts — OpenAPI spec, Python client, JS ES-module client (client.mjs), and HTML coverage report — every discovery run.
• Dark-theme HTML coverage report — endpoint table, methods, status codes, and gap candidates highlighting paths that didn't templatize.
• Zero-dep JavaScript client — works in Node 18+, Deno, Bun, and modern browsers. Bearer / API key / custom-header auth via createClient({ auth }).
• chunk command — fixed/sentence/heading strategies for .md, .txt, .pdf → JSONL ready for embeddings.
• search command — DuckDuckGo + optional --scrape, --extract, --chunk in one pipeline.
• PDF parsing — site2cli[rag] adds pdf_to_text, pdf_to_markdown, pdf_page_count via pdfplumber.
• Two long-standing bugs killed — chunk_fixed infinite loop when overlap >= chunk_size; brittle importlib.reload + patch.dict pattern in PDF tests.
• 559 tests (up from 500), all passing offline in

v0.5.0

• extract command — LLM-powered structured data extraction with JSON Schema validation, Pydantic model support, and batch processing
• scrape command — Web scraping with HTML-to-markdown/text/html conversion and main content extraction
• Proxy support — New --proxy flag on discover, run, extract, scrape
• --format flag on run — Output results as json, markdown, or text
• New content extra — pip install site2cli[content] for HTML conversion
• 417 tests (up from 357), all passing

v0.4.0

• OAuth Device Flow (RFC 8628) — site2cli auth login --provider github for GitHub, Google, Microsoft; token refresh, secure storage
• Multi-site orchestration — YAML/JSON pipelines that chain actions across sites with JSONPath data flow ($result.data[0].id)
• Pipeline management — site2cli orchestrate run/list/delete commands with on_error policies (fail/skip/retry)
• 357 tests (up from 306), all passing

v0.3.1

• Claude Code MCP integration — claude mcp add site2cli -- uvx --from 'site2cli[mcp]' site2cli --mcp works out of the box
• Live browser validation — Experiment 15: real Playwright browser → CDP capture → full pipeline tested against 5 public sites (4/5 pass)
• LLM-driven exploration validated — REST Countries: Claude found /v3.1/all endpoint in 8 browser steps
• Auto-probe for static sites — When homepage has no XHR, automatically discovers and probes API-like links (/posts, /users, etc.)
• Terminal demo GIF — assets/demo.gif shows the full discover → run → export flow

v0.3.0

• Cookie management — site2cli cookies list/set/clear/export/import with Playwright-compatible format
• Browser profile import — site2cli auth profile-import --browser chrome auto-detects Chrome/Firefox profiles
• Named browser sessions — --session flag on discover/run, site2cli session list/close/close-all
• Workflow recording — Record and replay browser workflows with parameterization
• Background browser daemon — site2cli daemon start/stop/status keeps a persistent browser
• Unified MCP server — site2cli --mcp serves ALL discovered sites as MCP tools
• 306 tests (up from 214), all passing

v0.2.5

• Cookie banner auto-dismissal — 3-strategy detection (30+ vendor selectors, multilingual text, a11y roles)
• Auth page detection — Detects login/SSO/OAuth/MFA/CAPTCHA pages
• Accessibility tree extraction — Better page representation for LLM-driven exploration
• Rich wait conditions — 9 condition types: network-idle, load, selector, stable, etc.
• Output filtering — --grep, --limit, --keys-only, --compact flags

---

Auto-Probe Discovery

Static homepage with no XHR? site2cli auto-discovers and probes REST-like links:

Community Spec Sharing

Share and reuse discovered API specs across teams:

---

Live Validation (8 Experiments, 15+ APIs)

Experiment #8: Core Pipeline (5 APIs)

| API | Endpoints | Spec | Client | MCP | Pipeline | |---|---|---|---|---|---| | JSONPlaceholder | 8 | Valid | Makes real calls | 8 tools | 157ms | | httpbin.org | 7 | Valid | Makes real calls | 7 tools | 179ms | | Dog CEO API | 5 | Valid | Makes real calls | 5 tools | 209ms | | Open-Meteo | 1 | Valid | Makes real calls | 1 tool | 686ms | | GitHub API | 4 | Valid | Makes real calls | 4 tools | 323ms | | Total | 25 | 5/5 | 5/5 | 25 tools | avg 310ms |

Experiment #9: API Breadth (10 APIs, 7 categories)

| API | Category | Endpoints | Spec | MCP Tools | |---|---|---|---|---| | PokeAPI | Structured REST | 5 | Valid | 5 | | CatFacts | Simple REST | 3 | Valid | 3 | | Chuck Norris | Simple REST | 3 | Valid | 3 | | SWAPI (Star Wars) | Nested Paths | 5 | Valid | 5 | | Open Library | Query Params | 2 | Valid | 2 | | USGS Earthquake | Government/Science | 2 | Valid | 2 | | NASA APOD | Government/Science | 1 | Valid | 1 | | Met Museum | Cultural | 3 | Valid | 3 | | Art Institute Chicago | Cultural | 4 | Valid | 4 | | REST Countries | Geographic | 5 | Valid | 5 | | Total | 7 categories | 33 | 10/10 | 33 |

Full Validati

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: lonexreb
• Source: lonexreb/site2cli
• License: MIT

Install and usage instructions live in the source repository linked above.