# Site2cli

> Turn any website into a CLI/API for AI agents

- **Type:** MCP server
- **Install:** `agentstack add mcp-lonexreb-site2cli`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [lonexreb](https://agentstack.voostack.com/s/lonexreb)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [lonexreb](https://github.com/lonexreb)
- **Source:** https://github.com/lonexreb/site2cli

## Install

```sh
agentstack add mcp-lonexreb-site2cli
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

Turn any website into a CLI/API for AI agents.
  Discover APIs automatically. Extract structured data like Firecrawl — but local, free, and open-source.

  
  
  
  
  
  
  

  Free, open-source alternative to Firecrawl, Browserbase browser-to-api, and Browserbase browser-to-cli — runs 100% locally, no API keys, no per-page billing.

---

## The Problem

AI agents interact with websites through browser automation, which is slow, expensive, and unreliable:

| | Without site2cli | With site2cli |
|---|---|---|
| **Speed** | 10-30s per action (browser) | 95% for discovered APIs |
| **Setup** | Write custom Playwright scripts | `site2cli discover ` |
| **Output** | Screenshots, raw HTML | Structured JSON, typed clients |

## CLI Overview

  

## What's New in v0.7.0 — `browser-to-api`+ for everyone

  

- **Offline trace replay** — `site2cli discover  --replay trace.json` rebuilds the spec without re-running the browser. Save real captures with `--save-trace`.
- **Four artifacts every run** — OpenAPI 3.1 (`.yaml` / `.json`), Python client, zero-dep **JavaScript ES-module client** (Node 18+ / Deno / Bun / browser), and a dark-theme **HTML coverage report** with gap candidates.
- **`chunk` command** — RAG-ready chunking (fixed / sentence / heading) for `.md`, `.txt`, and PDFs.
- **`search` command** — DuckDuckGo search piped into `--scrape`, `--extract`, or `--chunk` in one go.
- **PDF parsing** — `pdf_to_text`, `pdf_to_markdown`, `pdf_page_count` via the `[rag]` extra.
- **559 tests** (up from 500), all passing in 
  

```bash
# CLI
site2cli run kayak.com search_flights from=SFO to=JFK date=2025-04-01

# Or as MCP tools for AI agents
site2cli mcp generate kayak.com
site2cli mcp serve kayak.com
```

---

## Extract & Scrape — Open-Source Firecrawl Alternative

site2cli includes a complete web extraction pipeline — **no API keys for scraping, no pay-per-page pricing, runs 100% locally.**

### Comparison with Firecrawl

| Feature | Firecrawl | **site2cli** |
|---|---|---|
| Scrape to markdown | Yes (cloud) | **Yes (local)** |
| Structured extraction | Yes ($) | **Yes (local LLM)** |
| JSON Schema validation | Yes | **Yes** |
| Batch extraction | Yes | **Yes** |
| Main content extraction | Yes | **Yes** |
| Pricing | $0.001-0.004/page | **Free** |
| Runs locally | No (SaaS) | **Yes** |
| API discovery | No | **Yes** |
| MCP server generation | No | **Yes** |
| Progressive optimization | No | **Yes (3 tiers)** |
| Open source | Partial | **Yes (MIT)** |

### Extract Structured Data

LLM-powered extraction with natural language prompts and JSON Schema validation:

```bash
# Extract data using natural language
site2cli extract https://example.com -p "Extract the page title and all links"

# Extract with JSON Schema validation
site2cli extract https://news.ycombinator.com \
  -s '{"type":"object","properties":{"stories":{"type":"array"}}}'

# Use a Pydantic model as schema
site2cli extract https://example.com -s "myapp.models.Article"

# Batch extraction from multiple URLs
site2cli extract https://example.com -u https://example.org -p "Get the page title"

# Save results to file
site2cli extract https://example.com -p "Extract all headings" -o results.json
```

### Scrape Any Page to Markdown / Text / HTML

```bash
# Convert page to markdown (default)
site2cli scrape https://example.com

# Convert to plain text
site2cli scrape https://example.com --format text

# Extract just the main content (skip nav/footer/sidebar)
site2cli scrape https://example.com --main-content

# Save raw HTML of main content
site2cli scrape https://example.com --format html -o output.html
```

### Crawl an Entire Website

```bash
# Crawl a docs site to markdown
site2cli crawl https://docs.example.com -d 3 -n 100

# Stream pages as JSONL (great for piping)
site2cli crawl https://example.com --format jsonl --stream

# Generate a sitemap (URLs only, no content)
site2cli crawl https://example.com --sitemap

# Save all pages to a directory
site2cli crawl https://example.com -o output/

# Resume a previous crawl
site2cli crawl https://example.com --resume 
```

### Monitor Pages for Changes

```bash
# One-shot: compare against last snapshot
site2cli monitor https://example.com/pricing

# Continuous polling every 5 minutes
site2cli monitor https://example.com/api --interval 300

# Get notified via webhook on change
site2cli monitor https://example.com --webhook https://hooks.slack.com/xxx

# List all watches, show history
site2cli monitor --list
site2cli monitor --history 
```

### RAG-Ready Chunking (`chunk` command)

  

```bash
# Chunk a markdown file by headings (best for docs)
site2cli chunk README.md --strategy heading --output chunks.jsonl

# Chunk a PDF — requires site2cli[rag]
site2cli chunk paper.pdf --strategy sentence --size 500

# Fixed-size chunks with overlap (general fallback)
site2cli chunk notes.txt --strategy fixed --size 1000 --overlap 200

# Crawl + chunk in one shot — JSONL ready for any embedding pipeline
site2cli crawl https://docs.example.com --rag -o chunks.jsonl
```

Each chunk emits `{ text, index, total, url, title, section }` so you can pipe it straight into `pgvector`, `chroma`, `qdrant`, `weaviate`, or any other vector DB.

### Web Search → Scrape → Extract (`search` command)

```bash
# Plain search (DuckDuckGo, no API key)
site2cli search "Python web scraping 2026"

# Search + scrape top N results to markdown
site2cli search "pricing plans" --scrape --format markdown

# Search + scrape + chunk for RAG
site2cli search "API docs" --scrape --chunk heading -o chunks.jsonl

# Search + LLM-powered structured extraction
site2cli search "restaurant reviews austin" \
  --extract -p "extract name, rating, price tier"
```

### Capture Screenshots

```bash
# Full-page screenshot
site2cli screenshot https://example.com -o page.png

# Capture a specific element
site2cli screenshot https://example.com --selector ".pricing-table"

# Viewport-only JPEG with quality
site2cli screenshot https://example.com --viewport --format jpeg --quality 80
```

### Use a Proxy

```bash
# Any command supports --proxy
site2cli discover example.com --proxy http://proxy:8080
site2cli extract https://example.com -p "titles" --proxy socks5://proxy:1080
site2cli scrape https://example.com --proxy http://user:pass@proxy:8080
```

---

## Use with Claude Code / Claude Desktop

  

```bash
# Add site2cli as an MCP server for Claude Code
claude mcp add site2cli -- uvx --from 'site2cli[mcp]' site2cli --mcp

# Or add to Claude Desktop's config (~/.claude/claude_desktop_config.json):
# {
#   "mcpServers": {
#     "site2cli": {
#       "command": "uvx",
#       "args": ["--from", "site2cli[mcp]", "site2cli", "--mcp"]
#     }
#   }
# }
```

Once configured, Claude can call any discovered site's API as a tool:
> "Use site2cli to get data about the Pokemon Ditto"

**Note:** You need to run `site2cli discover ` first to populate the registry. The MCP server exposes all discovered sites as tools.

---

## Manage Browser Auth & Sessions

  

```bash
# Import a Chrome profile for authenticated discovery
site2cli auth profile-import --browser chrome

# Manage cookies
site2cli cookies list example.com
site2cli cookies export example.com

# Reuse browser sessions across commands
site2cli discover example.com --session my-session
site2cli run example.com search --session my-session

# Background browser daemon (persistent browser across CLI calls)
site2cli daemon start
site2cli daemon status
site2cli daemon stop

# Unified MCP server for ALL discovered sites
site2cli --mcp
# or: site2cli mcp serve-all
```

## Comparison

| Feature | browser-use 2.0 | Firecrawl | CLI-Anything | Stagehand v3 | **site2cli** |
|---|---|---|---|---|---|
| Works on any site | Yes | Yes | Yes | Yes | Yes |
| Structured output | No | Yes | Yes | Yes | Yes |
| Auto-discovery | No | No | No | No | **Yes** |
| Structured extraction | No | **Yes ($)** | No | No | **Yes (free)** |
| Scrape to markdown | No | **Yes ($)** | No | No | **Yes (free)** |
| MCP server generation | Acts as MCP | No | No | Yes | **Generates MCP** |
| Progressive optimization | No | No | No | Auto-cache | **Yes (3 tiers)** |
| Runs locally | Yes | No (SaaS) | Yes | Yes | **Yes** |
| Cookie banner handling | No | Yes | No | No | **Yes** |
| Auth page detection | No | No | No | No | **Yes** |
| Self-healing | No | No | No | Yes | **Yes** |
| No browser needed (after discovery) | No | N/A | No | No | **Yes** |
| Session persistence | **Yes** | No | No | No | **Yes** |
| Daemon mode | **Yes** (~50ms) | No | No | No | **Yes** |
| Full site crawling | No | **Yes** | No | No | **Yes** |
| Change detection/monitoring | No | **Yes ($)** | No | No | **Yes (free)** |
| Screenshot capture | No | Yes | No | Yes | **Yes** |
| Community spec sharing | No | No | No | No | **Yes** |

## How It Works

site2cli uses **Progressive Formalization** — a 3-tier system that automatically graduates interactions from slow-but-universal to fast-but-specific:

```mermaid
graph LR
    A["Tier 1: BrowserExploration"] -->|"Patterndetected"| B["Tier 2: CachedWorkflow"]
    B -->|"APIdiscovered"| C["Tier 3: DirectAPI Call"]
    style A fill:#ff6b6b,color:#fff
    style B fill:#ffd93d,color:#000
    style C fill:#6bcb77,color:#fff
```

The **Discovery Pipeline** captures browser traffic and converts it into structured interfaces:

```mermaid
graph TD
    A[Launch Browser + CDP] --> B[Capture Network Traffic]
    B --> C[Group by Endpoint Pattern]
    C --> D[LLM-Assisted Analysis]
    D --> E[OpenAPI 3.1 Spec]
    E --> F[Python Client]
    E --> G[CLI Commands]
    E --> H[MCP Server]
```

## What Gets Generated

From a single discovery session, site2cli produces:

| Output | Description |
|---|---|
| **OpenAPI 3.1 Spec** | Full API specification with schemas, parameters, auth |
| **Python Client** | Typed httpx client with methods for each endpoint |
| **CLI Commands** | Typer commands you can run from terminal |
| **MCP Server** | Tools that AI agents (Claude, etc.) can call directly |

## As a Python Library

```python
from site2cli.discovery.analyzer import TrafficAnalyzer
from site2cli.discovery.spec_generator import generate_openapi_spec
from site2cli.generators.mcp_gen import generate_mcp_server_code

# Analyze captured traffic
analyzer = TrafficAnalyzer(exchanges)
endpoints = analyzer.extract_endpoints()

# Generate OpenAPI spec
spec = generate_openapi_spec(api)

# Generate MCP server
mcp_code = generate_mcp_server_code(site, spec)
```

---

## What's New in v0.7.0

- **Full feature parity with Browserbase `browser-to-api` and `browser-to-cli`** — plus generates more (Python client, MCP tools, Typer CLI) than either.
- **`--replay` and `--save-trace`** on `discover` — offline trace I/O via the portable `Trace` JSON format.
- **`--spec-format yaml|json`** on `discover` — emit OpenAPI 3.1 in either format.
- **Auto-emit four artifacts** — OpenAPI spec, Python client, JS ES-module client (`client.mjs`), and HTML coverage report — every discovery run.
- **Dark-theme HTML coverage report** — endpoint table, methods, status codes, and gap candidates highlighting paths that didn't templatize.
- **Zero-dep JavaScript client** — works in Node 18+, Deno, Bun, and modern browsers. Bearer / API key / custom-header auth via `createClient({ auth })`.
- **`chunk` command** — fixed/sentence/heading strategies for `.md`, `.txt`, `.pdf` → JSONL ready for embeddings.
- **`search` command** — DuckDuckGo + optional `--scrape`, `--extract`, `--chunk` in one pipeline.
- **PDF parsing** — `site2cli[rag]` adds `pdf_to_text`, `pdf_to_markdown`, `pdf_page_count` via pdfplumber.
- **Two long-standing bugs killed** — `chunk_fixed` infinite loop when `overlap >= chunk_size`; brittle `importlib.reload + patch.dict` pattern in PDF tests.
- **559 tests** (up from 500), all passing offline in 
v0.5.0

- **`extract` command** — LLM-powered structured data extraction with JSON Schema validation, Pydantic model support, and batch processing
- **`scrape` command** — Web scraping with HTML-to-markdown/text/html conversion and main content extraction
- **Proxy support** — New `--proxy` flag on `discover`, `run`, `extract`, `scrape`
- **`--format` flag on `run`** — Output results as json, markdown, or text
- **New `content` extra** — `pip install site2cli[content]` for HTML conversion
- **417 tests** (up from 357), all passing

v0.4.0

- **OAuth Device Flow (RFC 8628)** — `site2cli auth login --provider github` for GitHub, Google, Microsoft; token refresh, secure storage
- **Multi-site orchestration** — YAML/JSON pipelines that chain actions across sites with JSONPath data flow (`$result.data[0].id`)
- **Pipeline management** — `site2cli orchestrate run/list/delete` commands with on_error policies (fail/skip/retry)
- **357 tests** (up from 306), all passing

v0.3.1

- **Claude Code MCP integration** — `claude mcp add site2cli -- uvx --from 'site2cli[mcp]' site2cli --mcp` works out of the box
- **Live browser validation** — Experiment 15: real Playwright browser → CDP capture → full pipeline tested against 5 public sites (4/5 pass)
- **LLM-driven exploration validated** — REST Countries: Claude found `/v3.1/all` endpoint in 8 browser steps
- **Auto-probe for static sites** — When homepage has no XHR, automatically discovers and probes API-like links (`/posts`, `/users`, etc.)
- **Terminal demo GIF** — `assets/demo.gif` shows the full discover → run → export flow

v0.3.0

- **Cookie management** — `site2cli cookies list/set/clear/export/import` with Playwright-compatible format
- **Browser profile import** — `site2cli auth profile-import --browser chrome` auto-detects Chrome/Firefox profiles
- **Named browser sessions** — `--session` flag on discover/run, `site2cli session list/close/close-all`
- **Workflow recording** — Record and replay browser workflows with parameterization
- **Background browser daemon** — `site2cli daemon start/stop/status` keeps a persistent browser
- **Unified MCP server** — `site2cli --mcp` serves ALL discovered sites as MCP tools
- **306 tests** (up from 214), all passing

v0.2.5

- **Cookie banner auto-dismissal** — 3-strategy detection (30+ vendor selectors, multilingual text, a11y roles)
- **Auth page detection** — Detects login/SSO/OAuth/MFA/CAPTCHA pages
- **Accessibility tree extraction** — Better page representation for LLM-driven exploration
- **Rich wait conditions** — 9 condition types: network-idle, load, selector, stable, etc.
- **Output filtering** — `--grep`, `--limit`, `--keys-only`, `--compact` flags

---

## Auto-Probe Discovery

Static homepage with no XHR? site2cli auto-discovers and probes REST-like links:

  

## Community Spec Sharing

Share and reuse discovered API specs across teams:

  

---

Live Validation (8 Experiments, 15+ APIs)

### Experiment #8: Core Pipeline (5 APIs)

| API | Endpoints | Spec | Client | MCP | Pipeline |
|---|---|---|---|---|---|
| JSONPlaceholder | 8 | Valid | Makes real calls | 8 tools | 157ms |
| httpbin.org | 7 | Valid | Makes real calls | 7 tools | 179ms |
| Dog CEO API | 5 | Valid | Makes real calls | 5 tools | 209ms |
| Open-Meteo | 1 | Valid | Makes real calls | 1 tool | 686ms |
| GitHub API | 4 | Valid | Makes real calls | 4 tools | 323ms |
| **Total** | **25** | **5/5** | **5/5** | **25 tools** | **avg 310ms** |

### Experiment #9: API Breadth (10 APIs, 7 categories)

| API | Category | Endpoints | Spec | MCP Tools |
|---|---|---|---|---|
| PokeAPI | Structured REST | 5 | Valid | 5 |
| CatFacts | Simple REST | 3 | Valid | 3 |
| Chuck Norris | Simple REST | 3 | Valid | 3 |
| SWAPI (Star Wars) | Nested Paths | 5 | Valid | 5 |
| Open Library | Query Params | 2 | Valid | 2 |
| USGS Earthquake | Government/Science | 2 | Valid | 2 |
| NASA APOD | Government/Science | 1 | Valid | 1 |
| Met Museum | Cultural | 3 | Valid | 3 |
| Art Institute Chicago | Cultural | 4 | Valid | 4 |
| REST Countries | Geographic | 5 | Valid | 5 |
| **Total** | **7 categories** | **33** | **10/10** | **33** |

### Full Validati

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [lonexreb](https://github.com/lonexreb)
- **Source:** [lonexreb/site2cli](https://github.com/lonexreb/site2cli)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** yes

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-lonexreb-site2cli
- Seller: https://agentstack.voostack.com/s/lonexreb
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.