# Scrapo

> AI-native, agent-first web scraping for Python. Cost-aware tiered fetching (HTTP, browser, stealth, agent), model-agnostic LLM extraction with self-healing selector caches, API-first resolution, deterministic replay and field-level diff, and an MCP server.

- **Type:** MCP server
- **Install:** `agentstack add mcp-vikast908-scrapo`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [vikast908](https://agentstack.voostack.com/s/vikast908)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [vikast908](https://github.com/vikast908)
- **Source:** https://github.com/vikast908/Scrapo
- **Website:** https://scrapo-agent.vercel.app

## Install

```sh
agentstack add mcp-vikast908-scrapo
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# 🕸️ Scrapo

**The web-scraping library agents deserve.**

*Selector-cheap. LLM-resilient. Replay-safe. Self-hosted.*

[](https://www.python.org/)
[](LICENSE)
[](https://github.com/vikast908/Scrapo)

[Quickstart](#quickstart) | [Architecture](#architecture) | [Features](#features) | [Why Scrapo](#why-scrapo) | [CLI](#cli) | [MCP](#use-as-an-mcp-server)

---

## What is Scrapo?

Scrapo is a Python library that fuses four worlds the rest of the market keeps separate:

**AI-native ingestion***markdown, schema JSON*

**Agentic browsing***observe / act / extract*

**Production crawling***queues, dedup, scaling*

**Managed access***api-first, proxies, anti-bot*

Plus a feature nobody else ships: **deterministic replay** of every fetch, so extraction drift is auditable.

> Not a developer? **[LAYMAN.md](LAYMAN.md)** explains what Scrapo does, and what it cannot do, in plain English.

---

## Why Scrapo

| **5-tier router** | **Hybrid extractor** | **Model pinning** |
|:---:|:---:|:---:|
| Auto-escalates HTTP, browser, stealth, agent on real failure signals only | Selector-cheap by default; falls back to LLM and self-heals | Strict mode refuses unpinned LLM extraction, so extraction cannot silently drift |

| **Provenance** | **Deterministic replay** | **Safe by default** |
|:---:|:---:|:---:|
| Every chunk carries URL, selector path, byte range, heading trail | Re-extract from archived HTML 6 months later; diff fields between any two runs | SSRF guard, opt-in robots gate, regex+Luhn PII, geo allow/deny, append-only audit log |

---

## Architecture

```
                 +---------------------------------------------+
                 |               scrapo.scrape(...)            |
                 +-------------------+-------------------------+
                                     |
        +----------------------------+----------------------------+
        |                            |                            |
   (1) Tier Router            (2) Extractor             (3) Document Shaper
   API-first (known APIs)     Embedded metadata (free)   Content-type dispatch
   T0 HTTP (retries)          Selector cache (host key)  HTML to Markdown
   T1 HTTP+session            -> if validation fails     Heading-aware chunks
   T2 Browser                 model-agnostic LLM         Per-chunk provenance
   T3 Browser+stealth         -> self-heal / evict       Cross-crawl dedup
   T4 Agent                   selector cache writeback
        |                            |                            |
        +---------+------------------+---------+-------------------+
                  |                            |
            (4) Replay store             (5) Policy gate
            SQLite (WAL) + gzip HTML      SSRF / robots / PII / geo / audit
                  |                            |
                  +-------------+--------------+
                                |
                        (6) Agent surface
                        MCP server + tool schemas
```

```
scrapo/
├── access/      # (1) API-first resolver (api_providers) + 5-tier router + pooled browser + request interception + agent driver + action cache + proxy adapters & rotating pool + Interact actions (incl. scroll_until / click_until)
├── extract/     # (2) embedded metadata (JSON-LD/OG/microdata) + hybrid selector + model-agnostic LLM (Anthropic/Gemini native, any OpenAI-compatible endpoint), model pinning, cost-aware budget
├── shape/       # (3) markdown + heading chunker + content-type dispatch (HTML / JSON / feed / PDF / text)
├── replay/      # (4) SQLite metadata + pluggable snapshot store (local or S3) + field-level diff
├── policy/      # (5) robots, PII (flag or redact), geo, append-only audit
├── crawl/       # persistent SQLite queue + async scheduler + sitemap discovery + rel=next pagination + batch
├── agent/       # (6) MCP server + tool schemas
├── server/      # (7) self-hosted watch control plane: persistent WatchStore + WatchScheduler + webhook/callback notifiers
├── results.py   # typed ScrapeResult / CrawlResult / ExtractionView
├── watch.py     # watch(url) -> Watch.refresh() -> ChangeSet (change tracking)
├── export.py    # to_jsonl / to_csv dataset writers
├── sync.py      # synchronous facade (scrape_sync / crawl_sync / ...)
├── security.py  # SSRF guard for fetch targets
├── _db.py       # tuned SQLite connections (WAL, busy timeout)
├── logging.py   # structlog setup for the CLI / MCP server
├── api.py       # public scrape / extract / crawl / crawl_stream
├── web.py       # local browser UI (scrapo serve)
└── cli.py       # Typer CLI
```

---

## Quickstart

```bash
pip install scrapo-ai          # distribution is "scrapo-ai"; in code you `import scrapo`

pip install "scrapo-ai[browser,anthropic,mcp]"
playwright install chromium
```

### 1. Scrape one URL

```python
import asyncio, scrapo

async def main():
    res = await scrapo.scrape("https://example.com/")   # res is a typed ScrapeResult
    print(res.markdown)
    print("run_id:", res.run_id)
    # res["markdown"] / res.get("status") still work too (back-compat with the 0.1 dict)

asyncio.run(main())
```

### 2. Typed extraction, including lists (LLM once, selectors forever)

```python
import asyncio, scrapo
from pydantic import BaseModel

class Offer(BaseModel):
    name: str
    price: str

class Listing(BaseModel):
    page_title: str
    offers: list[Offer] = []        # array fields become repeated-element extraction

async def main():
    res = await scrapo.scrape("https://example.com/shop", schema=Listing)
    print(res.extraction.data)      # {'page_title': '...', 'offers': [{'name': ..., 'price': ...}, ...]}
    print(res.extraction.method)    # 'llm' on the first run, 'selector' after
    print(res.cost_usd)             # 0.0 once selectors are cached

asyncio.run(main())
```

> First call uses the LLM and **caches the selectors it learns** (keyed by host + schema; for `list[Model]` fields it caches a container selector plus per-subfield selectors). Every subsequent call against that host + schema uses cached selectors and **zero LLM tokens**. When the layout drifts, validation fails, Scrapo falls back to the LLM, re-derives selectors, and self-heals; a cache entry that keeps failing is evicted automatically.

### 3. Recursive crawl

```python
await scrapo.crawl(
    seeds=["https://docs.python.org/3/"],
    max_depth=2,
    same_host_only=True,
)
```

### 4. Replay and diff

```bash
scrapo list                    # recent runs
scrapo replay          # re-extract from archived HTML, no network
scrapo diff      # field-level diff
```

---

## Features

Cost-aware tier router

| Tier | What it does | When |
|---|---|---|
| **T0** `HTTP` | `httpx` plain GET, with bounded retry/backoff on 429/5xx and transport errors | static HTML, JSON endpoints |
| **T1** `HTTP_SESSIONED` | + browser-like headers/cookies | soft anti-bot |
| **T2** `BROWSER` | Playwright headless | JS-rendered pages, SPA shells |
| **T3** `BROWSER_STEALTH` | + stealth + residential proxy | hard anti-bot |
| **T4** `AGENT` | LLM-driven multi-step browser via a pluggable `AgentDriver` (a reference `LLMAgentDriver` ships in; `SCRAPO_AGENT_DRIVER=llm`), with **action caching** so a repeated goal replays without the LLM | logins, captchas, flows |

Before T0, **API-first resolution** (see the feature section below) short-circuits the whole ladder for known sites (e.g. Wikipedia → its REST API), so the cheapest path of all is one the tier router never sees.

Escalation triggers: Cloudflare/Akamai/PerimeterX/DataDome/Distil fingerprints, HTTP `403 / 429 / 503`, empty body, missing required schema fields, and unrendered single-page-app shells (lots of script, almost no rendered text). `Budget(max_tier=..., max_llm_calls=..., max_cost_usd=...)` caps how far it goes. The browser tiers block images/fonts/media/css by default and capture JSON XHR/fetch responses onto the result. The Tier-4 driver records the action sequence it used to reach a goal on a host (`agent_actions.sqlite`) and replays it on later runs with zero LLM tokens, self-healing back to the model only when a recorded step no longer applies (`SCRAPO_AGENT_ACTION_CACHE=0` to disable).

API-first: known sites resolve to their public API

Some sites CAPTCHA every scraper yet publish the *same* content through a clean, unauthenticated API. When Scrapo recognises such a URL it fetches the API **before** the tier router runs — skipping the whole HTTP → browser → stealth → agent escalation and the bot wall that defeats it. Wikipedia is the headline case: it blocks scrapers aggressively but serves every article through its REST API (`/api/rest_v1/page/html/{title}`), and the same contract covers its Wikimedia sister projects (Wiktionary, Wikinews, Wikibooks, Wikiquote, Wikiversity, Wikivoyage, Wikisource).

```python
r = scrape_sync("https://en.wikipedia.org/wiki/Albert_Einstein")
r.via        # "api:wikipedia"  — served from the REST API, no CAPTCHA, no browser
r.url        # "https://en.wikipedia.org/wiki/Albert_Einstein"  — the page you asked for
r.markdown   # clean article text, through the normal markdown/chunk/extraction pipeline
```

The REST HTML runs through the same markdown / chunk / provenance / extraction pipeline as any page, and conditional-GET + replay still apply. On by default; turn it off per call with `scrape(api_first=False)` (CLI `--no-api-first`, MCP `api_first=false`) or globally with `SCRAPO_API_FIRST=0`. It's also suppressed automatically whenever you force a tier, pass `actions`, or ask for a `screenshot` — i.e. when you explicitly want the live page. Agents get this for free: the `scrapo_scrape` MCP tool documents it and the result's `via` field reports when it fired. The provider registry (`scrapo/access/api_providers.py`) is a plain tuple, so adding a site is a few lines.

Content-type aware: HTML, JSON, feeds, PDFs

A URL is not always an HTML page, so `scrape()` dispatches on `Content-Type` (with a little body sniffing):

| Content | What you get | `result.kind` |
|---|---|---|
| `text/html` | the normal selectolax + markdown + chunk pipeline | `html` |
| `application/json` / `ld+json` | pretty-printed JSON as markdown; parsed object on `result.data` | `json` |
| RSS / Atom | a markdown list of entries; parsed items on `result.data` | `feed` |
| `application/pdf` | extracted text (requires `pip install "scrapo-ai[pdf]"`) | `pdf` |
| `text/plain` | the body verbatim | `text` |

Zero-LLM extraction from embedded structured data

Before the selector cache or the LLM, Scrapo tries to satisfy your schema straight from data the page already hands out: schema.org JSON-LD (``), OpenGraph / Twitter / vertical `` tags, and microdata `itemprop` attributes. A huge fraction of commercial pages (products, articles, recipes, jobs, events) embed this, so the common case becomes free, deterministic, and immune to layout drift.

```python
class Product(BaseModel):
    name: str
    price: str | None = None

res = await scrapo.scrape("https://shop.example.com/widget", schema=Product)
print(res.extraction.method)   # 'metadata'  (no selector cache, no LLM)
print(res.cost_usd)            # 0.0
```

The extraction ladder is now: **embedded metadata -> selector cache -> LLM**. It is conservative: it only returns when every required field was sourced and the object validates, otherwise it falls through to the existing path, so it never costs correctness. The bare `` tag is not treated as a source (it is page chrome, not a structured annotation). On by default; `Config(metadata_extraction=False)` or `SCRAPO_METADATA_EXTRACTION=0` disables it.

Hybrid selector + LLM extractor (scalar and list fields)

```
cache hit + validates    -> return (method=selector, llm_calls=0, cost_usd=0)
miss / fail / over budget -> LLM with schema -> validate -> verify + persist selectors -> return (method=llm)
repeated cache failures   -> evict the stale entry, re-derive next run
```

The LLM is asked to return both the JSON payload *and* CSS selectors per field. A scalar field gets a string selector; a `list[Model]` field gets `{"__list__": "", "": "", ...}`, which Scrapo applies as `tree.css(container)` then per-subfield extraction inside each match. Returned selectors are verified against the live HTML before being cached, so a hallucinated selector never poisons the cache. The cache is keyed by host (not registered domain), so `blog.example.com` and `shop.example.com` never collide.

Model pinning (Zyte-style, but built in)

```python
from scrapo.extract.pinning import PinnedModel

pin = PinnedModel.make(
    provider="anthropic",
    model_id="claude-opus-4-7",
    prompt_template="(your prompt template)",
)

await scrapo.scrape(url, schema=Product, pin=pin, strict_pin=True)
```

`strict_pin=True` makes the extractor refuse to run if the configured LLM does not match the pin. Silent model drift cannot happen in production.

Per-chunk provenance

Every chunk Scrapo emits carries:
```python
{
    "url":           "https://example.com/page",
    "selector_path": "markdown://Features/Pricing",
    "byte_start":    8421,
    "byte_end":      9842,
    "heading_trail": ["Features", "Pricing"],
    "chunk_hash":    "ab12cd34...",
}
```

You can trace any LLM citation back to a specific section of a specific URL.

Deterministic replay and diff

Every fetch persists raw HTML, headers, screenshots, and the typed extraction:

```bash
scrapo replay 9f3e1c...        # re-extract from archived HTML, no network
scrapo diff 9f3e1c... abc123...  # field-level diff, with notes when model/schema changed
```

Sample diff output:
```
diff 9f3e1c...  vs  abc123...
  HTML changed
  ! model changed: anthropic:claude-opus-4-7 -> anthropic:claude-sonnet-4-6 (extraction may drift)
  field changes:
    - price: '$42' -> '$45'
    - in_stock: True -> False
```

`scrapo scrape  --diff-last` prints that diff against the previous run of the same URL in one step.

Watch a URL for changes (cheap re-scrapes)

Re-scraping a URL the HTTP tier fetched before sends a conditional GET (`If-None-Match` / `If-Modified-Since`). A `304 Not Modified` is rebuilt from the archived snapshot: **no body transfer, no LLM call** (the selector cache makes re-extraction free), and no duplicate snapshot is written. `scrape()` / `crawl()` get this automatically; `Config(conditional_requests=False)` (or `SCRAPO_CONDITIONAL_REQUESTS=0`) turns it off.

`watch()` builds the change-tracking loop on top of that:

```python
import scrapo

w = await scrapo.watch("https://example.com/pricing", schema=Pricing)
# ... later, or on a schedule of your choosing ...
change = await w.refresh()
if change.not_modified:
    print("unchanged (304)")
elif change.changed:
    print(change.summary())          # field-level diff vs. the previous run
    for d in change.field_changes:
        print(d)                     # e.g. price: '$42' -> '$45'
```

`Watch` is in-process: the run history (and the diff) live in the replay store. Persisting a *list* of watches with a built-in scheduler is a hosted-service concern and is intentionally left out.

Streaming crawl

```python
async for page in scrapo.crawl_stream(["https://blog.example.com/"], schema=Post):
    save(page)                       # process pages as they complete, not all at the end
```

Breaking out of the loop early stops the crawl and tears the shared browser down. `crawl()` remains the buffered convenience (returns aggregate stats + an `on_page` callback).

Main-content extraction (cleaner Markdown)

Turn on a readability-style pass that strips site furniture — nav, sidebars, footers, cookie banners, ads — and keeps just the article body before converting to Markdown. Output reads like a clean document, which is what you want for RAG/LLM ingestion.

```python
res = await scrapo.scrape("https://blog.example.com/post", main_content=True)
print(res.markdown)   # boilerplate removed; provenance/chunks still attached
```

Off by default (full-page conversion). Enab

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [vikast908](https://github.com/vikast908)
- **Source:** [vikast908/Scrapo](https://github.com/vikast908/Scrapo)
- **License:** MIT
- **Homepage:** https://scrapo-agent.vercel.app

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-vikast908-scrapo
- Seller: https://agentstack.voostack.com/s/vikast908
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.