Scrapo

🕸️ Scrapo

The web-scraping library agents deserve.

Selector-cheap. LLM-resilient. Replay-safe. Self-hosted.

[](https://www.python.org/) [](LICENSE) [](https://github.com/vikast908/Scrapo)

[Quickstart](#quickstart) | [Architecture](#architecture) | [Features](#features) | [Why Scrapo](#why-scrapo) | [CLI](#cli) | [MCP](#use-as-an-mcp-server)

---

What is Scrapo?

Scrapo is a Python library that fuses four worlds the rest of the market keeps separate:

AI-native ingestionmarkdown, schema JSON

Agentic browsingobserve / act / extract

Production crawlingqueues, dedup, scaling

Managed accessapi-first, proxies, anti-bot

Plus a feature nobody else ships: deterministic replay of every fetch, so extraction drift is auditable.

> Not a developer? [LAYMAN.md](LAYMAN.md) explains what Scrapo does, and what it cannot do, in plain English.

---

Why Scrapo

| 5-tier router | Hybrid extractor | Model pinning | |:---:|:---:|:---:| | Auto-escalates HTTP, browser, stealth, agent on real failure signals only | Selector-cheap by default; falls back to LLM and self-heals | Strict mode refuses unpinned LLM extraction, so extraction cannot silently drift |

| Provenance | Deterministic replay | Safe by default | |:---:|:---:|:---:| | Every chunk carries URL, selector path, byte range, heading trail | Re-extract from archived HTML 6 months later; diff fields between any two runs | SSRF guard, opt-in robots gate, regex+Luhn PII, geo allow/deny, append-only audit log |

---

Architecture

                 +---------------------------------------------+
                 |               scrapo.scrape(...)            |
                 +-------------------+-------------------------+
                                     |
        +----------------------------+----------------------------+
        |                            |                            |
   (1) Tier Router            (2) Extractor             (3) Document Shaper
   API-first (known APIs)     Embedded metadata (free)   Content-type dispatch
   T0 HTTP (retries)          Selector cache (host key)  HTML to Markdown
   T1 HTTP+session            -> if validation fails     Heading-aware chunks
   T2 Browser                 model-agnostic LLM         Per-chunk provenance
   T3 Browser+stealth         -> self-heal / evict       Cross-crawl dedup
   T4 Agent                   selector cache writeback
        |                            |                            |
        +---------+------------------+---------+-------------------+
                  |                            |
            (4) Replay store             (5) Policy gate
            SQLite (WAL) + gzip HTML      SSRF / robots / PII / geo / audit
                  |                            |
                  +-------------+--------------+
                                |
                        (6) Agent surface
                        MCP server + tool schemas

scrapo/
├── access/      # (1) API-first resolver (api_providers) + 5-tier router + pooled browser + request interception + agent driver + action cache + proxy adapters & rotating pool + Interact actions (incl. scroll_until / click_until)
├── extract/     # (2) embedded metadata (JSON-LD/OG/microdata) + hybrid selector + model-agnostic LLM (Anthropic/Gemini native, any OpenAI-compatible endpoint), model pinning, cost-aware budget
├── shape/       # (3) markdown + heading chunker + content-type dispatch (HTML / JSON / feed / PDF / text)
├── replay/      # (4) SQLite metadata + pluggable snapshot store (local or S3) + field-level diff
├── policy/      # (5) robots, PII (flag or redact), geo, append-only audit
├── crawl/       # persistent SQLite queue + async scheduler + sitemap discovery + rel=next pagination + batch
├── agent/       # (6) MCP server + tool schemas
├── server/      # (7) self-hosted watch control plane: persistent WatchStore + WatchScheduler + webhook/callback notifiers
├── results.py   # typed ScrapeResult / CrawlResult / ExtractionView
├── watch.py     # watch(url) -> Watch.refresh() -> ChangeSet (change tracking)
├── export.py    # to_jsonl / to_csv dataset writers
├── sync.py      # synchronous facade (scrape_sync / crawl_sync / ...)
├── security.py  # SSRF guard for fetch targets
├── _db.py       # tuned SQLite connections (WAL, busy timeout)
├── logging.py   # structlog setup for the CLI / MCP server
├── api.py       # public scrape / extract / crawl / crawl_stream
├── web.py       # local browser UI (scrapo serve)
└── cli.py       # Typer CLI

---

Quickstart

pip install scrapo-ai          # distribution is "scrapo-ai"; in code you `import scrapo`

pip install "scrapo-ai[browser,anthropic,mcp]"
playwright install chromium

1. Scrape one URL

import asyncio, scrapo

async def main():
    res = await scrapo.scrape("https://example.com/")   # res is a typed ScrapeResult
    print(res.markdown)
    print("run_id:", res.run_id)
    # res["markdown"] / res.get("status") still work too (back-compat with the 0.1 dict)

asyncio.run(main())

2. Typed extraction, including lists (LLM once, selectors forever)

import asyncio, scrapo
from pydantic import BaseModel

class Offer(BaseModel):
    name: str
    price: str

class Listing(BaseModel):
    page_title: str
    offers: list[Offer] = []        # array fields become repeated-element extraction

async def main():
    res = await scrapo.scrape("https://example.com/shop", schema=Listing)
    print(res.extraction.data)      # {'page_title': '...', 'offers': [{'name': ..., 'price': ...}, ...]}
    print(res.extraction.method)    # 'llm' on the first run, 'selector' after
    print(res.cost_usd)             # 0.0 once selectors are cached

asyncio.run(main())

> First call uses the LLM and caches the selectors it learns (keyed by host + schema; for list[Model] fields it caches a container selector plus per-subfield selectors). Every subsequent call against that host + schema uses cached selectors and zero LLM tokens. When the layout drifts, validation fails, Scrapo falls back to the LLM, re-derives selectors, and self-heals; a cache entry that keeps failing is evicted automatically.

3. Recursive crawl

await scrapo.crawl(
    seeds=["https://docs.python.org/3/"],
    max_depth=2,
    same_host_only=True,
)

4. Replay and diff

scrapo list                    # recent runs
scrapo replay          # re-extract from archived HTML, no network
scrapo diff      # field-level diff

---

Features

Cost-aware tier router

| Tier | What it does | When | |---|---|---| | T0 HTTP | httpx plain GET, with bounded retry/backoff on 429/5xx and transport errors | static HTML, JSON endpoints | | T1 HTTP_SESSIONED | + browser-like headers/cookies | soft anti-bot | | T2 BROWSER | Playwright headless | JS-rendered pages, SPA shells | | T3 BROWSER_STEALTH | + stealth + residential proxy | hard anti-bot | | T4 AGENT | LLM-driven multi-step browser via a pluggable AgentDriver (a reference LLMAgentDriver ships in; SCRAPO_AGENT_DRIVER=llm), with action caching so a repeated goal replays without the LLM | logins, captchas, flows |

Before T0, API-first resolution (see the feature section below) short-circuits the whole ladder for known sites (e.g. Wikipedia → its REST API), so the cheapest path of all is one the tier router never sees.

Escalation triggers: Cloudflare/Akamai/PerimeterX/DataDome/Distil fingerprints, HTTP 403 / 429 / 503, empty body, missing required schema fields, and unrendered single-page-app shells (lots of script, almost no rendered text). Budget(max_tier=..., max_llm_calls=..., max_cost_usd=...) caps how far it goes. The browser tiers block images/fonts/media/css by default and capture JSON XHR/fetch responses onto the result. The Tier-4 driver records the action sequence it used to reach a goal on a host (agent_actions.sqlite) and replays it on later runs with zero LLM tokens, self-healing back to the model only when a recorded step no longer applies (SCRAPO_AGENT_ACTION_CACHE=0 to disable).

API-first: known sites resolve to their public API

Some sites CAPTCHA every scraper yet publish the same content through a clean, unauthenticated API. When Scrapo recognises such a URL it fetches the API before the tier router runs — skipping the whole HTTP → browser → stealth → agent escalation and the bot wall that defeats it. Wikipedia is the headline case: it blocks scrapers aggressively but serves every article through its REST API (/api/rest_v1/page/html/{title}), and the same contract covers its Wikimedia sister projects (Wiktionary, Wikinews, Wikibooks, Wikiquote, Wikiversity, Wikivoyage, Wikisource).

r = scrape_sync("https://en.wikipedia.org/wiki/Albert_Einstein")
r.via        # "api:wikipedia"  — served from the REST API, no CAPTCHA, no browser
r.url        # "https://en.wikipedia.org/wiki/Albert_Einstein"  — the page you asked for
r.markdown   # clean article text, through the normal markdown/chunk/extraction pipeline

The REST HTML runs through the same markdown / chunk / provenance / extraction pipeline as any page, and conditional-GET + replay still apply. On by default; turn it off per call with scrape(api_first=False) (CLI --no-api-first, MCP api_first=false) or globally with SCRAPO_API_FIRST=0. It's also suppressed automatically whenever you force a tier, pass actions, or ask for a screenshot — i.e. when you explicitly want the live page. Agents get this for free: the scrapo_scrape MCP tool documents it and the result's via field reports when it fired. The provider registry (scrapo/access/api_providers.py) is a plain tuple, so adding a site is a few lines.

Content-type aware: HTML, JSON, feeds, PDFs

A URL is not always an HTML page, so scrape() dispatches on Content-Type (with a little body sniffing):

| Content | What you get | result.kind | |---|---|---| | text/html | the normal selectolax + markdown + chunk pipeline | html | | application/json / ld+json | pretty-printed JSON as markdown; parsed object on result.data | json | | RSS / Atom | a markdown list of entries; parsed items on result.data | feed | | application/pdf | extracted text (requires pip install "scrapo-ai[pdf]") | pdf | | text/plain | the body verbatim | text |

Zero-LLM extraction from embedded structured data

Before the selector cache or the LLM, Scrapo tries to satisfy your schema straight from data the page already hands out: schema.org JSON-LD (`), OpenGraph / Twitter / vertical tags, and microdata itemprop` attributes. A huge fraction of commercial pages (products, articles, recipes, jobs, events) embed this, so the common case becomes free, deterministic, and immune to layout drift.

class Product(BaseModel):
    name: str
    price: str | None = None

res = await scrapo.scrape("https://shop.example.com/widget", schema=Product)
print(res.extraction.method)   # 'metadata'  (no selector cache, no LLM)
print(res.cost_usd)            # 0.0

The extraction ladder is now: embedded metadata -> selector cache -> LLM. It is conservative: it only returns when every required field was sourced and the object validates, otherwise it falls through to the existing path, so it never costs correctness. The bare ` tag is not treated as a source (it is page chrome, not a structured annotation). On by default; Config(metadataextraction=False) or SCRAPOMETADATA_EXTRACTION=0` disables it.

Hybrid selector + LLM extractor (scalar and list fields)

cache hit + validates    -> return (method=selector, llm_calls=0, cost_usd=0)
miss / fail / over budget -> LLM with schema -> validate -> verify + persist selectors -> return (method=llm)
repeated cache failures   -> evict the stale entry, re-derive next run

The LLM is asked to return both the JSON payload and CSS selectors per field. A scalar field gets a string selector; a list[Model] field gets {"__list__": "", "": "", ...}, which Scrapo applies as tree.css(container) then per-subfield extraction inside each match. Returned selectors are verified against the live HTML before being cached, so a hallucinated selector never poisons the cache. The cache is keyed by host (not registered domain), so blog.example.com and shop.example.com never collide.

Model pinning (Zyte-style, but built in)

from scrapo.extract.pinning import PinnedModel

pin = PinnedModel.make(
    provider="anthropic",
    model_id="claude-opus-4-7",
    prompt_template="(your prompt template)",
)

await scrapo.scrape(url, schema=Product, pin=pin, strict_pin=True)

strict_pin=True makes the extractor refuse to run if the configured LLM does not match the pin. Silent model drift cannot happen in production.

Per-chunk provenance

Every chunk Scrapo emits carries:

{
    "url":           "https://example.com/page",
    "selector_path": "markdown://Features/Pricing",
    "byte_start":    8421,
    "byte_end":      9842,
    "heading_trail": ["Features", "Pricing"],
    "chunk_hash":    "ab12cd34...",
}

You can trace any LLM citation back to a specific section of a specific URL.

Deterministic replay and diff

Every fetch persists raw HTML, headers, screenshots, and the typed extraction:

scrapo replay 9f3e1c...        # re-extract from archived HTML, no network
scrapo diff 9f3e1c... abc123...  # field-level diff, with notes when model/schema changed

Sample diff output:

diff 9f3e1c...  vs  abc123...
  HTML changed
  ! model changed: anthropic:claude-opus-4-7 -> anthropic:claude-sonnet-4-6 (extraction may drift)
  field changes:
    - price: '$42' -> '$45'
    - in_stock: True -> False

scrapo scrape --diff-last prints that diff against the previous run of the same URL in one step.

Watch a URL for changes (cheap re-scrapes)

Re-scraping a URL the HTTP tier fetched before sends a conditional GET (If-None-Match / If-Modified-Since). A 304 Not Modified is rebuilt from the archived snapshot: no body transfer, no LLM call (the selector cache makes re-extraction free), and no duplicate snapshot is written. scrape() / crawl() get this automatically; Config(conditional_requests=False) (or SCRAPO_CONDITIONAL_REQUESTS=0) turns it off.

watch() builds the change-tracking loop on top of that:

import scrapo

w = await scrapo.watch("https://example.com/pricing", schema=Pricing)
# ... later, or on a schedule of your choosing ...
change = await w.refresh()
if change.not_modified:
    print("unchanged (304)")
elif change.changed:
    print(change.summary())          # field-level diff vs. the previous run
    for d in change.field_changes:
        print(d)                     # e.g. price: '$42' -> '$45'

Watch is in-process: the run history (and the diff) live in the replay store. Persisting a list of watches with a built-in scheduler is a hosted-service concern and is intentionally left out.

Streaming crawl

async for page in scrapo.crawl_stream(["https://blog.example.com/"], schema=Post):
    save(page)                       # process pages as they complete, not all at the end

Breaking out of the loop early stops the crawl and tears the shared browser down. crawl() remains the buffered convenience (returns aggregate stats + an on_page callback).

Main-content extraction (cleaner Markdown)

Turn on a readability-style pass that strips site furniture — nav, sidebars, footers, cookie banners, ads — and keeps just the article body before converting to Markdown. Output reads like a clean document, which is what you want for RAG/LLM ingestion.

res = await scrapo.scrape("https://blog.example.com/post", main_content=True)
print(res.markdown)   # boilerplate removed; provenance/chunks still attached

Off by default (full-page conversion). Enab

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: vikast908
• Source: vikast908/Scrapo
• License: MIT
• Homepage: https://scrapo-agent.vercel.app

Install and usage instructions live in the source repository linked above.