Anansi

> The spider that learns.

Every web scraper starts working. The question is how long before it breaks.

Anansi is built on a different assumption: the web is adversarial and unstable, and your scraper should handle that without your involvement.

When a site changes its layout, Anansi finds the data anyway and remembers the fix: CSS selectors are scored by confidence and healed automatically. When a page needs a browser to render, it switches to one silently. When bot detection gets in the way, it mimics Chrome's TLS fingerprint at the network level, the layer most scrapers never think about. When you re-crawl, unchanged pages are skipped before a request is even made. When extraction goes wrong, Pydantic validation catches it immediately instead of letting garbage accumulate in your database.

The result: a crawler that handles hostile sites, survives redesigns, and gets better the longer it runs. Ships with an MCP server so any LLM can drive a full crawl through a conversation.

---

Capabilities

| | | |---|---| | Self-healing parser | CSS selectors are stored with confidence scores. When one breaks, four healing strategies run — fuzzy class matching, text-pattern regex, structural context, XPath fallback — and the winner is persisted for next time. | | Structured data extraction | JSON-LD, Open Graph, and Microdata are extracted from every page automatically. Fields matched in schema.org markup skip CSS evaluation entirely — they're more stable and require no selector maintenance. | | TLS / HTTP-2 fingerprint mimicry | Enterprise bot-detection (Cloudflare, Akamai, DataDome) fingerprints your TLS ClientHello and HTTP/2 SETTINGS/frame ordering before inspecting a single header. With impersonate="chrome124", Anansi uses curl-cffi to reproduce both, plus per-host session warm-up and a graduated Akamai-block escalation ladder. Install the tls extra (see [Install](#install)); operator-gated, authorized use only. | | Auto browser upgrade | Every HTTP response is checked for SPA markers, noscript redirects, and suspiciously low text density. JS shells trigger a silent retry with a stealth Playwright browser. The decision is cached per domain for the crawl session. | | Anti-bot & Cloudflare bypass | The browser fetcher removes webdriver fingerprints, spoofs plugins, hardware concurrency, audio context, font measurements, battery API, and touch points, adds canvas/WebGL noise, auto-dismisses GDPR/cookie consent banners, and waits out Cloudflare Turnstile challenges automatically. | | Adaptive rate limiting | A per-domain sliding window tracks error rates. A 429 immediately doubles the request gap and activates a circuit breaker. Sustained 5xx errors increase the gap further. Clean windows slowly decay back toward the base delay. | | Incremental crawling | ETag, Last-Modified, and content MD5 are stored per URL. Re-crawls send conditional GET headers — 304 responses skip parsing entirely, and hash comparison catches changes even without server-side ETag support. Sitemap ` dates are used for a pre-flight filter that skips unchanged pages before a network request is even made. | | **URL canonicalization** | Tracking parameters (utm*, fbclid, gclid, and 25 others) are stripped before URLs enter the queue. Remaining parameters are sorted and fragments removed — so ?utmsource=twitter and ?utmsource=facebook are the same crawl target. | | **Item validation** | Set itemschema = MyPydanticModel on a Spider and every yielded item is validated before persistence. Type coercion is automatic ("49.99" → 49.99). Invalid items carry a validationerrors key; valid/invalid counts and error rate appear in live crawl metrics. | | **Concurrent crawler** | Pure asyncio, semaphore-gated workers, SQLite-backed URL queue. Crawls survive process restarts. Pause mid-run and resume days later with Crawler.resume(crawl_id, MySpider)`. | | Proxy rotation | HTTP/HTTPS/SOCKS5 with round-robin, random, or least-used strategies. Failed proxies are auto-quarantined and retested in the background. | | MCP server | FastMCP server exposes 17 scraping tools — fetch, extract, crawl, screenshot, train/validate selectors, cancel, cache control, and more — so any LLM or tool-calling agent can drive a full crawl through a conversation. |

Also includes: JS interaction (click, fill, scroll, infinite-scroll loop, wait), network request interception (capture JSON API responses from SPAs), robots.txt compliance, sitemap discovery, content deduplication, auth/cookie support, configurable retries with Retry-After support, CSV/JSON/JSONL export.

---

Install

The distribution name is anansi-scraper; the import package is anansi. It is installed from this Git repository (not yet published to PyPI), so the optional extras use pip's extras @ git+URL syntax:

# Core install
pip install "git+https://github.com/mdowis/anansi"

# For browser-based fetching (Cloudflare bypass, JS rendering):
playwright install chromium

# With the TLS-fingerprint-mimicry extra (curl-cffi impersonation):
pip install "anansi-scraper[tls] @ git+https://github.com/mdowis/anansi"

# With the OpenAI / ChatGPT Agents SDK extra:
pip install "anansi-scraper[openai] @ git+https://github.com/mdowis/anansi"

Once installed, the MCP server is available as the anansi-mcp console script or via python -m anansi.mcp_server.server, and the CLI as anansi.

Windows: pip is often not on PATH. Use py -m pip install ... instead. If py isn't found either, download Python from python.org and check "Add Python to PATH" during setup.

---

How it works

Extraction pipeline

    Per field:
         │
    ┌────▼──────────────────────────┐
    │  Structured data pre-pass     │  JSON-LD / Open Graph / Microdata
    │                               │  matched fields skip all CSS work
    └────┬──────────────────────────┘
         │ field not in structured data
    ┌────▼──────────────────────────┐
    │  Try known selectors          │  ordered by confidence score (SQLite)
    │  Try primary selector         │
    └────┬──────────────────────────┘
         │ all fail
    ┌────▼──────────────────────────┐
    │  Healing strategies           │
    │  1. Text-pattern match        │  regex on element text
    │  2. Attribute fuzzy match     │  Levenshtein-similar CSS classes
    │  3. Structural context        │  parent/sibling navigation
    │  4. XPath fallback            │  CSS→XPath conversion
    └────┬──────────────────────────┘
         │ winner (score ≥ 0.5)
    ┌────▼──────────────────────────┐
    │  Persist new selector         │  confidence stored in SQLite
    │  Success: score × 1.05 + 0.02 │  cap 1.0
    │  Failure: score × 0.85 − 0.05 │  floor 0.0
    │  Unused >7d: score × 0.99/day │
    └───────────────────────────────┘

Auto browser upgrade

    HTTP fetch
         │
    ┌────▼──────────────────────┐
    │  Domain cached as JS?     │──Yes──► BrowserFetcher directly
    └────┬──────────────────────┘
         │ No
    ┌────▼──────────────────────┐
    │  needs_browser(html)?     │  SPA markers (React/Vue/Next/Nuxt/Angular)
    │                           │  noscript redirect · text/HTML  30% ──► gap × 1.5
    │
    └── window full, error rate  **Note:** this only spoofs the User-Agent. Sites that verify crawlers by
> reverse DNS / source-IP ranges will still see a non-Google address — spoofing
> the UA does not place you on Google's network. Use responsibly and within the
> site's Terms of Service.

### CLI

```bash
# Fetch and print as markdown
anansi fetch https://example.com --output markdown

# Use browser (Cloudflare bypass, JS rendering)
anansi fetch https://protected-site.com --browser

# Fetch presenting as Googlebot
anansi fetch https://example.com --as-googlebot
anansi fetch https://example.com --bot-profile googlebot-mobile

# List all recorded crawls
anansi crawls

# Start the MCP server
anansi mcp

More examples in [/examples](examples/).

---

MCP Server (LLM Integration)

Anansi ships a FastMCP server that exposes all scraping capabilities as tools any LLM can call over stdio transport.

> Windows note: Claude Desktop and most MCP clients on Windows spawn the server with a restricted PATH that often excludes Python313\Scripts\, so anansi-mcp may not be found. Use python -m anansi.mcp_server.server in any config where it fails.

Start the server

anansi-mcp
# or
python -m anansi.mcp_server.server

Tools

| Tool | Description | |---|---| | fetch_url | Fetch a single page — HTML, text, or markdown; supports chunking, browser mode, and browser actions | | fetch_urls | Fetch multiple URLs concurrently in one call | | fetch_and_extract | Fetch and extract structured fields (CSS + structured data) in one call | | extract | Extract structured data from an HTML string with adaptive selectors | | crawl_site | Launch a background crawl; returns a crawl_id immediately | | get_crawl_items | Retrieve persisted items from a crawl (paginated) | | export_crawl | Export items as JSONL, JSON, or CSV | | crawl_metrics | Live stats: pages/sec, error rate, unchanged pages, queue depth, item validation counts | | pause_crawl | Pause a running crawl | | resume_crawl | Resume a paused crawl (same process) | | list_crawls | List all crawls and their state | | selector_health | Inspect learned selector confidence scores for a URL pattern | | cancel_crawl | Permanently cancel a running or paused crawl (irreversible; distinct from pause_crawl) | | screenshot_url | Capture a PNG screenshot of any page via headless browser; returns base64 or saves to file | | train_selector | Manually teach the parser a correct CSS/XPath/text selector for a URL pattern at confidence 1.0 | | validate_selector | Test CSS selectors against a live page without affecting stored confidence scores | | clear_cache | Invalidate the in-memory page cache (all entries, or a single URL) |

fetch_url parameters

| Parameter | Default | Description | |---|---|---| | url | required | The URL to fetch | | use_browser | false | Use headless browser (bypasses Cloudflare, renders JS) | | proxy | null | Proxy URL — "http://user:pass@host:port" | | wait_for_selector | null | Wait for this CSS selector before returning (browser only) | | timeout | 30.0 | Request timeout in seconds | | format | "html" | Output format: "html", "text", or "markdown" | | chunk_size | null | Max characters per chunk — null returns the full page | | chunk_index | 0 | Which chunk to return (0-indexed) | | actions | null | Browser interactions to run after page load (see below) | | impersonate | null | curl-cffi TLS/HTTP-2 fingerprint target (e.g. "chrome124"); falls back to ANANSI_IMPERSONATE env var; per-request, overrides the instance default | | bot_profile | null | Present as a known crawler: "googlebot" or "googlebot-mobile". Pins the crawler User-Agent + minimal headers (and, in a crawl, evaluates robots.txt as that crawler). Independent of impersonate. | | capture_network | false | Browser only. Intercept JSON API responses the page makes during load/actions. Returns raw payloads in captured_requests — ideal for API-first SPAs. Bypasses cache. | | capture_patterns | null | URL substrings to filter captured responses (e.g. ["/api/", "/graphql"]). Max 20 entries. Requires capture_network=true. |

Handling large pages

Raw HTML is often 500 kB–2 MB. Three strategies, simplest to most granular:

Switch format — strips markup (typically 5–10× smaller):

fetch_url(url="https://example.com/article", format="text")
fetch_url(url="https://example.com/docs",    format="markdown")

Chunk — splits at DOM or paragraph boundaries; page is cached 5 min so subsequent chunks cost nothing:

fetch_url(url="https://example.com", format="markdown", chunk_size=20000, chunk_index=0)
# → {content: "...", chunk_index: 0, total_chunks: 4}
fetch_url(url="https://example.com", format="markdown", chunk_size=20000, chunk_index=1)

Extract only what you need — target specific fields with fetch_and_extract or extract and never download the full page content.

fetch_and_extract example

fetch_and_extract(
    url="https://shop.example.com/product/1",
    selectors={"title": "h1.product-title", "price": ".price", "sku": ".sku"},
)
# → {
#     "url": "https://...", "status": 200, "elapsed": 0.42,
#     "data": {"title": "Widget Pro", "price": "$49.99", "sku": "WGT-001"},
#     "structured_data": {
#       "json_ld": [{"@type": "Product", "name": "Widget Pro", "price": "49.99"}],
#       "open_graph": {"title": "Widget Pro", "image": "https://..."},
#       "microdata": []
#     }
#   }

Fields matched in JSON-LD or Open Graph appear in data directly — CSS selectors are not evaluated for them. structured_data always contains the raw metadata.

Browser interactions (actions)

Pass an actions list with use_browser=true for dynamically loaded content. Actions execute in order after page load.

| Type | Required fields | Optional fields | Description | |---|---|---|---| | click | selector | — | Click a CSS-matched element | | fill | selector, value | — | Type text into an input | | press | selector, key | — | Press a key while an element is focused | | scroll_to_bottom | — | — | Scroll to the bottom of the page (single shot) | | scroll_until_stable | — | max_scrolls (1–30, default 10), scroll_delay (100–5000 ms, default 1500) | Scroll repeatedly until page height stops changing — handles infinite-scroll feeds, product listings, and lazy-loaded content. Stops when height is stable for 2 consecutive checks, or when the 60 s action budget is hit. | | wait | ms | — | Pause for N milliseconds | | wait_for_selector | selector | — | Wait until a CSS selector appears in the DOM |

# Infinite scroll — load all items automatically
fetch_url(url="https://example.com/feed", use_browser=true, actions=[
    {"type": "scroll_until_stable", "max_scrolls": 15, "scroll_delay": 1500},
])

# Submit a search form
fetch_url(url="https://example.com/search", use_browser=true, format="text", actions=[
    {"type": "fill", "selector": "input[name=q]", "value": "web scraping"},
    {"type": "press", "selector": "input[name=q]", "key": "Enter"},
    {"type": "wait_for_selector", "selector": ".results"},
])

Network request interception (capture_network)

Many modern sites (React, Next.js, Vue, Nuxt) render a minimal HTML shell and load all actual data via XHR/fetch API calls. capture_network=true registers a response listener before navigation and collects every JSON API response the page makes — bypassing HTML parsing entirely.

fetch_url(
    url="https://shop.example.com/products",
    use_browser=true,
    capture_network=true,
    capture_patterns=["/api/products", "/graphql"],
    actions=[{"type": "scroll_until_stable"}],
)
# → {
#     "url": "https://...", "status": 200, "via_browser": true,
#     "captured_requests": [
#       {"url": "https://shop.example.com/api/products?page=1", "status": 200,
#        "body": {"items": [...], "total": 240}},
#       ...
#     ],
#     "content": "...",   # HTML shell (often minimal)
#   }

• Capped at 50 responses, 200 KB each (larger responses are silently skipped)
• capture_patterns filters by URL substring; omit to capture all JSON responses
• Results bypass the page cache (each call re-fetches and re-intercepts)

Client configuration

Claude Code:

claude mcp add anansi -- anansi-mcp

Claude Desktop / Cursor / Windsurf — add to the client's MCP config file:

{ "mcpServers": { "anansi": { "command": "anansi-mcp" } } }

If anansi-mcp is not found (common on Windows where the Scripts directo

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mdowis
• Source: mdowis/anansi
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.