Ultimate Scraper

Ultimate Web Scraper

Quick Start

python scripts/scrape.py "https://example.com"

With AI extraction:

python scripts/scrape.py "https://example.com" \
  -e "Extract all product names and prices" -o json

Workflow

1. Basic Scrape

python scripts/scrape.py "URL"

Output: Markdown content to stdout.

2. With AI Extraction

python scripts/scrape.py "URL" \
  -e "Natural language instruction" -o json

Output: JSON with data field containing extracted info.

3. Protected Sites

python scripts/scrape.py "URL" \
  -m stealth -g us

Output: Content fetched via Camoufox + US residential proxy.

4. Handle Failures

If exit code 2 (premium proxy required), use your preferred unlocker/proxy service.

Decision Tree

User wants to scrape?
├─ Simple page, no anti-bot → python scripts/scrape.py "URL"
├─ Need specific data → python scripts/scrape.py "URL" -e "Extract X" -o json
├─ Protected site → python scripts/scrape.py "URL" -m stealth -g us
├─ Multiple URLs → python scripts/scrape.py URL1 URL2 -p 10
├─ URLs from file → python scripts/scrape.py --batch urls.jsonl -p 10
├─ Save to separate files → python scripts/scrape.py --batch urls.txt --output-dir ./articles/
├─ Long-running job → python scripts/scrape.py --batch FILE --output-dir ./ --checkpoint job.json
├─ Exit code 1 + NotFound → Page doesn't exist (404), no action needed
├─ Exit code 2 → Requires premium proxy/unlocker service
└─ Detect protection → python scripts/scrape.py "URL" --probe-only

CLI Options

| Option | Short | Description | Default | |--------|-------|-------------|---------| | --mode MODE | -m | auto\|static\|http\|browser\|agent\|stealth\|ai\|visual | auto | | --output FMT | -o | markdown\|json\|raw | markdown | | --json | -j | Alias for -o json | | | --extract PROMPT | -e | Natural language extraction instruction | | | --schema JSON | -s | JSON schema for structured extraction | | | --visual | | Enable visual extraction (screenshot + Vision LLM) | false | | --proxy-geo GEO | -g | us, uk, de, us-newyork, uk-london, etc. | | | --proxy-sticky | | Same IP across requests | false | | --session NAME | | Named session for cookie persistence | | | --max-tier N | | Limit escalation (0-5) | 4 | | --timeout SEC | -t | Request timeout in seconds | 30 | | --no-cache | | Bypass 24h cache | false | | --verbose | -v | Show tier progression | false | | --probe-only | | Detect site profile only | false | | --parallel N | -p | Concurrent scrapes (batch mode) | 5 | | --batch FILE | | Read URLs from file (JSONL or text) | | | --batch-output FILE | | JSONL output for batch (after completion) | | | --output-stream FILE | | Stream results to JSONL (as each completes) | | | --output-dir DIR | | Write each result to separate file in directory | | | --output-ext EXT | | File extension for --output-dir (default: md) | md | | --checkpoint FILE | | Resume file for long-running jobs | | | --import-state FILE | | Import agent-browser state | | | --export-state FILE | | Export state for agent-browser | | | --actions JSON | | Browser actions array | | | --wait-for SEL | | CSS selector to wait for | | | --behavior-intensity N | | Behavioral simulation intensity (0.5-2.0) | 1.0 | | --no-rate-limit | | Disable per-domain rate limiting | false | | --no-trackers | | Disable tracker/fingerprinter blocking | false | | --captcha-solve | | Force CAPTCHA solving attempt | false |

Architecture

| Component | Implementation | |-----------|----------------| | Static extraction | chompjs + extruct (__NEXTDATA_, JSON-LD) | | TLS spoofing | curl_cffi + BrowserForge headers | | Light stealth | CloakBrowser (C++ patches, preferred) + Patchright fallback | | Tracker blocking | CDP network interception (25 patterns, all browser tiers) | | Full anti-detect | Camoufox (C++ fingerprint spoofing) | | CAPTCHA solving | CapSolver (AI) + 2Captcha (human) with auto-detection | | Visual extraction | Screenshot + Vision LLM | | AI extraction | Crawl4AI + 3-tier LLM routing | | WebMCP extraction | Chrome 147+ navigator.modelContext tool discovery | | Proxy | Bring-your-own residential/mobile proxy | | Cache | SQLite, 24h TTL | | Sessions | SQLite-backed, auto-persist on anti-bot | | Fingerprint persistence | SQLite-backed, per-domain consistent identity | | Tier history | Per-domain tier success tracking for faster starts | | Behavioral simulation | Bezier curves, human typing, reading pauses | | Rate limiting | Per-domain sliding window (60s) for browser tiers | | Sensitive sites | Auto-detected (LinkedIn, X, etc.) with enhanced stealth | | Loop detection | Action loop detector (WARNING/STUCK/CRITICAL thresholds) | | Shadow DOM | Recursive deepQuery/deepQueryAll for piercing shadow roots | | JA4T detection | Transport-layer fingerprint detection, auto-skip Tier 1 |

Tier System

| Tier | Mode | Technology | Use Case | |------|------|------------|----------| | 0 | static | chompjs/extruct | __NEXTDATA_, JSON-LD (fastest) | | 1 | http | curl_cffi | TLS fingerprint spoofing | | 2 | browser | CloakBrowser/Patchright | Stealth browser (CAPTCHA solving) | | 2.5 | agent | agent-browser | CLI automation + tracker blocking | | 3 | stealth | Camoufox | Full anti-detect (Cloudflare bypass) | | 4 | ai | Crawl4AI + LLM | AI-powered extraction | | 5 | visual | Screenshot + Vision | Visual LLM extraction (bypasses DOM detection) |

Auto-escalation: Tier N fails → rotate proxy → retry → escalate to N+1.

JA4T Detection

Sites using transport-layer fingerprinting (JA4T) are automatically detected. When JA4T is detected:

• Tier 1 (HTTP with TLS spoofing) is skipped
• Starts at Tier 2 (browser) minimum
• Verbose mode shows: [JA4T] Skipping Tier 1 - transport-layer fingerprinting detected

CAPTCHA Solving

When a CAPTCHA is encountered at Tier 2 or 3:

1. Detects type (reCAPTCHA v2/v3, hCaptcha, Cloudflare Turnstile)
2. Extracts sitekey from page
3. Sends to CapSolver (AI, ~5s) or 2Captcha (human, ~30s) for solving
4. Injects token into page and continues

Requires CAPSOLVER_API_KEY or TWOCAPTCHA_API_KEY env var. Without keys, CAPTCHAs trigger tier escalation as before.

CloakBrowser

Tier 2 prefers CloakBrowser (C++ patched Chromium with 26 source-level patches) over Patchright. Falls back to Patchright if CloakBrowser is not installed. Set CLOAKBROWSER_ENABLED=0 to force Patchright.

Per-Domain Rate Limiting

Browser tiers (2+) enforce per-domain rate limits via sliding window (60s). Default: 8 req/min. Sensitive sites have lower limits (LinkedIn: 4, Instagram: 4, Facebook: 5). Disable with --no-rate-limit.

Sensitive Site Mode

Sites like LinkedIn, X/Twitter, Facebook, Instagram, TikTok are auto-detected as sensitive:

• Minimum tier 2 (browser) in auto mode
• Fingerprint rotation locked (no rotation on block)
• Behavior intensity boosted (minimum 1.3x)
• Rate limits enforced at lower thresholds

Tier History

Successful tier usage is tracked per domain. On repeat visits, auto mode starts at the lowest known-good tier (requires 3+ successes with 80%+ success ratio). This skips unnecessary lower-tier attempts.

Fingerprint Persistence

Consistent browser fingerprints per domain:

• Same browser/version for each domain (not randomized)
• Fingerprints stored in SQLite, linked to sessions
• Auto-rotation after blocks or 30+ days
• Market-share weighted browser selection by geo

LLM Routing (--extract)

The default fallback chain reflects the developer's setup. Models and providers are user preference — modify scripts/extraction/ai_router.py to wire in your own. Any OpenAI-compatible API works for Tier 1.

Recommended local model: GLM-4.7-Flash-UD Q4 via vLLM/llama.cpp, or any instruction-following model with JSON output (Qwen 2.5, Llama 3.1, Mistral, etc.).

1. Local LLM (any OpenAI-compatible API) → configure via LOCAL_LLM_URL
   ↓ unavailable
2. z.ai GLM-4.5-Air → configure via ZAI_API_KEY
   ↓ rate limited
3. Claude Haiku → configure via ANTHROPIC_API_KEY

Error Handling

| Error | Auto-Handled | Action | |-------|--------------|--------| | Blocked | Yes | Rotates proxy, escalates tier | | CaptchaRequired | Yes | Solves via CapSolver/2Captcha, then escalates | | PaywallDetected | No | Exit 2 → Use premium proxy/unlocker service | | NotFound | No | Exit 1 → Page doesn't exist (verified via HEAD request) | | RateLimited | Yes | Waits 5s, rotates proxy, retries | | Timeout | Yes | Retries once, then escalates | | LoginRequired | Informational | Not actionable |

404 Detection

Pages that don't exist are detected via:

1. HTTP 404 status code (99% confidence)
2. Content patterns ("page not found", "404", etc.) + HEAD request verification

When detected:

• Marked as failed with error_type="NotFound"
• Does NOT trigger fallback (page genuinely doesn't exist)
• Skipped in batch processing (won't waste retries)

Exit Codes

| Code | Meaning | Action | |------|---------|--------| | 0 | Success | Parse output | | 1 | Failed (all tiers exhausted or NotFound) | Report failure | | 2 | Premium proxy required | Use unlocker service | | 130 | Interrupted | User cancelled |

Output Formats

markdown (default)

Clean markdown converted from HTML via html2text. Best for reading/summarizing.

json

{
  "success": true,
  "url": "...",
  "final_url": "...",
  "tier_used": 1,
  "data": { ... },
  "static_data": { ... },
  "markdown": "...",
  "metadata": { ... }
}

raw

Raw HTML. Use when processing HTML directly.

Examples

E-commerce Product

python scripts/scrape.py \
  "https://amazon.com/dp/B0ABC123" \
  -m stealth -g us \
  -e "Extract product name, price, rating, review count" -j

News Article (Potential Paywall)

python scripts/scrape.py \
  "https://nytimes.com/article" \
  -e "Extract title, author, date, full text" -j

If exit code 2: use premium proxy/unlocker service.

Recipe (Structured Data)

python scripts/scrape.py \
  "https://recipe-site.com/cake" \
  -m static -j

Returns JSON-LD recipe schema if available.

Visual Extraction (Screenshot + Vision LLM)

For heavily protected pages or canvas-rendered content:

python scripts/scrape.py \
  "https://protected-site.com/data" \
  --visual -e "Extract all prices and product names visible on the page" -j

Visual extraction:

• Takes full-page screenshot using Tier 3 (Camoufox)
• Extracts data using Vision LLM
• Bypasses DOM monitoring/mutation detection
• Works on canvas-rendered or heavily obfuscated pages

Multi-page Session

# Login and save session
python scripts/scrape.py \
  "https://site.com/login" --session acct -m stealth \
  --actions '[{"type":"fill","selector":"#email","text":"user@example.com"},{"type":"fill","selector":"#pass","text":"xxx"},{"type":"click","selector":"#submit"}]'

# Reuse session
python scripts/scrape.py \
  "https://site.com/dashboard" --session acct -e "Extract user data" -j

Batch Processing

# From command line args
python scripts/scrape.py \
  URL1 URL2 URL3 -p 10 --batch-output results.jsonl

# From file (JSONL or plain text)
python scripts/scrape.py \
  --batch urls.jsonl -p 10 --batch-output results.jsonl

# Write each result to separate markdown file (filename from URL slug)
python scripts/scrape.py \
  --batch urls.txt --output-dir ./articles/ -p 10

# Write as JSON files instead
python scripts/scrape.py \
  --batch urls.txt --output-dir ./data/ --output-ext json -o json

Streaming Output (Long-Running Jobs)

# Stream results as they complete + checkpoint for resume
python scripts/scrape.py \
  --batch urls.jsonl \
  --output-stream results.jsonl \
  --checkpoint job1.json \
  -p 10 -v

# Resume interrupted job (skips already-processed URLs)
python scripts/scrape.py \
  --batch urls.jsonl \
  --output-stream results.jsonl \
  --checkpoint job1.json \
  -p 10

Cross-tool Workflow (agent-browser state import)

# Login with agent-browser (or any tool that exports state)
# ... login flow ...

# Scrape with imported state
python scripts/scrape.py \
  "https://site.com/dashboard" \
  --session imported --import-state ~/auth.json \
  -e "Extract user data" -j

Cache

• Location: ~/.cache/ultimate-scraper/cache.db
• TTL: 24 hours
• Bypass: --no-cache
• Key includes: URL + mode + extract_prompt (different prompts = different entries)

Dependencies

Core (all tiers):

• Python 3.8+
• httpx (pip install httpx) — async HTTP client
• beautifulsoup4 (pip install beautifulsoup4) — HTML parsing
• lxml (pip install lxml) — fast XML/HTML parser
• html2text (pip install html2text) — HTML→Markdown conversion
• pyyaml (pip install pyyaml) — YAML config loading
• python-dotenv (pip install python-dotenv) — .env file loading

Tier 0 — Static extraction:

• chompjs (pip install chompjs) — JavaScript object→Python dict parsing
• extruct (pip install extruct) — JSON-LD, Microdata, OpenGraph extraction

Tier 1 — HTTP with TLS spoofing:

• curl_cffi (pip install curl_cffi) — HTTP client with browser TLS fingerprint impersonation

Tier 2 — Stealth browser:

• scrapling (pip install scrapling) — Patchright-based stealth browser automation
• cloakbrowser (pip install cloakbrowser) — 26 C++ source-level Chromium patches (preferred over Patchright). Binary auto-downloads ~200MB on first use. Set CLOAKBROWSER_ENABLED=0 to force Patchright fallback.

Tier 3 — Anti-detect browser:

• camoufox (pip install camoufox[geoip] && python -m camoufox fetch) — C++ anti-detect Firefox with hardware-backed fingerprinting. First run downloads ~780MB browser package.

Tier 4 — AI extraction:

• crawl4ai (pip install crawl4ai) — AI-powered web crawling with LLM integration

Test dependencies:

• pytest, pytest-asyncio, pytest-cov, scipy, httpx

Install order:

# 1. Core
pip install httpx beautifulsoup4 lxml html2text pyyaml python-dotenv

# 2. Static extraction (Tier 0)
pip install chompjs extruct

# 3. HTTP tier (Tier 1)
pip install curl_cffi

# 4. Browser tiers (Tier 2-3)
pip install scrapling cloakbrowser
pip install 'camoufox[geoip]' && python -m camoufox fetch

# 5. AI tier (Tier 4)
pip install crawl4ai

# 6. Test dependencies (optional)
pip install pytest pytest-asyncio pytest-cov scipy

WSL2 Known Issues

| Issue | Tier | Symptom | Workaround | |-------|------|---------|------------| | Camoufox Turnstile failure | 3 | Cloudflare Turnstile never solves (90s poll, zero captures) | Run on native Linux or VM via SSH | | Virtual GPU fingerprinting | 3 | WSL2's synthetic GPU produces fingerprints Turnstile detects as non-human | Native Linux VM passes; WSL2 does not | | CloakBrowser display | 2 | Headed mode may fail without X server | Install VcXsrv or use headless mode |

Tier 3 on WSL2 is unreliable for Turnstile-protected sites. WSL2's virtual GPU produces inconsistent canvas, WebGL, and audio fingerprints. Tiers 0-2 and 4-5 work normally on WSL2.

Environment Variables

See .env.example for all configurable options. Key variables:

# Proxy (bring your own)
PROXY_HOST=                    # Proxy hostname
PROXY_PORT=                    # Proxy port
PROXY_USERNAME=                # Auth username
PROXY_PASSWORD=                # Auth password

# AI extraction (optional)
LOCAL_LLM_URL=                 # OpenAI-compatible endpoint
LOCAL_LLM_ENABLED=false        # Enable local LLM
ZAI_API_KEY=                   # z.ai API key
ANTHROPIC_API_KEY=             # Anthropic API key

# CAPTCHA solving (optional)
CAPSOLVER_API_KEY=             # AI-based solver (faster)
TWOCAPTCHA_API_KEY=            # Human workers fallback

# CloakBrowser — auto|1|0
CLOAKBROWSER_ENABLED=auto

# WebMCP — auto|1|0
WEBMCP_ENABLED=auto
CHROME_CHANNEL=                # c

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [yoloshii](https://github.com/yoloshii)
- **Source:** [yoloshii/ultimate-scraper-skill](https://github.com/yoloshii/ultimate-scraper-skill)
- **License:** MIT

Install and usage instructions live in the source repository linked above.