Web Scraping

Web Scraping (Firecrawl-first)

Goal

Extract LLM-ready web content without hacking around: clean markdown, structured JSON, anti-bot and JS-rendering handled. Firecrawl is the reference wrapper; fallback to Playwright or curl + html2text if unavailable.

When to trigger this skill

• "scrape this page / this site"
• "extract data from ..."
• "crawl site X"
• "fetch all articles from ..."
• "search the web and extract the content"
• "parse this dynamic page" (site with JS-rendering)
• "bypass the paywall / anti-bot" (legitimate use only)

When NOT to use this skill

• Quick web search without structured extraction -> WebSearch is enough
• A single static URL, simple page -> WebFetch is enough
• Visual test / browser interaction -> skill qa-chrome or agent-browser
• Form / login automation -> agent-browser or Playwright directly

Prerequisites

Option 1: Firecrawl cloud (recommended)

export FIRECRAWL_API_KEY="fc-xxx"      # https://firecrawl.dev
npm install -g firecrawl               # or pip install firecrawl-py

Option 2: Firecrawl self-hosted

Docker compose available on github.com/mendableai/firecrawl. Useful if data is sensitive or budget is limited.

Option 3: Fallback without Firecrawl

If Firecrawl is missing, degrade gracefully:

| Need | Fallback | Limitation | |--------|----------|------------| | Simple static page | curl -sL URL \| pandoc -f html -t markdown | No JS rendering | | JS-heavy page | npx playwright + page.content() + markdownify | Heavy, 300MB+ of deps | | Whole site | recursive filtered wget | No deduplication, no LLM-ready output |

IMPORTANT: always announce when degrading. The user must know if the content is partial (JS not rendered).

The 5 Firecrawl operations

1. Scrape (one URL)

firecrawl scrape https://example.com/article \
  --formats markdown,links \
  --only-main-content

Output: clean markdown (navigation / footers stripped), list of links, OG metadata.

2. Crawl (whole site)

firecrawl crawl https://docs.example.com \
  --limit 100 \
  --include-paths "/docs/**" \
  --exclude-paths "/docs/legacy/**" \
  --formats markdown

Output: one markdown per page + JSON manifest. Ask for confirmation before crawl > 50 pages (API costs + time).

3. Map (URL discovery)

firecrawl map https://example.com --search "pricing"

Output: list of relevant URLs. Useful BEFORE a crawl to target the right sections.

4. Extract (structured data via LLM)

firecrawl extract https://example.com/pricing \
  --prompt "Extract plans with name, price, features" \
  --schema '{"plans":[{"name":"str","price":"num","features":["str"]}]}'

Output: JSON conforming to the schema. Saves hours of fragile CSS selectors.

5. Search (search + extract in one pass)

firecrawl search "best pve proxmox backup strategies" \
  --limit 10 \
  --scrape-options '{"formats":["markdown"]}'

Output: top N results with extracted content. Replaces WebSearch + N WebFetch.

Recommended workflow

1. IDENTIFY the need
   - 1 page               -> scrape
   - N known pages        -> scrape in a loop with `xargs -P 4`
   - Whole site           -> map (recon) -> targeted crawl
   - Structured data      -> extract with schema
   - Search + extract     -> search

2. ESTIMATE costs
   - Firecrawl cloud: credits per page scraped
   - Ask for confirmation if > 50 pages or > 10 MB expected

3. RUN with limits on the first attempt
   - --limit 5 to test
   - Inspect the output
   - Re-run at full volume if OK

4. SAVE the result
   - `./scraped//.md` by convention
   - Commit if data is reusable (mind copyright)

5. CHECK legality / ethics
   - Respect robots.txt unless explicitly authorized
   - No personal data without consent (GDPR)
   - No commercial paywall bypass

Concrete examples

Extract a lib's docs for RAG

firecrawl crawl https://docs.terraform.io/language \
  --limit 200 --formats markdown \
  --output-dir ./rag-corpus/terraform

Compare pricings of 5 competitors

for url in url1 url2 url3 url4 url5; do
    firecrawl extract "$url" \
      --prompt "Extract pricing plans" \
      --schema pricing.schema.json >> pricing-compared.jsonl
done

Monitor a changelog

firecrawl scrape https://example.com/changelog \
  --formats markdown \
  | diff - last-changelog.md \
  && mv  `dev:dev-rag` | Build a corpus for RAG ingestion |
| `web-scraping` -> `biz:biz-competitor` | Factual competitive & market analysis on real data |
| `web-scraping` + `writing-skills` | Import third-party lib docs into a local skill |
| `qa-chrome` instead of `web-scraping` | Visual tests, DOM interaction, screenshots |

## Anti-patterns

- NEVER scrape without checking robots.txt AND Terms of Service
- NEVER commit scraped data without checking the rights
- NEVER launch a crawl > 50 pages without user confirmation
- NEVER use Firecrawl to replace `WebSearch` for a simple factual question (needlessly expensive)
- NEVER bruteforce a site in massive parallel (max 4 workers by default)

## Absolute rules

IMPORTANT: Always announce when degrading to a fallback (Playwright / curl) — the content may be partial.

IMPORTANT: Ask for confirmation before any crawl exceeding 50 pages or a site outside the user's control.

YOU MUST respect robots.txt and the target site's ToS.

YOU MUST save outputs in `./scraped//` with timestamp for traceability.

NEVER bypass an anti-bot system without documented legitimate justification.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [christopherlouet](https://github.com/christopherlouet)
- **Source:** [christopherlouet/claude-base](https://github.com/christopherlouet/claude-base)
- **License:** MIT
- **Homepage:** https://christopherlouet.github.io/claude-base/

Install and usage instructions live in the source repository linked above.