# Grabber Development

> >

- **Type:** Skill
- **Install:** `agentstack add skill-acaprino-claude-code-daodan-grabber-development`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [acaprino](https://agentstack.voostack.com/s/acaprino)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [acaprino](https://github.com/acaprino)
- **Source:** https://github.com/acaprino/claude-code-daodan/tree/master/plugins/grabber-development/skills/grabber-development

## Install

```sh
agentstack add skill-acaprino-claude-code-daodan-grabber-development
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Python Web Scraping

Knowledge base for building production-grade Python web scraping systems. Covers the full stack from target assessment through production observability.

## First Tool Call on Every Scraping Task

This section overrides everything else in this skill if there is any conflict. Read it first, act on it first.

When this skill activates on a scraping task, **your next non-question tool call MUST launch a visible browser with the capture surface attached**. Not `Write pyproject.toml`. Not `Write models.py`. Not "let me sketch the architecture first". Browser first, then code.

The default path is **user-driven navigation with live capture**, not Claude-clicks. The user knows their data and their portal better than you do, and authenticated SaaS sites need them anyway. Steps:

1. Ask the bare minimum to start: target URL, what data is wanted, authenticated yes/no. One short batch of questions, then stop asking.
2. Immediately invoke `playwright-skill` (preferred) or write an inline Patchright script via `Bash`. The script must run with `headless=False`, attach every handler in the Capture Surface below, and park on `input()` waiting for the user.
3. Tell the user verbatim: "Browser is open with full network capture (XHR + fetch + WebSocket + SSE + workers + cookies + main-frame navigations). Log in, navigate to the data, apply the filters you'd use day-to-day, then press Enter here so I can dump the capture and reason from real endpoints."
4. While the user navigates, you watch the capture stream. When they press Enter you have: real URLs, real endpoint paths, real field names, real WebSocket frames, real auth cookies. *Now* you can scaffold.

The Claude-drives variant is fine only when there is no login, no 2FA, and no UI-knowledge gap. Same launch, same capture handlers; you call `page.goto` / `page.click` yourself instead of parking on `input()`.

Writing project files (`pyproject.toml`, `src//...`, `models.py`) before the capture is in your hands is the failure mode this section exists to prevent. If you catch yourself drafting field-name `alias` tuples from "common patterns" (Italian + English, REST conventions, framework defaults), stop and launch the browser instead.

The full capture surface, output checklist, and anti-patterns are in the **Discovery Gate** section below. Read those too. But the imperative is here: **browser before code, every time, user navigates by default.**

## When to Use

- Assessing a target site's protection level and choosing the right tools
- Discovering API endpoints via network traffic interception (Playwright/Patchright)
- Extracting data from rendered pages when no API is available
- Bypassing anti-bot systems (Cloudflare, DataDome, PerimeterX)
- Configuring TLS/HTTP fingerprint impersonation (curl_cffi, primp)
- Setting up stealth browser automation (Patchright, Camoufox, Nodriver)
- Designing proxy architecture with tiered escalation
- Solving CAPTCHAs programmatically (CapSolver, 2Captcha, playwright-captcha)
- Building production scraping pipelines (Scrapy, Crawlee)
- Adding rate limiting and observability to scraping systems

## Discovery Gate (READ BEFORE WRITING ANY CODE)

**Phase 1 (Target Assessment) and Phase 2 (Data Discovery) are blocking gates, not optional steps.** You MUST execute them yourself and have their concrete outputs in hand before scaffolding any project file (`pyproject.toml`, modules, models, CLI). No exceptions.

**You always control the browser session and the capture.** The deliverable of discovery is not a script you hand over; it is a live capture you watched. Always launch the browser yourself (via `playwright-skill` or inline Patchright) with `headless=False` and the full capture surface attached, and keep the session open inside your turn.

**Who clicks depends on the task. The capture is yours either way:**

- **Claude clicks** when the target is reachable without the user: no login, no 2FA, you know the UI, you can guess the right filters.
- **User clicks, you watch in-loop** when the target needs the user: login, 2FA / OTP, navigation choices only the user can make, or the user wants to drive the filters / dates / screens visited. This is the gold-standard path for authenticated SaaS portals because the user knows exactly which screens hold the target data. Launch the browser visibly, park on an `input()` checkpoint, let the user navigate while the network capture streams live, then dump the capture when they signal "done".
- The actual anti-pattern is writing `scripts/discover.py` and telling the user "run this and paste the output back". That breaks the loop: by the time the user runs it, you have no eyes on the session and no chance to ask "wait, click that filter again, I lost the payload".

**Capture surface (attach all of these from page launch):**

- `page.on("request")` / `page.on("response")` for XHR + fetch (URL, method, status, headers, cookies, request body, response body when JSON or text)
- `page.on("websocket")` then `ws.on("framesent")` / `ws.on("framereceived")` for WebSocket traffic in both directions
- Response content-type sniff for `text/event-stream` (SSE) and chunked transfer
- `page.on("worker")` for service-worker- and dedicated-worker-initiated requests
- GraphQL detection: URL ends in `/graphql`, request body has `operationName` / `variables` / `extensions.persistedQuery.sha256Hash`
- `context.cookies()` after login, plus any anti-bot cookies (`cf_clearance`, `__cf_bm`, `datadome`, `_px3`, `ak_bmsc`, `incap_ses`)
- `page.on("framenavigated")` filtered to the main frame, to record every landing URL after redirects

Redact `Authorization`, `Cookie`, and password fields in anything saved to disk. Keep them in the in-memory capture you reason from.

**Discovery outputs you MUST collect before scaffolding** (treat as a checklist; if any item is still a guess, you have not finished discovery):
- Real URL of every page that holds target data (not assumed paths, not `/#/...` guesses)
- Real XHR/fetch endpoint URLs, methods, status codes, request headers, cookies
- Real field names and shapes from at least one captured JSON response (paste a redacted sample into the design notes)
- For any WebSocket the page opens: handshake URL, subprotocol, first frames in each direction (auth + subscribe), recurring message schema
- For any SSE / EventSource stream: endpoint URL, event types, payload shape
- For GraphQL: exact `operationName` and `variables`, persisted-query SHA if present
- For service-worker / dedicated-worker requests: the worker URL and the requests it issues
- Anti-bot fingerprint check: `cf_clearance`, `__cf_bm`, `datadome`, `_px3`, `ak_bmsc`, `incap_ses` present or absent
- SPA framework / DOM structure of any page where API discovery fails and a DOM fallback is needed

If any of those is still a guess, you have not finished discovery; do not proceed to scaffolding.

### Anti-Patterns (do not do these)

- Scaffolding `pyproject.toml` and module skeleton before observing one real network request from the target
- Inferring endpoint URLs from "common patterns" (e.g. `/api/invoices`, `/#/fatture-ricevute`) without observation
- Building a regex / filter list of "common field names" (e.g. `(fatture|invoice|received|ricevute|passive)`) as a substitute for the real endpoint name
- Writing a Pydantic model with `Field(alias=...)` tuples of "Italian + English likely names" instead of the names actually returned by the API
- Handing the user a `discover.py` script as the *first* discovery step when you could open the browser yourself
- Marking Phase 1 / Phase 2 as "skipped, will refine later" and proceeding to write code anyway

## Core Workflow

For every scraping task, follow this sequence (the Discovery Gate above governs steps 1 and 2):

### 1. Target Assessment (YOU execute this)
- Load the target URL in a stealth browser (via `playwright-skill` or inline Patchright)
- Identify: static HTML vs JS-rendered, anti-bot service (check for cf_clearance, DataDome cookies, px cookies), data volume needed, update frequency
- Capture: real landing URL after any redirects, SPA framework if any, presence/absence of anti-bot cookies

### 2. Data Discovery (API-first, YOU execute this)
- Navigate the site with Playwright/Patchright **yourself**
- Intercept all network traffic via `page.on("request")` and `page.on("response")`
- Filter for XHR/fetch requests returning JSON, GraphQL, or structured data
- Classify: REST endpoints, WebSocket streams, GraphQL queries, SSE feeds
- Record (do not guess): URLs, methods, headers, cookies, response shapes, real field names
- If structured API found: generate curl_cffi replay code (skip browser entirely)
- If the page is gated by credentials you have and 2FA you can prompt for, log in via the browser session and continue discovery in the authenticated area

### 3. DOM Fallback (only if no API)
- If data exists only in rendered HTML: extract via CSS/XPath selectors
- Check for JSON-LD, microdata, or inline `` JSON before parsing DOM
- For unstable DOM: use LLM-based extraction (Crawl4AI, ScrapeGraphAI) as final fallback
- Hybrid pattern: CSS selectors for stable fields ($0/extraction) + LLM for unstable fields (~$0.01/repair)

### 4. Stealth & Evasion (layer minimally)
- No protection: plain curl_cffi with `impersonate="chrome"` -- done
- Basic Cloudflare: Patchright persistent context + real Chrome channel
- Heavy Cloudflare/Turnstile: stealth browser + CAPTCHA solver + residential proxy
- DataDome: Camoufox + ghost-cursor behavioral simulation + residential proxy
- PerimeterX: organic navigation pattern + session warming + residential proxy

### 5. Production Hardening
- Rate limiting: pyrate-limiter v4 with Redis backend for distributed scraping
- Proxy rotation: tiered escalation (datacenter -> ISP -> residential -> mobile)
- Observability: structlog + Prometheus + Grafana
- Error handling: retry with backoff, proxy failover, session rotation

## Tool Selection Quick Reference

| Target Profile | HTTP Client | Browser | Framework |
|---------------|-------------|---------|-----------|
| No JS, no protection | curl_cffi | none | Scrapy / httpx |
| JS-rendered, no protection | none | Playwright | Crawlee |
| Basic Cloudflare | curl_cffi + cf_clearance | Patchright (for cookie) | Scrapy |
| Heavy Cloudflare | none | Patchright persistent | Crawlee |
| DataDome | none | Camoufox + ghost-cursor | custom |
| PerimeterX | none | Nodriver / Patchright | custom |
| AI extraction needed | none | Crawl4AI / Firecrawl | standalone |

## Proxy Tier Quick Reference

| Tier | Type | Price Range | Use When |
|------|------|-------------|----------|
| 0 | No proxy | free | Unprotected targets, development |
| 1 | Datacenter | $0.10-0.50/GB | Light protection, high volume |
| 2 | ISP (static residential) | $0.53-1.47/IP | Account management, login flows |
| 3 | Residential | $0.49-8.00/GB | Anti-bot bypass, geo-targeting |
| 4 | Mobile | $4-13/GB | Highest trust, last resort |

## Reference Materials

- `field-guide.md` -- full 2025-2026 Python web scraping field guide covering browser stealth, TLS fingerprinting, behavioral biometrics, anti-bot bypass, CAPTCHA solving, proxy landscape, frameworks, AI-assisted scraping, GraphQL reverse engineering, rate limiting, and observability

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [acaprino](https://github.com/acaprino)
- **Source:** [acaprino/claude-code-daodan](https://github.com/acaprino/claude-code-daodan)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-acaprino-claude-code-daodan-grabber-development
- Seller: https://agentstack.voostack.com/s/acaprino
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.