# Intelligent Web Scraper

> Self-learning intelligent web scraper agent - automatically analyzes page structure, handles pagination, anti-blocking, and discovers article series. No user configuration needed - AI decides everything.

- **Type:** Skill
- **Install:** `agentstack add skill-yrzhe-claude-skills-intelligent-web-scraper`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Yrzhe](https://agentstack.voostack.com/s/yrzhe)
- **Installs:** 0
- **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Yrzhe](https://github.com/Yrzhe)
- **Source:** https://github.com/Yrzhe/claude-skills/tree/main/plugins/intelligent-web-scraper/skills/intelligent-web-scraper

## Install

```sh
agentstack add skill-yrzhe-claude-skills-intelligent-web-scraper
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Intelligent Web Scraper Agent

A self-aware, self-learning web scraping agent that accumulates experience over time.

## Prerequisites

> **IMPORTANT**: Before using this skill, ensure these dependencies are installed.

### One-Click Setup (Recommended)

```bash
# Navigate to skill directory and run setup
cd ~/.claude/skills/intelligent-web-scraper
chmod +x setup.sh && ./setup.sh

# Or for VPS/headless environment:
./setup.sh --headless
```

The setup script will:
- Detect your environment (macOS/Linux, GUI/headless)
- Install all Python dependencies
- Install Playwright browsers
- Set up Crawl4AI
- Create necessary directories

### Check Dependencies

```bash
# Check if everything is installed correctly
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py

# Auto-install missing dependencies
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --install

# Output as JSON (for programmatic use)
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --json
```

### Manual Setup (Alternative)

If you prefer manual installation:

```bash
# 1. Install Python dependencies
pip install -r ~/.claude/skills/intelligent-web-scraper/requirements.txt

# 2. Install Playwright browser
python -m playwright install chromium

# 3. Set up Crawl4AI
crawl4ai-setup

# 4. For VPS/headless Linux, also run:
python -m playwright install-deps chromium
```

### Required Tools

| Tool | Purpose | Installation |
|------|---------|--------------|
| **Playwright MCP** | Browser automation | Must be configured in Claude Code MCP settings |
| **Python 3.9+** | Script execution | `brew install python` or system package manager |
| **Crawl4AI** | Advanced crawling | Included in setup.sh |

### Verify Installation

```bash
# Quick verification
python -c "from crawl4ai import AsyncWebCrawler; print('Crawl4AI OK')"
python -c "from playwright.sync_api import sync_playwright; print('Playwright OK')"

# Full check
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py
```

### VPS/Headless Deployment

For servers without GUI (VPS, Docker, CI):

```bash
# 1. Run setup in headless mode
./setup.sh --headless

# 2. The scraper will automatically use headless browser
# No GUI required - all scraping works via headless Chromium

# 3. Check configuration
python3 ~/.claude/skills/intelligent-web-scraper/scripts/config.py
```

**Note**: `local_browser_scraper.py` (CDP mode) requires a GUI and is not available in headless environments. Use `crawl4ai_wrapper.py` instead.

---

## Local Browser Scraping (CDP)

**Use this when the user wants to scrape using their local browser (Comet, Chrome, etc.)**

This allows scraping while preserving user's login sessions and cookies!

### Quick Start

```bash
# 1. Install websockets (one time)
pip install websockets

# 2. Launch Comet/Chrome with debugging (or use the script)
python ~/.claude/skills/intelligent-web-scraper/scripts/local_browser_scraper.py \
    --launch comet \
    --url "https://example.com"

# 3. Scrape data from current page
python ~/.claude/skills/intelligent-web-scraper/scripts/local_browser_scraper.py \
    --extract articles \
    --output data.json
```

### Using local_browser_scraper.py

**Launch browser with debugging:**
```bash
python scripts/local_browser_scraper.py --launch comet --url "https://douban.com"
python scripts/local_browser_scraper.py --launch chrome --url "https://example.com"
```

**Scrape from current tab:**
```bash
# Get page text
python scripts/local_browser_scraper.py --extract text

# Get all links
python scripts/local_browser_scraper.py --extract links

# Get articles
python scripts/local_browser_scraper.py --extract articles

# Custom JavaScript
python scripts/local_browser_scraper.py --extract "document.querySelectorAll('h1').length"
```

**Scrape specific tab (by URL pattern):**
```bash
python scripts/local_browser_scraper.py --url-pattern "douban.com" --extract articles
```

**Built-in extractors:** `text`, `html`, `title`, `links`, `images`, `tables`, `articles`, `metadata`

### Manual CDP (Alternative)

**CRITICAL: Always use user's EXISTING profile to preserve login sessions!**

```bash
# Comet - use existing profile
/Applications/Comet.app/Contents/MacOS/Comet \
    --remote-debugging-port=9222 \
    --user-data-dir="$HOME/Library/Application Support/Comet" \
    "https://example.com" &

# Chrome - use existing profile
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
    --remote-debugging-port=9222 \
    --user-data-dir="$HOME/Library/Application Support/Google/Chrome" \
    "https://example.com" &
```

### Browser Profile Locations (macOS)

| Browser | User Data Directory |
|---------|---------------------|
| Comet | `~/Library/Application Support/Comet` |
| Chrome | `~/Library/Application Support/Google/Chrome` |
| Edge | `~/Library/Application Support/Microsoft Edge` |
| Brave | `~/Library/Application Support/BraveSoftware/Brave-Browser` |

**NEVER use temp directory** like `/tmp/browser-debug` - this creates empty profile without logins!

**Why use existing profile:**
- Preserves all login sessions (no re-authentication needed)
- Browser extensions work normally
- User's bookmarks, history available

### When to Use Local Browser vs Playwright

| Use Local Browser | Use Playwright |
|-------------------|----------------|
| User requests specific browser | Automated bulk scraping |
| Need user's login session | Don't need authentication |
| User wants to see the page | Headless scraping OK |
| Site blocks headless browsers | Standard sites |

---

## Core Capabilities

### 1. Intelligent Page Analysis
- Auto-detect page type (product list, article, series, etc.)
- Identify data containers and selectors
- Recognize pagination mechanisms

### 2. Smart Pagination & Scroll Loading
- Page numbers, Next button, Infinite scroll, Load more, API pagination
- Auto-detect and handle appropriately

**CRITICAL: Scroll Loading Requirements**
- **MUST** automatically scroll down the page until all content is loaded
- Many modern websites use lazy loading/infinite scroll, initially showing only partial content
- Scroll strategy:
  1. Record current page height
  2. Scroll to bottom
  3. Wait 1-2 seconds for new content to load
  4. Check if page height increased
  5. Repeat until height stops changing (3 consecutive times)
- Use `browser_evaluate` to execute scrolling:
  ```javascript
  window.scrollTo(0, document.body.scrollHeight)
  ```

### 3. Detail Link Following

**CRITICAL: Detail Page Scraping Requirements**
- List pages may only show title/summary, full content is on detail pages
- **MUST** detect and follow detail links to get complete data
- Patterns to identify detail links:
  - "View more", "Read more", "Details", "Full article"
  - Title itself is a link
  - Arrow links like "→"
- Scraping workflow:
  1. First scrape all entries from list page
  2. Identify detail link for each entry
  3. Visit detail pages one by one to extract full content
  4. Merge data and save
- Note: Detail page scraping needs appropriate delays (2-5s) to avoid blocking

### 4. Anti-Blocking
- Adaptive delays (2-60s based on signals)
- User-Agent rotation
- Block detection and recovery

### 5. Series Discovery
- Find prev/next links
- Detect table of contents
- URL pattern analysis
- Build complete series from single article

### 6. **Self-Learning** (NEW)
- Records successful patterns for each domain
- Accumulates lessons learned
- Reuses knowledge on similar sites

---

## Self-Learning System

### How It Works

```
┌─────────────────────────────────────────────────────────────┐
│                    SCRAPE REQUEST                            │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 1: Check Experience Database                            │
│ - Read experiences/site_patterns.json                        │
│ - Look for matching domain/URL pattern                       │
│ - If found: Use learned selectors and strategies             │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 2: Execute Scraping                                     │
│ - Use learned patterns OR discover new ones                  │
│ - Apply anti-blocking strategies                             │
│ - Extract data                                               │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 3: Learn & Record                                       │
│ - If successful: Save patterns to site_patterns.json         │
│ - If failed: Record lesson in lessons_learned.md             │
│ - Update success rate and timestamps                         │
└─────────────────────────────────────────────────────────────┘
```

### Experience Database Structure

**Location**: `~/.claude/skills/intelligent-web-scraper/experiences/`

```
experiences/
├── site_patterns.json    # Learned patterns for each domain
├── lessons_learned.md    # Accumulated wisdom and gotchas
└── README.md             # How the learning system works
```

### Pattern Storage Format

```json
{
  "domain.com": {
    "url_patterns": {
      "/movie/*/comments": {
        "selectors": {
          "container": ".comment-list",
          "item": ".comment-item",
          "fields": {
            "author": ".comment-info a",
            "content": ".comment-content span",
            "rating": ".rating"
          }
        },
        "pagination": {
          "type": "page_number",
          "selector": ".paginator a"
        },
        "scroll_loading": {
          "required": true,
          "max_scrolls": 20,
          "scroll_delay_ms": 1500,
          "notes": "Page uses infinite scroll loading"
        },
        "detail_links": {
          "required": true,
          "selector": "a.read-more, a:has-text('→')",
          "fields_in_detail": ["fullContent", "images", "comments"],
          "notes": "List page only has titles, need to scrape detail pages for full content"
        },
        "anti_block": {
          "min_delay": 3,
          "max_delay": 8,
          "needs_login": false
        },
        "success_count": 15,
        "last_success": "2024-01-15",
        "notes": "Rate limit after 50 requests"
      }
    }
  }
}
```

---

## Execution Guide

### When You Receive a Scraping Request:

#### Phase 1: Check Existing Knowledge

```python
# ALWAYS do this first
experiences_path = "~/.claude/skills/intelligent-web-scraper/experiences/site_patterns.json"
```

1. Read `site_patterns.json`
2. Extract domain from target URL
3. Check if matching pattern exists
4. If yes: Report to user "I have experience with this site, using learned patterns"

#### Phase 2: Analyze Page

```bash
# 1. Navigate to URL
browser_navigate(url)

# 2. Wait for load
browser_wait_for(time=3)

# 3. [IMPORTANT] Scroll to load all content
browser_evaluate('''async () => {
  let prevHeight = 0;
  let currentHeight = document.body.scrollHeight;
  let noChangeCount = 0;

  while (noChangeCount  setTimeout(r, 1500));
    currentHeight = document.body.scrollHeight;

    if (currentHeight === prevHeight) {
      noChangeCount++;
    } else {
      noChangeCount = 0;
    }
  }
  return document.querySelectorAll('article, [class*="item"]').length;
}''')

# 4. Get snapshot (DOM structure)
browser_snapshot()

# 5. Take screenshot for visual analysis
browser_take_screenshot()
```

Analyze and identify:
- **Page Type**: product_list | article_list | article_series | single_page
- **Data Container**: Main wrapper selector
- **Data Items**: Individual item selector
- **Fields**: Specific data field selectors
- **Pagination**: Type and selectors
- **Detail Links**: Whether detail links need to be followed for more data

#### Phase 3: Confirm with User

```markdown
## Page Analysis Report

**URL**: [target]
**Domain Experience**: [Yes, used N times / No, first time]

### Structure Detected
- **Page Type**: [type]
- **Items Found**: ~[N] items on current page
- **Pagination**: [type] ([N] pages estimated)

### Extraction Plan
| Field | Selector | Sample |
|-------|----------|--------|
| title | .item-title | "Example Title" |
| price | .price | "$99.00" |

Continue scraping? [Y/n]
```

#### Phase 4: Execute Scraping

Apply appropriate strategy based on pagination type:
- **Page Numbers**: Iterate through URLs
- **Next Button**: Click and wait loop
- **Infinite Scroll**: Scroll and detect new content (see scroll code above)
- **Load More**: Click button until exhausted

#### Phase 4.5: Detail Page Scraping

**When list page content is incomplete, this step is required:**

```javascript
// 1. Extract all entries and their detail links
const entries = await browser_evaluate(() => {
  const items = document.querySelectorAll('article, .item');
  return Array.from(items).map(item => ({
    title: item.querySelector('h2, h3, .title')?.textContent,
    summary: item.querySelector('p, .desc')?.textContent,
    detailUrl: item.querySelector('a[href*="detail"], a[href*="view"], a:has-text("→")')?.href
  }));
});

// 2. Visit detail pages one by one
for (const entry of entries) {
  if (entry.detailUrl) {
    // Open new tab
    browser_tabs({ action: 'new' });
    browser_navigate(entry.detailUrl);
    browser_wait_for({ time: 2 });

    // Extract detail content
    const detail = await browser_evaluate(() => ({
      fullContent: document.querySelector('article, .content, main')?.innerText,
      images: Array.from(document.querySelectorAll('img')).map(i => i.src),
      metadata: { /* ... */ }
    }));

    entry.detail = detail;

    // Close tab, return to list
    browser_tabs({ action: 'close' });
    await sleep(2000); // Polite delay
  }
}
```

**When detail page scraping is needed:**
- List only shows title/date, content is on detail page
- Has "View more", "Read full article" links
- Data fields are obviously incomplete

#### Phase 5: Learn & Save

**CRITICAL**: After EVERY successful scrape, update the experience database.

```python
# Update site_patterns.json with:
{
  "selectors": discovered_selectors,
  "pagination": detected_pagination,
  "anti_block": {
    "min_delay": actual_delay_used,
    "max_delay": max_delay_used,
    "blocked_at": request_count_when_blocked or null
  },
  "success_count": previous_count + 1,
  "last_success": today,
  "notes": any_special_observations
}
```

**If scraping fails**, add entry to `lessons_learned.md`:
```markdown
## [Date] - domain.com - [URL Pattern]
**Issue**: [What went wrong]
**Cause**: [Why it happened]
**Solution**: [How to avoid next time]
```

---

## Anti-Blocking Strategy

### Delay Escalation Levels

| Level | Delay | Trigger |
|-------|-------|---------|
| 0 (Normal) | 2-5s | Default |
| 1 (Caution) | 5-10s | Single 429/503 |
| 2 (Careful) | 10-20s | Repeated warnings |
| 3 (Critical) | 30-60s | Multiple blocks |
| 4 (Pause) | STOP | Captcha detected |

### Block Signal Detection

```python
BLOCK_SIGNALS = [
    # HTTP Status
    (429, "rate_limited"),
    (403, "forbidden"),
    (503, "service_unavailable"),

    # Page Content
    ("captcha", "captcha_required"),
    ("verification code", "captcha_required"),
    ("access denied", "blocked"),
    ("please wait", "rate_limited"),
]
```

### Recovery Strategy

1. **Increase delay** exponentially
2. **Rotate User-Agent**
3. **Clear cookies** if needed
4. **Notify user** if captcha detected
5. **Record** in lessons_learned.md

---

## Series Discovery Algorithm

### Finding Related Articles

1. **Prev/Next Links**
   - Look for: prev, next, previous, following, «, »
   - Follow chain in both directions

2. **Table of Contents**
   - Look for

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Yrzhe](https://github.com/Yrzhe)
- **Source:** [Yrzhe/claude-skills](https://github.com/Yrzhe/claude-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-yrzhe-claude-skills-intelligent-web-scraper
- Seller: https://agentstack.voostack.com/s/yrzhe
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.