Intelligent Web Scraper

Intelligent Web Scraper Agent

A self-aware, self-learning web scraping agent that accumulates experience over time.

Prerequisites

> IMPORTANT: Before using this skill, ensure these dependencies are installed.

One-Click Setup (Recommended)

# Navigate to skill directory and run setup
cd ~/.claude/skills/intelligent-web-scraper
chmod +x setup.sh && ./setup.sh

# Or for VPS/headless environment:
./setup.sh --headless

The setup script will:

• Detect your environment (macOS/Linux, GUI/headless)
• Install all Python dependencies
• Install Playwright browsers
• Set up Crawl4AI
• Create necessary directories

Check Dependencies

# Check if everything is installed correctly
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py

# Auto-install missing dependencies
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --install

# Output as JSON (for programmatic use)
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --json

Manual Setup (Alternative)

If you prefer manual installation:

# 1. Install Python dependencies
pip install -r ~/.claude/skills/intelligent-web-scraper/requirements.txt

# 2. Install Playwright browser
python -m playwright install chromium

# 3. Set up Crawl4AI
crawl4ai-setup

# 4. For VPS/headless Linux, also run:
python -m playwright install-deps chromium

Required Tools

| Tool | Purpose | Installation | |------|---------|--------------| | Playwright MCP | Browser automation | Must be configured in Claude Code MCP settings | | Python 3.9+ | Script execution | brew install python or system package manager | | Crawl4AI | Advanced crawling | Included in setup.sh |

Verify Installation

# Quick verification
python -c "from crawl4ai import AsyncWebCrawler; print('Crawl4AI OK')"
python -c "from playwright.sync_api import sync_playwright; print('Playwright OK')"

# Full check
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py

VPS/Headless Deployment

For servers without GUI (VPS, Docker, CI):

# 1. Run setup in headless mode
./setup.sh --headless

# 2. The scraper will automatically use headless browser
# No GUI required - all scraping works via headless Chromium

# 3. Check configuration
python3 ~/.claude/skills/intelligent-web-scraper/scripts/config.py

Note: local_browser_scraper.py (CDP mode) requires a GUI and is not available in headless environments. Use crawl4ai_wrapper.py instead.

---

Local Browser Scraping (CDP)

Use this when the user wants to scrape using their local browser (Comet, Chrome, etc.)

This allows scraping while preserving user's login sessions and cookies!

Quick Start

# 1. Install websockets (one time)
pip install websockets

# 2. Launch Comet/Chrome with debugging (or use the script)
python ~/.claude/skills/intelligent-web-scraper/scripts/local_browser_scraper.py \
    --launch comet \
    --url "https://example.com"

# 3. Scrape data from current page
python ~/.claude/skills/intelligent-web-scraper/scripts/local_browser_scraper.py \
    --extract articles \
    --output data.json

Using localbrowserscraper.py

Launch browser with debugging:

python scripts/local_browser_scraper.py --launch comet --url "https://douban.com"
python scripts/local_browser_scraper.py --launch chrome --url "https://example.com"

Scrape from current tab:

# Get page text
python scripts/local_browser_scraper.py --extract text

# Get all links
python scripts/local_browser_scraper.py --extract links

# Get articles
python scripts/local_browser_scraper.py --extract articles

# Custom JavaScript
python scripts/local_browser_scraper.py --extract "document.querySelectorAll('h1').length"

Scrape specific tab (by URL pattern):

python scripts/local_browser_scraper.py --url-pattern "douban.com" --extract articles

Built-in extractors: text, html, title, links, images, tables, articles, metadata

Manual CDP (Alternative)

CRITICAL: Always use user's EXISTING profile to preserve login sessions!

# Comet - use existing profile
/Applications/Comet.app/Contents/MacOS/Comet \
    --remote-debugging-port=9222 \
    --user-data-dir="$HOME/Library/Application Support/Comet" \
    "https://example.com" &

# Chrome - use existing profile
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
    --remote-debugging-port=9222 \
    --user-data-dir="$HOME/Library/Application Support/Google/Chrome" \
    "https://example.com" &

Browser Profile Locations (macOS)

| Browser | User Data Directory | |---------|---------------------| | Comet | ~/Library/Application Support/Comet | | Chrome | ~/Library/Application Support/Google/Chrome | | Edge | ~/Library/Application Support/Microsoft Edge | | Brave | ~/Library/Application Support/BraveSoftware/Brave-Browser |

NEVER use temp directory like /tmp/browser-debug - this creates empty profile without logins!

Why use existing profile:

• Preserves all login sessions (no re-authentication needed)
• Browser extensions work normally
• User's bookmarks, history available

When to Use Local Browser vs Playwright

| Use Local Browser | Use Playwright | |-------------------|----------------| | User requests specific browser | Automated bulk scraping | | Need user's login session | Don't need authentication | | User wants to see the page | Headless scraping OK | | Site blocks headless browsers | Standard sites |

---

Core Capabilities

1. Intelligent Page Analysis

• Auto-detect page type (product list, article, series, etc.)
• Identify data containers and selectors
• Recognize pagination mechanisms

2. Smart Pagination & Scroll Loading

• Page numbers, Next button, Infinite scroll, Load more, API pagination
• Auto-detect and handle appropriately

CRITICAL: Scroll Loading Requirements

• MUST automatically scroll down the page until all content is loaded
• Many modern websites use lazy loading/infinite scroll, initially showing only partial content
• Scroll strategy:

1. Record current page height
2. Scroll to bottom
3. Wait 1-2 seconds for new content to load
4. Check if page height increased
5. Repeat until height stops changing (3 consecutive times)

• Use browser_evaluate to execute scrolling:

``javascript window.scrollTo(0, document.body.scrollHeight) ``

3. Detail Link Following

CRITICAL: Detail Page Scraping Requirements

• List pages may only show title/summary, full content is on detail pages
• MUST detect and follow detail links to get complete data
• Patterns to identify detail links:
• "View more", "Read more", "Details", "Full article"
• Title itself is a link
• Arrow links like "→"
• Scraping workflow:

1. First scrape all entries from list page
2. Identify detail link for each entry
3. Visit detail pages one by one to extract full content
4. Merge data and save

• Note: Detail page scraping needs appropriate delays (2-5s) to avoid blocking

4. Anti-Blocking

• Adaptive delays (2-60s based on signals)
• User-Agent rotation
• Block detection and recovery

5. Series Discovery

• Find prev/next links
• Detect table of contents
• URL pattern analysis
• Build complete series from single article

6. Self-Learning (NEW)

• Records successful patterns for each domain
• Accumulates lessons learned
• Reuses knowledge on similar sites

---

Self-Learning System

How It Works

┌─────────────────────────────────────────────────────────────┐
│                    SCRAPE REQUEST                            │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 1: Check Experience Database                            │
│ - Read experiences/site_patterns.json                        │
│ - Look for matching domain/URL pattern                       │
│ - If found: Use learned selectors and strategies             │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 2: Execute Scraping                                     │
│ - Use learned patterns OR discover new ones                  │
│ - Apply anti-blocking strategies                             │
│ - Extract data                                               │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 3: Learn & Record                                       │
│ - If successful: Save patterns to site_patterns.json         │
│ - If failed: Record lesson in lessons_learned.md             │
│ - Update success rate and timestamps                         │
└─────────────────────────────────────────────────────────────┘

Experience Database Structure

Location: ~/.claude/skills/intelligent-web-scraper/experiences/

experiences/
├── site_patterns.json    # Learned patterns for each domain
├── lessons_learned.md    # Accumulated wisdom and gotchas
└── README.md             # How the learning system works

Pattern Storage Format

{
  "domain.com": {
    "url_patterns": {
      "/movie/*/comments": {
        "selectors": {
          "container": ".comment-list",
          "item": ".comment-item",
          "fields": {
            "author": ".comment-info a",
            "content": ".comment-content span",
            "rating": ".rating"
          }
        },
        "pagination": {
          "type": "page_number",
          "selector": ".paginator a"
        },
        "scroll_loading": {
          "required": true,
          "max_scrolls": 20,
          "scroll_delay_ms": 1500,
          "notes": "Page uses infinite scroll loading"
        },
        "detail_links": {
          "required": true,
          "selector": "a.read-more, a:has-text('→')",
          "fields_in_detail": ["fullContent", "images", "comments"],
          "notes": "List page only has titles, need to scrape detail pages for full content"
        },
        "anti_block": {
          "min_delay": 3,
          "max_delay": 8,
          "needs_login": false
        },
        "success_count": 15,
        "last_success": "2024-01-15",
        "notes": "Rate limit after 50 requests"
      }
    }
  }
}

---

Execution Guide

When You Receive a Scraping Request:

Phase 1: Check Existing Knowledge

# ALWAYS do this first
experiences_path = "~/.claude/skills/intelligent-web-scraper/experiences/site_patterns.json"

1. Read site_patterns.json
2. Extract domain from target URL
3. Check if matching pattern exists
4. If yes: Report to user "I have experience with this site, using learned patterns"

Phase 2: Analyze Page

# 1. Navigate to URL
browser_navigate(url)

# 2. Wait for load
browser_wait_for(time=3)

# 3. [IMPORTANT] Scroll to load all content
browser_evaluate('''async () => {
  let prevHeight = 0;
  let currentHeight = document.body.scrollHeight;
  let noChangeCount = 0;

  while (noChangeCount  setTimeout(r, 1500));
    currentHeight = document.body.scrollHeight;

    if (currentHeight === prevHeight) {
      noChangeCount++;
    } else {
      noChangeCount = 0;
    }
  }
  return document.querySelectorAll('article, [class*="item"]').length;
}''')

# 4. Get snapshot (DOM structure)
browser_snapshot()

# 5. Take screenshot for visual analysis
browser_take_screenshot()

Analyze and identify:

• Page Type: productlist | articlelist | articleseries | singlepage
• Data Container: Main wrapper selector
• Data Items: Individual item selector
• Fields: Specific data field selectors
• Pagination: Type and selectors
• Detail Links: Whether detail links need to be followed for more data

Phase 3: Confirm with User

## Page Analysis Report

**URL**: [target]
**Domain Experience**: [Yes, used N times / No, first time]

### Structure Detected
- **Page Type**: [type]
- **Items Found**: ~[N] items on current page
- **Pagination**: [type] ([N] pages estimated)

### Extraction Plan
| Field | Selector | Sample |
|-------|----------|--------|
| title | .item-title | "Example Title" |
| price | .price | "$99.00" |

Continue scraping? [Y/n]

Phase 4: Execute Scraping

Apply appropriate strategy based on pagination type:

• Page Numbers: Iterate through URLs
• Next Button: Click and wait loop
• Infinite Scroll: Scroll and detect new content (see scroll code above)
• Load More: Click button until exhausted

Phase 4.5: Detail Page Scraping

When list page content is incomplete, this step is required:

// 1. Extract all entries and their detail links
const entries = await browser_evaluate(() => {
  const items = document.querySelectorAll('article, .item');
  return Array.from(items).map(item => ({
    title: item.querySelector('h2, h3, .title')?.textContent,
    summary: item.querySelector('p, .desc')?.textContent,
    detailUrl: item.querySelector('a[href*="detail"], a[href*="view"], a:has-text("→")')?.href
  }));
});

// 2. Visit detail pages one by one
for (const entry of entries) {
  if (entry.detailUrl) {
    // Open new tab
    browser_tabs({ action: 'new' });
    browser_navigate(entry.detailUrl);
    browser_wait_for({ time: 2 });

    // Extract detail content
    const detail = await browser_evaluate(() => ({
      fullContent: document.querySelector('article, .content, main')?.innerText,
      images: Array.from(document.querySelectorAll('img')).map(i => i.src),
      metadata: { /* ... */ }
    }));

    entry.detail = detail;

    // Close tab, return to list
    browser_tabs({ action: 'close' });
    await sleep(2000); // Polite delay
  }
}

When detail page scraping is needed:

• List only shows title/date, content is on detail page
• Has "View more", "Read full article" links
• Data fields are obviously incomplete

Phase 5: Learn & Save

CRITICAL: After EVERY successful scrape, update the experience database.

# Update site_patterns.json with:
{
  "selectors": discovered_selectors,
  "pagination": detected_pagination,
  "anti_block": {
    "min_delay": actual_delay_used,
    "max_delay": max_delay_used,
    "blocked_at": request_count_when_blocked or null
  },
  "success_count": previous_count + 1,
  "last_success": today,
  "notes": any_special_observations
}

If scraping fails, add entry to lessons_learned.md:

## [Date] - domain.com - [URL Pattern]
**Issue**: [What went wrong]
**Cause**: [Why it happened]
**Solution**: [How to avoid next time]

---

Anti-Blocking Strategy

Delay Escalation Levels

| Level | Delay | Trigger | |-------|-------|---------| | 0 (Normal) | 2-5s | Default | | 1 (Caution) | 5-10s | Single 429/503 | | 2 (Careful) | 10-20s | Repeated warnings | | 3 (Critical) | 30-60s | Multiple blocks | | 4 (Pause) | STOP | Captcha detected |

Block Signal Detection

BLOCK_SIGNALS = [
    # HTTP Status
    (429, "rate_limited"),
    (403, "forbidden"),
    (503, "service_unavailable"),

    # Page Content
    ("captcha", "captcha_required"),
    ("verification code", "captcha_required"),
    ("access denied", "blocked"),
    ("please wait", "rate_limited"),
]

Recovery Strategy

1. Increase delay exponentially
2. Rotate User-Agent
3. Clear cookies if needed
4. Notify user if captcha detected
5. Record in lessons_learned.md

---

Series Discovery Algorithm

Finding Related Articles

1. Prev/Next Links

• Look for: prev, next, previous, following, «, »
• Follow chain in both directions

1. Table of Contents

• Look for

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Yrzhe
• Source: Yrzhe/claude-skills
• License: MIT

Install and usage instructions live in the source repository linked above.