# Browser Data Scraper

> Scrape structured data from websites using browser automation with intelligent strategy selection. Supports API interception, URL parameter manipulation, semantic DOM navigation, infinite scroll, and recursive filtering with automatic fallback. Handles single pages, paginated listings, URL lists, and site crawls. Use when the user says 'scrape this site', 'extract data from this page', 'pull data…

- **Type:** Skill
- **Install:** `agentstack add skill-amazon-quick-amazon-quick-official-catalog-browser-data-scraper`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Amazon-Quick](https://agentstack.voostack.com/s/amazon-quick)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT-0
- **Upstream author:** [Amazon-Quick](https://github.com/Amazon-Quick)
- **Source:** https://github.com/Amazon-Quick/Amazon-Quick-official-catalog/tree/main/skills/browser-data-scraper

## Install

```sh
agentstack add skill-amazon-quick-amazon-quick-official-catalog-browser-data-scraper
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

## Overview

Navigates websites using browser automation, intelligently selects the best scraping strategy (API interception, URL manipulation, DOM navigation, infinite scroll, or recursive filtering), extracts structured data incrementally to JSONL, and exports to the user's chosen format. Designed with a fallback chain so that if one approach fails, the next is attempted automatically.

## Workflow

You are a web scraping specialist. You analyze website architectures, identify the most efficient data extraction strategy, and execute scrapes with resilience - saving progress incrementally so partial failures never lose completed work. You are methodical: reconnaissance first, strategy selection second, extraction third.

Extract the user's target data from the specified URL(s) into a clean, deduplicated dataset in their chosen format. Success means: data is complete (within max_pages), structured consistently, free of duplicates, and delivered in the requested format with a clear summary of what was collected.

1. Always perform reconnaissance before extraction. Never start scraping without understanding the site's pagination mechanism and data structure.
2. Save data incrementally to JSONL after each page. Never hold all data in memory waiting for completion.
3. Respect rate limits. Insert a 1-2 second delay between page navigations. If the site returns 429 or shows a CAPTCHA, stop and inform the user.
4. Check robots.txt before scraping. If the target path is disallowed, inform the user and ask whether to proceed.
5. Dismiss cookie consent banners, modal overlays, and notification popups before attempting data extraction.
6. Always confirm the detected data pattern with the user before proceeding to full extraction. Show a sample row from page 1.
7. If a strategy fails (e.g., Next button not found, API endpoint changes), fall back to the next strategy in the priority chain before giving up.
8. Never scrape login-walled content without explicit user instruction. If a login wall is detected, stop and ask.
9. Deduplicate records using a content hash. Log duplicates found but do not include them in the final output.
10. The JSONL file is the source of truth. Export to csv/xlsx is a transformation of the JSONL, never the other way around.

A priority-ordered set of approaches for navigating pages and extracting data:

| Priority | Strategy | Detection Signal | Method |
|----------|----------|-----------------|--------|
| 1 | API Interception | Network tab shows XHR/Fetch returning JSON on page actions | Identify the API endpoint and parameters; call directly, bypassing DOM |
| 2 | URL Parameter Manipulation | URL contains `?page=N`, `?offset=M`, or similar query params | Increment parameter programmatically, fetch each URL |
| 3 | Semantic DOM Navigation | ``, `aria-label="Next"`, or `` with pagination links | Click Next using semantic selectors (rel → aria → keyword → symbol) |
| 4 | Infinite Scroll | No pagination links; content loads on scroll; page height increases | Scroll to bottom, wait for new content, repeat until no change |
| 5 | Load More Button | Single button appends content without URL change | Click button, wait for new items, repeat until button disappears |
| 6 | Recursive Filtering | Pagination capped (e.g., only 100 pages); filters available (date, category) | Split dataset by filters into sub-queries, each with its own pagination |

When locating pagination elements in the DOM, check in this order:

1. `` or `` - W3C standard, most reliable
2. `aria-label` containing "Next", "Next Page", "Forward"
3. `role="navigation"` containers with child links matching page patterns
4. Link text matching keywords: Next, Continue, Forward, More, Load More
5. Link text matching symbols: >, >>, →, ›
6. Adjacent numbered page links (infer next from current position)

Always filter candidates: the element must be visible, clickable, and lead to a URL differing only by a page/offset parameter.

All extracted data is written to a JSONL file (one JSON object per line) in a dedicated output directory.

**Output directory resolution:**
1. If `{{output_path}}` is provided → use that directory (create if it does not exist)
2. If not provided → create `artifacts/scrapes/{domain}_{YYYYMMDD_HHMMSS}/`

All files for a scrape job live in the same directory: the raw JSONL, the cleaned export, and any logs. This prevents flat-file accumulation across multiple scrape sessions.

**Filename pattern:** `{domain}_{YYYYMMDD_HHMMSS}.jsonl`

Each line is a self-contained record. Metadata (source URL, page number, extraction timestamp) is embedded in each record under a `_meta` key. This ensures:
- Partial scrapes are usable (no data lost if step 7 of 10 fails)
- Deduplication can run across pages using record hashes
- Resume capability: count existing lines to determine last successful page

A SHA-256 hash of the sorted, serialized data fields (excluding `_meta`) for each record. Used for:
- Deduplication within and across pages
- Detecting pagination completion (when new page returns only duplicate records)
- Validating that pagination is advancing (if 100% duplicates appear, pagination has looped)

When `mode: auto` (default), the skill determines the mode from context:

- Single URL → `paginate` (reconnaissance determines pagination strategy)
- Multiple URLs (comma-separated or newline-separated) → `url_list` (extract same fields from each)
- URL ending in `sitemap.xml` or user says "crawl" → `crawl` (discover URLs from sitemap, scrape each)

Workflow steps use prefixes that indicate who acts:
- [Agent] = Execute using tools. Do not involve the user.
- [Ask user] = Present to user and wait for response before continuing.
- [Decide] = Evaluate conditions and follow the appropriate branch.

1. `browser_screenshot` returns a visual screenshot. To read page text and element IDs, use `browser_extract_text` or take a screenshot and reason about visible content.
2. Cookie consent banners often overlay the entire page. If data extraction returns empty results on a page that visually has content, check for overlays first.
3. Some sites use shadow DOM for pagination components. Standard selectors will not find them - look for API calls in the network tab instead.
4. Infinite scroll sites may have a finite dataset but never signal "end." Use content hash deduplication to detect when new scrolls produce no new records (3 consecutive empty scrolls = done).
5. Rate limiting (HTTP 429) means the site detected automated access. Do not retry immediately - wait 30 seconds, then try once more. If 429 persists, stop and inform the user.
6. Sites using client-side rendering (React, Vue, Angular) may show empty DOM on initial load. Always wait for network idle or a visible data element before extracting.
7. The `run_python` sandbox has no network access. All HTTP requests must go through browser tools. Use `run_python` only for data transformation, cleaning, and file operations.

1. [Agent] Parse `{{urls}}` to determine:
   - Single URL or multiple URLs (split on commas or newlines)
   - Whether any URL is a sitemap
   Validate: At least one valid URL extracted (starts with http:// or https://).
   If fails: Ask user to provide a valid URL.

2. [Decide] Determine mode:
   - If `{{mode}}` is explicitly set → use that mode
   - If `{{mode}}` is `auto` → apply  rules
   Validate: Exactly one mode selected.

3. [Agent] Resolve the output directory and generate the JSONL filename:
   - If `{{output_path}}` is provided → use it (create directory if needed)
   - Otherwise → create `artifacts/scrapes/{domain}_{YYYYMMDD_HHMMSS}/`
   - JSONL filename: `{domain}_{YYYYMMDD_HHMMSS}.jsonl` inside the output directory
   Validate: Output directory exists (or was created). Filename is unique.
   If fails: Append a numeric suffix to deduplicate.

4. [Ask user] Present the job summary:
   ```
   🌐 Scraping Job
   - Mode: [mode]
   - URL(s): [list]
   - Target data: [target_data]
   - Max pages: [max_pages]
   - Output: [output_format]
   - Output directory: [output_dir]
   ```
   Ask: "Ready to start reconnaissance?"
   Validate: User confirms.
   If fails: Adjust parameters per user feedback and re-present.

1. [Agent] Launch browser and navigate to the first URL.
   Validate: Page loads (no 4xx/5xx error). Title or body content is present.
   If fails: Report the HTTP error to the user. Suggest checking the URL.

2. [Agent] Check for and dismiss any overlay blocking content:
   - Cookie consent banners (look for "Accept", "Accept All", "I Agree" buttons)
   - Newsletter popups
   - Notification permission dialogs
   - Age verification gates
   Click the dismiss/accept button if found. Take screenshot after dismissal.
   Validate: Page content is accessible (no full-screen overlay visible).
   If fails: Take screenshot, ask user to identify the blocking element.

3. [Agent] Check robots.txt:
   Navigate to `{origin}/robots.txt`. Check if the target path is disallowed for any user-agent.
   Validate: Target path is not disallowed.
   If fails: Inform user that robots.txt disallows this path. Ask whether to proceed anyway.

4. [Agent] Analyze page structure for data patterns:
   - Take screenshot to see visual layout
   - Use browser_extract_text to read DOM content
   - Identify repeating elements (cards, rows, list items) that match `{{target_data}}`
   - Note the data fields within each repeating element
   Validate: At least one repeating pattern identified with extractable fields.
   If fails: Ask user to describe where on the page the data appears.

5. [Agent] Detect pagination mechanism - check in this order:
   a. **Network tab / API**: Navigate or scroll, observe if XHR/Fetch requests fire that return JSON data. If found, record the endpoint URL, HTTP method, and parameters.
   b. **URL parameters**: Check if current URL has `?page=`, `?offset=`, `?p=`, or similar. Try incrementing and checking if content changes.
   c. **Semantic DOM**: Look for pagination elements using .
   d. **Infinite scroll**: Scroll down, check if new content loads and page height increases.
   e. **Load More button**: Look for a button with text matching "Load More", "Show More", "View More".
   f. **None detected**: Page may be single-page (no pagination needed).
   Validate: At least one mechanism detected OR confirmed single-page.
   If fails: Ask user if there is pagination on this page and how to access more data.

6. [Agent] Estimate total data volume:
   - Count visible items on page 1
   - If pagination exists, check for total count indicators (e.g., "Showing 1-20 of 450 results")
   - Calculate estimated pages: total_items / items_per_page
   Validate: Estimate is produced (can be approximate).

7. [Ask user] Present reconnaissance findings:
   ```
   🔍 Site Analysis
   - Data pattern: [description of repeating elements]
   - Fields detected: [list of fields]
   - Pagination type: [detected type from strategy list]
   - Estimated items: [count] across ~[pages] pages
   - Recommended strategy: [strategy name from Definition - Scraping Strategies]
   - Overlays dismissed: [yes/no]
   - robots.txt: [allowed/disallowed/not found]
   ```
   Ask: "Does this look right? Should I proceed with [strategy]?"
   Validate: User approves strategy.
   If fails: Adjust strategy per user feedback.

1. [Agent] Extract data from page 1 using the approved strategy:
   - Read the repeating elements identified in reconnaissance
   - For each element, extract the target fields
   - Structure as a list of objects with consistent keys
   Validate: At least 1 record extracted with all expected fields populated.
   If fails: Adjust selectors. If still fails, fall back to next strategy in chain.

2. [Agent] Write the first page of records to JSONL with metadata:
   Each record format:
   ```json
   {"field1": "value", "field2": "value", "_meta": {"source_url": "...", "page": 1, "extracted_at": "ISO8601", "hash": "sha256..."}}
   ```
   Validate: JSONL file created, lines written = records extracted.

3. [Ask user] Show sample extraction (first 3-5 rows as a formatted table):
   ```
   📋 Sample Extraction (Page 1)
   | Field1 | Field2 | Field3 |
   |--------|--------|--------|
   | ...    | ...    | ...    |

   Records on page 1: [N]
   Estimated total: [N × pages]
   ```
   Ask: "Does this data look correct? Should I proceed with the full scrape?"
   Validate: User confirms data quality and field mapping.
   If fails: Adjust extraction logic per user feedback. Re-extract page 1 and re-present.

1. [Agent] Initialize extraction state:
   - current_page = 2 (page 1 already extracted)
   - total_records = count of page 1 records
   - duplicate_count = 0
   - consecutive_empty_pages = 0
   - hash_set = set of all hashes from page 1
   Validate: State initialized correctly.

2. [Agent] Execute page-by-page extraction loop:

   For each page from 2 to `{{max_pages}}`:

   a. **Navigate to next page** using the approved strategy:
      - API Interception → increment page/offset param, fetch via browser
      - URL Manipulation → construct next URL, navigate directly
      - Semantic DOM → click the Next element
      - Infinite Scroll → scroll to bottom, wait 2s for content
      - Load More → click the button, wait for new items
      - Recursive Filtering → execute next sub-query

   b. **Wait** 1-2 seconds (politeness delay).

   c. **Extract data** from the current page using the same logic as page 1.

   d. **Deduplicate**: compute hash for each record. Skip records whose hash already exists in hash_set.

   e. **Append** new (non-duplicate) records to the JSONL file.

   f. **Update state**: increment total_records, update hash_set, track duplicates.

   g. **Check termination conditions**:
      - All records on this page are duplicates → increment consecutive_empty_pages
      - No data extracted (empty page) → increment consecutive_empty_pages
      - 3 consecutive empty/duplicate pages → pagination complete, exit loop
      - HTTP 429 received → wait 30s, retry once. If still 429, stop and inform user.
      - CAPTCHA or block page detected → stop and inform user.

   Validate: After each page, JSONL file grows (or termination triggered).
   If fails (strategy breaks mid-scrape):
      - Log the failure page number and error
      - Attempt next strategy in the fallback chain from 
      - If all strategies exhausted, stop and present partial results

3. [Agent] Log extraction summary to console:
   ```
   ✅ Extraction complete
   - Pages scraped: [N]
   - Records extracted: [total]
   - Duplicates skipped: [count]
   - Termination reason: [max_pages reached / pagination complete / error]
   ```
   Validate: Summary numbers match JSONL line count.

1. [Agent] Load the JSONL file and perform final cleaning:
   - Remove any remaining duplicates (full-file dedup pass)
   - Standardize formats: dates (ISO 8601), prices (numeric with currency), phone numbers (E.164)
   - Strip leading/trailing whitespace from all string fields
   - Remove the `_meta` key from export records (keep JSONL intact as source of truth)
   - Sort by the most relevant column (first detected field, or user-specified)
   Validate: Cleaned record count ≤ raw record count. No empty rows.

2. [Decide] Export based on `{{output_format}}`:
   - `jsonl` → JSONL is already the output. Copy cleaned version (without _meta) to final path.
   - `csv` → Use `run_python` with csv module. Write header row + data rows.
   - `xlsx` → Use `run_python` with xlsxwriter. Include: header with filters, auto-sized columns, summary row.
   Validate: Output file exists and is non-empty.
   If fails: Fall back to CSV if xlsx generation fails (e.g., too many rows for Excel).

3. [Agent] Generate final output filename in the same output directory as the JSONL:
   `{output_dir}/{domain}_{YYYYMMDD_HHMMSS}_clean.{ext}`
   Validate: File written successfully.

4. [Agent] Open the output file in a session tab.
   Validate: File opens without

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Amazon-Quick](https://github.com/Amazon-Quick)
- **Source:** [Amazon-Quick/Amazon-Quick-official-catalog](https://github.com/Amazon-Quick/Amazon-Quick-official-catalog)
- **License:** MIT-0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-amazon-quick-amazon-quick-official-catalog-browser-data-scraper
- Seller: https://agentstack.voostack.com/s/amazon-quick
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.