Batch

> Latest version: if you have web access, fetch the newest copy of this skill at https://www.olostep.com/skills/batch/SKILL.md — it may include capabilities added since this copy was installed. Otherwise the instructions below are complete and current.

Olostep Batch Scrape

Scrape up to 10,000 URLs in parallel. All pages are scraped concurrently with full browser rendering, anti-bot bypass, and residential proxies — no rate limiting, no blocking, no setup.

Why this matters

Scraping 50 pages sequentially takes minutes and often fails halfway through (rate limits, CAPTCHAs, IP bans). Olostep batch processes them all in parallel through its browser infrastructure — you get every page back, fully rendered, in seconds.

When to use

• User has 3+ URLs they want content from (don't scrape them one-by-one)
• User has a list from /map and wants to pull content from matching pages
• User wants to compare multiple competitor pages, product pages, or pricing pages side-by-side
• User needs to build a dataset from a list of known URLs
• User wants to audit, analyse, or extract patterns across many pages

Workflow

1. Collect URLs from the user (or from a previous /map result).
2. Build the array: [{"url": "...", "custom_id": "competitor-1"}, ...].
3. Call batch_scrape_urls with the array.
4. Use custom_id on each URL so you can match results back to the source.
5. Default to markdown. Use json + parser for structured product data.
6. After batch completes, synthesise across all results — compare, extract patterns, build tables, generate code.

Real developer workflows

"Compare competitor pricing pages" > "Batch scrape the pricing pages of Vercel, Netlify, Render, Railway, and Fly.io — extract each plan name, price, and key limits" → Batch 5 URLs with custom_id per competitor → extract pricing into a comparison table.

"Build a product dataset from Amazon" > "Here are 30 Amazon product URLs — scrape them all and extract name, price, rating, and review count" → Batch with parser: "@olostep/amazon-product" → get clean structured JSON for every product.

"Audit all docs pages for completeness" > "I mapped our docs site and got 45 URLs. Batch scrape all of them and tell me which pages are missing code examples" → /map first → feed URLs into batch → scan each page's content for code blocks → report gaps.

"Extract every job listing" > "Scrape all 20 of these job posting URLs and give me a table of: title, company, salary range, required skills, location" → Batch all listings → extract fields from each → return as a markdown table or JSON array.

"Dependency changelog audit" > "Scrape the changelogs for these 12 npm packages and flag any breaking changes in their latest major releases" → Batch all changelog URLs → scan for "BREAKING", "removed", "deprecated" → report per package.

"Conference speaker research" > "Batch scrape these 20 speaker profile pages and extract name, company, talk title, and bio" → Batch with custom_id per speaker → extract structured profiles → generate a speakers.json file.

"Geo-targeted pricing comparison" > "Scrape this SaaS pricing page from US, UK, and Germany to see if prices differ by region" → Three batch calls with country: "US", country: "GB", country: "DE" → compare results.

Chain with other skills

• map → batch: The most powerful combo. /map discovers all URLs on a site, you filter to the ones you need, then /batch scrapes them all. Perfect for docs sites, blogs, product catalogs.
• search → batch: Use /search to find relevant URLs, then /batch scrape all results for full content.
• batch → code: Batch scrape API docs pages, then generate typed clients or integration code from the combined content.

Parameters

• urlstoscrape: Array of {url, custom_id?} objects, 1–10,000 (required)
• output_format: markdown (default), html, json, text
• waitbeforescraping: ms to wait per URL for JS rendering, 0–10000 (optional)
• country: Country code for geo-targeted content — e.g. US, GB, DE (optional)
• parser: Specialised parser e.g. @olostep/amazon-product (optional)

Tips

• Always use custom_id — it makes matching results to sources trivial (e.g. "custom_id": "vercel-pricing")
• For 1–2 URLs, use /scrape. For 3+, always use /batch — it's parallel and faster
• Combine with /map for the most powerful pattern: discover → filter → batch scrape → synthesise
• The country param routes through residential proxies in that country — great for comparing regional content
• Always synthesise across results (tables, comparisons, summaries) — don't dump raw content

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: olostep-api
• Source: olostep-api/CLI
• License: MIT
• Homepage: https://github.com/olostep-api/olostep-cli

Install and usage instructions live in the source repository linked above.