Crawl

> Latest version: if you have web access, fetch the newest copy of this skill at https://www.olostep.com/skills/crawl/SKILL.md — it may include capabilities added since this copy was installed. Otherwise the instructions below are complete and current.

Olostep Crawl

Autonomously discover and scrape an entire website by following links from a start URL. Every page is rendered in a full browser with anti-bot bypass — perfect for ingesting docs, blogs, and knowledge bases.

Why this matters

Manually collecting pages from a docs site is tedious and incomplete. Crawl does it automatically — it starts at a URL, follows links, renders every page (even JS-heavy ones), and returns clean content from each. You get an entire site's knowledge in one call.

When to use

• User wants to learn a library by ingesting its complete docs
• User wants all blog posts or articles from a site as context
• User is preparing content for a RAG pipeline or knowledge base
• User wants to audit all pages across a site (content completeness, consistency, SEO)
• User wants a full picture of a competitor's documentation or marketing

Workflow

1. Use create_crawl with the starting URL.
2. Set max_pages: 10 by default — always ask the user before going above 20.
3. Set follow_links: true (default) to discover pages automatically.
4. Default to markdown — best for AI reasoning.
5. After crawling, synthesise the collected content — don't dump it raw.

Real developer workflows

"Learn this library from its docs" > "Crawl https://tanstack.com/query/latest/docs/overview and give me a cheat sheet of the key hooks, patterns, and gotchas" → Crawl 15–20 pages → synthesise into a concise reference with code snippets.

"Generate working examples from docs" > "Crawl the Resend docs and write me 5 practical TypeScript examples: send email, send with template, add attachment, send to list, check delivery status" → Crawl → extract each endpoint's signature and parameters → write copy-paste-ready examples.

"Build a knowledge base for my team" > "Crawl our internal docs at https://docs.internal.co and create a structured summary of every service, its owner, and its API endpoints" → Crawl 30–50 pages → extract service names, team ownership, endpoint lists → generate a services.json or markdown index.

"Competitive docs analysis" > "Crawl https://docs.competitor.com/api and compare their API capabilities with ours — what do they have that we don't?" → Crawl → list all endpoints and features → compare against the user's API → identify gaps.

"Content audit before a redesign" > "Crawl our marketing site and give me every page URL, its title, word count, and whether it has a CTA" → Crawl up to 50 pages → extract metadata from each → return as a structured table.

"Prepare context for a refactor" > "Crawl our architecture docs and summarise the auth system before I refactor it" → Crawl → filter pages related to auth → extract design decisions, data flow, dependencies.

Chain with other skills

• map → crawl: Use /map first to see all URLs, then /crawl a specific section. Avoids wasting crawl budget on irrelevant pages.
• crawl → code: Crawl the docs, then write complete integration code from the collected knowledge (see /docs-to-code).
• crawl → batch: If the crawl is too broad, use /map + /batch for more targeted extraction.

Parameters

• start_url: Starting URL for the crawl (required)
• max_pages: Maximum pages to crawl — default 10, increase carefully (optional)
• follow_links: Whether to follow links on each page — default true (optional)
• output_format: markdown (default), html, json, text
• country: Country code for geo-targeted crawling — e.g. US, GB (optional)
• parser: Specialised parser ID for structured extraction (optional)

Tips

• Start small: 10 pages is usually enough for a focused docs section. Ask before going to 50+.
• Always synthesise after crawling — generate a summary, cheat sheet, comparison, or structured data.
• Use /map first on large sites to understand the structure, then /crawl the relevant section.
• For a known list of URLs, /batch is better — it's parallel and you control exactly which pages to scrape.
• For a single page, just use /scrape.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: olostep-api
• Source: olostep-api/CLI
• License: MIT
• Homepage: https://github.com/olostep-api/olostep-cli

Install and usage instructions live in the source repository linked above.