# Crawl

> Autonomously crawl an entire website by following links from a starting URL using Olostep. Use when the user wants to ingest an entire docs site, blog, knowledge base, or any multi-page site as context for coding, writing, analysis, or building a knowledge base.

- **Type:** Skill
- **Install:** `agentstack add skill-olostep-api-cli-crawl`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [olostep-api](https://agentstack.voostack.com/s/olostep-api)
- **Installs:** 0
- **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [olostep-api](https://github.com/olostep-api)
- **Source:** https://github.com/olostep-api/CLI/tree/main/skills/crawl
- **Website:** https://github.com/olostep-api/olostep-cli

## Install

```sh
agentstack add skill-olostep-api-cli-crawl
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

> **Latest version:** if you have web access, fetch the newest copy of this skill at https://www.olostep.com/skills/crawl/SKILL.md — it may include capabilities added since this copy was installed. Otherwise the instructions below are complete and current.

# Olostep Crawl

Autonomously discover and scrape an entire website by following links from a start URL. Every page is rendered in a full browser with anti-bot bypass — perfect for ingesting docs, blogs, and knowledge bases.

## Why this matters

Manually collecting pages from a docs site is tedious and incomplete. Crawl does it automatically — it starts at a URL, follows links, renders every page (even JS-heavy ones), and returns clean content from each. You get an entire site's knowledge in one call.

## When to use

- User wants to learn a library by ingesting its complete docs
- User wants all blog posts or articles from a site as context
- User is preparing content for a RAG pipeline or knowledge base
- User wants to audit all pages across a site (content completeness, consistency, SEO)
- User wants a full picture of a competitor's documentation or marketing

## Workflow

1. Use `create_crawl` with the starting URL.
2. Set `max_pages: 10` by default — **always ask the user before going above 20**.
3. Set `follow_links: true` (default) to discover pages automatically.
4. Default to `markdown` — best for AI reasoning.
5. After crawling, **synthesise the collected content** — don't dump it raw.

## Real developer workflows

**"Learn this library from its docs"**
> "Crawl https://tanstack.com/query/latest/docs/overview and give me a cheat sheet of the key hooks, patterns, and gotchas"
→ Crawl 15–20 pages → synthesise into a concise reference with code snippets.

**"Generate working examples from docs"**
> "Crawl the Resend docs and write me 5 practical TypeScript examples: send email, send with template, add attachment, send to list, check delivery status"
→ Crawl → extract each endpoint's signature and parameters → write copy-paste-ready examples.

**"Build a knowledge base for my team"**
> "Crawl our internal docs at https://docs.internal.co and create a structured summary of every service, its owner, and its API endpoints"
→ Crawl 30–50 pages → extract service names, team ownership, endpoint lists → generate a services.json or markdown index.

**"Competitive docs analysis"**
> "Crawl https://docs.competitor.com/api and compare their API capabilities with ours — what do they have that we don't?"
→ Crawl → list all endpoints and features → compare against the user's API → identify gaps.

**"Content audit before a redesign"**
> "Crawl our marketing site and give me every page URL, its title, word count, and whether it has a CTA"
→ Crawl up to 50 pages → extract metadata from each → return as a structured table.

**"Prepare context for a refactor"**
> "Crawl our architecture docs and summarise the auth system before I refactor it"
→ Crawl → filter pages related to auth → extract design decisions, data flow, dependencies.

## Chain with other skills

- **map → crawl**: Use `/map` first to see all URLs, then `/crawl` a specific section. Avoids wasting crawl budget on irrelevant pages.
- **crawl → code**: Crawl the docs, then write complete integration code from the collected knowledge (see `/docs-to-code`).
- **crawl → batch**: If the crawl is too broad, use `/map` + `/batch` for more targeted extraction.

## Parameters
- **start_url**: Starting URL for the crawl (required)
- **max_pages**: Maximum pages to crawl — default 10, increase carefully (optional)
- **follow_links**: Whether to follow links on each page — default `true` (optional)
- **output_format**: `markdown` (default), `html`, `json`, `text`
- **country**: Country code for geo-targeted crawling — e.g. `US`, `GB` (optional)
- **parser**: Specialised parser ID for structured extraction (optional)

## Tips
- **Start small**: 10 pages is usually enough for a focused docs section. Ask before going to 50+.
- Always **synthesise** after crawling — generate a summary, cheat sheet, comparison, or structured data.
- Use `/map` first on large sites to understand the structure, then `/crawl` the relevant section.
- For a **known list** of URLs, `/batch` is better — it's parallel and you control exactly which pages to scrape.
- For a **single page**, just use `/scrape`.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [olostep-api](https://github.com/olostep-api)
- **Source:** [olostep-api/CLI](https://github.com/olostep-api/CLI)
- **License:** MIT
- **Homepage:** https://github.com/olostep-api/olostep-cli

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-olostep-api-cli-crawl
- Seller: https://agentstack.voostack.com/s/olostep-api
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.