# Docpull > Convert the public web into AI-ready Markdown with a local Python CLI/SDK/MCP crawler. - **Type:** MCP server - **Install:** `agentstack add mcp-raintree-technology-docpull` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [raintree-technology](https://agentstack.voostack.com/s/raintree-technology) - **Installs:** 0 - **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [raintree-technology](https://github.com/raintree-technology) - **Source:** https://github.com/raintree-technology/docpull - **Website:** https://docpull.raintree.technology/ ## Install ```sh agentstack add mcp-raintree-technology-docpull ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # docpull **Keep AI agents synced with changing public docs. Browser-free by default.** [](https://www.python.org/downloads/) [](https://pypi.org/project/docpull/) [](https://pepy.tech/project/docpull) [](https://github.com/raintree-technology/docpull/stargazers) [](https://github.com/raintree-technology/docpull/blob/main/LICENSE) DocPull turns public docs and web sources into refreshable, cited, agent-ready context packs. Define source sets once, sync them over time, diff what changed, and export fresh context for Cursor, Claude, Codex, OpenAI, LlamaIndex, LangChain, MCP clients, and RAG pipelines. The original `docpull URL ...` workflow still works: fetch public or explicitly authorized static/server-rendered web pages and write clean Markdown, NDJSON, SQLite, or OKF outputs. Project mode adds the persistent evidence lifecycle on top: sources, runs, diffs, exports, evals, accounting, and local auditability. DocPull is local-first: direct fetching, sitemap/link discovery, extraction, indexing, pack intelligence, and local `agent-browser` rendering can run with no provider account and no required API spend. Tavily, Exa, Parallel, and cloud renderers are optional escalation paths when local and open-source routes are not enough. DocPull exposes the same core workflows through CLI, Python SDK, and MCP, with each surface optimized for its user. The [Surface Contract](docs/surface-contract.md) defines how those surfaces align and where they intentionally differ. Web-source ingestion is the core workflow. Documentation is one high-value lane, not the product boundary. It works best on static or server-rendered pages such as blogs, API references, OpenAPI specs, changelogs, vendor pages, product pages, filings, docs sites, and other pages where the useful content is available in HTML or embedded page data. docpull is browser-free by default. JS-only pages are skipped with a clear reason unless you explicitly opt into the local `agent-browser` renderer. See [Scraping Boundary](docs/scraping-boundary.md) and [Alternatives](docs/alternatives.md) for the full boundary. ## Install ```bash pip install docpull ``` ## Project Quickstart ```bash docpull init stripe-docs docpull add https://docs.stripe.com docpull sync docpull diff docpull export context-pack --target cursor ``` Example diff after a later sync: ```text Project diff: +4 -2 ~18 api=2 pricing=1 Changed pages: - /payments/payment-intents likely API behavior change - /billing/subscriptions pricing / billing change - /webhooks likely API behavior change 0 failed URLs 0 robots blocked 0 paid/cloud routes used ``` Install optional extras as needed: ```bash pip install 'docpull[llm]' # tiktoken for token-accurate chunking pip install 'docpull[trafilatura]' # alternative extractor for noisy pages pip install 'docpull[mcp]' # stdio MCP server pip install 'docpull[serve]' # local pack JSON server runner pip install 'docpull[parallel]' # Parallel context packs pip install 'docpull[observability]' # Raindrop benchmark tracing pip install 'docpull[e2b]' # E2B cloud sandbox renderer SDK pip install 'docpull[all]' # all optional extras ``` Browser rendering is an explicit external extension, not part of the base install. Install an `agent-browser` compatible CLI separately, put it on `PATH`, or set `DOCPULL_AGENT_BROWSER_BIN=/path/to/agent-browser`. Verify the runtime with `docpull render --check`. Render targets must use HTTPS except for localhost/loopback HTTP during local testing, and DocPull keeps renderer action permissions locked down to HTML retrieval only. Because the current `agent-browser` CLI contract cannot enforce redirect, subresource, or connect-time DNS allow-lists, network browser rendering fails closed unless the operator sets `DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1` for trusted targets. For localhost/loopback HTTP tests, set `DOCPULL_RENDER_ALLOW_LOCAL_TARGETS=1`. For stronger isolation, cloud runtimes are available explicitly: `docpull render URL --runtime vercel` uses the Vercel Sandbox CLI and Vercel auth, while `docpull render URL --runtime e2b` uses the E2B Python SDK and `E2B_API_KEY`. These are never enabled by default. All runtimes execute the same `agent-browser --json` renderer contract. Use `--cloud-max-estimated-cost` to set a local per-render budget guard, and use `--cloud-agent-browser-install skip` with a prebuilt sandbox/template that already includes `agent-browser`. For E2B, pass `--template` or set `DOCPULL_E2B_TEMPLATE` to use that prebuilt environment. ## Free-First Budgets Use `--budget 0` when a run must not make paid-capable provider or cloud calls: ```bash docpull https://docs.example.com --budget 0 -o ./docs/example docpull discover scan https://docs.example.com -o ./packs/discovery DOCPULL_RENDER_TRUSTED_BROWSER_TARGETS=1 docpull render https://example.com/app --runtime local --budget 0 docpull providers context-pack "Find official docs" --provider all --dry-run --budget 0 --json docpull benchmark quick --zero-dollar --target-set zero-dollar --provider all ``` Under a zero budget, local cache, direct HTTP, sitemap/static-link discovery, local extraction, local indexing, pack analysis, monitors, and local `agent-browser` rendering for trusted targets remain allowed. Live Tavily, Exa, Parallel, Vercel Sandbox, and E2B calls are blocked before execution. Runs involving a budget or paid-capable route write `run.accounting.json` with non-secret route, cost, HTTP/cache, browser, and blocked-action metadata. Use `docpull discover scan URL` to build a provider-free discovery pack from open site hints: `llms.txt`, RSS/Atom feeds, OpenAPI specs, sitemap indexes, and public GitHub docs trees. It writes the same `candidate_sources.ndjson` contract as provider imports and URL/sitemap files, so the next step is still `docpull discover select` or `docpull discover fetch`. When a zero-dollar benchmark or local run is partial, DocPull reports the lowest-friction escalation path before spending money: trusted-target local `--render fallback` first, BYOK providers next, and cloud rendering only when local rendering or infrastructure is the blocker. Benchmark reports include suggested commands, estimated paid request counts, and estimated paid cost guards before any provider or cloud call is made. The `zero-dollar` benchmark target set is the Phase 2 measurement matrix. It keeps the existing docs/provider targets and adds JS-heavy docs, pricing, filings, feeds, sitemaps, and search-to-evidence tasks. The report classifies each target as `complete_for_0`, `complete_with_local_browser`, `partial_for_0`, `requires_provider`, `requires_cloud_browser`, or `blocked_by_policy`. ## Open Source And Hosted Boundary The open-source package owns local fetching, local rendering adapters, provider-free discovery, extraction, indexing, packs, diffs, monitors, MCP, BYOK providers, budget policy, accounting, and benchmarks. A hosted DocPull product, if offered, should sell managed execution: always-on schedules, browser/proxy infrastructure, persistent auth profiles, queues, alerts, dashboards, collaboration, retention, SSO, audit logs, SLAs, and bundled provider billing. The hosted boundary does not change the OSS default: no hidden paid calls, no CAPTCHA bypass, no stealth scraping, and no claim of a proprietary web-scale index. ## Persistent Projects Use project mode when a source corpus needs to stay fresh over time. A project is a local `docpull.yaml` plus a `.docpull/` state directory containing run history, cache, manifests, context-pack exports, eval sets, and a SQLite index. ```bash docpull init stripe-docs docpull add https://docs.stripe.com docpull sync docpull diff docpull export context-pack --target cursor ``` Each sync writes a normal local DocPull pack under `.docpull/runs//`, including `run.json`, `documents.jsonl`, `chunks.jsonl`, `manifest.json`, `documents.ndjson`, `corpus.manifest.json`, `sources.md`, `source-health.json`, `local.pack.json`, and accounting metadata. ```bash # Inspect the latest project state docpull status # Show run history docpull history # Diff the latest two runs, with deterministic local categories by default docpull diff # Write a review summary for the latest run docpull review # Create a versioned context-pack release docpull release context-pack --target cursor --tag stripe-docs-v1 # Generate eval cases from changed or latest documents docpull eval-set --limit 25 # One-command project sync, diff, and export for one source docpull watch https://docs.stripe.com --export cursor --alert changes ``` `docpull diff` is hash-based and deterministic locally. Optional BYOK semantic summaries are advisory and skip cleanly when no model key is configured. Use `docpull add URL --discover` or `docpull sync --update-discovery` to refresh and persist discovered source URLs in `docpull.yaml`; sync then uses that stored URL set for repeatable exact refreshes. For authenticated sources, store only environment variable references in `docpull.yaml`; DocPull resolves values in memory at sync time and writes only masked auth type/readiness to status, manifests, reviews, releases, and webhooks: ```yaml sources: - name: internal-docs url: https://docs.example.com auth: type: bearer_env env: EXAMPLE_DOCS_TOKEN policy: explicit-private ``` The commercial control-plane shape is API-led but not API-only: the local CLI remains the trust-building engine, while hosted DocPull manages projects, sync jobs, diffs, exports, releases, and signed webhooks through `/v1` JSON endpoints. The hosted ASGI MVP lives in `docpull.hosted`; `docpull remote ...` stores hosted API connection metadata and calls the same project lifecycle remotely. Remote bearer tokens are sent only to HTTPS API URLs by default; `--allow-insecure-local-http` is limited to localhost/loopback development. The launch screenshot for this flow lives at [`docs/launch-assets/docpull-project-diff-demo.png`](docs/launch-assets/docpull-project-diff-demo.png). ## 30-Second Usage ```bash docpull https://www.python.org/blogs/ --single -o ./python-news ``` Example output: ```text python-news/ index.md corpus.manifest.json ``` Markdown includes source metadata and readable page content: ```markdown --- title: "Blogs" source: https://www.python.org/blogs/ source_type: "html" --- # Blogs News from the Python Software Foundation, Python core developers, and the wider Python community. ``` Stream chunked NDJSON for agents and RAG: ```bash docpull https://www.python.org/blogs/ \ --single \ --profile llm \ --stream | jq . ``` Each line is a JSON document: ```json {"schema_version":1,"document_id":"doc_...","chunk_id":"chunk_...","url":"https://www.python.org/blogs/","title":"Blogs","content":"News from the Python Software Foundation...","source_type":"html","chunk_index":0,"token_count":842} ``` ## Common Workflows ```bash # Crawl a public web section and write Markdown files docpull https://www.python.org/blogs/ -o ./python-news # Stream LLM-ready NDJSON chunks from a source docpull https://www.python.org/blogs/ --profile llm --stream | jq . # Write SQLite with an FTS5 search index docpull https://www.python.org/blogs/ --format sqlite -o ./python-news-db # Build an Open Knowledge Format (OKF) bundle for portable source packs docpull https://example.com --profile okf -o ./site-okf # Turn a source corpus into agent-ready skills/rules docpull https://sdk.vercel.ai \ --skill vercel-ai \ --skill-agent all \ --skill-description "Vercel AI SDK source reference" ``` Local-first parity workflows mirror common hosted search/extract/crawl/research API shapes while writing auditable files instead of relying on a hosted index: ```bash # Normalize candidate URLs without fetching content docpull map urls ./urls.txt -o ./packs/map # Extract known URLs into a local pack docpull extract-pack ./urls.txt -o ./packs/extract # Select mapped candidates and fetch them docpull crawl-pack ./packs/map --select top:10 -o ./packs/crawl # Answer/research from an existing local pack with lifecycle artifacts docpull research-pack ./packs/crawl \ --objective "Summarize auth and webhook behavior" \ --schema ./output.schema.json # Build a cited entity/list pack from existing evidence docpull entities-pack ./packs/crawl --limit 100 ``` More examples live in [CLI Recipes](docs/examples/README.md). With an explicit `--skill-agent`, docpull stores the scraped corpus under `.docpull/skills//references` and creates agent-specific wrappers that point at that corpus. `--skill-agent claude` writes a Claude Code skill under `.claude/skills//`, `--skill-agent codex` writes a Codex skill under `.agents/skills//` with `agents/openai.yaml`, and `--skill-agent cursor` writes a Cursor project rule at `.cursor/rules/.mdc`. Use `--skill-agent all` to create all three. If you pass `--output-dir`, docpull stages the generated corpus there; explicit `--skill-agent` targets still write their active agent wrappers. Use docpull when you need to: - Convert public web sources - docs, blogs, API references, vendor pages, product pages, changelogs, filings, and OpenAPI specs - into Markdown or chunked NDJSON for LLM and RAG pipelines. - Give an agent a local tool for fetching, caching, grepping, and reading web sources. - Build repeatable context packs with stable IDs, hashes, manifests, and source metadata. - Mirror public web content for offline work while preserving attribution. ## Why docpull? docpull is designed for agent and RAG workflows, not just downloading pages. | Need | docpull gives you | | --- | --- | | Clean Markdown | Article-focused extraction with source metadata | | LLM chunks | NDJSON streaming and optional token-aware chunking | | Repeatability | Stable document IDs, chunk IDs, hashes, and manifests | | Offline work | Cached archives and mirrored source artifacts | | Agent access | Local CLI, Python SDK, and stdio MCP server | | Downstream exports | JSONL, Sheets CSV/TSV, n8n JSON, Vercel AI JSON, CrewAI JSON, warehouse NDJSON, optional Parquet, and agent skills | | Safer fetching | HTTPS defaults, robots.txt compliance, SSRF protections, and redirect guards | ## Supported Sources docpull uses async HTTP instead of browser automation by default and includes special handling for common web, documentation, and API surfaces. | Source shape | Support | | --- | --- | | Static HTML / SSR pages | Extracts article, main, or document regions | | Next.js / Mintlify | Parses static HTML and `__NEXT_DATA__` when available | | OpenAPI / Swagger | Renders specs into Markdown | | Docusaurus / Sphinx / MkDocs | Extracts static article or document regions | | VitePress / VuePress / Astro Starlight | Extracts static content regions | | GitBook / ReadMe.io | Extracts available article or content regions | | Redoc / Scalar | Extracts static API reference regions | | JS-only apps | Skipped unless useful content is present in HTML or embedded data | Use `--strict-js-required` when an agent should treat JS-only pages as hard errors instead of normal skips. ## Output Formats | Output | Use it for | | --- | --- | | Markdown | Local readable source snapshots with YAML frontmatter | | NDJSON | Streamed records or chunked records for agents and RAG | | SQLite | Local retrieval with an FTS5 index | | OKF | Portable Open Knowledge Format bundles with indexes and manifests | | Archive / mirror | Cached offline source snapshots | Every file-backed run writes `corpus.manifest.json` with stable document IDs, chunk IDs, hashes, output paths, and chunk counts. See [Corpus Manifest](docs/corpus-manifest.md). ## Profiles ```bash docpull https://site.com --profile rag # Default. Dedup + metadata. docpull https://site.com --profile llm # NDJSON chunks for agents/RAG. docpull https://site.com --profile okf # Portable Open Knowledge Format bundle. do … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [raintree-technology](https://github.com/raintree-technology) - **Source:** [raintree-technology/docpull](https://github.com/raintree-technology/docpull) - **License:** MIT - **Homepage:** https://docpull.raintree.technology/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-raintree-technology-docpull - Seller: https://agentstack.voostack.com/s/raintree-technology - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.