# Crawlberg > High-performance web crawling engine with bindings for 11 languages - **Type:** MCP server - **Install:** `agentstack add mcp-xberg-io-crawlberg` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [xberg-io](https://agentstack.voostack.com/s/xberg-io) - **Installs:** 0 - **Category:** [Web & Browser](https://agentstack.voostack.com/c/web-and-browser) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [xberg-io](https://github.com/xberg-io) - **Source:** https://github.com/xberg-io/crawlberg - **Website:** https://docs.crawlberg.xberg.io ## Install ```sh agentstack add mcp-xberg-io-crawlberg ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Crawlberg High-performance Rust web crawling engine for structured data extraction. Scrape, crawl, and map websites with native bindings for 14 languages — same engine, identical results across every runtime. ## What and Why? Crawlberg is the crawling **substrate**: everything you need to scrape and crawl a site end-to-end from a single Rust core — HTML→Markdown, headless-Chrome fallback, robots/sitemap parsing, per-domain throttling, and an SSRF-safe policy — with identical results across 14 language bindings. Productization concerns (managed proxy pools, tuned WAF fingerprints, authenticated-session injection, scheduling, billing) live in [xberg-enterprise](https://github.com/xberg-io/xberg-enterprise), the reference operational implementation. Every extension point (`Frontier`, `RateLimiter`, `CrawlStore`, `EventEmitter`, `ContentFilter`, `WafClassifier`, …) is a trait you inject via `CrawlEngineBuilder::with_(...)`. ### Features | Feature | Description | | ------- | ----------- | | **Structured extraction** | Text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, response headers | | **Markdown conversion** | Clean Markdown with citations, document structure, and fit-content mode | | **Concurrent crawling** | Depth-first, breadth-first, or best-first traversal with configurable depth, page limits, and concurrency | | **14 language bindings** | Rust, Python, Node.js, Ruby, Go, Java, Kotlin (Android), C#, PHP, Elixir, Dart, Swift, Zig, and WebAssembly | | **Smart filtering** | BM25 relevance scoring, URL include/exclude patterns, robots.txt compliance, sitemap discovery | | **Browser rendering** | Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass | | **Batch & streaming** | Scrape or crawl hundreds of URLs concurrently; real-time crawl events via async streams | | **SSRF-safe by default** | Refuses loopback, private, link-local, and cloud-metadata addresses; opt out via env var or `CrawlConfig` | | **Auth & rate limiting** | HTTP Basic, Bearer, and custom-header auth with cookie jars; per-domain request throttling | | **MCP server & REST API** | Model Context Protocol integration for AI agents plus an HTTP server with OpenAPI spec | ### Supported Platforms Precompiled binaries for Linux (x86_64/aarch64), macOS (ARM64), and Windows (x64) across every binding. See the [platform support reference](https://docs.crawlberg.xberg.io) for the full matrix. ⭐ Star this repo to show your support — it helps others discover Crawlberg. ## Quick Start ### Language Packages Python ```sh pip install crawlberg ``` See [Python README](https://github.com/xberg-io/crawlberg/tree/main/packages/python) for full documentation. Node.js ```sh npm install @xberg-io/crawlberg ``` See [Node.js README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-node) for full documentation. Rust ```sh cargo add crawlberg ``` See [Rust README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg) for full documentation. Go ```sh go get github.com/xberg-io/crawlberg/packages/go ``` See [Go README](https://github.com/xberg-io/crawlberg/tree/main/packages/go) for full documentation. Java Available on Maven Central as `io.xberg.crawlberg:crawlberg`. See [Java README](https://github.com/xberg-io/crawlberg/tree/main/packages/java) for the dependency snippet and current version. C# ```sh dotnet add package Crawlberg ``` See [C# README](https://github.com/xberg-io/crawlberg/tree/main/packages/csharp) for full documentation. Ruby ```sh gem install crawlberg ``` See [Ruby README](https://github.com/xberg-io/crawlberg/tree/main/packages/ruby) for full documentation. PHP ```sh composer require xberg-io/crawlberg ``` See [PHP README](https://github.com/xberg-io/crawlberg/tree/main/packages/php) for full documentation. Elixir Add `{:crawlberg, "~> 0.3"}` to your `mix.exs` dependencies. See [Elixir README](https://github.com/xberg-io/crawlberg/tree/main/packages/elixir) for full documentation. Dart / Flutter ```sh dart pub add crawlberg ``` See [Dart README](https://github.com/xberg-io/crawlberg/tree/main/packages/dart) for full documentation. Kotlin (Android) Available on Maven Central as `io.xberg.crawlberg.android:crawlberg-android`. See [Kotlin README](https://github.com/xberg-io/crawlberg/tree/main/packages/kotlin-android) for the dependency snippet and current version. Swift Add via Swift Package Manager. See [Swift README](https://github.com/xberg-io/crawlberg/tree/main/packages/swift) for full documentation. Zig See [Zig README](https://github.com/xberg-io/crawlberg/tree/main/packages/zig) for installation and usage. WebAssembly ```sh npm install @xberg-io/crawlberg-wasm ``` See [WebAssembly README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-wasm) for full documentation. C/C++ (FFI) C header + shared library from [GitHub Releases](https://github.com/xberg-io/crawlberg/releases). See [FFI crate](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-ffi) for full documentation. CLI ```sh cargo install crawlberg-cli ``` ```sh brew install xberg-io/tap/crawlberg ``` See [CLI README](https://github.com/xberg-io/crawlberg/tree/main/crates/crawlberg-cli) for full documentation. ### AI Coding Assistants Install the Crawlberg plugin from the [`xberg-io/plugins`](https://github.com/xberg-io/plugins) marketplace. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the `crawlberg` MCP server, and works with every major coding agent — expand your harness below. Claude Code ```text /plugin marketplace add xberg-io/plugins /plugin install crawlberg@xberg ``` Codex CLI ```text /plugins add https://github.com/xberg-io/plugins ``` Then search for `crawlberg` and select **Install Plugin**. Cursor Settings → Plugins → Add from URL → `https://github.com/xberg-io/plugins`, then select **crawlberg**. Gemini CLI ```text gemini extensions install https://github.com/xberg-io/plugins ``` Factory Droid ```text droid plugin marketplace add https://github.com/xberg-io/plugins droid plugin install crawlberg@xberg ``` GitHub Copilot CLI ```text copilot plugin marketplace add https://github.com/xberg-io/plugins copilot plugin install crawlberg@xberg ``` opencode Add the package to `opencode.json`: ```json { "$schema": "https://opencode.ai/config.json", "plugin": ["@xberg-io/opencode-crawlberg"] } ``` ## Documentation Full guides, per-language API references, the substrate/operational model, antibot strategy, and observability live at **[docs.crawlberg.xberg.io](https://docs.crawlberg.xberg.io)**. ## Contributing Contributions are welcome! See our [Contributing Guide](https://github.com/xberg-io/crawlberg/blob/main/CONTRIBUTING.md). ## Part of Xberg.dev - [Xberg](https://github.com/xberg-io/xberg) — document intelligence: text, tables, metadata from 91+ formats with optional OCR. - [Xberg Enterprise](https://github.com/xberg-io/xberg-enterprise) — managed extraction API with SDKs, dashboards, and observability. - [crawlberg](https://github.com/xberg-io/crawlberg) — web crawling and scraping with HTML→Markdown and headless-Chrome fallback. - [html-to-markdown](https://github.com/xberg-io/html-to-markdown) — fast, lossless HTML→Markdown engine. - [liter-llm](https://github.com/xberg-io/liter-llm) — universal LLM API client with native bindings for 14 languages and 143 providers. - [tree-sitter-language-pack](https://github.com/xberg-io/tree-sitter-language-pack) — tree-sitter grammars and code-intelligence primitives. - [alef](https://github.com/xberg-io/alef) — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos. ## License [MIT License](https://github.com/xberg-io/crawlberg/blob/main/LICENSE) ## Links - [Documentation](https://docs.crawlberg.xberg.io) - [GitHub Repository](https://github.com/xberg-io/crawlberg) - [Issue Tracker](https://github.com/xberg-io/crawlberg/issues) - [Changelog](CHANGELOG.md) - [Discord](https://discord.gg/xt9WY3GnKR) — community, roadmap, announcements. ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [xberg-io](https://github.com/xberg-io) - **Source:** [xberg-io/crawlberg](https://github.com/xberg-io/crawlberg) - **License:** MIT - **Homepage:** https://docs.crawlberg.xberg.io Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-xberg-io-crawlberg - Seller: https://agentstack.voostack.com/s/xberg-io - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.