Crawlberg

⭐ Star this repo to show your support — it helps others discover Crawlberg.

Quick Start

Language Packages

Python

pip install crawlberg

See Python README for full documentation.

Node.js

npm install @xberg-io/crawlberg

See Node.js README for full documentation.

Rust

cargo add crawlberg

See Rust README for full documentation.

Go

go get github.com/xberg-io/crawlberg/packages/go

See Go README for full documentation.

Java

Available on Maven Central as io.xberg.crawlberg:crawlberg. See Java README for the dependency snippet and current version.

C#

dotnet add package Crawlberg

See C# README for full documentation.

Ruby

gem install crawlberg

See Ruby README for full documentation.

PHP

composer require xberg-io/crawlberg

See PHP README for full documentation.

Elixir

Add {:crawlberg, "~> 0.3"} to your mix.exs dependencies. See Elixir README for full documentation.

Dart / Flutter

dart pub add crawlberg

See Dart README for full documentation.

Kotlin (Android)

Available on Maven Central as io.xberg.crawlberg.android:crawlberg-android. See Kotlin README for the dependency snippet and current version.

Swift

Add via Swift Package Manager. See Swift README for full documentation.

Zig

See Zig README for installation and usage.

WebAssembly

npm install @xberg-io/crawlberg-wasm

See WebAssembly README for full documentation.

C/C++ (FFI)

C header + shared library from GitHub Releases. See FFI crate for full documentation.

CLI

cargo install crawlberg-cli

brew install xberg-io/tap/crawlberg

See CLI README for full documentation.

AI Coding Assistants

Install the Crawlberg plugin from the xberg-io/plugins marketplace. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the crawlberg MCP server, and works with every major coding agent — expand your harness below.

Claude Code

/plugin marketplace add xberg-io/plugins
/plugin install crawlberg@xberg

Codex CLI

/plugins add https://github.com/xberg-io/plugins

Then search for crawlberg and select Install Plugin.

Cursor

Settings → Plugins → Add from URL → https://github.com/xberg-io/plugins, then select crawlberg.

Gemini CLI

gemini extensions install https://github.com/xberg-io/plugins

Factory Droid

droid plugin marketplace add https://github.com/xberg-io/plugins
droid plugin install crawlberg@xberg

GitHub Copilot CLI

copilot plugin marketplace add https://github.com/xberg-io/plugins
copilot plugin install crawlberg@xberg

opencode

Add the package to opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["@xberg-io/opencode-crawlberg"]
}

Documentation

Full guides, per-language API references, the substrate/operational model, antibot strategy, and observability live at docs.crawlberg.xberg.io.

Contributing

Contributions are welcome! See our Contributing Guide.

Part of Xberg.dev

• Xberg — document intelligence: text, tables, metadata from 91+ formats with optional OCR.
• Xberg Enterprise — managed extraction API with SDKs, dashboards, and observability.
• crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
• html-to-markdown — fast, lossless HTML→Markdown engine.
• liter-llm — universal LLM API client with native bindings for 14 languages and 143 providers.
• tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
• alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.

License

MIT License

Links

• Documentation
• GitHub Repository
• Issue Tracker
• [Changelog](CHANGELOG.md)
• Discord — community, roadmap, announcements.

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: xberg-io
• Source: xberg-io/crawlberg
• License: MIT
• Homepage: https://docs.crawlberg.xberg.io

Install and usage instructions live in the source repository linked above.