# Acrawl

> Autonomous web crawler. 17 browser tools + goal-driven run_goal agent. Single binary, stealth.

- **Type:** MCP server
- **Install:** `agentstack add mcp-mingye-lu-agenticcrawler`
- **Verified:** Pending review
- **Seller:** [Mingye-Lu](https://agentstack.voostack.com/s/mingye-lu)
- **Installs:** 0
- **Latest version:** 0.10.0
- **License:** MIT
- **Upstream author:** [Mingye-Lu](https://github.com/Mingye-Lu)
- **Source:** https://github.com/Mingye-Lu/AgenticCrawler

## Install

```sh
agentstack add mcp-mingye-lu-agenticcrawler
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

█████╗  ██████╗██████╗  █████╗ ██╗    ██╗██╗     
 ██╔══██╗██╔════╝██╔══██╗██╔══██╗██║    ██║██║     
 ███████║██║     ██████╔╝███████║██║ █╗ ██║██║     
 ██╔══██║██║     ██╔══██╗██╔══██║██║███╗██║██║     
 ██║  ██║╚██████╗██║  ██║██║  ██║╚███╔███╔╝███████╗
 ╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═╝ ╚══╝╚══╝ ╚══════╝

  Browser automation agent that acts and observes. Navigate, click, and extract — or inspect network traffic, debug console errors, profile performance, and audit accessibility. One Rust binary.

  
  
  

  Single Rust binary. Full DevTools observability. 42 tools. 25 LLM providers. MCP server built-in.

---

## Why acrawl?

Most browser agents stop at navigate and click. acrawl goes further: the agent can also inspect network requests, analyze console errors, measure page performance, audit accessibility, and intercept network calls — the full DevTools surface, available as first-class agent tools.

It ships as a single Rust binary. No Python runtime, no Node runtime, no Docker. Drop it into any server or CI pipeline and describe a goal; the agent figures out what to visit, what to click, what to inspect, and when it's done.

- **One binary, zero runtimes.** `cargo build --release` produces a self-contained executable. No Python, no Node runtime — just Rust and a Chromium download for browser automation.
- **Acts and observes.** The agent has the full DevTools surface as first-class tools: inspect network requests with timing, analyze and deduplicate console logs, stream WebSocket messages, measure page performance (TTFB, resource breakdown), audit cookies and browser storage, measure JS/CSS coverage, run axe-core WCAG accessibility audits, and intercept or mock network calls. No other agent framework exposes this.
- **Deterministic where you can, AI where you must.** Define loops, conditionals, and parallel branches as JSON scripts — executed without any LLM calls. Fall back to the agent when pages behave unexpectedly. Best of both worlds.
- **No code required.** Describe the goal in plain English. The agent plans, navigates, and extracts.
- **Smart fetching.** Static pages are served over HTTP (fast). When JavaScript or interaction is needed, acrawl detects JS framework markers (`__next_data__`, `__nuxt`, `__vue`, `ng-app`, React roots), auth redirects, and short `` bodies — then transparently escalates to a headless browser.
- **Sub-agent parallelism.** Fork child agents onto separate browser tabs with independent state and step budgets. A URL-claiming registry prevents siblings from crawling the same page twice.
- **MCP client and server.** Extend the agent with custom tools via [Model Context Protocol](https://modelcontextprotocol.io) servers. Or flip it: `acrawl mcp` exposes 38 browser and DevTools tools plus `run_goal` to Claude Code, Cursor, VS Code, Zed, and 13 other clients.
- **25 LLM providers.** Anthropic, OpenAI, Google Gemini, DeepSeek, AWS Bedrock, Azure OpenAI, Vertex AI, GitHub Copilot, Groq, Mistral, xAI, Cohere, Alibaba DashScope, OpenRouter, and more. Or bring your own via any OpenAI-compatible endpoint.

### How does it compare?

#### vs. AI web agents and scraping tools

| | acrawl | browser-use | Stagehand | Skyvern | Firecrawl | Playwright MCP | Scrapy | Playwright scripts |
|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| No code needed | Yes | No | No | Partial | No | No | No | No |
| Single binary | Yes | No | No | No | No | No | No | No |
| JS rendering | Yes | Yes | Yes | Yes | Yes | Yes | No | Yes |
| LLM-powered navigation | Yes | Yes | Yes | Yes | Limited | No | No | No |
| No Python / Node needed | Yes | No | No | No | No | No | No | No |
| Form filling / interaction | Yes | Yes | Yes | Yes | No | Yes | No | Yes |
| Sub-agent parallelism | Yes | No | No | Partial | Partial | No | Partial | No |
| 25 LLM providers | Yes | Via LiteLLM | Partial | Partial | N/A | N/A | N/A | N/A |
| MCP client (use tools) | Yes | No | No | No | No | No | No | No |
| MCP server (expose as tools) | Yes | No | No | No | Yes | Yes | No | No |
| Stealth browser built-in | Yes | Cloud only | Via Browserbase | Cloud only | No | No | No | No |
| DevTools observability (network, console, perf, a11y) | Yes | No | No | No | No | No | No | No |
| Deterministic script layer (zero LLM calls) | Yes | No | Partial | No | No | No | Yes | Yes |
| Open source | Yes | Yes (MIT) | Yes (MIT) | Yes (Apache) | Engine only | Yes (MIT) | Yes (BSD) | Yes (Apache) |

Notes:
- **browser-use** (85k+ GitHub stars): Python + Playwright, DOM + screenshots, supports GPT/Claude/Gemini/Ollama via LiteLLM, 89.1% WebVoyager. No single binary — requires Python and `pip install`. Every action calls an LLM: 2-5s/step, ~$0.02-0.30/task. Cloud tier adds stealth; self-hosted is bare Playwright.
- **Stagehand** (Browserbase, 21k+ stars): TypeScript + CDP (v3), mixes deterministic Playwright with AI primitives (`act()`, `extract()`, `observe()`). Action caching reuses successful clicks without re-calling the LLM. Requires Node and, for production, Browserbase cloud hosting.
- **Skyvern** (21k+ stars, Apache 2.0): vision-first (screenshot-only, no DOM), handles legacy portals and government forms that DOM tools struggle with. No-code cloud UI available. Each step costs vision-model tokens — ~$0.10-0.50/task. 85.85% WebVoyager.
- **Firecrawl** (82k+ stars): managed scraping API. Returns LLM-ready Markdown, JSON extraction, site-wide crawl. Not an agentic tool — minimal multi-step interaction. Ships an official MCP server. Per-page pricing from $19/month.
- **Playwright MCP** (Microsoft, 29k+ stars): MCP server that exposes browser control via the accessibility tree. Sub-100ms actions, zero vision tokens. Drives an LLM client's browser rather than having its own reasoning — no autonomous goal navigation. Used in GitHub Copilot Agent.

#### vs. native LLM provider browsing

Most AI providers offer some form of browsing, but it is designed for **conversational information retrieval**, not programmatic web automation. Key constraints:

| | acrawl | ChatGPT Agent | Claude Computer Use | Claude in Chrome | Gemini Deep Research | Copilot / Edge |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| Real JS-rendered browser | Yes | Yes (sandboxed cloud VM) | Indirect (dev provides env) | Yes (your Chrome) | No (search API only) | Limited (Bing retrieval) |
| Click / fill forms | Yes | Yes (requires user confirmation) | Yes | Yes | No | Limited |
| Programmable / scriptable | Yes | No | Yes (API beta) | No | No | No |
| Sub-agent parallelism | Yes | No | No | No | No | No |
| MCP server (expose as tools) | Yes | No | No | No | No | No |
| Returns structured data | Yes | No (text summaries) | No (screenshots) | No | No | No |
| Stealth / anti-bot | Yes | No | No | No | No | No |
| No vendor lock-in | Yes (25 providers) | OpenAI only | Anthropic only | Anthropic only | Google only | OpenAI / Bing only |
| Runs without paid subscription | Yes (OSS; LLM key needed) | No (Plus/Pro/Business) | No (API cost) | No (Max plan) | Partial | Yes (free tier) |

Notes:
- **ChatGPT Agent** (OpenAI, July 2025): runs in a sandboxed cloud virtual machine with its own Chromium instance. Can browse, click, and fill forms but pauses for user confirmation on sensitive actions (purchases, logins). Uses two modes: a fast text browser for research queries and a visual browser for interaction. Cannot run code in the browser, install extensions, or access your local file system. Susceptible to prompt injection. Available to Plus/Pro/Business subscribers.
- **ChatGPT Atlas** (OpenAI, October 2025): a full Chromium browser with ChatGPT integrated as a sidebar + agent. Agent mode drives the same sandboxed cloud VM as ChatGPT Agent; core limitations are identical.
- **Claude Computer Use** (Anthropic API, beta since October 2024): screenshot + mouse/keyboard API for any desktop application, not just browsers. Vision-only — no DOM access. Developers must provide and manage the entire computing environment (typically a Docker container with Xvfb + Firefox). Not a ready-to-use binary. Requires significant infrastructure to operate in production.
- **Claude in Chrome** (Anthropic Chrome extension, beta November 2025+): lets Claude operate within your existing Chrome session using your real cookies and logins. Available to Max plan subscribers. Not an open API — no programmatic control. Good for interactive personal tasks; not suitable for batch automation.
- **Gemini / Deep Research** (Google): browsing is grounded via Google Search API calls, not a live browser session. Deep Research synthesizes across many searches but cannot interact with pages (click, fill forms, navigate dynamically). Project Mariner (experimental computer use) is a separate, limited research preview.
- **Copilot / Edge** (Microsoft): Edge's Copilot Mode uses Bing retrieval with some ability to navigate pages. Real-world tests show high latency (6+ minutes for multi-page comparison tasks) and frequent interruptions for user confirmation. Not a developer API.

## Quick Start

### Install

**Linux / macOS (x64 / ARM64):**
```bash
curl -fsSL https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.sh | bash
```

**Windows (x64, PowerShell):**
```powershell
irm https://raw.githubusercontent.com/Mingye-Lu/AgenticCrawler/main/install.ps1 | iex
```

This downloads the latest binary, verifies its SHA256 checksum, and sets up CloakBrowser for stealth browser automation. Requires Node.js 20+ for browser features.

acrawl checks for updates on startup and shows a notification when a new version is available.

Build from source

```bash
git clone https://github.com/Mingye-Lu/AgenticCrawler.git
cd AgenticCrawler
cargo build --release

# Install CloakBrowser (required for browser automation — binary auto-downloads on first use)
npm install
```

### Browser Extension (optional)

The acrawl Bridge extension lets acrawl control your real browser (with your sessions, cookies, and existing extensions) instead of a headless CloakBrowser instance. Download `acrawl-extension.zip` from the [latest release](https://github.com/Mingye-Lu/AgenticCrawler/releases/latest), unzip it, then load it into your browser:

| Browser | Extensions page | Developer mode toggle |
|---------|----------------|----------------------|
| Chrome | `chrome://extensions` | Top-right |
| Edge | `edge://extensions` | Bottom-left |
| Brave | `brave://extensions` | Top-right |
| Arc / Vivaldi / Opera | `://extensions` | Varies |

Enable **Developer mode**, click **Load unpacked**, and select the unzipped folder. Then run `/extension` in the acrawl REPL to connect. See [extension/README.md](extension/README.md) for full setup details.

### Configure

```bash
# Set up your LLM provider (interactive prompt)
./target/release/acrawl auth anthropic   # or: openai, other
```

Credentials are stored in `~/.acrawl/credentials.json`. Override the config directory with `ACRAWL_CONFIG_HOME`.

### Run

```bash
# Interactive REPL
./target/release/acrawl

# One-shot mode
./target/release/acrawl prompt "scrape all book titles and prices from books.toscrape.com"

# Resume a saved session
./target/release/acrawl --resume session.json /status /compact
```

## Examples

**Scrape a product catalog:**

```
acrawl > scrape all book titles, prices, and ratings from books.toscrape.com
```

The agent navigates to the site, reads the page, extracts the data, paginates through all 50 pages, and returns structured JSON.

**Fill and submit a form:**

```
acrawl > go to example.com/contact, fill in name "Jane Doe", email "jane@example.com",
         message "Hello", and submit the form
```

The agent locates form fields, fills them in, clicks submit, and confirms the result.

**Monitor a price:**

```
acrawl > check the current price of "Rust in Action" on books.toscrape.com
```

Single-page extraction — the agent fetches, reads, and returns the price without unnecessary navigation.

**Extract from JS-rendered pages:**

```
acrawl > get all repository names and star counts from github.com/trending
```

Static HTTP won't work here. acrawl detects React/Next.js markers and automatically escalates to a headless browser to render the JavaScript.

**Parallel multi-page crawl:**

```
acrawl > scrape the title, author, and price of every book across all 50 pages on books.toscrape.com.
         Fork a sub-agent for each page to speed this up.
```

The agent spawns up to 5 concurrent sub-agents, each on its own browser tab, to crawl pages in parallel. Results are merged when all sub-agents finish.

## Features

### 42-Tool Toolbox

#### Navigation

| Tool | Description |
|------|-------------|
| `navigate` | Go to a URL (supports `format`: markdown/text/html/fit_markdown). Uses HTTP first, auto-escalates to browser when JS is detected. Returns structured content with a `page_map`. `fit_markdown` prunes boilerplate DOM nodes before conversion, saving tokens. |
| `go_back` | Browser back button. Returns `page_state` with the resulting page structure. |
| `scroll` | Scroll up or down by pixel amount (`pixels`, default: 500). Returns `page_state` after scrolling. |
| `switch_tab` | Switch to a different browser tab by index. Returns `page_state` of the new tab. |
| `wait` | Wait for a CSS selector to reach a given state (`visible`, `hidden`, `attached`, `detached`) or a fixed timeout (up to 300s). Returns `page_state` after the condition is met. |
| `refresh` | Reload the current page. Returns `page_state` after reload. Use after setting intercept rules to replay the page load with rules active. Seq counter increments for temporal observation queries. |

#### Content Formats

The `navigate` tool's `format` parameter controls how the page is returned:

| Format | Description |
|--------|-------------|
| `markdown` | Full HTML → markdown conversion. All content preserved. |
| `fit_markdown` | **Recommended.** Prunes boilerplate before conversion, saving 30-60% tokens on typical pages. |
| `text` | Plain text, no markdown. |
| `html` | Raw HTML. |

`fit_markdown` works in two passes:

1. **Hard-block removal**, elements whose `class` or `id` attribute contains any of these strings are removed immediately: `nav`, `footer`, `header`, `sidebar`, `ads`, `comment`, `promo`, `advert`, `social`, `share`.
2. **Score-based pruning**, remaining elements are scored; anything below 0.48 is removed. The score is:

   ```
   0.4 × text_density
   + 0.2 × (1 − link_density)
   + 0.2 × tag_weight
   + 0.1 × class_id_score
   + 0.1 × ln(text_length + 1)
   ```

   Tag weights: `article` = 1.5 · `h1` = 1.2 · `h2` = 1.1 · `h3/p/section` = 1.0 · `h4` = 0.9 · `h5/table` = 0.8 · `h6` = 0.7 · `span` = 0.3 · `div/li/ul/ol` = 0.5.

**Use `markdown` instead of `fit_markdown` when:** the page has important content inside elements named `sidebar`, `nav`, or similar, for example, metadata panels, related-article links, or author info stored in a sidebar div.

If `fit_markdown` prunes all content (empty result), the tool automatically falls back to plain text.

#### Interaction

| Tool | Description |
|------|-------------|
| `click` | Click an element by CSS selector, `@eN` ref, or visible label text. Use `text` (with optional `role`/`region`) to activate a button, tab, or link by its label — handy for SPA admin UIs and modals where CSS paths are fragile. Returns `page_state` after the click. |
| `click_at` | Click at specific viewport coordinates (x, y). Use for canvas, maps, or SVGs. Returns `page_state`. |
| `fill_form` | Fill form fields by selector, name, `@eN` ref, or visible label text — labels resolve page-wide, so fields in modals and div-based UIs without a `` boundary work too. Optional auto-submit. Returns `page_state`. |
| `select_option` | Select a dropdown option by value, label, or index. Works on native `` and custom ARIA/portal dropdowns; omit value/label/index to list the available options without selecting. Returns `page_state`. |
| `hover` | Hover over an element to reveal tooltips o

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Mingye-Lu](https://github.com/Mingye-Lu)
- **Source:** [Mingye-Lu/AgenticCrawler](https://github.com/Mingye-Lu/AgenticCrawler)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Versions

- **0.10.0** — security scan: pending review — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-mingye-lu-agenticcrawler
- Seller: https://agentstack.voostack.com/s/mingye-lu
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.