# Spectrawl

> The unified web layer for AI agents. Search (8 engines), stealth browse, auth, and act on 24 platforms. One npm install, self-hosted.

- **Type:** MCP server
- **Install:** `agentstack add mcp-pyx-corp-spectrawl`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Pyx-Corp](https://agentstack.voostack.com/s/pyx-corp)
- **Installs:** 0
- **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Pyx-Corp](https://github.com/Pyx-Corp)
- **Source:** https://github.com/Pyx-Corp/spectrawl
- **Website:** https://www.npmjs.com/package/spectrawl

## Install

```sh
agentstack add mcp-pyx-corp-spectrawl
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Spectrawl

The unified web layer for AI agents. Search, browse, crawl, extract, and act on platforms — one package, self-hosted.

**5,000 free searches/month** via Gemini Grounded Search. Full page scraping, stealth browsing, multi-page crawling, structured extraction, AI browser agent, 24 platform adapters.

## What It Does

AI agents need to interact with the web — searching, browsing pages, crawling sites, logging into platforms, posting content. Today you wire together Playwright + a search API + cookie managers + platform-specific scripts. Spectrawl is one package that does all of it.

```
npm install spectrawl
```

## How It Works

Spectrawl searches via Gemini Grounded Search (Google-quality results), scrapes the top pages for full content, and returns everything to your agent. Your agent's LLM reads the actual sources and forms its own answer — no pre-chewed summaries.

## Quick Start

```bash
npm install spectrawl
export GEMINI_API_KEY=your-free-key  # Get one at aistudio.google.com
```

```js
const { Spectrawl } = require('spectrawl')
const web = new Spectrawl()

// Deep search — returns sources for your agent/LLM to process
const result = await web.deepSearch('how to build an MCP server in Node.js')
console.log(result.sources)   // [{ title, url, content, score }]

// With AI summary (opt-in — uses extra Gemini call)
const withAnswer = await web.deepSearch('query', { summarize: true })
console.log(withAnswer.answer)  // AI-generated answer with [1] [2] citations

// Fast mode — snippets only, skip scraping
const fast = await web.deepSearch('query', { mode: 'fast' })

// Basic search — raw results
const basic = await web.search('query')
```

> **Why no summary by default?** Your agent already has an LLM. If we summarize AND your agent summarizes, you're paying two LLMs for one answer. We return rich sources — your agent does the rest.

## Spectrawl vs Others

| | Tavily | Crawl4AI | Firecrawl | Stagehand | Spectrawl |
|---|---|---|---|---|---|
| Speed | ~2s | ~5s | ~3s | ~3s | ~6-10s |
| Free tier | 1,000/mo | Unlimited | 500/mo | None | 5,000/mo |
| Returns | Snippets + AI | Markdown | Markdown/JSON | Structured | Full page + structured |
| Self-hosted | No | Yes | Yes | Yes | Yes |
| Anti-detect | No | No | No | No | **Yes (Camoufox)** |
| Block detection | No | No | No | No | **8 services** |
| CAPTCHA solving | No | No | No | No | **Yes (Gemini Vision)** |
| Structured extraction | No | No | No | **Yes** | **Yes** |
| NL browser agent | No | No | No | **Yes** | **Yes** |
| Network capturing | No | Yes | No | No | **Yes** |
| Multi-page crawl | No | Yes | Yes | No | **Yes (+ sitemap)** |
| Platform posting | No | No | No | No | **24 adapters** |
| Auth management | No | No | No | No | **Cookie store + refresh** |

## Search

Two modes: **basic search** and **deep search**.

### Basic Search

```js
const results = await web.search('query')
```

Returns raw search results from the engine cascade. Fast, lightweight.

### Deep Search

```js
const results = await web.deepSearch('query', { summarize: true })
```

Full pipeline: query expansion → parallel search → merge/dedup → rerank → scrape top N → optional AI summary with citations.

### Search Engine Cascade

Default cascade: **Gemini Grounded → Tavily → Brave**

Gemini Grounded Search gives you Google-quality results through the Gemini API. Free tier: 5,000 grounded queries/month.

| Engine | Free Tier | Key Required | Default |
|--------|-----------|-------------|---------|
| **Gemini Grounded** | 5,000/month | `GEMINI_API_KEY` | ✅ Primary |
| Tavily | 1,000/month | `TAVILY_API_KEY` | ✅ 1st fallback |
| Brave | 2,000/month | `BRAVE_API_KEY` | ✅ 2nd fallback |
| DuckDuckGo | Unlimited | None | Available |
| Bing | Unlimited | None | Available |
| Serper | 2,500 trial | `SERPER_API_KEY` | Available |
| Google CSE | 100/day | `GOOGLE_CSE_KEY` | Available |
| Jina Reader | Unlimited | None | Available |
| SearXNG | Unlimited | Self-hosted | Available |

### Deep Search Pipeline

```
Query → Gemini Grounded + DDG (parallel)
  → Merge & deduplicate (12-19 results)
  → Source quality ranking (boost GitHub/SO/Reddit, penalize SEO spam)
  → Parallel scraping (Jina → readability → Playwright fallback)
  → Returns sources to your agent (AI summary opt-in with summarize: true)
```

### What you get without any keys

DDG-only search, raw results, no AI answer. Works from home IPs. Datacenter IPs get rate-limited by DDG — recommend at minimum a free Gemini key.

## Browse

Stealth browsing with anti-detection. Three tiers (auto-escalation):

1. **Playwright + stealth plugin** — default, works immediately
2. **Camoufox binary** — engine-level anti-fingerprint (`npx spectrawl install-stealth`)
3. **Remote Camoufox** — for existing deployments

If tier 1 gets blocked, Spectrawl automatically escalates to tier 2 (if installed) or tier 3 (if configured). No manual intervention needed.

### Browse Options

```js
const page = await web.browse('https://example.com', {
  screenshot: true,    // Take a PNG screenshot
  fullPage: true,      // Full page screenshot (not just viewport)
  html: true,          // Return raw HTML alongside markdown
  stealth: true,       // Force stealth mode
  camoufox: true,      // Force Camoufox engine
  noCache: true,       // Bypass cache
  auth: 'reddit'       // Use stored auth cookies for this platform
})
```

### Browse Response

```js
{
  content: "# Page Title\n\nExtracted markdown content...",
  url: "https://example.com",
  title: "Page Title",
  statusCode: 200,
  cached: false,
  engine: "camoufox",            // which engine was used
  screenshot: Buffer,        // PNG buffer (JS) or base64 (HTTP)
  html: "...",       // raw HTML (if html: true)
  blocked: false,                 // true if block page detected
  blockInfo: null                 // { type: 'cloudflare', detail: '...' }
}
```

### Block Page Detection

Spectrawl detects block/challenge pages from **8 anti-bot services** and reports them in the response instead of returning garbage HTML:

- **Cloudflare** (including RFC 9457 structured errors)
- **Akamai**
- **AWS WAF**
- **Imperva / Incapsula**
- **DataDome**
- **PerimeterX / HUMAN**
- **hCaptcha** challenges
- **reCAPTCHA** challenges
- Generic bot detection (403, "access denied", etc.)

When a block is detected, the response includes `blocked: true` and `blockInfo: { type, detail }`.

### Site-Specific Fallbacks

Some sites block all datacenter IPs regardless of stealth. Spectrawl automatically routes these through alternative APIs:

| Site | Problem | Fallback | Cost |
|------|---------|----------|------|
| **Reddit** | Blocks all datacenter IPs | [PullPush API](https://api.pullpush.io) — Reddit archive | Free |
| **Amazon** | CAPTCHA wall on product pages | [Jina Reader](https://r.jina.ai) — server-side rendering | Free |
| **X/Twitter** | Login wall on posts | [xAI Responses API](https://docs.x.ai) with `x_search` | ~$0.06/post |
| **LinkedIn** | HTTP 999, IP fingerprinting | Requires residential proxy (see below) | ~$7/GB |

These fallbacks activate automatically — just `browse()` the URL and Spectrawl picks the right path. No config needed for Reddit and Amazon. X requires `XAI_API_KEY` env var. LinkedIn requires a residential proxy.

#### LinkedIn: Why It's Different

LinkedIn fingerprints the IP where cookies were created. Even valid cookies get rejected from a different IP. Every free approach fails from datacenter servers:

- Direct browse: HTTP 999
- Voyager API with cookies: 401 (IP mismatch)
- Jina Reader: empty response
- Facebook/Googlebot UA: 317K of CSS, zero content

**The only working solution is a residential proxy.** We recommend [Bright Data](https://brightdata.com) for best results (72M+ residential IPs, ~99.7% success rate, dedicated social media unlockers). For budget use, [Smartproxy](https://smartproxy.com) ($7/GB, 55M IPs, 3-day free trial) works well at lower cost.

Setup:
```bash
# Bright Data (recommended)
npx spectrawl config set proxy '{"host":"brd.superproxy.io","port":22225,"username":"YOUR_ZONE_USER","password":"YOUR_PASS"}'

# Smartproxy (budget alternative)
npx spectrawl config set proxy '{"host":"gate.smartproxy.com","port":10001,"username":"YOUR_USER","password":"YOUR_PASS"}'

# Store your LinkedIn cookies (export from browser)
npx spectrawl login linkedin --account yourname --cookies ./linkedin-cookies.json

# Now browse LinkedIn normally
curl localhost:3900/browse -d '{"url":"https://www.linkedin.com/in/someone"}'
```

Other residential proxy providers that work:
- [IPRoyal](https://iproyal.com) — $7/GB, 32M IPs
- [Bright Data](https://brightdata.com) — premium quality, higher cost
- [Oxylabs](https://oxylabs.io) — enterprise-grade

> ⚠️ **Avoid WebShare** — recycled datacenter IPs marketed as residential, no HTTPS support.

### CAPTCHA Solving

Built-in CAPTCHA solver using **Gemini Vision** (free tier: 1,500 req/day):

- ✅ Image CAPTCHAs
- ✅ Text/math CAPTCHAs
- ✅ Simple visual challenges
- ❌ reCAPTCHA v2/v3 (requires token solving services)
- ❌ hCaptcha (requires token solving services)
- ❌ Cloudflare Turnstile (requires token solving services)

The solver automatically detects CAPTCHA type and attempts resolution before returning the page.

## Extract — Structured Data Extraction

Pull structured data from any page using LLM + optional CSS/XPath selectors. Like Stagehand's `extract()` but self-hosted and integrated with Spectrawl's anti-detect browsing.

### Basic Extraction

```js
const result = await web.extract('https://news.ycombinator.com', {
  instruction: 'Extract the top 3 story titles and their point counts',
  schema: {
    type: 'object',
    properties: {
      stories: {
        type: 'array',
        items: {
          type: 'object',
          properties: {
            title: { type: 'string' },
            points: { type: 'number' }
          }
        }
      }
    }
  }
})
// result.data = { stories: [{ title: "...", points: 210 }, ...] }
```

### HTTP API

```bash
curl -X POST http://localhost:3900/extract \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "instruction": "Extract the page title and main heading",
    "schema": {"type": "object", "properties": {"title": {"type": "string"}, "heading": {"type": "string"}}}
  }'
```

Response:
```json
{
  "data": { "title": "Example Domain", "heading": "Example Domain" },
  "url": "https://example.com",
  "title": "Example Domain",
  "contentLength": 129,
  "duration": 679
}
```

### Targeted Extraction with Selectors

Narrow extraction scope using CSS or XPath selectors — reduces tokens and improves accuracy:

```js
const result = await web.extract('https://news.ycombinator.com', {
  instruction: 'Extract all story titles',
  selector: '.titleline',  // CSS selector
  // or: selector: 'xpath=//table[@class="itemlist"]'
  schema: { type: 'object', properties: { titles: { type: 'array', items: { type: 'string' } } } }
})
```

### Relevance Filtering (BM25)

For large pages, filter content by relevance before sending to the LLM — saves tokens:

```js
const result = await web.extract('https://en.wikipedia.org/wiki/Node.js', {
  instruction: 'Extract the creator and release date',
  relevanceFilter: true   // BM25 scoring keeps only relevant sections
})
// Content reduced from 50K+ chars to ~2K relevant chars
```

### Extract from Content (No Browsing)

Already have the content? Skip the browse step:

```js
const result = await web.extractFromContent(markdownContent, {
  instruction: 'Extract all email addresses',
  schema: { type: 'object', properties: { emails: { type: 'array', items: { type: 'string' } } } }
})
```

Uses Gemini Flash (free) by default. Falls back to OpenAI if configured.

## Agent — Natural Language Browser Actions

Control a browser with natural language. Navigate, click, type, scroll — the LLM interprets the page and decides what to do.

```js
const result = await web.agent('https://example.com', 'click the More Information link', {
  maxSteps: 5,       // max actions to take
  screenshot: true   // screenshot after completion
})
// result.success = true
// result.url = "https://www.iana.org/domains/reserved" (navigated!)
// result.steps = [{ step: 1, action: "click", elementIdx: 0, result: "clicked" }, ...]
```

### HTTP API

```bash
curl -X POST http://localhost:3900/agent \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "instruction": "click the More Information link", "maxSteps": 3}'
```

Response:
```json
{
  "success": true,
  "url": "https://www.iana.org/domains/reserved",
  "title": "IANA — Reserved Domains",
  "steps": [
    { "step": 1, "action": "click", "elementIdx": 0, "reason": "clicking the More Information link", "result": "clicked" }
  ],
  "content": "...",
  "duration": 5200
}
```

### Supported Actions

The agent can: **click**, **type** (fill inputs), **select** (dropdowns), **press** (keyboard keys), **scroll** (up/down).

## Network Request Capturing

Capture XHR/fetch requests made by a page during browsing — useful for discovering hidden APIs:

```js
const result = await web.browse('https://example.com', {
  captureNetwork: true,
  captureNetworkHeaders: true,  // include request headers
  captureNetworkBody: true      // include response bodies ( **Note:** Screenshots bypass the cache — each request renders a fresh page.

## Crawl

Multi-page website crawler with automatic RAM-based parallelization.

```js
const result = await web.crawl('https://docs.example.com', {
  depth: 2,              // how many link levels to follow
  maxPages: 50,          // stop after N pages
  format: 'markdown',    // 'markdown' or 'html'
  scope: 'domain',       // 'domain' | 'subdomain' | 'path'
  concurrency: 'auto',   // auto-detect from available RAM, or set a number
  merge: true,           // merge all pages into one document
  includePatterns: [],   // regex patterns to include
  excludePatterns: [],   // regex patterns to skip
  delay: 300,            // ms between batch launches (politeness)
  stealth: true          // use anti-detect browsing
})
```

### Crawl Response

```js
{
  pages: [
    { url: 'https://docs.example.com/', content: '...', title: '...', statusCode: 200 },
    { url: 'https://docs.example.com/guide', content: '...', title: '...', statusCode: 200 },
    // ...
  ],
  stats: {
    pagesScraped: 23,
    duration: 45000,
    concurrency: 4
  }
}
```

### Sitemap-Based Crawling

Spectrawl auto-discovers `sitemap.xml` and pre-seeds the crawl queue — much faster than link-following for documentation sites:

```js
const result = await web.crawl('https://docs.example.com', {
  useSitemap: true,  // enabled by default
  maxPages: 20
})
// [crawl] Found sitemap at https://docs.example.com/sitemap.xml with 82 URLs
// [crawl] Pre-seeded 20 URLs from sitemap
```

Set `useSitemap: false` to disable and rely only on link discovery.

### Webhook Notifications

Get notified when a crawl completes:

```bash
curl -X POST http://localhost:3900/crawl \
  -d '{"url": "https://docs.example.com", "maxPages": 50, "webhook": "https://your-server.com/webhook"}'
```

Spectrawl will POST the full crawl result to your webhook URL when finished.

### Async Crawl Jobs

For large sites, use async mode to avoid HTTP timeouts:

```bash
# Start a crawl job (returns immediately)
curl -X POST http://localhost:3900/crawl \
  -d '{"url": "https://docs.example.com", "depth": 3, "maxPages": 100, "async": true}'
# Response: { "jobId": "abc123", "status": "running" }

# Check job status
curl http://localhost:3900/crawl/abc123

# List all jobs
curl http://localhost:3900/crawl/jobs

# Check system capacity
curl http://localhost:3900/crawl/capacity
```

### RAM-Based Auto-Parallelization

Spectrawl estimates ~250MB per browser tab and calculates safe concurrency from available system RAM:

- **8GB server:** ~4 concurrent tabs
- **16GB server:** ~8 concurrent tabs
- **32GB server:** 10 concurrent tabs (capped)

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Pyx-Corp](https://github.com/Pyx-Corp)
- **Source:** [Pyx-Corp/spectrawl](https://github.com/Pyx-Corp/spectrawl)
- **License:** MIT
- **Homepage:** https://www.npmjs.com/package/spectrawl

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** yes
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-pyx-corp-spectrawl
- Seller: https://agentstack.voostack.com/s/pyx-corp
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.