Librecrawl Technical Seo Audit Mcp

🕷️ librecrawl-technical-seo-audit-mcp

The AI-native technical SEO crawler.

Run a complete on-site SEO audit on any website — straight from Claude, Cursor, Codex, or any Model Context Protocol (MCP) client. Unlimited pages · 50+ checks · PDF + CSVs · MIT-licensed · self-hosted · ephemeral by design.

Built on the open-source LibreCrawl engine, exposed through 37 MCP tools your AI assistant calls directly.

[](LICENSE) [](https://modelcontextprotocol.io) [](https://python.org) [](https://github.com/adityaarsharma/librecrawl-technical-seo-audit-mcp/releases) [](https://github.com/adityaarsharma/librecrawl-technical-seo-audit-mcp/stargazers) [](https://github.com/PhialsBasement/LibreCrawl)

[](https://docs.anthropic.com/claude-code) [](https://claude.ai/download) [](https://cursor.com) [](https://github.com/openai/codex) [](https://codeium.com/windsurf) [](https://continue.dev)

[⚡ Install in 60s](#-install-in-60-seconds) · [🪄 What it does](#-the-whole-pitch-in-4-lines) · [🚀 50+ checks](#-50-checks-every-audit) · [🆚 Compare](#-feature-comparison-to-other-on-site-seo-crawlers) · [📖 Quick start](#-your-first-audit)

---

🤔 Don't know what an MCP is? Read this 30-second explainer

> Model Context Protocol (MCP) is the open standard that lets AI assistants like Claude, Cursor, or Codex call external tools. Think of it as "USB for AI assistants" — you plug a tool in, the AI can use it. librecrawl-technical-seo-audit-mcp is one of those tools. Once installed, you just ask your AI assistant to audit a site, and it does. No GUI. No dashboard. No exports.

New to all this?

• Don't have Claude Code yet? → Install Claude Code (free for individuals).
• Prefer Cursor? → Get Cursor.
• Already have one of those? → Skip to [Install in 60s](#-install-in-60-seconds).

---

🪄 The whole pitch in 4 lines

You:    Audit https://acme.com — full site, no caps, give me the zip
Agent:  → librecrawl_start_chunked_audit · polls until done · saves zip locally
You:    Show me broken pages + broken external links + hreflang errors
Agent:  → reads CSVs, prints filtered tables. Server already forgot the audit.

That's the product. Your AI assistant runs a full technical SEO audit for you. You get a branded PDF + 7 CSVs covering 50+ technical checks, ready to hand a client. The server wipes everything the moment you download.

---

✅ One command = a complete audit (v2.1.1)

No flags to remember, no caps to set. Anyone who runs:

librecrawl_start_chunked_audit(url="https://yoursite.com")

gets a full end-to-end audit by default:

• Every page — entire sitemap crawled, including orphan pages not linked from the homepage
• Every page's text — readability, AI-tells, boilerplate analyzed word-by-word
• Every outbound link — every domain (yours, third-party, social, CDN) HTTP-validated for broken links
• No silent dropping — per-page checks + link validation cover 100% of pages; the report states exactly what was covered
• Heavy sites safe — 4–5 MB pages crawl fine; Screaming-Frog-grade politeness never overloads the origin
• Zero history — the server forgets the entire audit the moment you download the zip; re-scan anytime, nothing persists

Verified on real production sites: 1,942-page and 709-page WordPress sites, full coverage, origins healthy throughout.

> Note: runs one audit at a time (single-tenant backend). For team use, queue audits sequentially — concurrent multi-audit routing is on the roadmap.

---

🔥 Why this exists

There are great desktop SEO crawlers (you know the ones). There are great cloud SEO suites. There was no AI-native crawler. librecrawl-technical-seo-audit-mcp fills that gap with five things no comparable open-source MCP server does:

⚡ It runs inside your AI assistant

37 MCP tools your agent calls directly. No GUI app to babysit, no SaaS dashboard to log into, no CSV exports to upload to ChatGPT. You just ask.

🚀 Chunked-progressive crawler that never times out

Most SEO MCP servers (SiteAudit MCP, AgentAEO, SE Ranking MCP) run synchronously and disconnect on sites over a few hundred pages. librecrawl-technical-seo-audit-mcp runs the crawl in a background worker thread, persists progress to SQLite WAL, and returns a session_id in under 2 seconds. Your agent polls a tiny status tool until done. 10,000-page enterprise sites work the same as 50-page blogs. Survives PM2 / MCP-client restarts mid-crawl.

🛡️ Catches WAF challenges other crawlers silently misreport

Cloudflare, Akamai, DataDome, Imperva, and PerimeterX challenge pages are served as 200 OK but contain a JavaScript challenge instead of your content. Most crawlers report these as "page OK, all good". librecrawl-technical-seo-audit-mcp fingerprints the challenge in the response body and flags bot_block_challenge_detected. You see what's actually broken.

🤖 An AIMD controller tunes crawl delay live

Additive-Increase / Multiplicative-Decrease — the same algorithm TCP congestion control uses. Error rate > 10% → halve chunk, double delay. p95 latency > 1.5× target → 1.5× delay. Clean signals → additive decrease. Polite by construction. No rate-limit blow-ups. No manual tuning. Respects robots.txt Crawl-Delay floor.

🧹 Ephemeral by design — the agency-safe default

Once you download the zip, the server deletes the session row, every artifact file on disk, AND the upstream LibreCrawl crawl record. Per-audit server footprint after cleanup: 0 bytes, 0 rows. Auditing 50 client sites? Zero data persists where another operator could see it.

📄 Branded PDF reports ready to hand a client

WeasyPrint, A4, page numbers, footer on every page. Open in any PDF viewer. No SaaS watermark. Hand it to a client as your work.

---

⚡ Install in 60 seconds

curl -fsSL https://raw.githubusercontent.com/adityaarsharma/librecrawl-technical-seo-audit-mcp/main/install.sh | bash

The installer asks 3 questions (target client, optional Google PageSpeed API key, optional GSC integration) and writes a ready-to-use MCP entry into your Claude / Cursor / Codex / Windsurf config. Done.

What if I'm not a developer?

You don't need to be. If you can:

1. Open a terminal (macOS: Cmd+Space → "Terminal" · Windows: Win+R → "powershell")
2. Paste the curl command above
3. Answer 3 yes/no questions

…you're done. The installer handles Python, Docker, the LibreCrawl backend, and your AI client config. First-audit-to-zip is under 10 minutes from cold start.

Manual install (Python 3.10+, Docker for LibreCrawl backend)

git clone https://github.com/adityaarsharma/librecrawl-technical-seo-audit-mcp.git
cd librecrawl-technical-seo-audit-mcp
python3 -m venv venv && source venv/bin/activate
pip install httpx mcp weasyprint markdown fpdf2
# Start LibreCrawl backend on :5080 (see install.sh for Docker compose)
python server.py

Add to your client config (Claude Desktop example):

{
  "mcpServers": {
    "librecrawl": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "http://127.0.0.1:5081/mcp"]
    }
  }
}

---

🚀 50+ checks every audit

🔒 Security & headers

missing_hsts · missing_csp · missing_x_frame_options · missing_x_content_type_options · missing_referrer_policy · x_robots_tag_vs_meta_mismatch · mixed_content

🛡️ WAF / bot-block detection

bot_block_challenge_detected — fingerprints Cloudflare · Akamai · DataDome · Imperva · PerimeterX

🗺️ Sitemap & robots

sitemap_url_noindex · sitemap_url_3xx · sitemap_url_disallowed_in_robots · sitemap_contains_canonicalized · sitemap_over_50k_urls · sitemap_over_50mb

🌍 Hreflang full audit

missing_return_tag · missing_self_reference · missing_x_default · invalid_codes · to_noindex · to_broken · conflicts_lang_attr

🔗 Canonical health

canonical_chain_depth · canonical_to_relative · canonical_to_redirect · canonical_outside_head · bad_canonical

🔁 Redirects (every flavour)

redirect_chains · meta_refresh_redirect · js_redirect · http_refresh_redirect

🏷️ Schema.org (16 types)

Article · Product · Recipe · FAQPage · BreadcrumbList · Event · JobPosting · VideoObject · HowTo · Organization · LocalBusiness · Person · Review · AggregateRating · Course · NewsArticle — validates schema.org spec AND Google Rich Results required fields. Handles @graph (Yoast / Rank Math / WPRM).

🔤 URL quality

url_contains_space · url_multiple_slashes · url_non_ascii · url_underscores · url_repetitive_path · long_urls · uppercase_urls · url_params_heavy

⚓ Anchor text

non_descriptive_anchor_text · empty_anchor_text · anchor_image_no_alt · broken_bookmarks

🕸️ Internal linking

internal_nofollow_outlinks · nofollow_only_inbound · follow_and_nofollow_mixed · orphan_pages

🖼️ Image performance + CLS

lazy_load_attr_missing · srcset_missing · image_dimensions_missing · next_gen_image_format · image_oversized_kb · missing_alt_pages · broken_img_pages

📐 HTML structure

html_over_2mb · noscript_in_head · broken_or_invalid_html · dom_size_excessive · lorem_ipsum_detected

♿ Accessibility / metadata

iframes_present · iframe_missing_title · missing_favicon · missing_html_lang · invalid_html_lang · missing_charset · missing_viewport

🪤 Crawl-budget killers

spider_trap_calendar · url_session_id_high_entropy · faceted_url_explosion

✍️ Content quality

low_readability (Flesch) · long_sentences · passive_voice_pct · missing_terminal_punctuation · boilerplate_ratio · ai_tell_tokens_found (delve · unlock · seamlessly · leverage) · has_lorem_ipsum

🚨 Dev leaks

outlinks_to_localhost (RFC1918 in production)

🔗 Every outbound URL HEAD/GET-validated into 17 status classes — ok · redirect · forbidden · not_found · timeout · dns_error · ssl_error · connection_refused · etc. Per-target: final URL after redirects, source pages, anchor text, response time, server header.

📈 GSC merge — pull Google Search Console data, call librecrawl_merge_gsc_data(crawl_id, gsc_data). URLs normalised before joining. Emits 4 extra CSVs: per-page-with-gsc · gsc-winners · gsc-losers (high impr + CTR This is a factual feature comparison. Prices were checked at publication and may have changed — see each vendor's site for current pricing. Brand names belong to their respective owners.

| Capability | Desktop crawler (Screaming Frog SEO Spider™)1 | Desktop+cloud crawler (Sitebulb™)2 | Cloud site-audit (Ahrefs™)3 | librecrawl-technical-seo-audit-mcp | |---|:---:|:---:|:---:|:---:| | Pricing model | Free tier (500 URLs) · paid annual licence | Paid monthly subscription | Bundled with main subscription | Free, MIT-licensed, self-hosted | | Page cap | 500 free / unlimited paid | Unlimited | Tiered by subscription plan | ♾️ Unlimited | | Runs inside your AI assistant | ❌ | ❌ | ❌ | ✅ | | Chunked / background crawl (no timeout) | ❌ | ❌ | Cloud only | ✅ | | Auto-adaptive crawl delay (AIMD) | ❌ | Manual | Hidden | ✅ | | WAF / bot-block detection on 200-OK pages | ❌ | ❌ | ❌ | ✅ | | Sitemap-orphan fill (URLs not internally linked) | ❌ | ❌ | ❌ | ✅ | | Ephemeral by default (zero server footprint) | N/A | N/A | N/A | ✅ | | Broken links (4xx/5xx/timeout/DNS/SSL) | ✅ | ✅ | ✅ | ✅ | | Redirect chains with destination | ✅ | ✅ | ✅ | ✅ | | Title / meta / H1 + duplicates | ✅ | ✅ | ✅ | ✅ | | Canonical full audit | ✅ | ✅ | ✅ | ✅ | | Hreflang full audit (incl. return-tag graph) | ✅ | ✅ | Partial | ✅ | | Sitemap full cross-checks | ✅ | ✅ | Partial | ✅ | | Schema.org validation (16 types + Rich Results) | ✅ | ✅ | Partial | ✅ | | Soft-404 fingerprinting | ✅ | ✅ | ✅ | ✅ | | Mixed content (HTTPS → HTTP) | ✅ | ✅ | ✅ | ✅ | | Security headers pack | ✅ | ✅ | Partial | ✅ | | Image performance + CLS | ✅ | ✅ | ✅ | ✅ | | Content quality (Flesch · AI-tells · boilerplate) | ❌ | Partial | ❌ | ✅ | | Crawl-budget traps (calendar · session-id · facets) | ✅ | ✅ | ✅ | ✅ | | Branded PDF report | ❌ | ✅ | ❌ | ✅ | | GSC clicks/impressions merge | Paid add-on | Paid add-on | Native | ✅ | | JavaScript rendering | ✅ | ✅ | Cloud only | 🛣️ Roadmap |

1 Screaming Frog SEO Spider is a trademark of Screaming Frog Ltd, UK. We are not affiliated. 2 Sitebulb is a trademark of Sitebulb Ltd, UK. We are not affiliated. 3 Ahrefs is a trademark of Ahrefs Pte. Ltd., Singapore. We are not affiliated.

Reading guide: if you currently use a paid on-site crawler and your workflow is "crawl → export CSVs → analyse", librecrawl-technical-seo-audit-mcp covers that flow inside your AI assistant for £0 with no page caps. If your workflow depends on JavaScript-rendered SPAs, that's on the [roadmap](#-roadmap) but not shipped yet — use the desktop tool for now.

---

📊 What every audit produces

Single zip, 8 files:

| File | Use | |---|---| | SUMMARY.txt | One-page orientation | | -.pdf | Branded human-readable PDF (open in any viewer) | | -.md | Markdown source of the PDF (grep-friendly) | | per-page.csv | 1 row per URL × 30 columns of check booleans + failed_checks_list | | sitemap-recon.csv | Sitemap-vs-crawl diff | | external-links.csv | Every outbound URL + status | | content-audit.csv | Per-page readability + AI-tells | | extended-checks.csv | 1 row per (URL × check × severity × detail) — all 50+ checks |

---

📖 Your first audit

You:   Audit https://example.com — full site, no caps

Agent: → librecrawl_start_chunked_audit(url=..., total_max_pages=10000)
         returns session_id in  **Not planned:** keyword research, backlink analysis, SERP tracking. Those are different problems with different MCP servers (DataForSEO, etc.). This tool is laser-focused on **technical on-site SEO crawling**.

[Open an issue](https://github.com/adityaarsharma/librecrawl-technical-seo-audit-mcp/issues/new) to bump priorities or request a check.

---

## 🏗️ Architecture

┌─────────────────────────────────────────────────────────────┐ │ MCP client (Claude Code / Desktop / Cursor / Codex …) │ └────────────────────────────┬────────────────────────────────┘ │ streamable HTTP or stdio ▼ ┌─────────────────────────────────────────────────────────────┐ │ librecrawl-technical-seo-audit-mcp wrapper (server.py — FastMCP, 37 tools) │ │ ┌─────────────────┐ ┌──────────────────────────────┐ │ │ │ runner.py │ │ externallinks / schema / │ │ │ │ background │ │ contentaudit / extended │ │ │ │ worker thread │ │ checks / sitemapfill / │ │ │ │ AIMD controller │ │ pdf_report │ │ │ └────────┬────────┘ └──────────────────────────────┘ │ │ │ │ │ ┌────────▼────────┐ ┌──────────────────────────────┐ │ │ │ state.py │ │ libreclient.py — typed │ │ │ │ SQLite WAL │ │ wrapper to upstream API │ │ │ │ session state │ └──────────────┬───────────────┘ │ │ └─────────────────┘ │ │ └─────────────────────────────────────────┼────────────────────┘ │ ▼ ┌──────────────────────────────┐ │ LibreCrawl Flask backend │ │ :5080 — single-tenant │ │ crawls + extracts SEO data │ └──────────────────────────────┘

---

## ⚙️ Configuration

| Env var | Default | Purpose |
|---|---|---|
| `LIBRECR

…

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [adityaarsharma](https://github.com/adityaarsharma)
- **Source:** [adityaarsharma/librecrawl-technical-seo-audit-mcp](https://github.com/adityaarsharma/librecrawl-technical-seo-audit-mcp)
- **License:** MIT
- **Homepage:** https://github.com/adityaarsharma/librecrawl-mcp

Install and usage instructions live in the source repository linked above.