Palena Websearch Mcp

Palena WebSearch MCP

Enterprise-grade web search for AI — with compliance boundaries built in.

[](LICENSE) [](https://go.dev/) [](https://modelcontextprotocol.io/) [](https://opentelemetry.io/)

Palena WebSearch MCP is a Go MCP server that turns the noisy public web into LLM-ready context — searched, scraped, de-PII'd, reranked, and hash-chained for audit — so your agents can browse without tripping every compliance officer on the floor.

Built for fintech, healthtech, and govtech teams who cannot send raw third-party HTML directly to a foundation model, but still need their AI to have fresh, cited, reproducible knowledge from the live internet.

> Palena — Hawaiian for boundary, limit, border.

---

Why Palena

| Problem | What most "web search" tools do | What Palena does | |---|---|---| | Raw pages leak PII into prompts | Ship HTML as-is | Presidio audit/redact/block before the LLM sees it | | Bot-protected sites return nothing | Fail silently | Tiered L0→L1→L2 escalation (readability → headless → stealth+proxy) | | "We scraped this" has no receipts | No trace | Three-stage SHA-256 hash chain per document | | Results are ranked by a generic search engine | Trust the top 10 | Pluggable reranker (KServe GPU, FlashRank CPU, RankLLM, or none) | | Compliance only gets shown a demo | Hope for the best | Domain allow/blocklists, robots.txt enforcement, per-domain rate limits, OTel traces on every span |

---

The Pipeline

                  MCP tools/call web_search
                            │
                            ▼
 ┌──────────────────────────────────────────────────────────────┐
 │  1. SEARCH      SearXNG metasearch                           │
 │                 query expansion · engine routing · dedup     │
 ├──────────────────────────────────────────────────────────────┤
 │  2. POLICY      domain allow/block · robots.txt · rate limit │
 ├──────────────────────────────────────────────────────────────┤
 │  3. SCRAPE      L0  HTTP + readability       (fast path)     │
 │                 L1  Playwright headless      (needs JS)      │
 │                 L2  Playwright + stealth/proxy (anti-bot)    │
 ├──────────────────────────────────────────────────────────────┤
 │  4. PII         Microsoft Presidio · audit / redact / block  │
 ├──────────────────────────────────────────────────────────────┤
 │  5. RERANK      FlashRank · KServe cross-encoder · RankLLM   │
 ├──────────────────────────────────────────────────────────────┤
 │  6. FORMAT      markdown · citations · SHA-256 provenance    │
 └──────────────────────────────────────────────────────────────┘
                            │
                            ▼
                     MCP tool_result

Every stage is an OpenTelemetry span. Every document gets three hashes (raw HTML → extracted markdown → final delivered content). The audit trail is ready to ship to ClickHouse out of the box.

---

Live Example

Real output from the MCP server answering "latest AI hot topics 2026" (category: news):

{
  "query": "latest AI hot topics 2026",
  "result_count": 3,
  "reranker_used": "flashrank",
  "pii_mode": "audit",
  "pii_checked": true,
  "search_engines": ["google news", "duckduckgo", "bing news"],
  "total_duration_ms": 4414,
  "results": [
    { "title": "OpenClaw Exposes the Real Cybersecurity Risks of Agentic AI",
      "score": 0.984, "scraper_level": 0,
      "content_hash": "c3b4d690…" },
    { "title": "Comprehensive AI Conference List for 2026: Dates, Locations, and Keynotes",
      "score": 0.973, "scraper_level": 0,
      "content_hash": "7af1205e…" },
    { "title": "Can AI infrastructure costs be a value driver? — Oracle AI World 2026",
      "score": 0.002, "scraper_level": 0,
      "content_hash": "9e48c311…" }
  ]
}

Note how FlashRank pushes the substantive pieces to the top (0.98 / 0.97) and demotes the vendor-conference writeup that happens to contain the right keywords (0.002) — signal the stock SearXNG ranking does not surface.

---

Quick Start

Full stack (all sidecars)

docker compose -f deploy/docker-compose.yml up --build

Brings up Palena + SearXNG + Presidio Analyzer + Presidio Anonymizer + Playwright + FlashRank.

Minimal (L0 scraping only)

docker compose -f deploy/docker-compose.minimal.yml up --build

Palena + SearXNG only. ~200 MB image, no browser, no PII, no reranking.

Smoke-test the server

# Sidecar health
curl http://localhost:8080/health
# → {"status":"ok","sidecars":{"searxng":"ok","presidio-analyzer":"ok",...}}

# List the exposed tool over MCP Streamable HTTP
SESS=$(curl -sS -X POST http://localhost:8080/mcp \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"curl","version":"0.1"}}}' \
  -D - -o /dev/null | awk '/Mcp-Session-Id/ {print $2}' | tr -d '\r')

curl -sS -X POST http://localhost:8080/mcp \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -H "Mcp-Session-Id: $SESS" \
  -d '{"jsonrpc":"2.0","method":"notifications/initialized"}'

curl -sS -X POST http://localhost:8080/mcp \
  -H 'content-type: application/json' \
  -H 'accept: application/json, text/event-stream' \
  -H "Mcp-Session-Id: $SESS" \
  -d '{"jsonrpc":"2.0","id":2,"method":"tools/list"}'

---

MCP Client Integration

LibreChat

# librechat.yaml
mcpServers:
  palena:
    type: streamableHttp      # or: sse
    url: http://palena:8080/mcp

Claude Desktop

{
  "mcpServers": {
    "palena": { "url": "http://localhost:8080/sse" }
  }
}

Cursor / Windsurf / custom agents

Any MCP-compatible client works — connect to /sse (legacy event stream) or /mcp (Streamable HTTP).

---

The web_search Tool

{
  "name": "web_search",
  "inputSchema": {
    "type": "object",
    "properties": {
      "query":      { "type": "string",  "description": "The search query" },
      "category":   { "type": "string",  "description": "general | news | code | science (default general)" },
      "language":   { "type": "string",  "description": "ISO code — en, de, fr… (default en)" },
      "timeRange":  { "type": "string",  "description": "day | week | month | year" },
      "maxResults": { "type": "integer", "description": "1–20 (default 5)" }
    },
    "required": ["query"]
  }
}

The response is formatted markdown with numbered results, relevance scores, source citations, and structured metadata (PII status, reranker, engines hit, total latency).

---

Configuration

Start from the annotated example:

cp config/palena.example.yaml config/palena.yaml

| Section | Controls | Key env override | |---|---|---| | search | SearXNG URL, engines per category, default language | PALENA_SEARCH_SEARXNG_URL | | scraper | Concurrency, timeouts, Playwright WS endpoint, proxy pool | PALENA_SCRAPER_PLAYWRIGHT_ENDPOINT | | pii | Mode (audit / redact / block), Presidio URLs, entities | PALENA_PII_MODE | | reranker | Provider (kserve / flashrank / rankllm / none) | PALENA_RERANKER_PROVIDER | | policy | Domain allow/blocklists, robots.txt cache, rate limits | PALENA_POLICY_DOMAIN_MODE | | provenance | Hash chain, ClickHouse export | PALENA_PROVENANCE_ENABLED | | otel | Trace + metric exporters | PALENA_OTEL_ENABLED |

Full annotated reference: [config/palena.example.yaml](config/palena.example.yaml) · deep-dive: [Configuration guide](docs/configuration.md).

---

Sidecar Matrix

Palena is a Go orchestrator. Every external capability runs as its own container and is optional except SearXNG.

| Sidecar | Image | Protocol | Required | Purpose | |---|---|---|---|---| | SearXNG | searxng/searxng | HTTP JSON | Yes | Metasearch aggregation | | Presidio Analyzer | mcr.microsoft.com/presidio-analyzer | HTTP JSON | No | PII entity detection | | Presidio Anonymizer | mcr.microsoft.com/presidio-anonymizer | HTTP JSON | No | PII masking / replacement | | Playwright | mcr.microsoft.com/playwright | Playwright WS | No | JS-rendered page extraction (L1/L2) | | FlashRank | Flask + ONNX (bundled) | HTTP JSON | No | CPU cross-encoder reranking | | KServe | Your own InferenceService | HTTP JSON | No | GPU cross-encoder reranking |

Missing sidecars trigger graceful degradation, not failure — L1/L2 disabled, PII reported as "not checked," reranker falls back to search-engine order.

Deployment Profiles

| Profile | Components | Footprint | |---|---|---| | Minimal | Palena + SearXNG | ~200 MB | | Standard | + Presidio + Playwright + FlashRank | ~2–3 GB | | Enterprise | + KServe GPU reranker (e.g. mxbai-rerank) | Variable (GPU) |

Helm (Kubernetes / OpenShift)

# Standard
helm install palena deploy/helm/palena-websearch-mcp/

# Minimal — disable all optional sidecars
helm install palena deploy/helm/palena-websearch-mcp/ \
  --set presidio.enabled=false \
  --set playwright.enabled=false \
  --set flashrank.enabled=false

# With your own GPU reranker
helm install palena deploy/helm/palena-websearch-mcp/ \
  --set reranker.provider=kserve \
  --set reranker.endpoint=http://mxbai-rerank.kserve.svc:8080

---

HTTP Endpoints

| Method | Path | Description | |---|---|---| | GET | /sse | MCP SSE transport (legacy event stream) | | POST | /mcp | MCP Streamable HTTP transport | | GET | /health | Sidecar reachability probe | | GET | /metrics | Prometheus metrics (when OTel metrics enabled) |

---

Project Layout

palena-websearch-mcp/
├── cmd/palena/           # Binary entry point, config + server wiring
├── internal/
│   ├── config/           # YAML parsing, validation, env overrides
│   ├── search/           # SearXNG client, query expansion, dedup
│   ├── scraper/          # L0 readability · L1/L2 Playwright · stealth · proxy pool
│   ├── pii/              # Presidio client, policy modes, PII-free audit records
│   ├── reranker/         # Pluggable interface · KServe · FlashRank · RankLLM · no-op
│   ├── policy/           # Domain filter, robots.txt, per-domain rate limit
│   ├── output/           # Markdown formatting, provenance hash chain
│   ├── transport/        # MCP SSE + Streamable HTTP, tool handler
│   └── otel/             # Tracing + metrics setup
├── config/               # Default + annotated example YAML
├── deploy/               # Dockerfile, Compose (full + minimal), Helm chart
└── docs/                 # User documentation

---

Documentation

Full user documentation lives in [docs/](docs/README.md).

| Topic | Where to read | |---|---| | Start here | [Getting Started](docs/getting-started.md) · [Concepts](docs/concepts.md) · [FAQ](docs/faq.md) | | Using the server | [Tool Reference](docs/tool-reference.md) · [Integrations](docs/integrations.md) · [Configuration](docs/configuration.md) | | Pipeline stages | [Scraping](docs/scraping.md) · [PII & Compliance](docs/pii-and-compliance.md) · [Reranking](docs/reranking.md) · [Provenance](docs/provenance.md) | | Running in production | [Deployment](docs/deployment.md) · [Observability](docs/observability.md) |

---

Technology Stack

• Core: Go 1.26+, single static binary
• Search: SearXNG (self-hosted metasearch)
• Extraction: go-shiori/go-readability (L0), playwright-community/playwright-go against Microsoft's official Playwright image (L1/L2)
• PII: Microsoft Presidio Analyzer + Anonymizer
• Reranking: Mixedbread mxbai-rerank via KServe · FlashRank ONNX/CPU · RankLLM against any inference endpoint
• Observability: OpenTelemetry traces + Prometheus-compatible metrics
• Transport: MCP SSE, MCP Streamable HTTP
• Config: YAML + environment variable overrides
• Deploy: Docker Compose (dev), Helm chart (Kubernetes / OpenShift)

---

License

Copyright © 2026 bitkaio LLC.

Licensed under the [Apache License, Version 2.0](LICENSE). You may use, modify, and redistribute Palena under the terms of that license. For commercial support, custom reranker models, or production SLAs, contact bitkaio LLC.

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: PalenaAI
• Source: PalenaAI/palena-websearch-mcp
• License: Apache-2.0
• Homepage: https://websearch.palena.ai/

Install and usage instructions live in the source repository linked above.