Ai Gateway

English | 中文

Ferro Labs AI Gateway

High-performance AI gateway in Go. Route LLM requests across 30 providers via a single OpenAI-compatible API.

Deploy templates

[](https://railway.com/deploy/ferro-labs-ai-sqlite-storage?referralCode=KblxKX&utmmedium=integration&utmsource=template&utmcampaign=generic) [](https://railway.com/deploy/ferro-labs-ai-postgresql-storage?referralCode=KblxKX&utmmedium=integration&utmsource=template&utmcampaign=generic) [](https://render.com/deploy?repo=https://github.com/ferro-labs/ai-gateway)

[](https://go.dev) [](https://pkg.go.dev/github.com/ferro-labs/ai-gateway) [](https://codecov.io/gh/ferro-labs/ai-gateway) [](LICENSE) [](https://github.com/ferro-labs/ai-gateway/stargazers) [](https://github.com/ferro-labs/ai-gateway/actions/workflows/ci.yml) [](https://github.com/ferro-labs/ai-gateway/actions/workflows/code-scanning.yml) [](https://deepwiki.com/ferro-labs/ai-gateway) [](https://artifacthub.io/packages/search?org=ferro-labs) [](https://discord.gg/yCAeYvJeDV)

🔀 30 providers, 2,500+ models — one API ⚡ 13,925 RPS at 1,000 concurrent users 📦 Single binary, zero dependencies, 32 MB base memory

---

Quick Start

Get from zero to first request in under 2 minutes.

Option A — Binary (fastest)

curl -fsSL https://github.com/ferro-labs/ai-gateway/releases/download/v1.0.6/ferrogw_1.0.6_linux_amd64.tar.gz | tar xz
chmod +x ferrogw
./ferrogw init          # generates config.yaml + MASTER_KEY
./ferrogw               # starts the server

Option B — Docker

docker pull ghcr.io/ferro-labs/ai-gateway:latest
docker run -p 8080:8080 \
  -e OPENAI_API_KEY=sk-your-key \
  -e MASTER_KEY=fgw_your-master-key \
  ghcr.io/ferro-labs/ai-gateway:latest

Option C — Go

go install github.com/ferro-labs/ai-gateway/cmd/ferrogw@latest
ferrogw init            # first-run setup
ferrogw                 # start the server

First-time setup

ferrogw init generates a master key and writes a minimal config.yaml:

$ ferrogw init

  Master key (set as MASTER_KEY env var):
  fgw_a3f2e1d4c5b6a7f8e9d0c1b2a3f4e5d6

  Config written to: ./config.yaml

  Next steps:
    export MASTER_KEY=fgw_a3f2e1d4c5b6a7f8e9d0c1b2a3f4e5d6
    export OPENAI_API_KEY=sk-...
    ferrogw

The master key is shown once — store it in your .env file or secret manager. It is never written to disk.

Minimal config

Create config.yaml (or use ferrogw init):

strategy:
  mode: fallback

targets:
  - virtual_key: openai
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]
  - virtual_key: anthropic

aliases:
  fast: gpt-4o-mini
  smart: claude-3-5-sonnet-20241022

First request

export OPENAI_API_KEY=sk-your-key
export MASTER_KEY=fgw_your-master-key   # set by ferrogw init

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Hello from Ferro Labs AI Gateway"}]
  }' | jq

---

Why Ferro Labs

Most AI gateways are Python proxies that crack under load or JavaScript services that eat memory. Ferro Labs AI Gateway is written in Go from the ground up for real-world throughput — a single binary that routes LLM requests with predictable latency and minimal resource usage.

| Feature | Ferro Labs | LiteLLM | Bifrost | Kong AI | |:-----------------|:------------|:--------|:-----------|:------------| | Language | Go | Python | Go | Go/Lua | | Single binary | ✅ | ❌ | ✅ | ❌ | | Providers | 30 | 100+ | 20+ | 10+ | | MCP support | ✅ | ❌ | ✅ | ❌ | | Response cache | ✅ | ✅ | ✅ | ❌ (paid) | | Guardrails | ✅ | ✅ | ❌ | ❌ (paid) | | OSS license | Apache 2.0 | MIT | Apache 2.0 | Apache 2.0 | | Managed cloud | Coming Soon | ✅ | ✅ | ✅ |

---

Performance

Benchmarked against Kong OSS, Bifrost, LiteLLM, and Portkey on GCP n2-standard-8 (8 vCPU, 32 GB RAM) using a 60ms fixed-latency mock upstream — results reflect gateway overhead only.

Ferro Labs Latency Profile

| VU | RPS | p50 | p99 | Memory | |---:|---:|---:|---:|---:| | 50 | 813 | 61.3ms | 64.1ms | 36 MB | | 150 | 2,447 | 61.2ms | 63.4ms | 47 MB | | 300 | 4,890 | 61.2ms | 64.4ms | 72 MB | | 500 | 8,014 | 61.5ms | 72.9ms | 89 MB | | 1,000 | 13,925 | 68.1ms | 111.9ms | 135 MB |

At 1,000 VU: 13,925 RPS, p50 overhead 8.1ms, memory 135 MB. No connection pool failures. No throughput ceiling.

Live Upstream Overhead (OpenAI API)

Measured against live OpenAI API (gpt-4o-mini) using two independent methods: the gateway's X-Gateway-Overhead-Ms response header (precise internal timing) and paired direct-vs-gateway requests (external black-box validation).

| Configuration | Overhead p50 | Overhead p99 | |:---|---:|---:| | No plugins (bare proxy) | 0.002ms (2 microseconds) | 0.03ms | | With plugins (word-filter, max-token, logger, rate-limit) | 0.025ms (25 microseconds) | 0.074ms |

The gateway adds 25 microseconds of processing overhead per request in a typical production configuration. LLM API calls take 500ms-2s — the gateway is 20,000x faster than the provider it proxies.

How to Reproduce

git clone https://github.com/ferro-labs/ai-gateway-performance-benchmarks
cd ai-gateway-performance-benchmarks
make setup && make bench

Full methodology, raw results, and flamegraph analysis: ferro-labs/ai-gateway-performance-benchmarks

---

Features

🔀 Routing

• 8 routing strategies: single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional
• Provider failover with configurable retry policies and status code filters
• Cost-optimized routing can explicitly fallback, skip, or allow providers with unknown catalog prices
• Per-request model aliases (fast → gpt-4o-mini, smart → claude-3-5-sonnet)

🔌 Providers (30)

| OpenAI & Compatible | Anthropic & Google | Cloud & Enterprise | Open Source & Inference | |:---|:---|:---|:---| | OpenAI | Anthropic | AWS Bedrock | Ollama, Ollama Cloud | | Azure OpenAI | Google Gemini | Azure Foundry | Hugging Face | | OpenRouter | Vertex AI | Databricks | Replicate | | DeepSeek | | Cloudflare Workers AI | Together AI | | Perplexity | | | Fireworks | | xAI (Grok) | | | DeepInfra | | Mistral | | | NVIDIA NIM | | Groq | | | SambaNova | | Cohere | | | Novita AI | | AI21 | | | Cerebras | | Moonshot / Kimi | | | Qwen / DashScope |

🛡️ Guardrails & Plugins

• Word/phrase filtering — block sensitive terms before they reach providers
• Token and message limits — enforce maxtokens and maxmessages per request
• Response caching — in-memory cache with configurable TTL and entry limits
• Rate limiting — global RPS plus per-API-key and per-user RPM limits
• Budget controls — per-API-key USD spend tracking with configurable token pricing
• Request logging — structured logs with optional SQLite/PostgreSQL persistence

⚡ Performance

• Per-provider HTTP connection pools with optimized settings
• sync.Pool for JSON marshaling buffers and streaming I/O
• Zero-allocation stream detection, async hook dispatch batching
• Single binary, ~32 MB base memory, linear scaling to 1,000+ VUs

🤖 MCP (Model Context Protocol)

• Agentic tool-call loop — the gateway drives tool_calls automatically
• Streamable HTTP transport (MCP 2025-11-25 spec)
• Tool filtering with allowed_tools and bounded max_call_depth
• Multiple MCP servers with cross-server tool deduplication

📊 Observability

• OpenTelemetry tracing (v1.1.0+) — OTLP gRPC/HTTP exporter, W3C traceparent propagation, GenAI semantic conventions (gen_ai.*) plus ferro.* extensions for cost, routing, MCP, and stream timings; privacy_level enforced on error recording; configurable shutdown_grace
• Prometheus metrics at /metrics
• Deep health checks at /health with per-provider status
• Structured JSON request logging with SQLite/PostgreSQL persistence (trace ID unified across logs, OTel spans, and X-Request-ID response header)
• Admin API with usage stats, request logs, and config history/rollback
• Built-in dashboard UI at /dashboard
• HTTP-level connection tracing with DNS, TLS, and first-byte latency

---

Examples

Integration examples for common use cases are in ferro-labs/ai-gateway-examples:

| Example | Description | |:--------|:------------| | basic | Single chat completion to the first configured provider | | fallback | Fallback strategy — try providers in order with retries | | loadbalance | Weighted load balancing across targets (70/30 split) | | with-guardrails | Built-in word-filter and max-token guardrail plugins | | with-mcp | Local MCP server with tool-calling integration | | embedded | Embed the gateway as an HTTP handler inside an existing server |

---

Configuration

Full annotated example — copy to config.yaml and customize:

# Routing strategy
strategy:
  mode: fallback  # single | fallback | loadbalance | conditional
                  # least-latency | cost-optimized | content-based | ab-test
  # cost-optimized only: fallback (default) | skip | allow
  # unpriced_strategy: fallback

# Provider targets (tried in order for fallback mode)
targets:
  - virtual_key: openai
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]
      initial_backoff_ms: 100
  - virtual_key: anthropic
    retry:
      attempts: 2
  - virtual_key: gemini

# Model aliases — resolved before routing
aliases:
  fast: gpt-4o-mini
  smart: claude-3-5-sonnet-20241022
  cheap: gemini-1.5-flash

# Plugins — executed in order at the configured stage
plugins:
  - name: word-filter
    type: guardrail
    stage: before_request
    enabled: true
    config:
      blocked_words: ["password", "secret"]
      case_sensitive: false

  - name: max-token
    type: guardrail
    stage: before_request
    enabled: true
    config:
      max_tokens: 4096
      max_messages: 50

  - name: rate-limit
    type: guardrail
    stage: before_request
    enabled: true
    config:
      requests_per_second: 100
      key_rpm: 60

  - name: request-logger
    type: logging
    stage: before_request
    enabled: true
    config:
      level: info
      persist: true
      backend: sqlite
      dsn: ferrogw-requests.db

# MCP tool servers (optional)
mcp_servers:
  - name: my-tools
    url: https://mcp.example.com/mcp
    headers:
      Authorization: Bearer ${MY_TOOLS_TOKEN}
    allowed_tools: [search, get_weather]
    max_call_depth: 5
    timeout_seconds: 30

See [config.example.yaml](config.example.yaml) and [config.example.json](config.example.json) for the full template with all options.

---

Observability

Ferro Labs AI Gateway ships first-class OpenTelemetry support in v1.1.0+. When OTel is disabled (the default) the gateway runs with a zero-allocation no-op provider — there is no cost to leaving it off. When you set an OTLP endpoint, every request emits a gateway.request root span with rich GenAI semantic conventions plus Ferro-specific extensions for cost, routing, and stream timings.

Enable in one step

Either set the standard OTel environment variable:

export OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317
ferrogw serve

…or add an observability block to config.yaml:

observability:
  tracing:
    enabled: true
    endpoint: localhost:4317   # or leave blank to read OTEL_EXPORTER_OTLP_ENDPOINT
    protocol: grpc             # grpc | http/protobuf
    service_name: ferrogw
    sample_ratio: 1.0
    privacy_level: metadata    # none | metadata | full  (see below)
    shutdown_grace: 10s        # per OTel shutdown stage; total can take up to 2x this value
    # headers:                        # OTLP export headers for authenticated backends
    #   dd-api-key: "${DATADOG_API_KEY}"  # values support ${ENV_VAR} interpolation

  # exporters wires plugin observability exporters (see "Plugin exporters" below).
  # exporters:
  #   - name: langsmith
  #     enabled: true
  #     config:
  #       api_key: "${LANGSMITH_API_KEY}"

Standard OTEL_* environment variables (e.g. OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_TRACES_SAMPLER) always take precedence over the config file — this matches the OTel SDK convention and is required for predictable container deployments.

observability.tracing.headers lets you send OTLP traces to authenticated managed backends (Datadog, New Relic, Honeycomb, Grafana Cloud) by setting vendor-specific headers such as API keys. Values support ${ENV_VAR} interpolation so secrets are never stored literally in the config file. The standard OTEL_EXPORTER_OTLP_HEADERS environment variable also applies per OTel convention.

The endpoint scheme selects transport security: an https:// endpoint uses TLS, while an http:// endpoint or a bare host:port (e.g. localhost:4317) connects in plaintext. Managed backends require the https:// form.

What gets emitted

The following attributes are currently emitted on the gateway.request root span. Attributes marked "Planned" are reserved but not yet wired.

• gateway.request root span per request (SERVER kind) with gen_ai.system, gen_ai.operation.name, gen_ai.request.model, gen_ai.response.model, gen_ai.usage.{input,output}_tokens
• HTTP {GET,POST} child span per outbound provider call (CLIENT kind, via otelhttp transport wrapping) — propagates traceparent to upstream providers
• ferro.* emitted attributes: ferro.cost.{usd,input_usd,output_usd,cache_read_usd,cache_write_usd,reasoning_usd,model_found}, ferro.routing.{strategy,target_key}, ferro.stream.time_to_{first,last}_token_ms, ferro.gateway.trace_id, ferro.plugin.{name,kind,stage,outcome,reason}, ferro.mcp.{server,tool,latency_ms}
• W3C TraceContext + Baggage propagation: inbound traceparent is honoured; outbound requests carry it forward
• Unified trace ID: the OTel trace_id, the X-Request-ID response header, and the trace_id field on every log line are guaranteed equal per request for all requests served through the gateway's HTTP stack. (Embedders that bypass logging.Middleware receive a consistent-but-independent span trace ID.)

Try it locally with Jaeger

docker run --rm -p 16686:16686 -p 4317:4317 jaegertracing/all-in-one
OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4317 ferrogw serve
# fire a request, then open http://localhost:16686

Privacy levels

privacy_level controls how error messages are recorded on spans. No prompt or response content is exported at any level — that requires a future L3 exporter plugin.

| Level | Error recording on spans | Default | |:------|:------|:------| | none | Status and exception carry only the static string "redacted" — no content or internal type exposed | — | | metadata | Error message is redacted (email / JWT / AWS keys replaced by tokens) before being attached | ✅ | | full | Raw error text recorded without redaction — for trusted self-hosted debugging only | — |

Invalid values are rejected at startup by config validation.

Plugin exporters

The `observabil

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: ferro-labs
• Source: ferro-labs/ai-gateway
• License: Apache-2.0
• Homepage: https://docs.ferrolabs.ai

Install and usage instructions live in the source repository linked above.