Llm Mcp Gateway Rs

MCP LLM Gateway / MCP 大模型网关

An OpenAI Responses API–compatible gateway that integrates MCP servers (local docs/PDF KB + database MCP examples) and works with big-agi.com. 一个 OpenAI Responses API 兼容的网关，集成 MCP servers（本地 docs/PDF 知识库 + 数据库 MCP 示例），并兼容 big-agi.com。

> Provider-agnostic direction > Today the gateway speaks the OpenAI-style Responses API upstream (OpenAI or any compatible endpoint). > The design is intended to be extensible to additional upstream providers (e.g., Claude) via adapters in the future. > > 当前网关上游使用 OpenAI 风格的 Responses API**（OpenAI 或任何兼容 endpoint）。 > 设计上希望未来可通过适配层扩展更多上游（例如 Claude 等）。

---

Overview / 简介

本项目是一个 OpenAI API 风格的网关（重点支持 /v1/responses），并在本地通过 stdio 拉起一个或多个 MCP Server，在需要时执行 tool loop（工具调用循环）：

• Docs/PDF 知识库 MCP（内置）：docs-kb-mcp-rs（Rust），用于检索本地索引（RAG / KB）。
• 数据库 MCP（示例）：以 Oracle MCP 为例演示数据库工具链（schema 查询 + 只读 SQL），但网关本身不绑定特定数据库，未来可扩展更多 MCP server。
• Big-AGI 兼容：对 Responses streaming/事件结构做了更严格的兼容（尤其 function_call schema）。
• OpenAI API 兼容：暴露 /v1/responses、/v1/models 等端点，便于 OpenAI SDK / 兼容客户端对接。

> 注意：目前主要面向 Responses API。如果你的客户端只调用 /v1/chat/completions，需要额外适配或改用支持 Responses 的客户端。

This project is an OpenAI-style gateway (focused on /v1/responses). It spawns one or more MCP servers via stdio and runs a tool loop when needed:

• Docs/PDF KB MCP (bundled): docs-kb-mcp-rs (Rust) for local indexed retrieval (RAG / KB).
• Database MCP (example): Oracle MCP is used as an example integration (schema inspection + read-only SQL), but the gateway is not tied to any single database; more MCP servers can be added later.
• Big-AGI compatible: stricter Responses streaming/event compatibility (especially function_call schema).
• OpenAI API compatible: exposes /v1/responses, /v1/models, etc. for OpenAI SDKs and compatible clients.

> Note: This repo primarily targets the Responses API. If your client only uses /v1/chat/completions, you’ll need additional adaptation or use a Responses-capable client.

---

Features / 功能特性

• OpenAI 风格端点：
• POST /v1/responses
• GET /v1/models
• GET /health
• 三种模式（环境变量默认 + 单请求覆盖）：
• proxy：尽量“原样”透传上游 Responses（流式逐事件透传）
• compat：启用 MCP tools + tool loop（流式为 buffered 兼容输出）
• auto：根据请求特征自动选择 proxy/compat（默认）
• MCP 集成：
• 内置 docs KB MCP（stdio）
• 数据库 MCP（以 Oracle 为例；可扩展更多 MCP server）
• 安全护栏（以数据库 run_sql_query 工具为例）：
• 只允许单条 SELECT/WITH
• 自动包裹 ROWNUM 如果你只运行网关 + 已存在的索引文件，可能不会直接调用 pdftotext`；但为了完整能力（重新索引 PDF），建议安装。

The indexer (kb-indexer-rs) relies on the external pdftotext command (usually from Poppler tools) to extract text from PDFs.

> If you only run the gateway with an existing index, you may not directly need pdftotext. Still recommended for full PDF indexing capability.

---

Repository Layout / 仓库结构

这是一个 Cargo workspace，包含三个 crate：

• crates/gateway-openai：网关服务（HTTP，OpenAI Responses API 兼容 + MCP tool loop）
• crates/docs-kb-mcp-rs：本地 docs KB MCP server（stdio）
• crates/kb-indexer-rs：索引器（把 docs/PDF 等生成索引目录）

This is a Cargo workspace with three crates:

• crates/gateway-openai: gateway server (HTTP, OpenAI Responses compatibility + MCP tool loop)
• crates/docs-kb-mcp-rs: local docs KB MCP server (stdio)
• crates/kb-indexer-rs: index builder (builds the on-disk index directory)

---

Build / 编译

Build release binaries / 构建 release 二进制

cargo build --release

Binaries are typically located at:

• target/release/gateway-openai
• target/release/docs-kb-mcp-rs
• target/release/kb-indexer-rs

---

Configuration / 配置

1) Copy .env.example / 复制 .env.example

cp .env.example .env

2) Key environment variables / 关键环境变量

> 完整示例请看 .env.example。这里只强调最关键的几项。

• OPENAI_API_KEY（必填 / required）
• INDEX_DIR（必填 / required）
• DOCS_MCP_COMMAND（必填 / required）
• BIND_ADDR（建议本地配合 HTTPS 反代时用 127.0.0.1:8000 / recommended with local HTTPS proxy: 127.0.0.1:8000）

---

HTTPS for Big-AGI (Caddy + mkcert) / Big-AGI 使用 HTTPS（Caddy + mkcert）

> 本章节放在 Run / 运行 前面，方便你按顺序配置。 > This section is placed before Run so you can follow setup steps in order.

Why HTTPS is recommended / 为什么推荐 HTTPS

当你在浏览器里使用 big-agi.com（HTTPS 页面）去访问本地网关时，现代浏览器可能会触发以下限制：

• 从 HTTPS 页面访问本地 http://127.0.0.1:8000 可能出现混合内容/安全上下文限制
• Private Network Access（PNA）相关的预检与限制更严格

最稳妥的做法：网关仍然跑 HTTP（绑定 BIND_ADDR=127.0.0.1:8000），再用 Caddy 在本机提供 HTTPS 入口，并用 mkcert 生成并信任自签证书。

When using big-agi.com (an HTTPS page) to call your local gateway, modern browsers may enforce:

• mixed-content / secure-context restrictions when calling http://127.0.0.1:8000 from an HTTPS page
• stricter Private Network Access (PNA) preflight rules

The most reliable approach: keep the gateway on HTTP (bind BIND_ADDR=127.0.0.1:8000), then use Caddy to expose a local HTTPS endpoint, with mkcert providing a trusted local certificate.

---

Step 0: Ensure .env matches the proxy plan / 第 0 步：确认 .env 配置与反代一致

Make sure your .env contains (same as .env.example recommendation):

BIND_ADDR=127.0.0.1:8000
CORS_ALLOW_ORIGINS=https://app.big-agi.com

> 中文：CORS origin 需要允许 Big-AGI 的网页来源。 > English: CORS must allow Big-AGI’s web origin.

---

Step 1: Install mkcert + Caddy / 第 1 步：安装 mkcert + Caddy（macOS 示例）

brew install mkcert caddy

(Optional) If you use Firefox and want it to trust mkcert certs:

brew install nss

---

Step 2: Install the local CA (trust it) / 第 2 步：安装并信任本地 CA

mkcert -install

• 中文：这一步会把 mkcert 的本地 CA 安装到 macOS Keychain 并设为受信任。
• English: This installs mkcert’s local CA into macOS Keychain and marks it as trusted.

---

Step 3: Generate certificates / 第 3 步：生成证书

From your project root:

mkdir -p certs
cd certs
mkcert localhost 127.0.0.1 ::1
cd ..

You should get files similar to:

• certs/localhost+2.pem
• certs/localhost+2-key.pem

> 提醒 / Note > mkcert 的输出文件名可能随 host 数量变化；如果你想固定文件名，可使用 -cert-file / -key-file 参数自行指定。

---

Step 4: Caddy reverse proxy / 第 4 步：Caddy 反代配置

Create a Caddyfile at repo root:

# Listen on https://localhost:8787
https://localhost:8787 {
 tls certs/localhost+2.pem certs/localhost+2-key.pem
 # Reverse proxy to the Rust gateway
 reverse_proxy 127.0.0.1:8000
}

Run Caddy:

caddy run --config Caddyfile

Verify (should NOT require -k once trusted):

curl -s https://localhost:8787/health

---

Security note / 安全说明

• 中文：mkcert 只适合本机/内网开发环境，不建议用于公网生产环境。
• English: mkcert is for local/dev environments, not for public production deployments.

---

Run / 运行

1) Build (if not built) / 1) 编译（若未编译）

cargo build --release

2) Prepare .env / 2) 准备 .env

cp .env.example .env

Edit .env and set at least:

• OPENAI_API_KEY
• INDEX_DIR
• DOCS_MCP_COMMAND

3) Start the gateway / 3) 启动网关

./target/release/gateway-openai

Then you can access:

• GET /health
• GET /v1/models
• POST /v1/responses

If you enabled Caddy HTTPS proxy as above, your browser-friendly endpoint is:

• https://localhost:8787 (via Caddy)
• and the gateway remains on http://127.0.0.1:8000 (internal)

---

Using with Big-AGI (big-agi.com) / 与 Big-AGI 对接（big-agi.com）

在 Big-AGI 的 provider 设置里选择 OpenAI（或兼容 OpenAI 的 provider），然后：

• Base URL：建议填 https://localhost:8787/v1（如果你启用了 Caddy HTTPS 反代）
• API Key：可随意填写（网关默认不校验入站 key；上游访问使用 OPENAI_API_KEY）
• Model：选择 UPSTREAM_MODEL 或 UPSTREAM_MODELS 中的一个

In Big-AGI, choose an OpenAI (or OpenAI-compatible) provider and set:

• Base URL: recommended https://localhost:8787/v1 (if using the Caddy HTTPS proxy)
• API Key: any value (the gateway does not validate inbound keys by default; upstream uses OPENAI_API_KEY)
• Model: pick one from UPSTREAM_MODEL / UPSTREAM_MODELS

---

API Behavior Notes / API 行为说明

Supported endpoints / 支持的端点

• POST /v1/responses
• GET /v1/models
• GET /health

Stream Mode (auto / proxy / compat)

You can set it in two ways:

1) env default: GATEWAY_STREAM_MODE=auto|compat|proxy 2) per-request override: include gateway_stream_mode in the JSON body

Example:

{
  "model": "gpt-5.2",
  "input": "hello",
  "stream": true,
  "gateway_stream_mode": "compat"
}

• proxy mode:
• does not inject MCP tools
• performs request allowlist + input sanitization + optional web_search restrictions + forces store=true
• streaming: passthrough SSE events as-is
• compat mode:
• injects MCP function tools
• runs tool loop (extract function_call → call MCP tool → send function_call_output back upstream)
• streaming: buffered compatibility event sequence (+ keepalive)

---

Database MCP safety / 数据库 MCP 安全建议（泛化）

即使网关做了工具 allowlist、SQL 过滤等限制，仍强烈建议：

• 数据库账号使用只读权限
• 网关与 DB MCP 部署在受控网络（内网/防火墙）
• 如果需要暴露给更多客户端，务必在 Caddy/Nginx 侧做鉴权（Basic Auth / Token / IP allowlist）

Even with allowlists and SQL filtering, strongly recommended:

• use read-only database credentials
• deploy gateway + DB MCP in a controlled network (LAN/firewall)
• enforce auth at Caddy/Nginx if exposed (Basic Auth / token / IP allowlist)

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: eztakesin
• Source: eztakesin/llm-mcp-gateway-rs
• License: MIT

Install and usage instructions live in the source repository linked above.