# Llm Mcp Gateway Rs

> Rust LLM gateway compatible with the OpenAI Responses API, integrating MCP (local Docs/PDF KB + extensible tool/database MCP servers) with a tool-loop, and compatible with Big-AGI. Rust 实现的 LLM 网关：兼容 OpenAI Responses API，集成 MCP（本地 Docs/PDF 知识库 + 可扩展的工具/数据库 MCP），支持 tool loop，并适配 Big-AGI。

- **Type:** MCP server
- **Install:** `agentstack add mcp-eztakesin-llm-mcp-gateway-rs`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [eztakesin](https://agentstack.voostack.com/s/eztakesin)
- **Installs:** 0
- **Category:** [Databases](https://agentstack.voostack.com/c/databases)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [eztakesin](https://github.com/eztakesin)
- **Source:** https://github.com/eztakesin/llm-mcp-gateway-rs

## Install

```sh
agentstack add mcp-eztakesin-llm-mcp-gateway-rs
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# MCP LLM Gateway / MCP 大模型网关

**An OpenAI Responses API–compatible gateway that integrates MCP servers (local docs/PDF KB + database MCP examples) and works with `big-agi.com`.**
**一个 OpenAI Responses API 兼容的网关，集成 MCP servers（本地 docs/PDF 知识库 + 数据库 MCP 示例），并兼容 `big-agi.com`。**

> **Provider-agnostic direction
> Today the gateway speaks the **OpenAI-style Responses API** upstream (OpenAI or any compatible endpoint).
> The design is intended to be extensible to additional upstream providers (e.g., Claude) via adapters in the future.
>
> 当前网关上游使用 **OpenAI 风格的 Responses API**（OpenAI 或任何兼容 endpoint）。
> 设计上希望未来可通过适配层扩展更多上游（例如 Claude 等）。

---

## Overview / 简介

本项目是一个 **OpenAI API 风格的网关**（重点支持 `/v1/responses`），并在本地通过 **stdio 拉起一个或多个 MCP Server**，在需要时执行 **tool loop（工具调用循环）**：

- **Docs/PDF 知识库 MCP（内置）**：`docs-kb-mcp-rs`（Rust），用于检索本地索引（RAG / KB）。
- **数据库 MCP（示例）**：以 Oracle MCP 为例演示数据库工具链（schema 查询 + 只读 SQL），但网关本身不绑定特定数据库，未来可扩展更多 MCP server。
- **Big-AGI 兼容**：对 Responses streaming/事件结构做了更严格的兼容（尤其 `function_call` schema）。
- **OpenAI API 兼容**：暴露 `/v1/responses`、`/v1/models` 等端点，便于 OpenAI SDK / 兼容客户端对接。

> 注意：目前主要面向 **Responses API**。如果你的客户端只调用 `/v1/chat/completions`，需要额外适配或改用支持 Responses 的客户端。

This project is an **OpenAI-style gateway** (focused on `/v1/responses`). It spawns **one or more MCP servers via stdio** and runs a **tool loop** when needed:

- **Docs/PDF KB MCP (bundled)**: `docs-kb-mcp-rs` (Rust) for local indexed retrieval (RAG / KB).
- **Database MCP (example)**: Oracle MCP is used as an example integration (schema inspection + read-only SQL), but the gateway is not tied to any single database; more MCP servers can be added later.
- **Big-AGI compatible**: stricter Responses streaming/event compatibility (especially `function_call` schema).
- **OpenAI API compatible**: exposes `/v1/responses`, `/v1/models`, etc. for OpenAI SDKs and compatible clients.

> Note: This repo primarily targets the **Responses API**. If your client only uses `/v1/chat/completions`, you’ll need additional adaptation or use a Responses-capable client.

---

## Features / 功能特性

- OpenAI 风格端点：
  - `POST /v1/responses`
  - `GET /v1/models`
  - `GET /health`
- 三种模式（环境变量默认 + 单请求覆盖）：
  - **proxy**：尽量“原样”透传上游 Responses（流式逐事件透传）
  - **compat**：启用 MCP tools + tool loop（流式为 buffered 兼容输出）
  - **auto**：根据请求特征自动选择 proxy/compat（默认）
- MCP 集成：
  - 内置 docs KB MCP（stdio）
  - 数据库 MCP（以 Oracle 为例；可扩展更多 MCP server）
- 安全护栏（以数据库 `run_sql_query` 工具为例）：
  - 只允许单条 `SELECT/WITH`
  - 自动包裹 `ROWNUM  如果你只运行网关 + 已存在的索引文件，可能不会直接调用 `pdftotext`；但为了完整能力（重新索引 PDF），建议安装。

The indexer (`kb-indexer-rs`) relies on the external `pdftotext` command (usually from Poppler tools) to extract text from PDFs.

> If you only run the gateway with an existing index, you may not directly need `pdftotext`. Still recommended for full PDF indexing capability.

---

## Repository Layout / 仓库结构

这是一个 Cargo workspace，包含三个 crate：

- `crates/gateway-openai`：网关服务（HTTP，OpenAI Responses API 兼容 + MCP tool loop）
- `crates/docs-kb-mcp-rs`：本地 docs KB MCP server（stdio）
- `crates/kb-indexer-rs`：索引器（把 docs/PDF 等生成索引目录）

This is a Cargo workspace with three crates:

- `crates/gateway-openai`: gateway server (HTTP, OpenAI Responses compatibility + MCP tool loop)
- `crates/docs-kb-mcp-rs`: local docs KB MCP server (stdio)
- `crates/kb-indexer-rs`: index builder (builds the on-disk index directory)

---

## Build / 编译

### Build release binaries / 构建 release 二进制

```bash
cargo build --release
```

Binaries are typically located at:

- `target/release/gateway-openai`
- `target/release/docs-kb-mcp-rs`
- `target/release/kb-indexer-rs`

---

## Configuration / 配置

### 1) Copy `.env.example` / 复制 `.env.example`

```bash
cp .env.example .env
```

### 2) Key environment variables / 关键环境变量

> 完整示例请看 `.env.example`。这里只强调最关键的几项。

- `OPENAI_API_KEY`（必填 / required）
- `INDEX_DIR`（必填 / required）
- `DOCS_MCP_COMMAND`（必填 / required）
- `BIND_ADDR`（建议本地配合 HTTPS 反代时用 `127.0.0.1:8000` / recommended with local HTTPS proxy: `127.0.0.1:8000`）

---

## HTTPS for Big-AGI (Caddy + mkcert) / Big-AGI 使用 HTTPS（Caddy + mkcert）

> 本章节放在 **Run / 运行** 前面，方便你按顺序配置。  
> This section is placed before **Run** so you can follow setup steps in order.

### Why HTTPS is recommended / 为什么推荐 HTTPS

当你在浏览器里使用 `big-agi.com`（HTTPS 页面）去访问本地网关时，现代浏览器可能会触发以下限制：

- 从 HTTPS 页面访问本地 `http://127.0.0.1:8000` 可能出现混合内容/安全上下文限制
- Private Network Access（PNA）相关的预检与限制更严格

最稳妥的做法：**网关仍然跑 HTTP**（绑定 `BIND_ADDR=127.0.0.1:8000`），再用 **Caddy 在本机提供 HTTPS 入口**，并用 **mkcert 生成并信任自签证书**。

When using `big-agi.com` (an HTTPS page) to call your local gateway, modern browsers may enforce:

- mixed-content / secure-context restrictions when calling `http://127.0.0.1:8000` from an HTTPS page
- stricter Private Network Access (PNA) preflight rules

The most reliable approach: keep the gateway on **HTTP** (bind `BIND_ADDR=127.0.0.1:8000`), then use **Caddy** to expose a local **HTTPS** endpoint, with **mkcert** providing a trusted local certificate.

---

### Step 0: Ensure `.env` matches the proxy plan / 第 0 步：确认 `.env` 配置与反代一致

Make sure your `.env` contains (same as `.env.example` recommendation):

```dotenv
BIND_ADDR=127.0.0.1:8000
CORS_ALLOW_ORIGINS=https://app.big-agi.com
```

> 中文：CORS origin 需要允许 Big-AGI 的网页来源。  
> English: CORS must allow Big-AGI’s web origin.

---

### Step 1: Install mkcert + Caddy / 第 1 步：安装 mkcert + Caddy（macOS 示例）

```bash
brew install mkcert caddy
```

(Optional) If you use Firefox and want it to trust mkcert certs:

```bash
brew install nss
```

---

### Step 2: Install the local CA (trust it) / 第 2 步：安装并信任本地 CA

```bash
mkcert -install
```

- 中文：这一步会把 mkcert 的本地 CA 安装到 macOS Keychain 并设为受信任。  
- English: This installs mkcert’s local CA into macOS Keychain and marks it as trusted.

---

### Step 3: Generate certificates / 第 3 步：生成证书

From your project root:

```bash
mkdir -p certs
cd certs
mkcert localhost 127.0.0.1 ::1
cd ..
```

You should get files similar to:

- `certs/localhost+2.pem`
- `certs/localhost+2-key.pem`

> 提醒 / Note
> mkcert 的输出文件名可能随 host 数量变化；如果你想固定文件名，可使用 `-cert-file` / `-key-file` 参数自行指定。

---

### Step 4: Caddy reverse proxy / 第 4 步：Caddy 反代配置

Create a `Caddyfile` at repo root:

```caddyfile
# Listen on https://localhost:8787
https://localhost:8787 {
 tls certs/localhost+2.pem certs/localhost+2-key.pem
 # Reverse proxy to the Rust gateway
 reverse_proxy 127.0.0.1:8000
}
```

Run Caddy:

```bash
caddy run --config Caddyfile
```

Verify (should NOT require `-k` once trusted):

```bash
curl -s https://localhost:8787/health
```

---

### Security note / 安全说明

- 中文：mkcert 只适合本机/内网开发环境，不建议用于公网生产环境。
- English: mkcert is for local/dev environments, not for public production deployments.

---

## Run / 运行

### 1) Build (if not built) / 1) 编译（若未编译）

```bash
cargo build --release
```

### 2) Prepare `.env` / 2) 准备 `.env`

```bash
cp .env.example .env
```

Edit `.env` and set at least:

- `OPENAI_API_KEY`
- `INDEX_DIR`
- `DOCS_MCP_COMMAND`

### 3) Start the gateway / 3) 启动网关

```bash
./target/release/gateway-openai
```

Then you can access:

- `GET /health`
- `GET /v1/models`
- `POST /v1/responses`

If you enabled Caddy HTTPS proxy as above, your browser-friendly endpoint is:

- `https://localhost:8787` (via Caddy)
- and the gateway remains on `http://127.0.0.1:8000` (internal)

---

## Using with Big-AGI (`big-agi.com`) / 与 Big-AGI 对接（`big-agi.com`）

在 Big-AGI 的 provider 设置里选择 OpenAI（或兼容 OpenAI 的 provider），然后：

- Base URL：建议填 `https://localhost:8787/v1`（如果你启用了 Caddy HTTPS 反代）
- API Key：可随意填写（网关默认不校验入站 key；上游访问使用 `OPENAI_API_KEY`）
- Model：选择 `UPSTREAM_MODEL` 或 `UPSTREAM_MODELS` 中的一个

In Big-AGI, choose an OpenAI (or OpenAI-compatible) provider and set:

- Base URL: recommended `https://localhost:8787/v1` (if using the Caddy HTTPS proxy)
- API Key: any value (the gateway does not validate inbound keys by default; upstream uses `OPENAI_API_KEY`)
- Model: pick one from `UPSTREAM_MODEL` / `UPSTREAM_MODELS`

---

## API Behavior Notes / API 行为说明

### Supported endpoints / 支持的端点

- `POST /v1/responses`
- `GET /v1/models`
- `GET /health`

### Stream Mode (auto / proxy / compat)

You can set it in two ways:

1) env default: `GATEWAY_STREAM_MODE=auto|compat|proxy`
2) per-request override: include `gateway_stream_mode` in the JSON body

Example:

```json
{
  "model": "gpt-5.2",
  "input": "hello",
  "stream": true,
  "gateway_stream_mode": "compat"
}
```

- **proxy mode**:
  - does not inject MCP tools
  - performs request allowlist + input sanitization + optional web_search restrictions + forces `store=true`
  - streaming: passthrough SSE events as-is
- **compat mode**:
  - injects MCP function tools
  - runs tool loop (extract `function_call` → call MCP tool → send `function_call_output` back upstream)
  - streaming: buffered compatibility event sequence (+ keepalive)

---

## Database MCP safety / 数据库 MCP 安全建议（泛化）

即使网关做了工具 allowlist、SQL 过滤等限制，仍强烈建议：

- 数据库账号使用**只读权限**
- 网关与 DB MCP 部署在受控网络（内网/防火墙）
- 如果需要暴露给更多客户端，务必在 Caddy/Nginx 侧做鉴权（Basic Auth / Token / IP allowlist）

Even with allowlists and SQL filtering, strongly recommended:

- use **read-only** database credentials
- deploy gateway + DB MCP in a controlled network (LAN/firewall)
- enforce auth at Caddy/Nginx if exposed (Basic Auth / token / IP allowlist)

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [eztakesin](https://github.com/eztakesin)
- **Source:** [eztakesin/llm-mcp-gateway-rs](https://github.com/eztakesin/llm-mcp-gateway-rs)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-eztakesin-llm-mcp-gateway-rs
- Seller: https://agentstack.voostack.com/s/eztakesin
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.