# Bitmcp Eval > Evaluate how LLM agents actually use your MCP server — recording proxy, claude/codex, optional LLM judge, HTML report. - **Type:** MCP server - **Install:** `agentstack add mcp-bitmovin-bitmcp-eval` - **Verified:** Pending review - **Seller:** [bitmovin](https://agentstack.voostack.com/s/bitmovin) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [bitmovin](https://github.com/bitmovin) - **Source:** https://github.com/bitmovin/bitmcp-eval - **Website:** https://bitmovin.github.io/bitmcp-eval/ ## Install ```sh agentstack add mcp-bitmovin-bitmcp-eval ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # bitmcp-eval [](https://github.com/bitmovin/bitmcp-eval/actions/workflows/ci.yml) [](LICENSE) **Evaluate how LLM agents actually use your MCP server.** You wrote an MCP server — but do agents _use_ it the way you intended? Do they pick the right tool for a given question? Every time, or only in 7 out of 10 runs? `bitmcp-eval` answers that. It runs your prompts through a real chat agent, intercepts all traffic to your MCP server with a recording proxy, and validates the observed tool calls against your expectations — multiple times per prompt, because agent behavior has spread. ``` ┌─────────────┐ prompt ┌──────────────┐ MCP (StreamableHTTP) ┌─────────────────┐ ┌────────────────┐ │ bitmcp-eval │ ───────▶ │ chat agent │ ──────────────────────▶ │ recording proxy │ ─────▶ │ your MCP │ │ (TUI) │ │ claude/codex │ ◀────────────────────── │ tools/call ✓ │ ◀───── │ server │ └─────────────┘ └──────────────┘ └─────────────────┘ └────────────────┘ │ │ └───────────── validate expected vs. recorded tool calls ◀──────────┘ → live TUI results + static HTML report ``` The proxy operates on the transport layer, so it needs **zero changes to your server**: anything that speaks the MCP StreamableHTTP protocol can be put under test — running locally or remote, with authentication (headers or OAuth) handled transparently. ## Contents - [Requirements](#requirements) - [Quickstart](#quickstart-2-minutes-bundled-demo-server) - [Evaluating your own MCP server](#evaluating-your-own-mcp-server) — test cases, config, running - [Configuration reference](#configuration-reference) - [Choosing the agents](#choosing-the-agents) — claude vs codex - [OAuth-protected MCP servers](#oauth-protected-mcp-servers) — the `login` command, token cache - [The LLM judge](#the-llm-judge-optional) — an independent semantic verdict per iteration - [Security & safety](#security--safety) - [Architecture](#architecture) - [Development](#development) · [Roadmap](#roadmap) ## Requirements - macOS or Linux (Windows is untested) - Node.js ≥ 20 with corepack enabled (Yarn Berry) - A chat agent CLI, installed and authenticated — a **recent version**, since the harness relies on newer flags (claude: `--strict-mcp-config`, headless `--resume`; codex: `exec resume`, `--json`): - [Claude Code](https://claude.com/claude-code) (`claude`) — the default; tested with 2.1.x - [OpenAI Codex](https://github.com/openai/codex) (`codex`, `npm install -g @openai/codex`) — tested with 0.142.x; select via `run.agents: [codex]` or run both for comparison - **The MCP server you want to evaluate**, reachable over StreamableHTTP (running locally or remote, auth headers supported). The bundled demo weather server exists only to verify your toolchain setup end-to-end — evaluating it tells you nothing about your own server's tool design. > **Cost note:** every iteration is a real agent conversation billed against your > Anthropic/OpenAI account or subscription quota. A run executes > `test cases × iterations × agents` conversations — plus one extra turn per scripted > answer that gets used. ## Quickstart (2 minutes, bundled demo server) ```sh corepack enable yarn install yarn build # terminal 1: start the demo weather MCP server on :3210 yarn demo-server # terminal 2: run the evaluation against it yarn start -c examples/eval.yaml ``` > **Running the CLI.** bitmcp-eval isn't published to npm yet, so every command in this > README is invoked as **`yarn start …`** from the repo. If you'd rather have a global > `bitmcp-eval` command, run `yarn build && yarn link` once inside `packages/cli`; then > `bitmcp-eval …` works anywhere. The two forms are interchangeable everywhere below. You'll watch every test case execute live, and at the end you get a link to a self-contained HTML report: ``` ✓ compare two forecasts ✓✓ (2/2 passed) ✓ current weather lookup ✓✓ (2/2 passed) ✓ list supported cities ✓✓ (2/2 passed) ╭──────────────────────────────────────────────────────────────────╮ │ Run finished │ │ 3 test cases · 6 iterations · 6 passed · 0 failed · 100% pass │ │ Report: file:///…/reports/bitmcp-eval-2026-07-06T10-00-00.html │ ╰──────────────────────────────────────────────────────────────────╯ ``` ## Evaluating your own MCP server ### 1. Write test cases A test case is one YAML file in a directory: ```yaml # testcases/current-weather.yaml name: current weather lookup # optional, defaults to the file name prompt: 'What is the current weather in Vienna?' expectedTools: - get_current_weather ``` **Validation semantics** - Every tool in `expectedTools` must be called **successfully** at least once per iteration. A call whose result is an error — a JSON-RPC error or an MCP result with `isError` (e.g. an upstream 401 wrapped by the server) — does not count; it is shown in the report as a failed call instead. - Listing a name N times means "at least N successful calls" — e.g. `get_forecast` twice for a two-city comparison prompt. - Extra calls of expected tools are fine. Calls of unlisted tools are reported in the HTML report but don't fail the iteration — agents legitimately explore. **Answering clarifying questions (`answers`)** Well-designed MCP servers often instruct the agent to ask the user before acting ("which license should I query?"). In a headless eval that question would end the conversation and the test would fail even though the server behaves correctly. Give the test case scripted replies (illustrative — these tools belong to an analytics server, not the weather demo): ```yaml prompt: 'Show me the ad completion funnel for last week.' expectedTools: - peekAllLicenses - queryTotal answers: - 'Use the license with the highest play volume, no need to ask me again.' ``` After each agent turn the harness checks the expectations; while they are unmet and answers remain, it sends the next answer **into the same agent session** (full conversation context) and re-validates. Answers that aren't needed are never sent. The HTML report shows the complete conversation for multi-turn iterations. ### 2. Write a config ```yaml # eval.yaml envFile: ./.env # optional; defaults to a .env next to this config, when present mcp: url: http://127.0.0.1:3210/mcp # your server (local or remote) headers: # injected into every request by the proxy - name: x-api-key value: ${MY_API_KEY} # ${VAR} pulls from the environment / envFile testcases: source: filesystem path: ./testcases # relative to this config file run: iterations: 3 # run each test case 3× to see the behavioral spread agents: [claude] # add codex to compare agents: [claude, codex] timeoutSeconds: 300 report: outDir: ./reports ``` ### 3. Run ```sh yarn start -c eval.yaml # full run yarn start -c eval.yaml -i 10 # override iterations from the CLI yarn start -c eval.yaml --debug # additionally log every proxied request's headers # to /bitmcp-eval-debug.log — useful for # diagnosing auth issues. Contains secrets, do not share. yarn start login -c eval.yaml # OAuth servers only: authorize + cache a token up front # (see "OAuth-protected MCP servers" below) ``` The process exits non-zero when the run itself fails (bad config, unreachable server). Evaluation outcomes — pass rates per test case and iteration — are in the TUI summary and the HTML report, which includes each iteration's recorded tool calls with arguments, durations, missing expectations, and the agent's full conversation. The report is written **incrementally**: after the first finished test case the TUI shows a "Live report" link you can open right away — the page carries an in-progress banner and auto-refreshes every 10 seconds until the run completes. ## Configuration reference | Key | Type | Default | Description | | -------------------- | --------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `envFile` | string | `.env` next to config, if present | Dotenv file loaded before `${VAR}` interpolation. Shell environment always takes precedence. | | `mcp.url` | URL, required | — | StreamableHTTP endpoint of the MCP server under test. | | `mcp.headers[]` | `{name, value}` | `[]` | Headers injected into every proxied request (e.g. API keys). `${VAR}` placeholders are interpolated. | | `mcp.oauth` | object | — | Only for OAuth servers **without** dynamic client registration: `clientId`, `clientSecret` (both `${VAR}`-interpolated), optional `scopes`, `redirectPort`. OAuth itself is auto-detected — see "OAuth-protected MCP servers". | | `testcases.source` | `filesystem` | `filesystem` | Test case provider. `s3` and `git` are planned. | | `testcases.path` | string, required | — | Directory with `*.yaml` test cases. Relative to the config file. `~` is expanded. | | `run.iterations` | int 1–100 | `3` | Executions per test case, to measure behavioral spread. | | `run.agents` | list of `claude` \| `codex` | `[claude]` | Agents to evaluate — the whole suite runs once per agent, with per-agent pass rates in the report. `run.agent: ` works as a single-agent alias. | | `run.timeoutSeconds` | int | `300` | Hard limit per agent invocation. | | `report.outDir` | string | `./reports` | Output directory for the HTML report. Relative to the config file. | | `judge` | object | — | Optional LLM judge: `baseUrl` (OpenAI-compatible), `model`, optional `apiKey` (`${VAR}`-interpolated), `timeoutSeconds` (default 60). See "The LLM judge". | ## Choosing the agents Agents are a property of the run (`run.agents` in `eval.yaml`). With more than one agent the whole test suite executes once per agent, and both the TUI summary and the HTML report ("Agents compared") break the pass rates down per agent — ideal for questions like "does codex use my server as reliably as claude?". **`claude`** (default) — each turn runs `claude -p … --strict-mcp-config`, so the server under test is the agent's _only_ MCP server and only its tools are auto-allowed. Follow-up `answers` resume the session via `--resume `. **`codex`** — each turn runs `codex exec --json …` with the proxy injected via a config override; follow-up `answers` use `codex exec resume `. The harness disables codex's built-in web search (`web_search="disabled"`) and ChatGPT apps/connectors (`features.apps=false` — codex was observed answering from the logged-in account's connectors, with that account's credentials, instead of the server under test), and runs each session in a neutral temp directory with an `AGENTS.md` steering it to answer only via the server under test. Caveats inherent to today's codex CLI: - `--dangerously-bypass-approvals-and-sandbox` is passed automatically because headless MCP tool calls are otherwise auto-cancelled ([openai/codex#16685](https://github.com/openai/codex/issues/16685), [openai/codex#24135](https://github.com/openai/codex/issues/24135)). This lifts the sandbox for shell commands, and codex's shell tool cannot be switched off by config — the `AGENTS.md` guardrail steers away from it, but that is instruction, not enforcement. **Only evaluate trusted test cases against trusted MCP servers, or run the harness in a container.** - Because it cannot be hard-restricted, every shell command and web search codex performs is recorded as an **escape from the MCP binding** and shown per iteration in the HTML report ("⚠ Left the MCP binding"). Escapes don't fail an iteration by themselves — answering _instead of_ calling the expected tools already fails validation, while e.g. shell-based math on top of proper tool results is legitimate. - codex has no isolation flag equivalent to `--strict-mcp-config` (its `--ignore-user-config` also drops the override that injects the proxy), so MCP servers from your own `~/.codex/config.toml` stay visible to the agent during the eval. Keep the eval machine's codex config clean for unpolluted results. ## OAuth-protected MCP servers Most hosted MCP servers sit behind OAuth. bitmcp-eval handles this **automatically** — you never configure endpoints or paste tokens. The chat agent stays entirely out of the OAuth loop; the recording proxy is the authenticated client and injects the bearer token on every request it forwards. ### How it works, end to end 1. **Detect.** At the start of every run the harness probes the MCP server. A normal response → no OAuth, nothing happens. A `401` challenge → OAuth is required, and the harness follows the advertised metadata (protected-resource → authorization-server) to discover the endpoints. 2. **Authorize (once).** If there is no usable token yet, the harness runs the standard OAuth **authorization-code + PKCE** flow: it opens your browser to the provider's login page and waits on a local loopback listener (`http://127.0.0.1:8765/callback`) for the redirect. You log in as yourself; the harness exchanges the code for an access token (and a refresh token). 3. **Cache.** The tokens are written to `~/.bitmcp-eval/tokens/` (file mode `0600`), keyed by authorization-server + resource — so one login serves **every** config pointing at that server, and survives across runs and restarts. 4. **Inject.** For the whole run the proxy adds `Authorization: Bearer ` to each request … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [bitmovin](https://github.com/bitmovin) - **Source:** [bitmovin/bitmcp-eval](https://github.com/bitmovin/bitmcp-eval) - **License:** MIT - **Homepage:** https://bitmovin.github.io/bitmcp-eval/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: flagged — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-bitmovin-bitmcp-eval - Seller: https://agentstack.voostack.com/s/bitmovin - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.