Bitmcp Eval

bitmcp-eval

[](https://github.com/bitmovin/bitmcp-eval/actions/workflows/ci.yml) [](LICENSE)

Evaluate how LLM agents actually use your MCP server.

You wrote an MCP server — but do agents use it the way you intended? Do they pick the right tool for a given question? Every time, or only in 7 out of 10 runs?

bitmcp-eval answers that. It runs your prompts through a real chat agent, intercepts all traffic to your MCP server with a recording proxy, and validates the observed tool calls against your expectations — multiple times per prompt, because agent behavior has spread.

┌─────────────┐  prompt  ┌──────────────┐  MCP (StreamableHTTP)   ┌─────────────────┐        ┌────────────────┐
│ bitmcp-eval │ ───────▶ │  chat agent  │ ──────────────────────▶ │ recording proxy │ ─────▶ │ your MCP       │
│    (TUI)    │          │ claude/codex │ ◀────────────────────── │  tools/call ✓   │ ◀───── │ server         │
└─────────────┘          └──────────────┘                         └─────────────────┘        └────────────────┘
       │                                                                   │
       └───────────── validate expected vs. recorded tool calls ◀──────────┘
                      → live TUI results + static HTML report

The proxy operates on the transport layer, so it needs zero changes to your server: anything that speaks the MCP StreamableHTTP protocol can be put under test — running locally or remote, with authentication (headers or OAuth) handled transparently.

Contents

• [Requirements](#requirements)
• [Quickstart](#quickstart-2-minutes-bundled-demo-server)
• [Evaluating your own MCP server](#evaluating-your-own-mcp-server) — test cases, config, running
• [Configuration reference](#configuration-reference)
• [Choosing the agents](#choosing-the-agents) — claude vs codex
• [OAuth-protected MCP servers](#oauth-protected-mcp-servers) — the login command, token cache
• [The LLM judge](#the-llm-judge-optional) — an independent semantic verdict per iteration
• [Security & safety](#security--safety)
• [Architecture](#architecture)
• [Development](#development) · [Roadmap](#roadmap)

Requirements

• macOS or Linux (Windows is untested)
• Node.js ≥ 20 with corepack enabled (Yarn Berry)
• A chat agent CLI, installed and authenticated — a recent version, since the harness

relies on newer flags (claude: --strict-mcp-config, headless --resume; codex: exec resume, --json):

• Claude Code (claude) — the default; tested with 2.1.x
• OpenAI Codex (codex, npm install -g @openai/codex) —

tested with 0.142.x; select via run.agents: [codex] or run both for comparison

• The MCP server you want to evaluate, reachable over StreamableHTTP (running locally

or remote, auth headers supported). The bundled demo weather server exists only to verify your toolchain setup end-to-end — evaluating it tells you nothing about your own server's tool design.

> Cost note: every iteration is a real agent conversation billed against your > Anthropic/OpenAI account or subscription quota. A run executes > test cases × iterations × agents conversations — plus one extra turn per scripted > answer that gets used.

Quickstart (2 minutes, bundled demo server)

corepack enable
yarn install
yarn build

# terminal 1: start the demo weather MCP server on :3210
yarn demo-server

# terminal 2: run the evaluation against it
yarn start -c examples/eval.yaml

> Running the CLI. bitmcp-eval isn't published to npm yet, so every command in this > README is invoked as yarn start … from the repo. If you'd rather have a global > bitmcp-eval command, run yarn build && yarn link once inside packages/cli; then > bitmcp-eval … works anywhere. The two forms are interchangeable everywhere below.

You'll watch every test case execute live, and at the end you get a link to a self-contained HTML report:

✓ compare two forecasts  ✓✓ (2/2 passed)
✓ current weather lookup ✓✓ (2/2 passed)
✓ list supported cities  ✓✓ (2/2 passed)

╭──────────────────────────────────────────────────────────────────╮
│ Run finished                                                     │
│ 3 test cases · 6 iterations · 6 passed · 0 failed · 100% pass    │
│ Report: file:///…/reports/bitmcp-eval-2026-07-06T10-00-00.html   │
╰──────────────────────────────────────────────────────────────────╯

Evaluating your own MCP server

1. Write test cases

A test case is one YAML file in a directory:

# testcases/current-weather.yaml
name: current weather lookup # optional, defaults to the file name
prompt: 'What is the current weather in Vienna?'
expectedTools:
  - get_current_weather

Validation semantics

• Every tool in expectedTools must be called successfully at least once per

iteration. A call whose result is an error — a JSON-RPC error or an MCP result with isError (e.g. an upstream 401 wrapped by the server) — does not count; it is shown in the report as a failed call instead.

• Listing a name N times means "at least N successful calls" — e.g. get_forecast twice

for a two-city comparison prompt.

• Extra calls of expected tools are fine. Calls of unlisted tools are reported in the

HTML report but don't fail the iteration — agents legitimately explore.

Answering clarifying questions (answers)

Well-designed MCP servers often instruct the agent to ask the user before acting ("which license should I query?"). In a headless eval that question would end the conversation and the test would fail even though the server behaves correctly. Give the test case scripted replies (illustrative — these tools belong to an analytics server, not the weather demo):

prompt: 'Show me the ad completion funnel for last week.'
expectedTools:
  - peekAllLicenses
  - queryTotal
answers:
  - 'Use the license with the highest play volume, no need to ask me again.'

After each agent turn the harness checks the expectations; while they are unmet and answers remain, it sends the next answer into the same agent session (full conversation context) and re-validates. Answers that aren't needed are never sent. The HTML report shows the complete conversation for multi-turn iterations.

2. Write a config

# eval.yaml
envFile: ./.env # optional; defaults to a .env next to this config, when present

mcp:
  url: http://127.0.0.1:3210/mcp # your server (local or remote)
  headers: # injected into every request by the proxy
    - name: x-api-key
      value: ${MY_API_KEY} # ${VAR} pulls from the environment / envFile

testcases:
  source: filesystem
  path: ./testcases # relative to this config file

run:
  iterations: 3 # run each test case 3× to see the behavioral spread
  agents: [claude] # add codex to compare agents: [claude, codex]
  timeoutSeconds: 300

report:
  outDir: ./reports

3. Run

yarn start -c eval.yaml          # full run
yarn start -c eval.yaml -i 10    # override iterations from the CLI
yarn start -c eval.yaml --debug  # additionally log every proxied request's headers
#                                  to /bitmcp-eval-debug.log — useful for
#                                  diagnosing auth issues. Contains secrets, do not share.
yarn start login -c eval.yaml    # OAuth servers only: authorize + cache a token up front
#                                  (see "OAuth-protected MCP servers" below)

The process exits non-zero when the run itself fails (bad config, unreachable server). Evaluation outcomes — pass rates per test case and iteration — are in the TUI summary and the HTML report, which includes each iteration's recorded tool calls with arguments, durations, missing expectations, and the agent's full conversation.

The report is written incrementally: after the first finished test case the TUI shows a "Live report" link you can open right away — the page carries an in-progress banner and auto-refreshes every 10 seconds until the run completes.

Configuration reference

| Key | Type | Default | Description | | -------------------- | --------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | envFile | string | .env next to config, if present | Dotenv file loaded before ${VAR} interpolation. Shell environment always takes precedence. | | mcp.url | URL, required | — | StreamableHTTP endpoint of the MCP server under test. | | mcp.headers[] | {name, value} | [] | Headers injected into every proxied request (e.g. API keys). ${VAR} placeholders are interpolated. | | mcp.oauth | object | — | Only for OAuth servers without dynamic client registration: clientId, clientSecret (both ${VAR}-interpolated), optional scopes, redirectPort. OAuth itself is auto-detected — see "OAuth-protected MCP servers". | | testcases.source | filesystem | filesystem | Test case provider. s3 and git are planned. | | testcases.path | string, required | — | Directory with *.yaml test cases. Relative to the config file. ~ is expanded. | | run.iterations | int 1–100 | 3 | Executions per test case, to measure behavioral spread. | | run.agents | list of claude \| codex | [claude] | Agents to evaluate — the whole suite runs once per agent, with per-agent pass rates in the report. run.agent: works as a single-agent alias. | | run.timeoutSeconds | int | 300 | Hard limit per agent invocation. | | report.outDir | string | ./reports | Output directory for the HTML report. Relative to the config file. | | judge | object | — | Optional LLM judge: baseUrl (OpenAI-compatible), model, optional apiKey (${VAR}-interpolated), timeoutSeconds (default 60). See "The LLM judge". |

Choosing the agents

Agents are a property of the run (run.agents in eval.yaml). With more than one agent the whole test suite executes once per agent, and both the TUI summary and the HTML report ("Agents compared") break the pass rates down per agent — ideal for questions like "does codex use my server as reliably as claude?".

claude (default) — each turn runs claude -p … --strict-mcp-config, so the server under test is the agent's only MCP server and only its tools are auto-allowed. Follow-up answers resume the session via --resume .

codex — each turn runs codex exec --json … with the proxy injected via a config override; follow-up answers use codex exec resume . The harness disables codex's built-in web search (web_search="disabled") and ChatGPT apps/connectors (features.apps=false — codex was observed answering from the logged-in account's connectors, with that account's credentials, instead of the server under test), and runs each session in a neutral temp directory with an AGENTS.md steering it to answer only via the server under test. Caveats inherent to today's codex CLI:

• --dangerously-bypass-approvals-and-sandbox is passed automatically because headless

MCP tool calls are otherwise auto-cancelled (openai/codex#16685, openai/codex#24135). This lifts the sandbox for shell commands, and codex's shell tool cannot be switched off by config — the AGENTS.md guardrail steers away from it, but that is instruction, not enforcement. Only evaluate trusted test cases against trusted MCP servers, or run the harness in a container.

• Because it cannot be hard-restricted, every shell command and web search codex performs

is recorded as an escape from the MCP binding and shown per iteration in the HTML report ("⚠ Left the MCP binding"). Escapes don't fail an iteration by themselves — answering instead of calling the expected tools already fails validation, while e.g. shell-based math on top of proper tool results is legitimate.

• codex has no isolation flag equivalent to --strict-mcp-config (its

--ignore-user-config also drops the override that injects the proxy), so MCP servers from your own ~/.codex/config.toml stay visible to the agent during the eval. Keep the eval machine's codex config clean for unpolluted results.

OAuth-protected MCP servers

Most hosted MCP servers sit behind OAuth. bitmcp-eval handles this automatically — you never configure endpoints or paste tokens. The chat agent stays entirely out of the OAuth loop; the recording proxy is the authenticated client and injects the bearer token on every request it forwards.

How it works, end to end

1. Detect. At the start of every run the harness probes the MCP server. A normal

response → no OAuth, nothing happens. A 401 challenge → OAuth is required, and the harness follows the advertised metadata (protected-resource → authorization-server) to discover the endpoints.

1. Authorize (once). If there is no usable token yet, the harness runs the standard

OAuth authorization-code + PKCE flow: it opens your browser to the provider's login page and waits on a local loopback listener (http://127.0.0.1:8765/callback) for the redirect. You log in as yourself; the harness exchanges the code for an access token (and a refresh token).

1. Cache. The tokens are written to ~/.bitmcp-eval/tokens/ (file mode 0600), keyed

by authorization-server + resource — so one login serves every config pointing at that server, and survives across runs and restarts.

1. Inject. For the whole run the proxy adds Authorization: Bearer to each

request

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: bitmovin
• Source: bitmovin/bitmcp-eval
• License: MIT
• Homepage: https://bitmovin.github.io/bitmcp-eval/

Install and usage instructions live in the source repository linked above.