Mcpbr

mcpbr

# One-liner install (installs + runs quick test)
curl -sSL https://raw.githubusercontent.com/greynewell/mcpbr/main/install.sh | bash

# Or install and run manually
pip install mcpbr && mcpbr run -n 1

Benchmark your MCP server against real GitHub issues. One command, hard numbers.

---

Model Context Protocol Benchmark Runner

[](https://pypi.org/project/mcpbr/) [](https://www.npmjs.com/package/mcpbr-cli) [](https://www.python.org/downloads/) [](https://github.com/greynewell/mcpbr/actions/workflows/ci.yml) [](https://opensource.org/licenses/MIT) [](https://doi.org/10.5281/zenodo.18627369) [](https://mcpbr.org/)

[](https://github.com/greynewell/mcpbr/labels/good%20first%20issue) [](https://github.com/greynewell/mcpbr/labels/help%20wanted) [](https://github.com/users/greynewell/projects/2)

> Stop guessing if your MCP server actually helps. Get hard numbers comparing tool-assisted vs. baseline agent performance on real GitHub issues.

---

⭐ Star the Supermodel Ecosystem

If this is useful, please star our tools — it helps us grow:

[](https://github.com/supermodeltools/mcp)  [](https://github.com/supermodeltools/mcpbr)  [](https://github.com/supermodeltools/typescript-sdk)  [](https://github.com/supermodeltools/arch-docs)  [](https://github.com/supermodeltools/dead-code-hunter)  [](https://github.com/supermodeltools/Uncompact)  [](https://github.com/supermodeltools/narsil-mcp)

---

What You Get

Real metrics showing whether your MCP server improves agent performance on SWE-bench tasks. No vibes, just data.

Why mcpbr?

MCP servers promise to make LLMs better at coding tasks. But how do you prove it?

mcpbr runs controlled experiments: same model, same tasks, same environment - the only variable is your MCP server. You get:

• Apples-to-apples comparison against a baseline agent
• Real GitHub issues from SWE-bench (not toy examples)
• Reproducible results via Docker containers with pinned dependencies

Blog

• SWE-bench Verified Is Broken: 5 Things I Found in the Source Code
• SWE-bench Tests Run 6x Faster on ARM64 with Native Containers

Research Paper

mcpbr: Benchmarking Model Context Protocol Servers on Software Engineering Tasks Grey Newell, Georgia Institute of Technology, 2026

We evaluated a code graph analysis MCP server on all 500 tasks from SWE-bench Verified using Claude Sonnet as the base agent. Key findings:

| Metric | Baseline | MCP-Augmented | Change | |--------|----------|---------------|--------| | Resolution Rate | 49.8% | 42.4% | -14.9% | | Tool Calls | — | — | -42.3% | | Tokens Used | — | — | -14.0% | | Cost per Task | — | — | -15.2% |

MCP tools alter the agent's exploration strategy, trading general-purpose search for opinionated shortcuts. The effect varies by codebase: the server helped on 1 of 12 repositories and hurt on 10, revealing an efficiency-resolution tradeoff that developers should evaluate before deploying MCP tools in production.

Methodology: Paired comparison experiments with Docker-isolated task environments, pinned dependencies, and identical model configurations. The only variable is the presence of MCP tools.

Cite this work

@software{newell2025mcpbr,
  author    = {Newell, Grey},
  title     = {mcpbr: Benchmarking Model Context Protocol Servers on Software Engineering Tasks},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.18627369},
  url       = {https://doi.org/10.5281/zenodo.18627369}
}

Supported Benchmarks

mcpbr supports 30+ benchmarks across 10 categories through a flexible abstraction layer:

| Category | Benchmarks | |----------|-----------| | Software Engineering | SWE-bench (Verified/Lite/Full), APPS, CodeContests, BigCodeBench, LeetCode, CoderEval, Aider Polyglot | | Code Generation | HumanEval, MBPP | | Math & Reasoning | GSM8K, MATH, BigBench-Hard | | Knowledge & QA | TruthfulQA, HellaSwag, ARC, GAIA | | Tool Use & Agents | MCPToolBench++, ToolBench, AgentBench, WebArena, TerminalBench, InterCode | | ML Research | MLAgentBench | | Code Understanding | RepoQA | | Multimodal | MMMU | | Long Context | LongBench | | Safety & Adversarial | Adversarial (HarmBench) | | Security | CyberGym | | Custom | User-defined benchmarks via YAML |

Featured Benchmarks

SWE-bench (Default) - Real GitHub issues requiring bug fixes. Three variants: Verified (500 manually validated), Lite (300 curated), and Full (2,294 complete). Pre-built Docker images available.

CyberGym - Security vulnerabilities requiring PoC exploits. 4 difficulty levels controlling context. Uses AddressSanitizer for crash detection.

MCPToolBench++ - Large-scale MCP tool use evaluation across 45+ categories. Tests tool discovery, selection, invocation, and result interpretation.

GSM8K - Grade-school math word problems testing chain-of-thought reasoning with numeric answer matching.

# Run SWE-bench Verified (default)
mcpbr run -c config.yaml

# Run any benchmark
mcpbr run -c config.yaml --benchmark humaneval -n 20
mcpbr run -c config.yaml --benchmark gsm8k -n 50
mcpbr run -c config.yaml --benchmark cybergym --level 2

# List all available benchmarks
mcpbr benchmarks

See the benchmarks guide for details on each benchmark and how to configure them.

Overview

This harness runs two parallel evaluations for each task:

1. MCP Agent: LLM with access to tools from your MCP server
2. Baseline Agent: LLM without tools (single-shot generation)

By comparing these, you can measure the effectiveness of your MCP server for different software engineering tasks. See the MCP integration guide for tips on testing your server.

Regression Detection

mcpbr includes built-in regression detection to catch performance degradations between MCP server versions:

Key Features

• Automatic Detection: Compare current results against a baseline to identify regressions
• Detailed Reports: See exactly which tasks regressed and which improved
• Threshold-Based Exit Codes: Fail CI/CD pipelines when regression rate exceeds acceptable limits
• Multi-Channel Alerts: Send notifications via Slack, Discord, or email

How It Works

A regression is detected when a task that passed in the baseline now fails in the current run. This helps you catch issues before deploying new versions of your MCP server.

# First, run a baseline evaluation and save results
mcpbr run -c config.yaml -o baseline.json

# Later, compare a new version against the baseline
mcpbr run -c config.yaml --baseline-results baseline.json --regression-threshold 0.1

# With notifications
mcpbr run -c config.yaml --baseline-results baseline.json \
  --regression-threshold 0.1 \
  --slack-webhook https://hooks.slack.com/services/YOUR/WEBHOOK/URL

Use Cases

• CI/CD Integration: Automatically detect regressions in pull requests
• Version Comparison: Compare different versions of your MCP server
• Performance Monitoring: Track MCP server performance over time
• Team Notifications: Alert your team when regressions are detected

Example Output

======================================================================
REGRESSION DETECTION REPORT
======================================================================

Total tasks compared: 25
Regressions detected: 2
Improvements detected: 5
Regression rate: 8.0%

REGRESSIONS (previously passed, now failed):
----------------------------------------------------------------------
  - django__django-11099
    Error: Timeout
  - sympy__sympy-18087
    Error: Test suite failed

IMPROVEMENTS (previously failed, now passed):
----------------------------------------------------------------------
  - astropy__astropy-12907
  - pytest-dev__pytest-7373
  - scikit-learn__scikit-learn-25570
  - matplotlib__matplotlib-23913
  - requests__requests-3362

======================================================================

For CI/CD integration, use --regression-threshold to fail the build when regressions exceed an acceptable rate:

# .github/workflows/test-mcp.yml
- name: Run mcpbr with regression detection
  run: |
    mcpbr run -c config.yaml \
      --baseline-results baseline.json \
      --regression-threshold 0.1 \
      -o current.json

This will exit with code 1 if the regression rate exceeds 10%, failing the CI job.

Installation

> Full installation guide with detailed setup instructions.

Prerequisites

• Python 3.11+
• Docker (running)
• ANTHROPIC_API_KEY environment variable
• Claude Code CLI (claude) installed
• Network access (for pulling Docker images and API calls)

Supported Models (aliases or full names):

• Claude Opus 4.5: opus or claude-opus-4-5-20251101
• Claude Sonnet 4.5: sonnet or claude-sonnet-4-5-20250929
• Claude Haiku 4.5: haiku or claude-haiku-4-5-20251001

Run mcpbr models to see the full list.

via npm

[](https://www.npmjs.com/package/mcpbr-cli)

# Run with npx (no installation)
npx mcpbr-cli run -c config.yaml

# Or install globally
npm install -g mcpbr-cli
mcpbr run -c config.yaml

> Package: mcpbr-cli on npm > > Note: The npm package requires Python 3.11+ and the mcpbr Python package (pip install mcpbr)

via pip

# Install from PyPI
pip install mcpbr

# Or install from source
git clone https://github.com/greynewell/mcpbr.git
cd mcpbr
pip install -e .

# Or with uv
uv pip install -e .

> Note for Apple Silicon users: The harness automatically uses x86_64 Docker images via emulation. This may be slower than native ARM64 images but ensures compatibility with all SWE-bench tasks.

Quick Start

Option 1: Use Example Configurations (Recommended)

Get started in seconds with our example configurations:

# Set your API key
export ANTHROPIC_API_KEY="your-api-key"

# Run your first evaluation using an example config
mcpbr run -c examples/quick-start/getting-started.yaml -v

This runs 5 SWE-bench tasks with the filesystem server. Expected runtime: 15-30 minutes, cost: $2-5.

Explore 25+ example configurations in the [examples/](examples/) directory:

• Quick Start: Getting started, testing servers, comparing models
• Benchmarks: SWE-bench Lite/Full, CyberGym basic/advanced
• MCP Servers: Filesystem, GitHub, Brave Search, databases, custom servers
• Scenarios: Cost-optimized, performance-optimized, CI/CD, regression detection

See the [Examples README](examples/README.md) for the complete guide.

Option 2: Generate Custom Configuration

1. Set your API key:

export ANTHROPIC_API_KEY="your-api-key"

1. Run mcpbr (config auto-created if missing):

# Config is auto-created on first run
mcpbr run -n 1

# Or explicitly generate a config file first
mcpbr init

1. Edit the configuration to point to your MCP server:

mcp_server:
  command: "npx"
  args:
    - "-y"
    - "@modelcontextprotocol/server-filesystem"
    - "{workdir}"
  env: {}

provider: "anthropic"
agent_harness: "claude-code"

model: "sonnet"  # or full name: "claude-sonnet-4-5-20250929"
dataset: "SWE-bench/SWE-bench_Lite"
sample_size: 10
timeout_seconds: 300
max_concurrent: 4

# Optional: disable default logging (logs are saved to output_dir/logs/ by default)
# disable_logs: true

1. Run the evaluation:

mcpbr run --config config.yaml

Infrastructure Modes

mcpbr supports running evaluations on different infrastructure platforms, allowing you to scale evaluations or offload compute-intensive tasks to cloud VMs.

Local (Default)

Run evaluations on your local machine:

infrastructure:
  mode: local  # default

This is the default mode - evaluations run directly on your machine using local Docker containers.

Azure VM

Run evaluations on Azure Virtual Machines with automatic provisioning and cleanup:

infrastructure:
  mode: azure
  azure:
    resource_group: mcpbr-benchmarks
    location: eastus
    cpu_cores: 10
    memory_gb: 40

Key features:

• Zero manual VM setup - provisioned automatically from config
• Automatic Docker, Python, and mcpbr installation
• Test task validation before full evaluation
• Auto-cleanup after completion (configurable)
• Cost-optimized with automatic VM deletion

Example usage:

# Run evaluation on Azure VM
mcpbr run -c azure-config.yaml

# VM is automatically created, evaluation runs, results are downloaded, VM is deleted

See [docs/infrastructure/azure.md](docs/infrastructure/azure.md) for full documentation including:

• Prerequisites and authentication
• VM sizing and cost estimation
• Debugging with preserve_on_error
• Troubleshooting guide

Side-by-Side Server Comparison

Compare two MCP servers head-to-head in a single evaluation run to see which implementation performs better.

Quick Example

# comparison-config.yaml
comparison_mode: true

mcp_server_a:
  name: "Task Queries"
  command: node
  args: [build/index.js]
  cwd: /path/to/task-queries

mcp_server_b:
  name: "Edge Identity"
  command: node
  args: [build/index.js]
  cwd: /path/to/edge-identity

benchmark: swe-bench-lite
sample_size: 10

mcpbr run -c comparison-config.yaml -o results.json

Results Output

Side-by-Side MCP Server Comparison

┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┓
┃ Metric            ┃ Task Queries ┃ Edge Identity┃ Δ (A - B)┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━┩
│ Resolved Tasks    │ 4/10         │ 2/10         │ +2       │
│ Resolution Rate   │ 40.0%        │ 20.0%        │ +100.0%  │
└───────────────────┴──────────────┴──────────────┴──────────┘

✓ Task Queries unique wins: 2 tasks
  - django__django-12286
  - astropy__astropy-7606

Use cases:

• A/B testing: Compare optimized vs. baseline implementations
• Tool evaluation: Test different MCP tool sets
• Version comparison: Benchmark v2.0 vs. v1.5

See [docs/comparison-mode.md](docs/comparison-mode.md) for complete documentation.

Claude Code Integration

[](https://claude.ai/download)

mcpbr includes a built-in Claude Code plugin that makes Claude an expert at running benchmarks correctly. The plugin provides specialized skills and knowledge about mcpbr configuration, execution, and troubleshooting.

Installation Options

You have three ways to enable the mcpbr plugin in Claude Code:

Option 1: Clone Repositor

…

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: greynewell
• Source: greynewell/mcpbr
• License: MIT

Install and usage instructions live in the source repository linked above.