# Mcpbr > Benchmark your MCP server. - **Type:** MCP server - **Install:** `agentstack add mcp-greynewell-mcpbr` - **Verified:** Pending review - **Seller:** [greynewell](https://agentstack.voostack.com/s/greynewell) - **Installs:** 0 - **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [greynewell](https://github.com/greynewell) - **Source:** https://github.com/greynewell/mcpbr ## Install ```sh agentstack add mcp-greynewell-mcpbr ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # mcpbr ```bash # One-liner install (installs + runs quick test) curl -sSL https://raw.githubusercontent.com/greynewell/mcpbr/main/install.sh | bash # Or install and run manually pip install mcpbr && mcpbr run -n 1 ``` Benchmark your MCP server against real GitHub issues. One command, hard numbers. --- **Model Context Protocol Benchmark Runner** [](https://pypi.org/project/mcpbr/) [](https://www.npmjs.com/package/mcpbr-cli) [](https://www.python.org/downloads/) [](https://github.com/greynewell/mcpbr/actions/workflows/ci.yml) [](https://opensource.org/licenses/MIT) [](https://doi.org/10.5281/zenodo.18627369) [](https://mcpbr.org/) [](https://github.com/greynewell/mcpbr/labels/good%20first%20issue) [](https://github.com/greynewell/mcpbr/labels/help%20wanted) [](https://github.com/users/greynewell/projects/2) > Stop guessing if your MCP server actually helps. Get hard numbers comparing tool-assisted vs. baseline agent performance on real GitHub issues. --- ## ⭐ Star the Supermodel Ecosystem If this is useful, please star our tools — it helps us grow: [](https://github.com/supermodeltools/mcp)  [](https://github.com/supermodeltools/mcpbr)  [](https://github.com/supermodeltools/typescript-sdk)  [](https://github.com/supermodeltools/arch-docs)  [](https://github.com/supermodeltools/dead-code-hunter)  [](https://github.com/supermodeltools/Uncompact)  [](https://github.com/supermodeltools/narsil-mcp) --- ## What You Get Real metrics showing whether your MCP server improves agent performance on SWE-bench tasks. No vibes, just data. ## Why mcpbr? MCP servers promise to make LLMs better at coding tasks. But how do you *prove* it? mcpbr runs controlled experiments: same model, same tasks, same environment - the only variable is your MCP server. You get: - **Apples-to-apples comparison** against a baseline agent - **Real GitHub issues** from SWE-bench (not toy examples) - **Reproducible results** via Docker containers with pinned dependencies ## Blog - [SWE-bench Verified Is Broken: 5 Things I Found in the Source Code](https://greynewell.com/blog/swe-bench-verified-broken-5-things-source-code/) - [SWE-bench Tests Run 6x Faster on ARM64 with Native Containers](https://greynewell.com/blog/swe-bench-arm64-native-containers-6x-faster/) ## Research Paper **[mcpbr: Benchmarking Model Context Protocol Servers on Software Engineering Tasks](https://doi.org/10.5281/zenodo.18627369)** *Grey Newell, Georgia Institute of Technology, 2026* We evaluated a code graph analysis MCP server on all 500 tasks from SWE-bench Verified using Claude Sonnet as the base agent. Key findings: | Metric | Baseline | MCP-Augmented | Change | |--------|----------|---------------|--------| | Resolution Rate | 49.8% | 42.4% | -14.9% | | Tool Calls | — | — | -42.3% | | Tokens Used | — | — | -14.0% | | Cost per Task | — | — | -15.2% | MCP tools alter the agent's exploration strategy, trading general-purpose search for opinionated shortcuts. The effect varies by codebase: the server helped on 1 of 12 repositories and hurt on 10, revealing an efficiency-resolution tradeoff that developers should evaluate before deploying MCP tools in production. **Methodology:** Paired comparison experiments with Docker-isolated task environments, pinned dependencies, and identical model configurations. The only variable is the presence of MCP tools. Cite this work ```bibtex @software{newell2025mcpbr, author = {Newell, Grey}, title = {mcpbr: Benchmarking Model Context Protocol Servers on Software Engineering Tasks}, year = {2026}, publisher = {Zenodo}, doi = {10.5281/zenodo.18627369}, url = {https://doi.org/10.5281/zenodo.18627369} } ``` ## Supported Benchmarks mcpbr supports 30+ benchmarks across 10 categories through a flexible abstraction layer: | Category | Benchmarks | |----------|-----------| | **Software Engineering** | [SWE-bench](https://mcpbr.org/benchmarks/swe-bench/) (Verified/Lite/Full), [APPS](https://mcpbr.org/benchmarks/apps/), [CodeContests](https://mcpbr.org/benchmarks/codecontests/), [BigCodeBench](https://mcpbr.org/benchmarks/bigcodebench/), [LeetCode](https://mcpbr.org/benchmarks/leetcode/), [CoderEval](https://mcpbr.org/benchmarks/codereval/), [Aider Polyglot](https://mcpbr.org/benchmarks/aider-polyglot/) | | **Code Generation** | [HumanEval](https://mcpbr.org/benchmarks/humaneval/), [MBPP](https://mcpbr.org/benchmarks/mbpp/) | | **Math & Reasoning** | [GSM8K](https://mcpbr.org/benchmarks/gsm8k/), [MATH](https://mcpbr.org/benchmarks/math/), [BigBench-Hard](https://mcpbr.org/benchmarks/bigbench-hard/) | | **Knowledge & QA** | [TruthfulQA](https://mcpbr.org/benchmarks/truthfulqa/), [HellaSwag](https://mcpbr.org/benchmarks/hellaswag/), [ARC](https://mcpbr.org/benchmarks/arc/), [GAIA](https://mcpbr.org/benchmarks/gaia/) | | **Tool Use & Agents** | [MCPToolBench++](https://mcpbr.org/benchmarks/mcptoolbench/), [ToolBench](https://mcpbr.org/benchmarks/toolbench/), [AgentBench](https://mcpbr.org/benchmarks/agentbench/), [WebArena](https://mcpbr.org/benchmarks/webarena/), [TerminalBench](https://mcpbr.org/benchmarks/terminalbench/), [InterCode](https://mcpbr.org/benchmarks/intercode/) | | **ML Research** | [MLAgentBench](https://mcpbr.org/benchmarks/mlagentbench/) | | **Code Understanding** | [RepoQA](https://mcpbr.org/benchmarks/repoqa/) | | **Multimodal** | MMMU | | **Long Context** | LongBench | | **Safety & Adversarial** | Adversarial (HarmBench) | | **Security** | [CyberGym](https://mcpbr.org/benchmarks/cybergym/) | | **Custom** | User-defined benchmarks via YAML | ### Featured Benchmarks **SWE-bench** (Default) - Real GitHub issues requiring bug fixes. Three variants: Verified (500 manually validated), Lite (300 curated), and Full (2,294 complete). Pre-built Docker images available. **CyberGym** - Security vulnerabilities requiring PoC exploits. 4 difficulty levels controlling context. Uses AddressSanitizer for crash detection. **MCPToolBench++** - Large-scale MCP tool use evaluation across 45+ categories. Tests tool discovery, selection, invocation, and result interpretation. **GSM8K** - Grade-school math word problems testing chain-of-thought reasoning with numeric answer matching. ```bash # Run SWE-bench Verified (default) mcpbr run -c config.yaml # Run any benchmark mcpbr run -c config.yaml --benchmark humaneval -n 20 mcpbr run -c config.yaml --benchmark gsm8k -n 50 mcpbr run -c config.yaml --benchmark cybergym --level 2 # List all available benchmarks mcpbr benchmarks ``` See the **[benchmarks guide](https://mcpbr.org/benchmarks/)** for details on each benchmark and how to configure them. ## Overview This harness runs two parallel evaluations for each task: 1. **MCP Agent**: LLM with access to tools from your MCP server 2. **Baseline Agent**: LLM without tools (single-shot generation) By comparing these, you can measure the effectiveness of your MCP server for different software engineering tasks. See the **[MCP integration guide](https://mcpbr.org/mcp-integration/)** for tips on testing your server. ## Regression Detection mcpbr includes built-in regression detection to catch performance degradations between MCP server versions: ### Key Features - **Automatic Detection**: Compare current results against a baseline to identify regressions - **Detailed Reports**: See exactly which tasks regressed and which improved - **Threshold-Based Exit Codes**: Fail CI/CD pipelines when regression rate exceeds acceptable limits - **Multi-Channel Alerts**: Send notifications via Slack, Discord, or email ### How It Works A regression is detected when a task that passed in the baseline now fails in the current run. This helps you catch issues before deploying new versions of your MCP server. ```bash # First, run a baseline evaluation and save results mcpbr run -c config.yaml -o baseline.json # Later, compare a new version against the baseline mcpbr run -c config.yaml --baseline-results baseline.json --regression-threshold 0.1 # With notifications mcpbr run -c config.yaml --baseline-results baseline.json \ --regression-threshold 0.1 \ --slack-webhook https://hooks.slack.com/services/YOUR/WEBHOOK/URL ``` ### Use Cases - **CI/CD Integration**: Automatically detect regressions in pull requests - **Version Comparison**: Compare different versions of your MCP server - **Performance Monitoring**: Track MCP server performance over time - **Team Notifications**: Alert your team when regressions are detected ### Example Output ``` ====================================================================== REGRESSION DETECTION REPORT ====================================================================== Total tasks compared: 25 Regressions detected: 2 Improvements detected: 5 Regression rate: 8.0% REGRESSIONS (previously passed, now failed): ---------------------------------------------------------------------- - django__django-11099 Error: Timeout - sympy__sympy-18087 Error: Test suite failed IMPROVEMENTS (previously failed, now passed): ---------------------------------------------------------------------- - astropy__astropy-12907 - pytest-dev__pytest-7373 - scikit-learn__scikit-learn-25570 - matplotlib__matplotlib-23913 - requests__requests-3362 ====================================================================== ``` For CI/CD integration, use `--regression-threshold` to fail the build when regressions exceed an acceptable rate: ```yaml # .github/workflows/test-mcp.yml - name: Run mcpbr with regression detection run: | mcpbr run -c config.yaml \ --baseline-results baseline.json \ --regression-threshold 0.1 \ -o current.json ``` This will exit with code 1 if the regression rate exceeds 10%, failing the CI job. ## Installation > **[Full installation guide](https://mcpbr.org/installation/)** with detailed setup instructions. Prerequisites - Python 3.11+ - Docker (running) - `ANTHROPIC_API_KEY` environment variable - Claude Code CLI (`claude`) installed - Network access (for pulling Docker images and API calls) **Supported Models (aliases or full names):** - Claude Opus 4.5: `opus` or `claude-opus-4-5-20251101` - Claude Sonnet 4.5: `sonnet` or `claude-sonnet-4-5-20250929` - Claude Haiku 4.5: `haiku` or `claude-haiku-4-5-20251001` Run `mcpbr models` to see the full list. ### via npm [](https://www.npmjs.com/package/mcpbr-cli) ```bash # Run with npx (no installation) npx mcpbr-cli run -c config.yaml # Or install globally npm install -g mcpbr-cli mcpbr run -c config.yaml ``` > **Package**: [`mcpbr-cli`](https://www.npmjs.com/package/mcpbr-cli) on npm > > **Note**: The npm package requires Python 3.11+ and the mcpbr Python package (`pip install mcpbr`) ### via pip ```bash # Install from PyPI pip install mcpbr # Or install from source git clone https://github.com/greynewell/mcpbr.git cd mcpbr pip install -e . # Or with uv uv pip install -e . ``` > **Note for Apple Silicon users**: The harness automatically uses x86_64 Docker images via emulation. This may be slower than native ARM64 images but ensures compatibility with all SWE-bench tasks. ## Quick Start ### Option 1: Use Example Configurations (Recommended) Get started in seconds with our example configurations: ```bash # Set your API key export ANTHROPIC_API_KEY="your-api-key" # Run your first evaluation using an example config mcpbr run -c examples/quick-start/getting-started.yaml -v ``` This runs 5 SWE-bench tasks with the filesystem server. Expected runtime: 15-30 minutes, cost: $2-5. **Explore 25+ example configurations** in the [`examples/`](examples/) directory: - **Quick Start**: Getting started, testing servers, comparing models - **Benchmarks**: SWE-bench Lite/Full, CyberGym basic/advanced - **MCP Servers**: Filesystem, GitHub, Brave Search, databases, custom servers - **Scenarios**: Cost-optimized, performance-optimized, CI/CD, regression detection See the **[Examples README](examples/README.md)** for the complete guide. ### Option 2: Generate Custom Configuration 1. **Set your API key:** ```bash export ANTHROPIC_API_KEY="your-api-key" ``` 2. **Run mcpbr (config auto-created if missing):** ```bash # Config is auto-created on first run mcpbr run -n 1 # Or explicitly generate a config file first mcpbr init ``` 3. **Edit the configuration** to point to your MCP server: ```yaml mcp_server: command: "npx" args: - "-y" - "@modelcontextprotocol/server-filesystem" - "{workdir}" env: {} provider: "anthropic" agent_harness: "claude-code" model: "sonnet" # or full name: "claude-sonnet-4-5-20250929" dataset: "SWE-bench/SWE-bench_Lite" sample_size: 10 timeout_seconds: 300 max_concurrent: 4 # Optional: disable default logging (logs are saved to output_dir/logs/ by default) # disable_logs: true ``` 4. **Run the evaluation:** ```bash mcpbr run --config config.yaml ``` ## Infrastructure Modes mcpbr supports running evaluations on different infrastructure platforms, allowing you to scale evaluations or offload compute-intensive tasks to cloud VMs. ### Local (Default) Run evaluations on your local machine: ```yaml infrastructure: mode: local # default ``` This is the default mode - evaluations run directly on your machine using local Docker containers. ### Azure VM Run evaluations on Azure Virtual Machines with automatic provisioning and cleanup: ```yaml infrastructure: mode: azure azure: resource_group: mcpbr-benchmarks location: eastus cpu_cores: 10 memory_gb: 40 ``` **Key features:** - Zero manual VM setup - provisioned automatically from config - Automatic Docker, Python, and mcpbr installation - Test task validation before full evaluation - Auto-cleanup after completion (configurable) - Cost-optimized with automatic VM deletion **Example usage:** ```bash # Run evaluation on Azure VM mcpbr run -c azure-config.yaml # VM is automatically created, evaluation runs, results are downloaded, VM is deleted ``` See [docs/infrastructure/azure.md](docs/infrastructure/azure.md) for full documentation including: - Prerequisites and authentication - VM sizing and cost estimation - Debugging with `preserve_on_error` - Troubleshooting guide ## Side-by-Side Server Comparison Compare two MCP servers head-to-head in a single evaluation run to see which implementation performs better. ### Quick Example ```yaml # comparison-config.yaml comparison_mode: true mcp_server_a: name: "Task Queries" command: node args: [build/index.js] cwd: /path/to/task-queries mcp_server_b: name: "Edge Identity" command: node args: [build/index.js] cwd: /path/to/edge-identity benchmark: swe-bench-lite sample_size: 10 ``` ```bash mcpbr run -c comparison-config.yaml -o results.json ``` ### Results Output ```text Side-by-Side MCP Server Comparison ┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━┓ ┃ Metric ┃ Task Queries ┃ Edge Identity┃ Δ (A - B)┃ ┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━┩ │ Resolved Tasks │ 4/10 │ 2/10 │ +2 │ │ Resolution Rate │ 40.0% │ 20.0% │ +100.0% │ └───────────────────┴──────────────┴──────────────┴──────────┘ ✓ Task Queries unique wins: 2 tasks - django__django-12286 - astropy__astropy-7606 ``` **Use cases:** - **A/B testing**: Compare optimized vs. baseline implementations - **Tool evaluation**: Test different MCP tool sets - **Version comparison**: Benchmark v2.0 vs. v1.5 See [docs/comparison-mode.md](docs/comparison-mode.md) for complete documentation. ## Claude Code Integration [](https://claude.ai/download) mcpbr includes a built-in Claude Code plugin that makes Claude an expert at running benchmarks correctly. The plugin provides specialized skills and knowledge about mcpbr configuration, execution, and troubleshooting. ### Installation Options You have three ways to enable the mcpbr plugin in Claude Code: #### Option 1: Clone Repositor … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [greynewell](https://github.com/greynewell) - **Source:** [greynewell/mcpbr](https://github.com/greynewell/mcpbr) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** yes - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: flagged — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-greynewell-mcpbr - Seller: https://agentstack.voostack.com/s/greynewell - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.