# Mcp Coder Bench

> Benchmark tool for measuring MCP server effectiveness in LLM-assisted development

- **Type:** MCP server
- **Install:** `agentstack add mcp-mpecan-mcp-coder-bench`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [mpecan](https://agentstack.voostack.com/s/mpecan)
- **Installs:** 0
- **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [mpecan](https://github.com/mpecan)
- **Source:** https://github.com/mpecan/mcp-coder-bench

## Install

```sh
agentstack add mcp-mpecan-mcp-coder-bench
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# mcp-coder-bench

A Rust-based benchmarking tool for measuring MCP (Model Context Protocol) server effectiveness in LLM-assisted development. Compare how different MCP servers impact token usage, cost, and task completion when using Claude Code.

## Features

- **MCP Server Benchmarking**: Compare baseline Claude Code performance against MCP-enhanced scenarios
- **Parallel Execution**: Run multiple benchmark containers simultaneously with configurable concurrency
- **Multi-Runtime Support**: Works with Docker and Podman (auto-detection)
- **Real-time Progress**: Live streaming of tool calls and MCP usage during execution
- **Statistical Analysis**: Confidence intervals, significance testing (Welch's t-test), effect sizes (Cohen's d)
- **Multiple Output Formats**: JSON, CSV, Markdown (with charts), HTML (with SVG visualizations), terminal tables
- **Workspace Isolation**: Each run gets a fresh workspace copy to ensure reproducibility
- **Structured Logging**: Tracing-based logging to file and console

## Installation

```bash
cargo install --path .
```

Or build from source:

```bash
cargo build --release
```

## Requirements

- Docker or Podman
- Rust 1.75+ (for building)
- `ANTHROPIC_API_KEY` environment variable

## Quick Start

### 1. Initialize Configuration

```bash
# Basic configuration
mcp-coder-bench init > mcp-coder-bench.yaml

# With Boarder MCP server example
mcp-coder-bench init --with-boarder > mcp-coder-bench.yaml
```

### 2. Validate Configuration

```bash
# Check config, workspace, and Docker connectivity
mcp-coder-bench validate

# Also verify container image exists
mcp-coder-bench validate --check-image
```

### 3. Run Benchmarks

```bash
# Run with default configuration
mcp-coder-bench run

# Run with custom settings
mcp-coder-bench run -p 4 -n 5  # 4 parallel containers, 5 runs each

# Run a single scenario
mcp-coder-bench run --scenario baseline

# Dry-run to validate without executing
mcp-coder-bench run --dry-run
```

### 4. Analyze Results

```bash
# Terminal table output
mcp-coder-bench analyze results/20260123/

# With statistical analysis
mcp-coder-bench analyze results/20260123/ --stats

# Export as Markdown with charts
mcp-coder-bench analyze results/20260123/ --format markdown --stats -o report.md

# Export as HTML with SVG visualizations
mcp-coder-bench analyze results/20260123/ --format html -o report.html

# Exclude outliers from analysis
mcp-coder-bench analyze results/20260123/ --exclude-outliers
```

### 5. Compare Scenarios

```bash
# Basic comparison
mcp-coder-bench compare results/baseline results/with-mcp

# With significance testing
mcp-coder-bench compare results/baseline results/with-mcp --significance

# Multi-scenario comparison
mcp-coder-bench compare results/v1 results/v2 results/v3 --format markdown
```

## Configuration

Example configuration file:

```yaml
name: "MCP Benchmark"

scenarios:
  - name: "baseline"
    description: "No MCP servers"
    mcp_config: {}

  - name: "with-boarder"
    description: "With Boarder MCP server"
    mcp_config:
      mcpServers:
        boarder:
          command: "/usr/local/bin/boarder"
          args: ["mcp"]

task:
  prompt_file: "prompts/task.md"
  workspace: "test-repo/"
  timeout_seconds: 600
  reset_strategy: "copy"  # copy, git, or none

execution:
  runs_per_scenario: 3
  parallelism: 2
  container_runtime: "auto"

container:
  image: "mcp-coder-bench:latest"  # Optional custom image
  builder: "docker"                 # Optional: docker, podman, or auto

output:
  directory: "results/"
  formats: ["json", "markdown"]
```

### Workspace Reset Strategies

- **copy**: Creates isolated workspace copies for each run (recommended for reproducibility)
- **git**: Resets workspace using `git checkout` and `git clean`
- **none**: Workspace persists between runs (useful for incremental tasks)

## Commands

### `run`

Execute benchmark scenarios.

```
mcp-coder-bench run [OPTIONS]

Options:
  -c, --config        Configuration file [default: mcp-coder-bench.yaml]
  -p, --parallelism      Number of parallel containers [default: 1]
  -n, --runs             Number of runs per scenario
  -s, --scenario      Run only a specific scenario
  -o, --output         Output directory for results
      --runtime    Container runtime (auto, docker, podman) [default: auto]
      --rebuild             Rebuild container image
      --dry-run             Validate without running
  -v, --verbose             Verbose output
```

### `analyze`

Analyze results from a benchmark run.

```
mcp-coder-bench analyze  [OPTIONS]

Options:
  -f, --format      Output format (json, csv, markdown, html, table) [default: table]
  -o, --output        Output file (stdout if not specified)
      --stats               Include statistical analysis
      --exclude-outliers    Exclude outlier runs from analysis
      --include-raw         Include raw output in results
```

### `compare`

Compare benchmark result sets.

```
mcp-coder-bench compare ... [OPTIONS]

Options:
  -f, --format      Output format (json, csv, markdown, html, table) [default: table]
  -o, --output        Output file (stdout if not specified)
      --significance        Include statistical significance testing
      --confidence   Confidence level for tests [default: 0.95]
```

### `validate`

Validate configuration without running benchmarks.

```
mcp-coder-bench validate [OPTIONS]

Options:
  -c, --config        Configuration file [default: mcp-coder-bench.yaml]
      --runtime    Container runtime [default: auto]
      --check-image         Also verify container image exists
  -v, --verbose             Verbose output
```

### `init`

Generate a sample configuration file.

```
mcp-coder-bench init [OPTIONS]

Options:
  -o, --output        Output file (stdout if not specified)
      --with-boarder        Include example Boarder MCP configuration
```

## Metrics Collected

- **Token Usage**: Input, output, cache creation, and cache read tokens
- **Cost**: Estimated USD cost based on Claude pricing
- **Tool Calls**: All tools used, with MCP tools tracked separately
- **Wall Time**: Actual execution duration
- **Success Rate**: Task completion status

## Statistical Features

- **Confidence Intervals**: 95% CI for token usage and cost (t-distribution for small samples)
- **Significance Testing**: Welch's t-test for comparing scenarios
- **Effect Sizes**: Cohen's d with labels (negligible/small/medium/large)
- **Outlier Detection**: IQR-based outlier identification and filtering
- **Distribution Analysis**: Histograms and percentiles

## Output Examples

### Terminal Table
```
╭──────────────┬──────┬────────────┬──────────┬──────────┬─────────╮
│ Scenario     │ Runs │ Avg Tokens │ Avg Cost │ Avg Time │ Success │
├──────────────┼──────┼────────────┼──────────┼──────────┼─────────┤
│ baseline     │ 5    │     125.3K │    $1.45 │   342.1s │    100% │
│ with-boarder │ 5    │      98.7K │    $1.12 │   287.3s │    100% │
╰──────────────┴──────┴────────────┴──────────┴──────────┴─────────╯
```

### MCP Tool Detection
```
MCP Tool Usage:
  mcp__boarder__cut_from_source - 2 calls
  mcp__boarder__clear_buffer - 1 call
```

## Logging

Logs are written to `~/.mcp-coder-bench/logs/` with rotation. Set `RUST_LOG` for console output:

```bash
RUST_LOG=debug mcp-coder-bench run
```

## License

MIT License - see [LICENSE](LICENSE) for details.

## Contributing

Contributions welcome! Please ensure:
- All tests pass (`cargo test`)
- Code is formatted (`cargo fmt`)
- No clippy warnings (`cargo clippy`)

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [mpecan](https://github.com/mpecan)
- **Source:** [mpecan/mcp-coder-bench](https://github.com/mpecan/mcp-coder-bench)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-mpecan-mcp-coder-bench
- Seller: https://agentstack.voostack.com/s/mpecan
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.