Calibra

Stop guessing which models and settings are best. Measure them.

Free, open-source benchmarking for coding agents. Test models, prompts, skills, and MCP servers - side by side, at scale.

---

Calibra is a benchmarking harness that tells you exactly how your coding agent performs across models, instructions, skills, MCP servers, and environments. Define a matrix, run it, and get hard numbers - pass rates, token costs, timing, failure breakdowns, and statistical rankings - all in an interactive web dashboard you can share with your team.

It works with any provider: OpenAI, HuggingFace, or your own self-hosted models via LM Studio, Ollama, or any OpenAI-compatible endpoint. Run thousands of evaluations against local models for free.

Built for Swival.

Why Calibra

You're flying blind without it. Switching models? Adding an MCP server? Changing your agent's system prompt? You need to know if that actually made things better. Calibra gives you a controlled experiment instead of a gut feeling.

• Five-dimensional testing - vary model, agent instructions, skills, MCP servers, and environment overlays in any combination. Calibra runs the full matrix automatically.
• Statistically rigorous - repeat trials, confidence intervals, Pareto fronts, effect sizes. Not just "it seemed faster."
• Works with open models - bring your own LM Studio, Ollama, or any OpenAI-compatible endpoint. Run thousands of evals for zero API cost.
• Your data stays yours - runs entirely on your machine. No results, prompts, or code are sent to third-party eval platforms.
• Completely free and open source - no license keys, no usage limits, no telemetry.

The Web Dashboard

Calibra ships with an interactive dashboard that makes results actually useful.

Campaign overview - see pass rates, variant counts, and trial totals at a glance. KPI tiles highlight what matters: median turns, failure rate, token efficiency.

Variant rankings - a sortable, filterable table ranked by pass rate, token cost, and speed. Instantly spot which model + skill + MCP combo wins.

Pass rate charts - horizontal bar charts color-coded by performance. A scatter plot maps token cost against pass rate, with the Pareto front drawn on top so you can see the efficiency frontier.

Task heatmap - a full matrix of tasks vs. variants, colored from red to teal. Click any cell to drill into that specific combination.

Variant deep dive - per-task breakdowns with outcome dots, box plots of turn distributions, failure category pie charts (infra / provider / tool / timeout / task), and tool usage bar charts showing success vs. failure rates per tool.

Trial inspector - a full chronological timeline of a single trial: every LLM call (with duration and token count), every tool invocation (with arguments and pass/fail), compactions, guardrail interventions, and reviewer feedback. Expand any event to see the raw details.

Campaign comparison - pick two runs and see deltas: pass rate changes, Cliff's delta effect sizes, and a bar chart of improvements and regressions across all common variants.

Dark mode included. The whole thing exports to static HTML - share results without running a server.

Quick Start

uv sync

Create a task:

mkdir -p tasks/hello-world/env

cat > tasks/hello-world/task.md  tasks/hello-world/verify.sh               # check config without running
calibra run  [--workers N]     # run trials in parallel
                     [--resume]        # skip completed trials
                     [--filter EXPR]   # limit variants at runtime
                     [--dry-run]       # show plan without executing
calibra analyze           # aggregate metrics and write reports
calibra show              # inspect a single trial
calibra compare          # side-by-side comparison
calibra diff           # diff two trial reports in the browser
calibra web serve         # launch interactive dashboard
calibra web build         # export static HTML

Task Format

tasks/my-task/
  task.md       # prompt sent to the agent (required)
  env/          # starter workspace files (required)
  verify.sh     # exit-code pass/fail check (optional)
  meta.toml     # arbitrary metadata (optional)

Documentation

• [Quick Start](docs.md/quickstart.md)
• [Writing Tasks](docs.md/tasks.md)
• [Campaign Configuration](docs.md/configuration.md)
• [Running Campaigns](docs.md/running.md)
• [Analyzing Results](docs.md/analysis.md)
• [Web Dashboard](docs.md/web-dashboard.md)
• [Advanced Topics](docs.md/advanced.md)
• [CLI Reference](docs.md/cli-reference.md)

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Swival
• Source: Swival/calibra
• License: MIT
• Homepage: https://calibra.swival.dev

Install and usage instructions live in the source repository linked above.