# Calibra

> A benchmarking harness for coding agents.

- **Type:** MCP server
- **Install:** `agentstack add mcp-swival-calibra`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Swival](https://agentstack.voostack.com/s/swival)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Swival](https://github.com/Swival)
- **Source:** https://github.com/Swival/calibra
- **Website:** https://calibra.swival.dev

## Install

```sh
agentstack add mcp-swival-calibra
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

Stop guessing which models and settings are best. Measure them.

Free, open-source benchmarking for coding agents.
Test models, prompts, skills, and MCP servers - side by side, at scale.

---

Calibra is a benchmarking harness that tells you *exactly* how your coding agent performs across models, instructions, skills, MCP servers, and environments. Define a matrix, run it, and get hard numbers - pass rates, token costs, timing, failure breakdowns, and statistical rankings - all in an interactive web dashboard you can share with your team.

It works with any provider: OpenAI, HuggingFace, or your own self-hosted models via LM Studio, Ollama, or any OpenAI-compatible endpoint. Run thousands of evaluations against local models for free.

Built for [Swival](https://github.com/swival/swival).

## Why Calibra

**You're flying blind without it.** Switching models? Adding an MCP server? Changing your agent's system prompt? You need to know if that actually made things better. Calibra gives you a controlled experiment instead of a gut feeling.

- **Five-dimensional testing** - vary model, agent instructions, skills, MCP servers, and environment overlays in any combination. Calibra runs the full matrix automatically.
- **Statistically rigorous** - repeat trials, confidence intervals, Pareto fronts, effect sizes. Not just "it seemed faster."
- **Works with open models** - bring your own LM Studio, Ollama, or any OpenAI-compatible endpoint. Run thousands of evals for zero API cost.
- **Your data stays yours** - runs entirely on your machine. No results, prompts, or code are sent to third-party eval platforms.
- **Completely free and open source** - no license keys, no usage limits, no telemetry.

## The Web Dashboard

Calibra ships with an interactive dashboard that makes results actually useful.

**Campaign overview** - see pass rates, variant counts, and trial totals at a glance. KPI tiles highlight what matters: median turns, failure rate, token efficiency.

**Variant rankings** - a sortable, filterable table ranked by pass rate, token cost, and speed. Instantly spot which model + skill + MCP combo wins.

**Pass rate charts** - horizontal bar charts color-coded by performance. A scatter plot maps token cost against pass rate, with the Pareto front drawn on top so you can see the efficiency frontier.

**Task heatmap** - a full matrix of tasks vs. variants, colored from red to teal. Click any cell to drill into that specific combination.

**Variant deep dive** - per-task breakdowns with outcome dots, box plots of turn distributions, failure category pie charts (infra / provider / tool / timeout / task), and tool usage bar charts showing success vs. failure rates per tool.

**Trial inspector** - a full chronological timeline of a single trial: every LLM call (with duration and token count), every tool invocation (with arguments and pass/fail), compactions, guardrail interventions, and reviewer feedback. Expand any event to see the raw details.

**Campaign comparison** - pick two runs and see deltas: pass rate changes, Cliff's delta effect sizes, and a bar chart of improvements and regressions across all common variants.

**Dark mode** included. The whole thing exports to **static HTML** - share results without running a server.

## Quick Start

```sh
uv sync
```

Create a task:

```sh
mkdir -p tasks/hello-world/env

cat > tasks/hello-world/task.md  tasks/hello-world/verify.sh               # check config without running
calibra run  [--workers N]     # run trials in parallel
                     [--resume]        # skip completed trials
                     [--filter EXPR]   # limit variants at runtime
                     [--dry-run]       # show plan without executing
calibra analyze           # aggregate metrics and write reports
calibra show              # inspect a single trial
calibra compare          # side-by-side comparison
calibra diff           # diff two trial reports in the browser
calibra web serve         # launch interactive dashboard
calibra web build         # export static HTML
```

## Task Format

```text
tasks/my-task/
  task.md       # prompt sent to the agent (required)
  env/          # starter workspace files (required)
  verify.sh     # exit-code pass/fail check (optional)
  meta.toml     # arbitrary metadata (optional)
```

## Documentation

- [Quick Start](docs.md/quickstart.md)
- [Writing Tasks](docs.md/tasks.md)
- [Campaign Configuration](docs.md/configuration.md)
- [Running Campaigns](docs.md/running.md)
- [Analyzing Results](docs.md/analysis.md)
- [Web Dashboard](docs.md/web-dashboard.md)
- [Advanced Topics](docs.md/advanced.md)
- [CLI Reference](docs.md/cli-reference.md)

## Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Swival](https://github.com/Swival)
- **Source:** [Swival/calibra](https://github.com/Swival/calibra)
- **License:** MIT
- **Homepage:** https://calibra.swival.dev

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/mcp-swival-calibra
- Seller: https://agentstack.voostack.com/s/swival
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.