Agent Bench

agent-bench

> Does that MCP server or skill actually make your coding agent better? Measure it. Deterministic gates, A/B arms, your own models, reproducible numbers instead of vibes.

[](LICENSE) [](https://nodejs.org) [](https://opencode.ai) [](#reproduce) [](results/)

What

agent-bench runs a coding agent through the same tasks twice, once with an "enhancement" (an MCP server, a skill prompt, a model setting) and once without, on the models you run, and scores every run with deterministic gates instead of model-graded vibes.

[bench] serena/ts-ambiguous  models=qwen,mistral  arms=baseline,serena  N=3  total=12
[1] qwen/baseline #1  success=true tools=19 tokIn=138864 diff=4f/8l 69264ms
[4] qwen/serena #1    success=true tools=8  tokIn=99506  diff=4f/8l 20071ms
[7] mistral/baseline #1  success=false tools=23 tokIn=240278 diff=8f/18l   ts-rename

# 3. full matrix
EXPERIMENT=serena ARMS=baseline,serena TASK_NAME=ts-rename,ts-callers,ts-ambiguous N=5 node runner/bench.js

# 4. aggregate
node runner/aggregate.js 

Raw data for every published number is in results/*.jsonl.

Layout

Series

The methodology article (sovgrid.org/blog/agent-bench-pillar/) explains the design. Published spokes so far: Serena · caveman. The series continues through the Claude marketplace top charts — submit a benchmark result via issue or PR.

runner/            bench.js (matrix) · aggregate.js · smoke.js · arms.js · tasks.js
runner/lib/        opencode invoker · JSONL parser · per-arm config writer
runner/claude-chat.js   same A/B on Claude models via the claude CLI (subscription auth)
fixture/           TS fixtures (ts-rename, ts-callers, ts-ambiguous) + chat
tasks/             per-task gate scripts
scripts/quality/   diff-stat · regression · lint-clean
prompts/           injected skill prompts, verbatim from upstream
results/           runs-*.jsonl (raw) · summaries · VERDICT.md · FINDINGS-*.md
article/           the published write-ups (canonical versions on sovgrid.org)

Design constraints

1. Deterministic gates only. A run passes a build, a typecheck, or a frozen fact checklist. No LLM grades another LLM.
2. A/B or it did not happen. Every claim is a within-model comparison against a baseline arm on identical prompts.
3. Negative results ship. A tool that measures as overhead gets published with the same template as a winner.
4. One change per arm. Toggle exactly one variable; everything else stays fixed and documented.
5. Raw data in the repo. Every table is recomputable from results/*.jsonl.
6. Benchmark hygiene is part of the method. One matrix here was thrown away because a health-check timer resurrected an idle model mid-run; the clean redo flipped a finding. Contaminated data dies, it does not get footnoted.

Adding an experiment

A new MCP: add an arm in runner/arms.js with its mcp block (__WORKDIR__ substitutes the per-run path). A new skill: add an arm with agentsFile pointing at the prompt. A new task: fixture dir + gate script + entry in runner/tasks.js. No harness changes.

Two opencode footguns the harness already handles: opencode run must get stdin from /dev/null (it blocks forever on an open stdin), and per-arm isolation goes through the OPENCODE_CONFIG env var; prompt-injection arms write AGENTS.md into the working copy (project-level opencode.json instructions is not picked up).

Honest limits

Small N (3-5), rates not significance. TypeScript fixtures only so far. Published numbers are from self-hosted models (Qwen3.6-35b on vLLM, Mistral-Small-4 on SGLang) on a DGX Spark plus Claude models via CLI; your stack will differ, which is exactly why the harness is a clone away.

License

MIT.

Source & license

This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: cipherfoxie
• Source: cipherfoxie/agent-bench
• License: MIT
• Homepage: https://sovgrid.org/blog/agent-bench-pillar/

Install and usage instructions live in the source repository linked above.