# Ai Agent Bench

> Use when the user wants to benchmark or compare AI agents (Claude Code, Codex, OpenCode) on a refactoring, perf, or code-change task in the current repo. Use when user says compare agents, benchmark Claude vs Codex, agent eval, measure agent, AI agent comparison, agent trial, /ai-agent-bench.

- **Type:** Skill
- **Install:** `agentstack add skill-reidemeister94-development-skills-ai-agent-bench`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [reidemeister94](https://agentstack.voostack.com/s/reidemeister94)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [reidemeister94](https://github.com/reidemeister94)
- **Source:** https://github.com/reidemeister94/development-skills/tree/main/skills/ai-agent-bench

## Install

```sh
agentstack add skill-reidemeister94-development-skills-ai-agent-bench
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# AI Agent Bench

Benchmark one or more AI agents on a real coding task in the current repo. Per agent, the harness:
1. Creates a git worktree at `start_commit` on a fresh `eval--run-` branch.
2. Runs `outer_check` once (baseline — live e2e correctness + wall time).
3. Launches the agent with the user's prompt; the agent uses `inner_check` for fast iteration.
4. Runs `outer_check` once again (post — same gate + wall time).
5. Captures transcript, diff, exit codes, timings under `eval-results///run--/`.

The branch survives after the trial; the worktree directory is removed. `outer_check` is both gate and measure — there are no pre_hooks, measure_repetitions, or sufficiency checks.

## TOML schema (`/.agent-bench.toml`)

```toml
prompt       = "prompts/.md"        # task prompt (markdown)
start_branch = "main"                      # override with start_commit = "" to pin
agents       = ["claude"]                  # subset of ["claude", "codex", "opencode"]
outer_check  = "./scripts/full_check.sh"   # live e2e: PASS/FAIL + wall-time, run once before/after
inner_check  = "pytest tests/integration/test_x.py -q"  # fast iteration test for the agent
```

The harness stages nothing — if the task needs fixtures/env, make `outer_check`/`inner_check` self-sufficient and commit (or gitignore + regenerate) anything they read.

## Step 0 — Preflight

```bash
REPO=$(git rev-parse --show-toplevel) || exit 1
[ -z "$(git -C "$REPO" status --porcelain)" ] || { echo "uncommitted changes — commit/stash first"; exit 1; }
[ -f "$REPO/.agent-bench.toml" ] || { echo "missing $REPO/.agent-bench.toml — see schema above"; exit 1; }
```

For each `agent`: `which $agent` must succeed. Python ≥ 3.11.

## Step 1 — Validate `outer_check` on HEAD

Run `outer_check` once in the repo before any trial. Exit 0 = baseline reference. Exit ≠ 0 = STOP, fix the code or the command; do NOT proceed.

## Step 2 — Confirm runtime params (numbered options, STOP and wait)

1. `agents` — confirm the TOML value or pick a subset.
2. `run_id` — default `1` if `eval-results//` is empty, else next integer.

To change the prompt or commands, the user edits the TOML and re-invokes.

## Step 3 — Launch trials sequentially

```bash
for AGENT in "${AGENTS[@]}"; do
    python "${CLAUDE_PLUGIN_ROOT}/skills/ai-agent-bench/scripts/run_trial.py" \
        --repo "$REPO" --config "$REPO/.agent-bench.toml" --agent "$AGENT" --run "$RUN_ID"
done
```

Sequential, never parallel (wall time is a measure).

`${CLAUDE_PLUGIN_ROOT}` is set by Claude Code; under Codex resolve via Glob `**/skills/ai-agent-bench/scripts/run_trial.py` or use the absolute path.

`run_trial.py` spawns `monitor.py` as a sidecar that polls `run_dir/status.txt` and tails `session.jsonl` every 3 min into `run_dir/progress.md`. Read `progress.md` on every user message to surface the heartbeat. Hard timeouts: warn at 150 min wall time, recommend terminating at 240 min if `status.txt` still says `agent:running`.

## Step 4 — Aggregate

```bash
python "${CLAUDE_PLUGIN_ROOT}/skills/ai-agent-bench/scripts/parse_transcript.py" \
    --aggregate "$REPO/eval-results/"/*/run-*/ \
    --output "$REPO/eval-results//comparison.json" \
    --render-report "$REPO/eval-results//comparison.md"
```

Print run dirs, branch names (`git checkout eval--run-` to inspect each diff), `outer_check` exit codes, baseline-vs-post wall-time delta, and cost USD per agent.

## Anomaly log

Append anything unexpected in real time to `/ai-agent-bench-anomalies.md`. Append-only across runs (one `## Run …` header per run dir, that path is the dedupe marker). Format and trigger list in `references/anomalies.md`.

## Rules

- **Never commit on the user's branch.** Agent works in a worktree on `eval--run-`; the snapshot commit is harness-owned.
- Re-running on the same `(task, agent, run_id)` creates a new timestamped run dir and branch; previous metrics stay intact.

Add a new agent: `references/agents.md`.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [reidemeister94](https://github.com/reidemeister94)
- **Source:** [reidemeister94/development-skills](https://github.com/reidemeister94/development-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-reidemeister94-development-skills-ai-agent-bench
- Seller: https://agentstack.voostack.com/s/reidemeister94
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.