Agent Eval Harness

Agent Eval Harness

Purpose

CLI tool for capturing trajectories from headless CLI agents, optimized for TypeScript/JavaScript projects using Bun.

The harness captures. You score.

| Harness Provides | You Provide | |------------------|-------------| | Prompt execution via headless adapters | Scoring logic (Braintrust, custom scripts) | | Full trajectory capture (thoughts, tools, plans) | Pass/fail determination via graders | | Structured JSONL output | LLM-as-judge prompts | | Reproducible execution environment | CI integration, golden file comparison |

Use this when:

• Capturing trajectories for downstream evaluation
• Generating training data (SFT/DPO) with full context
• Building regression test fixtures for agent behavior
• Comparing agent responses across configurations

Installation

# Run without installing (recommended)
bunx @plaited/agent-eval-harness capture prompts.jsonl --schema ./claude.json -o results.jsonl

# Or install as project dependency
bun add @plaited/agent-eval-harness

Core Principle: Capture Once, Derive Many Views

flowchart LR
    Prompts["prompts.jsonl"] --> Capture["capture/trials"]
    Schema["headless schema"] --> Capture
    Capture --> Results["results.jsonl (full trajectory)"]
    Results --> Summarize["summarize"]
    Results --> Calibrate["calibrate"]
    Results --> Custom["(your tools)"]
    Summarize --> Views["summary.jsonl / .md"]
    Calibrate --> Report["calibration.md"]
    Custom --> Pipeline["any scoring platform"]

Single output format: Full trajectory JSONL (always) No --format flag: Derive views with separate commands Schema exports: Zod schemas + JSON Schema for any tooling

Commands

Core Commands

| Command | Input | Output | Purpose | |---------|-------|--------|---------| | capture | prompts.jsonl + schema | results.jsonl | Trajectory capture (full) | | trials | prompts.jsonl + schema | trials.jsonl | Multi-run + optional metrics | | summarize | results.jsonl | summary.jsonl or .md | Derive compact views | | calibrate | results.jsonl | calibration.md | Sample failures for review | | validate-refs | prompts.jsonl | validation.jsonl | Check reference solutions | | balance | prompts.jsonl | balance.json | Analyze test set coverage | | schemas | (none) | JSON Schema | Export schemas for non-TS users |

Pipeline Commands (Unix-style)

| Command | Input | Output | Purpose | |---------|-------|--------|---------| | run | prompts.jsonl + schema | raw.jsonl | Execute prompts, raw output | | extract | raw.jsonl + schema | extracted.jsonl | Parse trajectories | | grade | extracted.jsonl + grader | graded.jsonl | Apply grader scoring | | format | results.jsonl | jsonl/markdown/csv | Convert output format | | compare | multiple results.jsonl | comparison.json | Compare runs (aggregate report) |

All commands support optional --grader ./grader.ts for scoring.

Capture Command

Basic Usage

bunx @plaited/agent-eval-harness capture  --schema  [options]

Arguments

| Argument/Flag | Description | Default | |------|-------------|---------| | prompts.jsonl | Input file with prompts to execute | Required | | -s, --schema | Path to headless adapter schema | Required | | -o, --output | Output file/path | stdout | | -c, --cwd | Working directory for agent | current | | -t, --timeout | Request timeout in ms | 60000 | | -j, --concurrency | Number of concurrent workers | 1 | | --workspace-dir | Base directory for per-prompt workspace isolation | none | | --progress | Show progress to stderr | false | | --append | Append to output file | false | | -g, --grader | Path to grader module | none | | --debug | Show detailed CLI output for debugging | false |

Examples

# Basic capture
bunx @plaited/agent-eval-harness capture prompts.jsonl --schema ./claude.json -o results.jsonl

# Parallel execution (4x faster with 4 workers)
bunx @plaited/agent-eval-harness capture prompts.jsonl --schema ./claude.json -j 4 -o results.jsonl

# With workspace isolation for code generation tasks
bunx @plaited/agent-eval-harness capture prompts.jsonl --schema ./claude.json \
  -j 4 --workspace-dir ./workspaces -o results.jsonl

# Using a local adapter script
bunx @plaited/agent-eval-harness capture prompts.jsonl bun ./my-adapter.ts -o results.jsonl

# With grader (adds score to each result)
bunx @plaited/agent-eval-harness capture prompts.jsonl --schema ./claude.json --grader ./grader.ts -o results.jsonl

Trials Command

Run each prompt multiple times for pass@k/pass^k analysis.

# Capture only (no grader)
bunx @plaited/agent-eval-harness trials prompts.jsonl --schema ./claude.json -k 5 -o trials.jsonl

# With grader (computes pass@k, pass^k)
bunx @plaited/agent-eval-harness trials prompts.jsonl --schema ./claude.json -k 5 --grader ./grader.ts -o trials.jsonl

# Parallel execution (4 prompts' trials run concurrently)
bunx @plaited/agent-eval-harness trials prompts.jsonl --schema ./claude.json -k 5 -j 4 -o trials.jsonl

# With workspace isolation (each trial gets its own directory)
bunx @plaited/agent-eval-harness trials prompts.jsonl --schema ./claude.json -k 5 -j 4 \
  --workspace-dir ./workspaces -o trials.jsonl

Parallelization notes:

• -j/--concurrency parallelizes across prompts (not trials within a prompt)
• Each prompt's k trials still run sequentially (required for aggregation)
• With 151 prompts and -j 4, you get 4 prompts running trials concurrently
• --workspace-dir creates {workspace-dir}/prompt-{id}-trial-{n}/ for each trial
• Progress logging shows aggregate completion (e.g., 12/50 prompts completed)

Workspace cleanup: Directories persist after completion for debugging. Clean up manually:

# After capture
rm -rf ./workspaces

# In CI (add as post-step)
- run: rm -rf ./workspaces
  if: always()

Output

Without grader:

{"id":"search-001","input":"Find the CEO","k":5,"trials":[{"trialNum":1,"output":"...","trajectory":[...],"duration":1234},...]}

With grader:

{"id":"search-001","input":"Find the CEO","k":5,"passRate":0.8,"passAtK":0.99,"passExpK":0.33,"trials":[{"trialNum":1,"output":"...","pass":true,"score":1.0},...]}

Summarize Command

Derive compact views from full trajectory results.

# Summary JSONL (for jq analysis)
bunx @plaited/agent-eval-harness summarize results.jsonl -o summary.jsonl

# Markdown (for LLM-as-judge)
bunx @plaited/agent-eval-harness summarize results.jsonl --markdown -o results.md

Calibrate Command

Sample failures for grader review. Calibration helps you distinguish between agent failures (agent did wrong thing) and grader bugs (agent was correct, grader too strict).

# Sample failures for human review
bunx @plaited/agent-eval-harness calibrate results.jsonl --sample 10 -o calibration.md

# Re-score with different grader to compare
bunx @plaited/agent-eval-harness calibrate results.jsonl --grader ./loose-grader.ts --sample 10 -o comparison.md

See [eval-concepts.md](references/eval-concepts.md#grader-calibration) for why calibration matters.

Validate-Refs Command

Check that reference solutions pass your grader before evaluating agents.

# Validate reference solutions
bunx @plaited/agent-eval-harness validate-refs prompts.jsonl --grader ./grader.ts -o validation.jsonl

# Check for failures
cat validation.jsonl | jq 'select(.pass == false)'

Why Use This?

If your reference solution fails your own grader:

• The task definition is ambiguous
• The grader is too strict
• The hint is wrong

Fix the eval before evaluating the agent.

Input Format

Prompts must include a reference field:

{"id":"test-001","input":"Create a button component","hint":"","reference":"export const Button = () => Click"}

Output Format

{"id":"test-001","input":"Create a button component","reference":"export const Button = () => Click","pass":true,"score":1.0,"reasoning":"Contains hint content"}

Balance Command

Analyze test set coverage to ensure balanced evaluation.

# Analyze prompt distribution
bunx @plaited/agent-eval-harness balance prompts.jsonl -o balance.json

# Pretty print
bunx @plaited/agent-eval-harness balance prompts.jsonl | jq .

Why Use This?

An eval with only "make X work" misses "don't break Y". Balance analysis shows:

• Category distribution (from metadata.category)
• Positive/negative case ratio
• Coverage gaps

Output Format

{
  "totalCases": 50,
  "categories": [
    { "name": "ui", "count": 20, "percentage": 40 },
    { "name": "logic", "count": 15, "percentage": 30 },
    { "name": "api", "count": 10, "percentage": 20 },
    { "name": "edge-case", "count": 5, "percentage": 10 }
  ],
  "underrepresented": ["edge-case"],
  "suggestions": ["Consider adding more test cases for: edge-case"]
}

Balanced Eval Design

Include both positive and negative cases:

| Type | Example | Purpose | |------|---------|---------| | Positive | "Add a login button" | Agent should succeed | | Negative | "Add a button without breaking tests" | Agent should not break things | | Edge case | "Handle empty input gracefully" | Agent should be robust |

See [eval-concepts.md](references/eval-concepts.md#test-set-balance) for more on balanced test sets.

Pipeline Workflow

The pipeline commands enable Unix-style composition for flexible evaluation workflows.

Full Pipeline Example

# Execute → Extract → Grade → Format in one pipeline
cat prompts.jsonl | \
  bunx @plaited/agent-eval-harness run -s claude.json | \
  bunx @plaited/agent-eval-harness extract -s claude.json | \
  bunx @plaited/agent-eval-harness grade -g ./grader.ts | \
  bunx @plaited/agent-eval-harness format -f markdown > report.md

Run Command

Execute prompts and output raw results. Three modes available:

# Schema mode (recommended)
bunx @plaited/agent-eval-harness run prompts.jsonl --schema claude.json

# Simple mode: {} placeholder substitution
bunx @plaited/agent-eval-harness run prompts.jsonl --simple "claude -p {} --output-format stream-json"

# Shell mode: $PROMPT environment variable
bunx @plaited/agent-eval-harness run prompts.jsonl --shell 'claude -p "$PROMPT" --output-format stream-json'

> ⚠️ Security Warning: The --simple and --shell modes execute prompts via shell commands. Prompts are escaped but do not use untrusted prompt content with these modes. Malicious prompt text could potentially escape the quoting and execute arbitrary commands. Use --schema mode (headless adapter) for untrusted inputs.

Extract Command

Parse raw output into structured trajectories:

# From file
bunx @plaited/agent-eval-harness extract raw.jsonl --schema claude.json -o extracted.jsonl

# Piped from run
bunx @plaited/agent-eval-harness run prompts.jsonl -s claude.json | \
  bunx @plaited/agent-eval-harness extract -s claude.json

Grade Command

Apply grader to extracted results:

bunx @plaited/agent-eval-harness grade extracted.jsonl --grader ./grader.ts -o graded.jsonl

Format Command

Convert results to different output formats:

# Markdown report
bunx @plaited/agent-eval-harness format results.jsonl --style markdown -o report.md

# CSV for spreadsheets
bunx @plaited/agent-eval-harness format results.jsonl --style csv -o results.csv

# JSONL (pass-through, default)
bunx @plaited/agent-eval-harness format results.jsonl --style jsonl

Compare Command

Compare multiple runs of the same prompts. Supports both CaptureResult (single-run) and TrialResult (multi-run reliability) formats with auto-detection.

# Default: auto-detect format, weighted strategy, JSON output
bunx @plaited/agent-eval-harness compare run1.jsonl run2.jsonl -o comparison.json

# Statistical significance strategy
bunx @plaited/agent-eval-harness compare run1.jsonl run2.jsonl --strategy statistical -o comparison.json

# Custom weights via environment variables (CaptureResult)
COMPARE_QUALITY=0.7 COMPARE_LATENCY=0.2 COMPARE_RELIABILITY=0.1 \
  bunx @plaited/agent-eval-harness compare run1.jsonl run2.jsonl -o comparison.json

# Markdown report format
bunx @plaited/agent-eval-harness compare run1.jsonl run2.jsonl --format markdown -o report.md

# Custom grader (LLM-as-Judge)
bunx @plaited/agent-eval-harness compare run1.jsonl run2.jsonl \
  --strategy custom --grader ./my-llm-judge.ts -o comparison.json

# With explicit labels
bunx @plaited/agent-eval-harness compare \
  --run "with-mcp:results-mcp.jsonl" \
  --run "vanilla:results-vanilla.jsonl" \
  -o comparison.json

Use cases for compare:

• Same agent, different MCP servers
• Same agent, different skills enabled
• Same agent, different model versions
• Different agents entirely

Trials Comparison (pass@k Analysis)

Compare TrialResult files for reliability analysis:

# Auto-detect trials format
bunx @plaited/agent-eval-harness compare trials1.jsonl trials2.jsonl -o comparison.json

# Explicit format (skip auto-detection)
bunx @plaited/agent-eval-harness compare trials1.jsonl trials2.jsonl --input-format trials -o comparison.json

# Custom weights for trials comparison
COMPARE_CAPABILITY=0.5 COMPARE_RELIABILITY=0.3 COMPARE_CONSISTENCY=0.2 \
  bunx @plaited/agent-eval-harness compare trials1.jsonl trials2.jsonl -o comparison.json

Trials metrics:

| Metric | Description | Formula | |--------|-------------|---------| | Capability (passAtK) | Can solve at least once in K tries | 1 - (1-p)^k | | Reliability (passExpK) | Solves consistently every time | p^k | | Flakiness | Gap between capability and reliability | passAtK - passExpK | | Quality (scores) | Aggregate grader scores across trials | avg/median/p25/p75 (only with grader) | | Performance (latency) | Aggregate trial durations | p50/p90/p99/mean/min/max (always present) |

Built-in Comparison Strategies

For CaptureResult (single-run):

| Strategy | Description | Env Vars | |----------|-------------|----------| | weighted (default) | Quality, latency, reliability | COMPARE_QUALITY, COMPARE_LATENCY, COMPARE_RELIABILITY | | statistical | Bootstrap for confidence intervals | COMPARE_BOOTSTRAP_ITERATIONS | | custom | Your own grader | --grader path |

For TrialResult (multi-run):

| Strategy | Description | Env Vars | |----------|-------------|----------| | weighted (default) | Capability, reliability, consistency | COMPARE_CAPABILITY, COMPARE_RELIABILITY, COMPARE_CONSISTENCY | | statistical | Bootstrap passAtK confidence intervals | COMPARE_BOOTSTRAP_ITERATIONS | | custom | Your own grader | --grader path |

Comparison Report Output

CaptureResult format outputs ComparisonReport:

{
  "meta": { "generatedAt": "...", "runs": ["baseline", "variant"], "promptCount": 100 },
  "quality": { "baseline": { "avgScore": 0.85, "passRate": 0.82 }, "variant": { ... } },
  "performance": { "baseline": { "latency": { "p50": 1200, "p90": 3400 } }, ... },
  "reliability": { "baseline": { "type": "run", "toolErrors": 5, "completionRate": 0.99 }, ... },
  "headToHead": { "pairwise": [{ "runA": "baseline", "runB": "variant", "aWins": 35, "bWins": 55 }] }
}

With --strategy statistical, quality and performance metrics include 95% confidence intervals:

{
  "quality": {
    "baseline": {
      "avgScore": 0.85,
      "passRate": 0.82,
      "confidenceIntervals": {
        "avgScore": [0.82, 0.88],
        "passRate": [0.79, 0.85]
      }
    }
  },
  "performance": {
    "baseline": {
      "latency": { "p50": 1200, "mean": 1350 },
      "confidenceIntervals": {
        "latencyMean": [1280, 1420]
      }
    }
  }
}

TrialResult format outputs TrialsComparisonReport:

{
  "meta": { "generatedAt": "...", "runs": ["claude", "gemini"], "p

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [youdotcom-oss](https://github.com/youdotcom-oss)
- **Source:** [youdotcom-oss/agent-skills](https://github.com/youdotcom-oss/agent-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.