Agentv Bench

AgentV Bench

A skill for evaluating agents and iteratively improving them through data-driven optimization.

At a high level, the process goes like this:

• Understand what the agent does and what "good" looks like
• Write evaluation test cases (EVAL.yaml or evals.json)
• Run the agent on those test cases, grade the outputs
• Analyze the results — what's working, what's failing, and why
• Improve the agent's prompts/skills/config based on the analysis
• Repeat until you're satisfied

Your job when using this skill is to figure out where the user is in this process and then jump in and help them progress. Maybe they want to start from scratch — help them write evals, run them, and iterate. Maybe they already have results — jump straight to analysis and improvement.

Be flexible. If the user says "I don't need a full benchmark, just help me debug this failure", do that instead.

After the agent is working well, you can also run description optimization to improve skill triggering accuracy (see references/description-optimization.md).

Communicating with the user

This skill is used by people across a wide range of familiarity with evaluation tooling. Pay attention to context cues:

• "evaluation" and "benchmark" are borderline but OK in most cases
• For "YAML", "grader", "assertion", "deterministic judge" — see serious cues from the user that they know what those mean before using them without explanation
• Briefly explain terms if in doubt

When presenting results, default to summary tables. Offer detail on request. In CI/headless mode, skip interactive prompts and exit with status codes.

---

Step 1: Understand the Agent

Before running or optimizing, understand what you're working with.

1. Read the agent's artifacts — prompts, skills, configs, recent changes. Understand the full picture: what tools are available, what the expected input/output looks like, what constraints exist.

1. Identify success criteria — what does "good" look like for this agent? What are the edge cases? What would a failure look like? Talk to the user if this isn't clear from the artifacts alone.

1. Understand the target harness — which provider runs the agent (Claude, GPT, Copilot CLI, Gemini, custom CLI)? This affects what grader types are available and how to run tests. Targets are configured in .agentv/targets.yaml (canonical location, searched from the eval file directory upward). Sensitive values like api_key must use ${{ ENV_VAR }} syntax — literal secrets are rejected as a security guardrail.

1. Challenge assumptions — if evals already exist, review their quality before running:

• Are the test cases testing the right things?
• Are assertions specific enough to catch real failures?
• Are there ambiguous or contradictory test cases?
• Flag eval issues before proceeding — running bad evals wastes time.

1. Check integrity — ensure task prompts (what the agent receives) are not also used as grader prompts (how outputs are scored). If a prompt file appears in both locations, note the overlap and optimize only for the task purpose.

---

Step 2: Write Evaluations

AgentV supports two evaluation formats:

EVAL.yaml (native, full features) — supports workspaces, code graders, multi-turn conversations, tool trajectory scoring, workspace file tracking, multi-provider targets. Use this for agent evaluation.

# example.eval.yaml
tests:
  - id: basic-code-review
    input: "Review this TypeScript file for bugs and suggest improvements"
    criteria: "Identifies the null pointer bug on line 12 and suggests a fix"
    assertions:
      - type: contains
        value: "null"
      - Review identifies the null pointer bug and suggests a concrete fix

workspace:
  template: ./workspace-template
  hooks:
    before_each:
      reset: fast

Multi-skill evaluation is handled naturally via input messages — describe the task in the test input, and the agent uses whatever skills it needs.

evals.json (skill-creator compatible) — auto-promoted to EVAL-equivalent format:

• prompt → input messages
• expected_output → reference answer
• assertions → graders
• files[] paths resolved relative to the evals.json location

{
  "skill_name": "my-agent",
  "evals": [
    {
      "id": 1,
      "prompt": "User's task prompt",
      "expected_output": "Description of expected result",
      "assertions": ["Output includes error handling", "Uses async/await"]
    }
  ]
}

Writing good test cases

Start with 2-3 realistic test cases — the kind of thing a real user would actually say. Share them with the user before running: "Here are a few test cases I'd like to try. Do these look right, or do you want to add more?"

Good assertions are objectively verifiable and have descriptive names. Subjective quality ("the output is good") is better evaluated qualitatively — don't force assertions onto things that need human judgment.

Grader types (cheapest to most expensive): exact, contains, regex, is-json, field-accuracy, composite, code-grader, tool-trajectory, llm-grader. See references/eval-yaml-spec.md for full config and grading recipes for each type.

Prefer deterministic graders over LLM graders whenever possible. If an assertion can be checked with contains or regex, don't use llm-grader.

---

Step 3: Run and Grade

This section is one continuous sequence — don't stop partway through.

Each run produces a new .agentv/results/default// directory automatically. Use timestamps to identify iterations when comparing runs.

Choosing a run mode

User instruction takes priority. If the user says "run in subagent mode", "use subagent mode", or "use CLI mode", use that mode directly.

If the user has not specified a mode, default to AgentV CLI mode.

| Mode | When to use | How | |------|-------------|-----| | AgentV CLI (default) | Normal eval runs, multi-provider benchmarking, CI, artifact generation, and Dashboard-compatible results. | agentv eval | | Subagent mode | Explicit user request, no usable CLI/runtime, or rare provider/request-cost constraints. | Read references/subagent-pipeline.md first. |

Do not read AGENT_EVAL_MODE or another environment variable to decide the default mode. User instruction always overrides the default.

Subagent mode — Parses eval.yaml directly, spawns executor subagents to run each test case in the current workspace, then spawns grader subagents to evaluate assertion types natively. This is an opt-in fallback. Read references/subagent-pipeline.md for the detailed procedure before using it.

AgentV CLI mode — AgentV CLI handles execution, grading, and artifact generation end-to-end. Works with all providers. Use this unless the user explicitly asks for subagent mode or the CLI path is unavailable.

Running evaluations

AgentV CLI mode (default, end-to-end, EVAL.yaml):

agentv eval 

Subagent mode — read references/subagent-pipeline.md for the detailed procedure. In brief: use pipeline input to extract inputs, dispatch one executor subagent per test case (all in parallel), then proceed to grading below.

Spawn all runs in the same turn. For each test case that needs both a "with change" and a "baseline" run, launch them simultaneously. Don't run one set first and come back for the other — launch everything at once so results arrive around the same time.

Multi-target benchmarking:

agentv eval  --target claude --target gpt --target copilot

Baseline strategy:

• New agent: baseline is "no prompt" or minimal prompt — same eval, no agent-specific configuration
• Improving existing: snapshot the current version before editing (cp -r /prompt-snapshot/), use as baseline throughout
• Multi-target: each target is its own baseline — no need for a separate "without" run

While runs are in progress, draft graders

Don't just wait for runs to finish — use this time productively. If assertions don't exist yet, draft them now. If they exist, review them and explain what they check to the user.

Good assertions are discriminating — they pass when the agent genuinely succeeds and fail when it doesn't. An assertion that passes for both good and bad outputs is worse than no assertion.

As runs complete, capture timing data

When each subagent task completes, you receive a notification containing total_tokens and duration_ms. Save this data immediately to timing.json in the run directory. See references/schemas.md for the timing.json schema.

This is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives.

After all executors complete

Once all executor subagent notifications arrive, read response.md directly from disk. Do NOT use read_agent to fetch executor results — they are already written to the run directory. Verify all responses exist before proceeding:

for d in //*/; do
  echo "$(basename $d): $(ls "$d"/response.md 2>/dev/null && echo OK || echo MISSING)"
done

If any response.md is MISSING, re-run that specific executor subagent. Do not proceed to grading until all responses are present.

Grading

In CLI mode, agentv eval handles all grading end-to-end — no manual phases needed.

In subagent mode, grading has three phases. All three are required — do not stop after phase 1.

Phase 1: Code graders (deterministic, zero-cost)

agentv pipeline grade 

This evaluates all deterministic assertions against response.md files. Two types are handled:

• code-grader scripts — external scripts executed against the response (arbitrary logic, any language)
• Built-in assertion types — evaluated in-process: contains, contains-any, contains-all, icontains, regex, equals, starts-with, ends-with, is-json, and variants

Both types are configured by pipeline input into code_graders/.json and graded by pipeline grade. Results are written to /code_grader_results/.json. Alternatively, pass --grader-type code to pipeline run to run these inline.

Do not dispatch LLM grader subagents for tests that only have contains, regex, or other built-in assertions — pipeline grade handles them entirely, at zero cost. To detect which tests need Phase 2, check whether /llm_graders/ contains any .json config files — pipeline input only writes there for llm-grader assertions. Tests with an empty (or missing) llm_graders/ directory are done after Phase 1.

Phase 2: LLM grading (semantic — do NOT skip this phase)

Dispatch one grader subagent per (test × LLM grader) pair, all in parallel. Do not write a script to call an LLM API instead — the grader subagents use their own reasoning, which IS the LLM grading. Example: 5 tests × 2 LLM graders = 10 grader subagents launched simultaneously.

Do NOT dispatch a single grader for multiple tests. Each subagent grades exactly one (test, grader) pair.

Before dispatching graders, read agents/grader.md and embed its full content as the system instructions in every grader subagent prompt. The grader is a general-purpose task agent — there is no auto-resolved "grader" type. Without agents/grader.md embedded verbatim, the subagent has no grading process, no output format, and no file-path knowledge, and will produce empty or incorrect output.

Each grader subagent (operating under agents/grader.md instructions):

1. Reads /llm_graders/.json for the grading prompt
2. Reads /response.md for the candidate output
3. Grades the response against the prompt criteria
4. Writes its result to disk: ///llm_grader_results/.json
5. Returns score (0.0–1.0) and per-assertion evidence to the orchestrator

Writing to disk is critical. Assertion arrays are lost if accumulated only in the orchestrator's context across multiple batches (context summarization drops detail). Writing per-test results to llm_grader_results/.json makes grading resumable and assertion evidence durable.

The result file format is:

{ "score": 0.85, "assertions": [{"text": "...", "passed": true, "evidence": "..."}] }

After all grader subagents complete, run Phase 3 directly.

Phase 3: Merge and validate

agentv pipeline bench 
agentv results validate 

pipeline bench reads LLM grader results from llm_grader_results/.json per test automatically, merges with code-grader scores, computes weighted pass_rate, and writes grading.json + index.jsonl + summary.json.

> Diagnosing pass_rate=0: If pipeline bench reports pass_rate=0 across the board, do not assume the tests genuinely failed. First verify the grading pipeline ran correctly: check that /llm_grader_results/.json exists and is non-empty for each test. If these files are absent or empty, the grader subagents failed to produce output (most common cause: agents/grader.md was not embedded in the subagent prompts — see Phase 2). Treat pass_rate=0 as a real signal only after confirming grader results exist.

Artifacts

All artifacts use established schemas — see references/schemas.md for the full definitions. Do not modify the structure. Key artifacts per run:

• grading.json: per-test assertions with {text, passed, evidence}, plus summary
• timing.json: {total_tokens, duration_ms, total_duration_seconds}
• summary.json: per-target aggregate {pass_rate, time_seconds, tokens}

Write artifacts to .agentv/artifacts/ or the iteration directory.

Workspace features (EVAL.yaml only)

• Workspace isolation — clone repos, run setup/teardown hooks (beforeall, beforeeach, aftereach, afterall)
• Materialization modes — pooled (reuse slots), temp (fresh per run), static (existing dir)
• Multi-repo — clone multiple repos with sparse checkout and shallow clone support
• File change tracking — grade by diffing workspace files before/after agent execution

---

Step 4: Analyze Results

Once all runs are graded, analyze the results before attempting improvements.

Pattern analysis

Read the JSONL results and look for:

• Always-pass tests — assertion too loose or non-discriminating. If it passes for both good and bad outputs, it's not testing anything.
• Always-fail tests — task impossible, eval broken, or assertion misconfigured. Don't optimize against broken evals.
• Flaky tests — non-deterministic results across runs. Investigate before treating failures as real.
• Systematic failures — same failure pattern across multiple tests. This usually points to a missing instruction or wrong approach.
• Deterministic upgrade candidates — llm-grader assertions that could be replaced with contains, regex, or is-json (cheaper, faster, more reliable).

Dispatch subagents

• Dispatch analyzer (read agents/analyzer.md) for a structured quality audit: deterministic upgrade suggestions, weak assertion detection, cost/quality flags, and benchmark pattern analysis.

• Dispatch comparator (read agents/comparator.md) for blind N-way comparison between iterations or targets. The comparator blinds provider identities, generates task-specific rubrics, scores each output, then unblinds and attributes improvements.

Trace analysis

Use CLI tools for deeper investigation:

agentv inspect           # Detailed execution trace inspection
agentv compare       # Structured diff between runs

Look for: tool call patterns, error recovery behavior, conversation flow, wasted steps.

Present results to the user

Show a summary table:

| Test ID          | Score | Pass/Fail | Delta | Notes                    |
|------------------|-------|-----------|-------|--------------------------|
| basic-code-review| 0.85  | ✓ PASS    | +0.15 | Found the bug this time  |
| edge-case-empty  | 0.0

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [EntityProcess](https://github.com/EntityProcess)
- **Source:** [EntityProcess/agentv](https://github.com/EntityProcess/agentv)
- **License:** MIT
- **Homepage:** https://agentv.dev

Install and usage instructions live in the source repository linked above.