Anthropic Evaluations

Anthropic Evaluations

Build rigorous evaluations for AI agents using Anthropic's proven patterns.

Quick Reference

You MUST read the reference files for detailed guidance:

• [Grader Types](./references/grader-types.md) - Code-based, model-based, human graders
• [Agent Type Patterns](./references/agent-type-patterns.md) - Coding, conversational, research, computer use
• [Roadmap](./references/roadmap.md) - Steps 0-8 for building evals from scratch
• [Frameworks](./references/frameworks.md) - Harbor, Promptfoo, Braintrust, etc.

YAML Templates:

• [coding-agent-eval.yaml](./references/coding-agent-eval.yaml) - Coding agent template
• [conversational-agent-eval.yaml](./references/conversational-agent-eval.yaml) - Support agent template

Annotated Examples:

• [Example: Coding Agent](./references/example-coding-agent.md) - Auth bypass fix walkthrough
• [Example: Conversational](./references/example-conversational.md) - Refund handling walkthrough

Core Definitions

| Term | Definition | |------|------------| | Task | Single test with defined inputs and success criteria | | Trial | One attempt at a task (run multiple for consistency) | | Grader | Logic that scores agent performance; tasks can have multiple | | Transcript | Complete record of a trial (outputs, tool calls, reasoning) | | Outcome | Final state in environment (not just what agent said) | | Evaluation harness | Infrastructure that runs evals end-to-end | | Agent harness | System enabling model to act as agent (scaffold) | | Evaluation suite | Collection of tasks measuring specific capabilities |

Grader Types (Quick Reference)

| Type | Methods | Best For | |------|---------|----------| | Code-based | String match, unit tests, static analysis, state checks | Fast, cheap, objective verification | | Model-based | Rubric scoring, assertions, pairwise comparison | Nuanced, open-ended tasks | | Human | SME review, A/B testing, spot-check sampling | Gold standard calibration |

See [Grader Types](./references/grader-types.md) for detailed comparison.

Capability vs Regression Evals

| Type | Question | Target Pass Rate | |------|----------|------------------| | Capability | "What can this agent do well?" | Start low, hill-climb | | Regression | "Does it still handle what it used to?" | Near 100% |

Capability evals with high pass rates "graduate" to regression suites.

Non-Determinism Metrics

| Metric | Measures | Use When | |--------|----------|----------| | pass@k | At least 1 success in k attempts | One success matters (coding) | | pass^k | All k attempts succeed | Consistency essential (customer-facing) |

Example: 75% per-trial success rate

• pass@3 ≈ 98% (likely to get at least one)
• pass^3 ≈ 42% (0.75³ all succeed)

Tracked Metrics

tracked_metrics:
  - type: transcript
    metrics: [n_turns, n_toolcalls, n_total_tokens]
  - type: latency
    metrics: [time_to_first_token, output_tokens_per_sec, time_to_last_token]

Attribution

Based on Demystifying evals for AI agents by Anthropic (January 2026).

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: dwmkerr
• Source: dwmkerr/claude-toolkit
• License: MIT
• Homepage: https://www.skills.sh/dwmkerr/claude-toolkit

Install and usage instructions live in the source repository linked above.