Agent Eval Framework

Agent Evaluation Framework

When to Use

• Before deploying an agent to production
• After changing an agent's system prompt or skills
• When agent output quality seems to degrade
• During periodic quality reviews
• When comparing two agent configurations

Step 1: Define Evaluation Criteria

Choose criteria relevant to your agent's purpose:

Universal Criteria

| Criterion | Question | Score | |-----------|----------|-------| | Correctness | Is the output factually/technically correct? | 0-10 | | Completeness | Does it cover all required aspects? | 0-10 | | Relevance | Is every part relevant to the request? | 0-10 | | Safety | Does it avoid harmful/insecure patterns? | 0-10 |

Code-Specific Criteria

| Criterion | Question | Score | |-----------|----------|-------| | Functionality | Does the code work as intended? | 0-10 | | Edge Cases | Are edge cases handled? | 0-10 | | Style | Does it match project conventions? | 0-10 | | Security | Are there vulnerabilities? | 0-10 |

Content-Specific Criteria

| Criterion | Question | Score | |-----------|----------|-------| | Accuracy | Are claims supported by evidence? | 0-10 | | Tone | Does it match the intended audience? | 0-10 | | Structure | Is it well-organized? | 0-10 | | Originality | Does it avoid generic/cliche content? | 0-10 |

Step 2: Choose Evaluation Method

A. Assertion-Based (Automated)

Define pass/fail conditions:

ASSERT: output contains "disclaimer"
ASSERT: output does NOT contain "TODO"
ASSERT: code compiles without errors
ASSERT: response length = 90% | Ship |
| Good | 70-89% | Ship with monitoring |
| Marginal | 50-69% | Fix before shipping |
| Failing | 10% week-over-week
- Pass rate drops below threshold
- Token cost increases >20% without scope change
- New failure modes not in original test suite

## Output Format

AGENT EVAL REPORT Agent: {name} Date: {ISO-8601} Test cases: {n} Method: {assertion|reference|rubric}

Results: Pass: {n} ({%}) Fail: {n} ({%}) Average score: {x}/10

Per-criterion: Correctness: {x}/10 Completeness: {x}/10 Safety: {x}/10

Verdict: {PASS|MARGINAL|FAIL} Recommendation: {ship|fix|block}

## What This Skill Does NOT Do

- Does not test the LLM model itself (tests agent in context)
- Does not perform adversarial red-teaming (different discipline)
- Does not replace user feedback (complements it)
- Does not measure latency or throughput (APM tools do this)

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [BuilderCed](https://github.com/BuilderCed)
- **Source:** [BuilderCed/agent-skills](https://github.com/BuilderCed/agent-skills)
- **License:** MIT
- **Homepage:** https://skills.sh/BuilderCed/agent-skills

Install and usage instructions live in the source repository linked above.