Skill Eval

Skill Eval — Agent Skill Evaluation Framework

Evaluate Agent Skills across four dimensions: safety (audit), quality (functional), reliability (trigger), and cost efficiency (Pareto classification).

Quick Start

skill-eval audit /path/to/skill          # Is it safe?
skill-eval report /path/to/skill         # Full grade (audit + functional + trigger)
skill-eval functional /path/to/skill     # Quality: with-skill vs without-skill
skill-eval trigger /path/to/skill        # Reliability: activation precision

Decision Tree

• "Is this skill safe?" → skill-eval audit
• "Full evaluation with grade" → skill-eval report
• "Full repo security review" → skill-eval audit --include-all
• "Write eval cases" → skill-eval init , then edit evals/
• "Compare two versions" → skill-eval compare
• "Check for regressions" → skill-eval snapshot , then skill-eval regression
• "Track changes" → skill-eval lifecycle --save --label v1.0

Commands

| Command | Purpose | |---------|---------| | audit | Security & structure scan (secrets, permissions, spec compliance) | | functional | Quality eval — runs prompts with and without skill, grades output | | trigger | Reliability eval — tests activation precision for relevant/irrelevant queries | | report | Unified grade combining audit (40%) + functional (40%) + trigger (20%) | | compare | Side-by-side comparison of two skills on the same eval cases | | snapshot | Save current audit as regression baseline | | regression | Check for score regressions against baseline | | lifecycle | Version tracking and change detection | | init | Generate eval scaffold from SKILL.md frontmatter |

For detailed flags and examples, see references/cli-reference.md.

Eval File Format

Functional evals (evals/evals.json):

[{"id": "case-1", "prompt": "...", "assertions": ["contains 'expected'"], "files": ["files/input.csv"]}]

Trigger queries (evals/eval_queries.json):

[{"query": "relevant question", "should_trigger": true}, {"query": "unrelated question", "should_trigger": false}]

Scoring

Grades: A (90+), B (80-89), C (70-79), D (60-69), F (<60). Findings deduct: CRITICAL −25, WARNING −10, INFO −2.

For the full security check reference and OWASP mapping, see references/security-checks.md.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: aws-samples
• Source: aws-samples/sample-agent-skill-eval
• License: MIT-0

Install and usage instructions live in the source repository linked above.