Evaluating Skills With Models
Evaluate skills by executing them across sonnet, opus, and haiku models using sub-agents. Use when testing if a skill works correctly, comparing model performance, or finding the cheapest compatible model. Returns numeric scores (0-100) to differentiate model capabilities.
Install
$ agentstack add skill-taisukeoe-agentic-ai-skills-creator-evaluating-skills-with-models ✓ scanned · ✓ verified — works with Claude Code, Cursor, and more.
Security review
✓ PassedNo issues found. Passed automated security review. · v0.1.0 How review works →
- ✓ Prompt-injection patterns
- ✓ Secret / credential exfiltration
- ✓ Dangerous shell & filesystem operations
- ✓ Untrusted network calls
- ✓ Known-malicious package signatures
What it can access
- ✓ Network access No
- ✓ Filesystem access No
- ✓ Shell / process execution No
- ✓ Environment & secrets No
- ✓ Dynamic code execution No
From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.
About
Evaluating Skills with Models
Evaluate skills across multiple Claude models using sub-agents with quality-based scoring.
> Requirement: Claude Code CLI only. Not available in Claude.ai.
Why Quality-Based Scoring
Binary pass/fail ("did it do X?") fails to differentiate models - all models can "do the steps." The difference is how well they do them. This skill uses weighted scoring to reveal capability differences.
Workflow
Step 1: Load Test Scenarios
Check for tests/scenarios.md in the target skill directory.
Default to difficult scenarios: When multiple scenarios exist, prioritize Hard or Medium difficulty scenarios for evaluation. Easy scenarios often don't show meaningful differences between models and aren't realistic for production use.
Required scenario format:
## Scenario: [Name]
**Difficulty:** Easy | Medium | Hard | Edge-case
**Query:** User request that triggers this skill
**Expected behaviors:**
1. [Action description]
- **Minimum:** What counts as "did it"
- **Quality criteria:** What "did it well" looks like
- **Haiku pitfall:** Common failure mode
- **Weight:** 1-5
**Output validation:** (optional)
- Pattern: `regex`
- Line count: ` **Note:** Token usage includes both skill execution AND reporting overhead. Sonnet tends to produce detailed reports, which inflates token count. Compare tool uses for execution efficiency.
## Quick Reference
Load scenarios (prioritize Hard) → Execute (parallel) → Score behaviors → Calculate totals → Add to README → Output summary
For detailed scoring guidelines, see [references/evaluation-structure.md](references/evaluation-structure.md).
## Source & license
This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.
- **Author:** [taisukeoe](https://github.com/taisukeoe)
- **Source:** [taisukeoe/agentic-ai-skills-creator](https://github.com/taisukeoe/agentic-ai-skills-creator)
- **License:** Apache-2.0
Install and usage instructions live in the source repository linked above.
Reviews
No reviews yet — be the first.
Write a review
Versions
- v0.1.0 Imported from the upstream source.