Eval Harness Design
Use when agent evals need graders, pass@k, regression gates, or reliability evidence.
Install
$ agentstack add skill-jukrap-ai-agent-playbook-eval-harness-design ✓ scanned · ✓ verified — works with Claude Code, Cursor, and more.
Security review
✓ PassedNo issues found. Passed automated security review. · v0.1.0 How review works →
- ✓ Prompt-injection patterns
- ✓ Secret / credential exfiltration
- ✓ Dangerous shell & filesystem operations
- ✓ Untrusted network calls
- ✓ Known-malicious package signatures
What it can access
- ✓ Network access No
- ✓ Filesystem access No
- ✓ Shell / process execution No
- ✓ Environment & secrets No
- ✓ Dynamic code execution No
From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.
About
Eval Harness Design
Use this as the primary delivery skill for eval-driven changes to agent workflows, prompts, MCP tools, skills, or automation surfaces.
Workflow
- Define the target behavior, risk class, baseline, failure modes, and release decision before changing the harness.
- Choose deterministic code, schema, or rule graders before model or human graders.
- Separate capability evals from regression evals, then set pass@k, pass^k, cost, latency, and repeatability expectations.
- Store eval definitions and run reports as runtime evidence until reviewed and promoted.
Reference
Read references/eval-artifact-contract.md for eval definition, run report, evidence envelope, and storage boundaries.
Read references/grader-and-metric-rubric.md for grader choice, metric thresholds, pass@k usage, and anti-overfitting checks.
Source & license
This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.
- Author: jukrap
- Source: jukrap/ai-agent-playbook
- License: MIT
- Homepage: https://www.npmjs.com/package/ai-agent-playbook
Install and usage instructions live in the source repository linked above.
Reviews
No reviews yet — be the first.
Write a review
Versions
- v0.1.0 Imported from the upstream source.