Quality Check Eval

Quality Check Eval

Use this when an AI workflow has quality checks, review notes, rubric items, or acceptance criteria, and you need to decide how each one should be handled.

The skill sorts each check into one of four buckets: automate in code, judge with an LLM, keep for human review, or remove.

This is useful when reviews are too manual, evals are vague, or a team is unsure which quality checks are worth automating.

Load references/eval-check-routing.md when auditing. That file owns the stable recommendation taxonomy, scoring fields, deterministic alternatives, and report template.

Use This When

Use this skill when the user asks to:

• audit type: judge, manual, rubric, or subjective eval checks
• decide whether to build an LLM-judge runner
• review skipped eval coverage
• rank eval checks by frequency, stakes, and automation value
• convert subjective checks into deterministic checks where possible

Contract

Produces: a markdown audit report in the current repo, plus a short chat summary.

Does not produce: runner implementation, schema migration, scheduler changes, or production writes unless the user explicitly asks after the audit.

Assumptions To State

Before acting, state:

• which repo/worktree you are auditing
• where eval checks live
• where eval logs or recent artifacts live
• what report path you will write

If any of these are unclear and not discoverable from local files, ask one concise question.

Audit Workflow

1. Find eval checks.

Use rg and local file inspection. Look for terms such as type: judge, type: manual, success_criteria, rubric, judge, manual, criteria, and eval.

1. Catalog eval checks.

Build a table with: check id, parent job/file, type, rubric/check text, expected evidence, and current evaluator path if any.

1. Scan execution signal.

Read eval logs, run reports, or recent output artifacts. Count how often each check is skipped, manually reviewed, failed, or would have been relevant. Prefer actual logs over speculation.

1. Check deterministic alternatives.

Use the deterministic alternatives in references/eval-check-routing.md.

1. Spot-check real artifacts.

Inspect 2 to 5 recent representative outputs. Ask whether each check would have caught a real quality issue. If it would not have fired, call that out.

1. Score each check.

Use the scoring fields in references/eval-check-routing.md.

1. Recommend one action.

For each check, choose exactly one action from references/eval-check-routing.md.

1. Write the report.

Save inside the audited repo. Prefer an existing eval/report folder. If none exists, use research/ or reports/. Use the report template in references/eval-check-routing.md.

Report Template

Use references/eval-check-routing.md.

Output Rules

• Be direct. Do not automate checks just because they exist.
• Prefer deterministic checks over LLM judges when the signal can be verified mechanically.
• Separate "valuable review question" from "worth automating."
• Include file paths and line references for important claims.
• End with a short summary: top promotions, auto-conversions, drops, and rough implementation effort.
• Provide a computer:// link to the saved report.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mattdweigand-sketch
• Source: mattdweigand-sketch/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.