Evaluate

Evaluate

Turn judgement into a frozen, repeatable assurance result. Deterministic tests still run; this skill covers behaviour whose quality cannot be proven by an exit code alone.

Contract

Freeze the canonical evaluation plan before seeing results:

• decision the evaluation informs and unacceptable failure modes;
• hash-bound dataset/corpus, provenance, consent or licence, data policy and

holdout boundary;

• candidate and applicable comparator manifests, paired sampling, seeds,

repetitions, timeouts, retries and exclusions;

• metric ranges, aggregation, thresholds and per-metric regression margins;
• deterministic preflight, rubric, blinded independent graders and

disagreement handling;

• safety, bias, privacy and adversarial cases proportionate to risk;
• enclosing delivery run when this evidence supports a non-trivial outcome.

Use [EVALUATION.template.json](templates/EVALUATION.template.json) and the [receipt contract](references/receipt.md). Keep raw examples outside the hot receipt; link safe relative artifacts and SHA-256 digests.

Run

1. Validate fixtures, hashes, schemas and leakage controls before any judgement

attempt. Deterministic failure stops or explicitly skips the frozen schedule.

1. Run each planned arm/family/repetition. Retain every attempt, retry and case

row with actual adapter/provider/model/effort lineage and usage disposition.

1. Account for passes, failures, omissions, skips, exclusions, timeouts, invalid

output and tool/provider errors. A retry appends; it never erases its parent.

1. Aggregate from retained rows. Report raw numerators/denominators,

distributions and failure clusters, rather than averages alone.

1. Blind independent graders to treatment identity. Record criterion evidence;

send disagreements to a fresh adjudicator rather than majority vote.

1. Compare with frozen thresholds and applicable arms. A post-hoc change creates

a fresh run; it does not rewrite the completed evidence.

1. Preserve bounded failures and route product defects to implement, unclear

requirements to scope, and operational regressions to diagnose.

Validate the receipt with:

${AGENTS_HOME:-$HOME/.agents}/skills/evaluate/scripts/validate_evaluation.py \
  EVALUATION.json --verify-hashes --require-pass \
  --expected-evaluation-id "$EVALUATION_ID" \
  --expected-plan-digest "$FROZEN_PLAN_DIGEST" \
  --expected-delivery-run-id "$DELIVERY_RUN_ID"

pass is a machine assurance result for the named distribution, not general safety or user acceptance. Attach it to the canonical delivery receipt; only that lifecycle's user gate can mark the outcome accepted. Unsupported receipt shapes fail closed; rerun them from a freshly frozen plan instead of migrating or guessing missing evidence.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mblauberg
• Source: mblauberg/provenant
• License: MIT

Install and usage instructions live in the source repository linked above.