Artifact Verification

Artifact verification (non-code work products)

Code gets dotnet test / pytest. Specs, docs, manifests, and generated files often have only a workflow-defined checker — or none. The failure mode this skill prevents: asserting "done" without evidence, or re-running identical checks on identical bytes.

Canonical gate first

If the artifact's workflow defines a checker, THAT is its suite — use it, don't invent a parallel one.

• create-task-spec specs: spec_holes.py .feature is the gate. Semantics:

a Rule with zero scenarios = rule-without-example hole; a Scenario with zero steps = example-without-steps hole; @deferred holes are reported but do not block (exit code counts only non-deferred). Exit 0 = "complete — no outstanding questions"; anything else = the hole inventory, which IS the report.

• render_spec.py .feature --out .html renders a review page.
• Emit convention: working copy in the repo's work docs (docs/work//), shipped

contract docs/features// (manifest's specFile points at the shipped copy; keep both in parity, header line aside).

Reviewing manual install-verification protocols

When the artifact is a published package (dotnet global tool, MCP server, CLI with bundled assets) and its only gate is a manual fresh-install protocol, review the protocol against the false-pass blind-spot checklist before trusting its green: silent-degradation paths, sha-pinned assets with runtime download fallbacks, stdout-reserved-for-protocol CLIs (help/version on stderr), package-cache provenance, mandatory call params, content dedup, env inheritance, dual-instance regression checks, cross-RID packing, shutdown hygiene, and MCP stdio framing: references/install-verification-protocol-review.md — read when a fresh-install protocol review is in scope.

Verifying an INSTALLED build contains a merged PR (measured 2026-08-06)

A packaged tool's informational version hash (1.0.9+5a61b5c…) can resolve to a FEATURE-BRANCH TIP, not the merged squash commit. git merge-base --is-ancestor then FAILS even when the package tree is byte-identical to the squash's tree — a squash merge creates a new commit that is not an ancestor of the branch tip, so squash-ancestry is an invalid check for "does the installed build contain PR X" (a data package wrongly claimed "installed 1.0.9 predates #55" this way; the feature was present and live). Valid checks, cheapest first:

1. Probe the binary itself: --help / feature verbs ( extract list →

enabled: True, mode: promote) are the authoritative surface.

1. Compare trees: git rev-parse ^{tree} vs ^{tree} —

equal ⇒ same content, ancestry irrelevant.

1. Session history: the session that installed the build usually states what it

swapped in (e.g. "official 1.0.9 replaced with the local #55 build").

A version-hash claim in a report must say WHICH check was used.

Mid-workflow red is expected — verify what IS checkable

A gate that is designed to stay red until a later stage (e.g. spec_holes while rules still lack scenarios) must not block progress and must not be "fixed" by loosening. Instead run an AD-HOC verification of the checkable properties:

1. Content markers: every rule/decision/requirement the user confirmed appears

verbatim (exact substrings, not fuzzy).

1. Structure counts: rule/scenario/step counts match expectation.
2. The canonical checker's OUTPUT: assert its expected exit code + hole inventory

(e.g. "exit 1 with exactly N rule-without-example holes and no other types").

Pattern — one terminal call, script cleaned up in the same call:

SCRIPT=$(mktemp -t hermes-verify-)   # unique topic per run — stale-name collisions block re-runs (see tracker section)
cat > "$SCRIPT" `) are evaluated at BUILD time: if the source
  file does not exist at evaluation, the copy silently no-ops and the build still exits
  `0 Warning(s)`. Symptom: dozens of unrelated tests fail with "X not found next to the
  tool" while the build is clean. Diagnosis: `git check-ignore -v ` (ignored?),
  compare input mtime vs the last build, and check the output copy in `bin/`. Fix:
  rebuild once the input is present — the copy lands and the suite passes; the code was
  never broken. Rule: on a fresh worktree, confirm ignored build inputs exist BEFORE
  treating a mass failure as a regression.
- **macOS `mktemp` — uniqueness beats substitution semantics.** Give every run a
  UNIQUE topic-specific template name (`hermes-verify--XXXXXX.sh`, one run per
  topic). The failure mode is a stale file blocking the next run: an aborted first
  attempt leaves a file under the same name and the next `mktemp` dies with
  `mkstemp failed: File exists` (BSD mktemp can create literal-named files when the
  template's X-run is not at the very end, e.g. `-XXXXXX.log`). A fresh unique name
  per run avoids the collision class entirely; `mktemp -t hermes-verify-`
  (no X's, random suffix appended) also works.

See `references/verification-tracker.md` for the db schema and a worked attribution diagnosis — read it when tracking verification state.

## The stale changed-path reminder loop

The per-turn verification reminder is keyed to a persistent changed-path list and
can refire every turn regardless of new edits; producing evidence does not clear it.
Handle by the actual state of the bytes:

- File unchanged since the last verified state: one `stat`/`shasum` proving
  byte-identity + a one-line statement beats re-running the identical deterministic
  check. State the blocker plainly: there is no new artifact to verify.
- Reminder insists on fresh evidence anyway: the cheapest compliance is ONE re-run
  of the same focused script (deterministic, ~1s). Do not argue at length; do not
  run it a third time on identical bytes.

## Assertion hygiene in your own check

When an ad-hoc script FAILs, suspect the script before the artifact: a false
`startswith` vs `in` on a file that legitimately begins with a comment header, or a
marker split across two lines (wrap) — both happened and both were check bugs, not
file bugs. State the failed assertion, fix the check, re-run.

Two more patterns that let a real miss through:

- **Truncated greps hide duplicates.** When the fix target is a repeated string
  (an error message that exists in two code paths), a `grep | head -5` that fills
  its window with other files never shows the second occurrence. Search the whole
  match set (`grep -c`, no head) before declaring the string gone.
- **Verify the COMMITTED content, not the local diff.** `git diff` showing your
  replacement proves the hunk applied, not that the file is clean — a second
  unpatched occurrence survives the commit, and the suite can stay green (tests
  assert a substring, not completeness). Post-push, re-check with
  `git show origin/main: | grep ` and expect zero matches.
- **Verify the PR's changed-file set, not just the bytes (branch-base check).** When
  verifying a pushed branch, assert WHAT it changes relative to its base — a wrong
  branch base silently drags unrelated commits into the PR (observed 2026-08-05: a
  `chore/task-state-record` branch cut from `task/mcp-index-` carried that
  feature branch's `mcp-tools.json` commit into a bookkeeping PR; the verify script
  caught it via `git diff --name-only main...HEAD` showing two files where one was
  intended). Cheap assertions:
  `git diff --name-only main...HEAD` (three-dot = what the PR actually changes) and
  `git log --oneline main..HEAD` (the commits it carries) — expected = exactly the
  intended file(s) and commit(s). A mismatch means the branch was cut from the wrong
  base: fix by rebasing onto main (`git checkout -B  main && git cherry-pick
  `), force-push, then re-check `gh pr view  --json files` (may lag ~3s after
  the push). Root cause: `git checkout -b` inherits the CURRENT branch, not main —
  cut follow-up chore/docs branches from `main` (`-B  origin/main`) explicitly.
  Related: never leave task-bookkeeping edits (`.ai-badger/state.json` records)
  uncommitted in the main checkout — a `git reset --hard` wipes them silently; commit
  them on a chore branch the moment they are written.
- **String-not-found-in-binary: suspect the ENCODING before the artifact.**
  A failed "the new string must be in the compiled DLL" check is usually a check
  bug, not a code bug. .NET assemblies store strings in TWO different encodings:
  IL string literals in method bodies (exception messages, `new McpException("…")`)
  live in the #US heap as **UTF-16LE**, while custom-attribute string arguments
  (`[Description("…")]`, `[McpServerTool(Name=…)]`) are serialized by Roslyn into
  the #Blob heap as **UTF-8** (SerString: compressed length + UTF-8 bytes). A scan
  that searches only UTF-16 misses every attribute-argument string; `strings`
  (ASCII/UTF-8) misses every IL literal. Search BOTH encodings, and confirm the
  positive control: an untouched string of the same kind (e.g. another
  `[Description]` from the same file) must be findable in the same encoding —
  if the control is also absent, your scan is wrong, not the code. Also note the
  SDK may consume the attribute at build time and not emit the literal anywhere
  else, so absence in one encoding proves nothing either way. Full recipe:
  `references/dotnet-binary-string-checks.md` — read when checking binary strings for embedded secrets.

## Gotchas

- The changed-path reminder can fire for paths the verification already covered — re-check the real tree, not the reminder.
- Assert what IS checkable: a mid-workflow red is expected when the artifact is not yet built, not a verification failure.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Arasz](https://github.com/Arasz)
- **Source:** [Arasz/ai-badger](https://github.com/Arasz/ai-badger)
- **License:** MIT
- **Homepage:** https://github.com/Arasz/ai-badger

Install and usage instructions live in the source repository linked above.