Verify Feature

Verify Feature

Verification is runtime observation. You build the app, run it, drive it to where the changed code executes, and capture what you see. That capture is your evidence. Nothing else is.

This is the difference between "I ran the tests" and "I drove the actual changed code through its real interface and watched it behave." Only the second is verification.

What verification is NOT

• Not running tests. Running them proves you can run CI, not that the change

works. Not as a warm-up, not "just to be sure," not as a regression sweep after. The time goes to running the app instead.

• Not typechecking. Same reason.
• Not import-and-call. from src.foo import bar; print(bar(x)) is a unit test

you just wrote. Calling a function in isolation only proves the function works in isolation. Find the real caller that reaches a user-facing surface.

• Not re-reading the code or re-running CI.

Step 1 — Find the change

The diff is ground truth. Any description of it is a claim. Read both; if they disagree, that's a finding.

git diff --stat origin/HEAD...   # committed vs base
git diff HEAD --stat             # uncommitted working tree
gh pr diff                    # PR context

State the commit count. If the diff is large, redirect to a file and read it. If there's no repo diff, say so and use the scope the user named.

Step 2 — Identify the runtime surface

The surface is where a user or program interacts with the changed behavior. Map the change to its surface and drive it there:

| Change reaches | Surface | What you do | |----------------|---------|-------------| | CLI / TUI | terminal | type the command, capture the pane (ShellExec; interactive-cli-driving for prompts) | | Server / API | socket | send the request, capture the response (curl/httpie) | | Web / UI (browser) | pixels | drive it and capture with shot-scraper — mandatory, see below | | iOS-native GUI | pixels | drive it (ios-simulator), screenshot |

Single component the full app can't build/reach? When the change is one frontend component and the production build OOMs or the surface sits behind auth/a paywall, mount just that component in a throwaway Vite harness and screenshot the REAL component (not a mock). Recipe in references/isolated-component-harness.md. | Library | package boundary | exercise the public export — import pkg, NOT import ./src/... | | Prompt / agent config | the agent | run the agent, capture its behavior | | CI workflow | Actions | dispatch it, read the run |

Internal functions are not surfaces. Something in the repo calls the changed function, and that caller ends at one of the rows above. Follow it there. A bash auth gate is verified by observing the CLI prompt/auto-allow behavior, not by inspecting the function's return value.

Step 3 — Drive the changed code

Use the smallest path that makes the changed code execute:

• Changed a flag → run with that flag.
• Changed a handler → hit that route.
• Changed error handling → trigger that error.
• Changed an internal function → find the CLI/request/render path that reaches it.

Read your plan back before running: does this path actually execute the changed lines? If not, pick a different entry point.

Destructive path? If the change touches code that deletes, publishes, sends, or writes outside the workspace and there's no dry-run or safe target, don't drive it live. Verify what you can around it and say in the report which path you did NOT exercise and why.

Web/UI surface — capture is mandatory, via shot-scraper

When the changed code renders in a browser (a web page, a React/Vue component, any HTML surface), a PASS is not valid without an attached shot-scraper capture of the changed UI. "It built" and "the console is clean" are not evidence a human can see; a screenshot is. Screenshot is the floor; record a video when the change is an interaction (a flow, an animation, a state transition), not just a static render.

command -v shot-scraper >/dev/null || uv tool install shot-scraper
shot-scraper install   # bundled chromium; no-op if already present

# static render → screenshot
shot-scraper "http://localhost:/" -o /tmp/verify-.png --wait 1500

# interaction → video (drive it with a JS/selector script; shot-scraper
# supports a YAML-config form for multi-step interactions)
shot-scraper video /tmp/verify-.yml --mp4 --timeout 60000

Attach the PNG (or MP4) to the report/PR. If the surface is behind auth or a paywall or the full build won't come up, mount just the changed component in a throwaway Vite harness (see references/isolated-component-harness.md) and shot-scraper THAT — the real component, never a mock. A UI change reported PASS with no attached capture is not verified; downgrade it to BLOCKED and say the capture step didn't run.

An interaction capture must perform the interaction it claims

If the change is a CLICK (a link becoming clickable, a button, a flow), the video/capture has to actually PERFORM that click and show its RESULT — not reveal, scroll, and stop. A recording whose script only expands a ``, toggles a "show all", or scrolls to bring the element into frame is a screenshot in motion; it proves the element is visible, never that it works. Before attaching a video, reread its own driver script: does a step click the exact target the claim is about, and does a later step capture what that click produced (the new page, the opened file, the changed state)? If not, the capture does not support a PASS for the interaction — it supports "renders", nothing more.

Burned once (CPE diligence source pills, PR #259): the "click flow" video only clicked the show all toggle and parked the cursor on the one INERT pill, never clicking the three working evidence links. It was submitted as proof the links open their evidence. It wasn't.

target="_blank" / new-tab links defeat naive browser-click tools

An anchor with target="_blank" opens a NEW tab/window. A browser_click on it commonly hangs and times out (the tool waits on same-tab navigation that never happens) or errors, because the result landed in a tab the tool isn't driving. That timeout is a TOOL limitation, not evidence the link is broken — and it is NOT permission to fall back to curl and then report "the browser click reached the route." curl proves the ROUTE serves; it says nothing about whether the ANCHOR the user clicks is wired to it. Keep the two claims separate. To actually observe the click-through, do one of:

• assert the new tab/page: after the click, switch to the newly-opened target and capture

ITS content (the raw JSON/markdown/page the link points at);

• or drive it same-tab: in the record script, read the anchor's href and page.goto it (or

temporarily strip target="_blank" in the DOM before clicking) so the navigation is observable and recordable;

• or, if you genuinely cannot make the browser follow it, say so explicitly: "link → route

wiring confirmed by the anchor's href matching a curl-verified endpoint; the in-browser click-through was NOT observed (new-tab tooling limitation)." That is an honest partial, not a PASS on "clicking the link opens the evidence."

Never merge a curl result and a failed browser click into a single sentence that implies the click succeeded.

When you ARE the evaluator in a generator/evaluator loop

If verification is happening inside an agent loop (a separate evaluator judging a generator's work), the rule sharpens: the evaluator's verification surface is the RUNNING APP, never the diff. Anthropic's long-running-agent harness earns its keep exactly here — its two highlighted catches were a FastAPI route-ordering bug that returned 422 and a delete-key handler with AND where it needed OR. Both passed every unit test and would have passed CI; they only surfaced because the evaluator drove the live app (Playwright / computer-use) and observed the actual behavior. An evaluator that reads diffs is doing code review, not verification. Drive the app, or you miss the class of bug the loop exists to catch.

The inert-feature trap (green tests, zero production effect)

A whole class of "verified" change does nothing in production even though every test passes. The tests pass because they exercise the new code with synthetic inputs the author constructed; the real corpus or pipeline never feeds it, so it sits dead. Green CI is not evidence the feature helps — only that the code runs when called. The user's standard is "try it on real data" / "does it actually help", not "do the tests pass."

Three recurring shapes (all seen shipping as PRs with passing tests, all inert):

1. Heuristic that never fires on real input. A markdown-link/relationship

extractor inferred edge types from keywords on the link's own line. Tests fed prose like "Apple competes with Samsung" → typed edges. Real memos put the relationship in the ## Competitors / ## Suppliers header and list bare links under it, so inference returned the mentions fallback for 100% of real edges. Tests green; feature useless.

1. Validator/gate guarding a marker nothing uses. A CI gate enforced that a

sentinel comment appears only in allowlisted files — but the marker was in ZERO real files, and a bad actor adding a rogue write won't self-incriminate with the comment. It fires only on someone who labels their own violation. Pure theater.

1. Correct engine with no fuel line. A contradiction-probe eval had real

logic and a working LLM judge, but nothing built its input objects from actual retrieval (zero constructors outside its own tests) and nothing in CI or the pipeline called it. An orphan module.

The probe that catches all three — run it before claiming a feature helps:

• Feed it the REAL corpus, not the test fixture. Run the new function over

actual production data (real repo files, a real DB row, a live payload) and read the OUTPUT. For the extractor above: real output showed every edge came back as the mentions fallback — that one line was the whole verdict.

• Count the population the feature acts on. "Validates pages with a

## Timeline" → grep -rln '^## Timeline' real/ returned 0 of 58. A feature whose trigger population is zero is inert by construction.

• Trace the wiring both directions. Does anything PRODUCE the input

(grep "TheInputType(" outside tests)? Does anything CONSUME/CALL it (grep the CI workflows, the pipeline entrypoint, the eval runner)? An orphan with no producer and no caller is scaffolding, not a feature.

• Adversary test for gates/validators. A gate that only catches a violator

who voluntarily labels their violation catches nothing. Ask: "what does the person doing the wrong thing actually do, and does this fire on THAT?"

If the feature is inert, say so plainly and either wire it or recommend closing the PR — do not report it as "verified, tests pass." See references/inert-feature-probe.md for the reusable shell recipe.

The stale-evidence trap (a confident finding measured against old data)

The inert-feature trap is "the code runs but never fires on real input." Its mirror image: you measure a finding against artifacts on disk (a fixture, a sample workspace, a data dump, a cached export) and assert a conclusion, but the artifacts are STALE — so the conclusion is confidently wrong and flips the moment you check fresh data. This bites hardest when the thing you're measuring (a schema, a vocabulary, a label set, an output format) is itself regenerated each run, so any single snapshot is a sample of one, not the rule.

Real shape (seen this session): a "should we add a deterministic check on appendix H's reliability tiers" question. The conclusion flipped FOUR times, each flip forced by checking a fresher / more authoritative source:

• #1 "the two tier vocabularies don't align" — measured against a 3-week-old test

fixture. WRONG.

• #2 "a constrained tier-1 check ships with zero false-fails" — measured against ONE

real but 3-week-old workspace. WRONG.

• #3 "the taxonomy is LLM-authored fresh every run, no stable schema, BLOCKED" —

measured across 2-3 older runs whose section headers genuinely drifted. Still WRONG, because those runs predated a format change.

• #4 (correct) — only when I read the NEWEST output by mtime AND the CANONICAL SPEC

(the compiler/evaluator SKILL.md that defines the lint-gated H table + fixed five-tier vocabulary) did it resolve: the schema IS pinned, the older "drift" was pre-pinning, and the check is shippable. Proven by cross-checking the freshest fixture: 28/30 rows matched, 0 inflation.

Four flipped conclusions, every flip because the evidence wasn't current, wasn't plural, OR wasn't anchored to the source-of-truth spec. The user had to push three times ("check the latest workspace", "isn't there one from today", "just look for the latest report.md") to force each correction. Lesson beyond freshness: when the artifact is generated by a process that has a SPEC (a prompt, a schema definition, a skill that dictates the format), the spec is the ground truth — a drifting sample means the sample is old or the spec changed, NOT that there's no rule. Read the spec before concluding "no stable structure exists."

Before asserting a finding measured from on-disk artifacts, run the freshness gate:

• Date the artifact. ls -lt / check the dir name / read the timestamp. If it's

a fixtures/ or evals/ file, assume it's frozen and possibly months behind prod — fixtures are pinned on purpose. Never generalize from a fixture to "what the pipeline does now."

• Find the actually-newest instance. find . -name -newermt '',

or sort the candidate dirs by mtime. "Newest I assumed" != "newest that exists" — check by real mtime, not by the path that came to mind first.

• Check more than one. If the artifact is regenerated per run, a single snapshot

can't tell you what's stable vs incidental. Compare 2-3 recent ones; a property that differs across them is NOT something you can write a deterministic check against.

• Know where the real artifacts live. If runs execute remotely (Modal/CI/cloud)

and only land in object storage or a DB, the local copies are whatever someone happened to pull down — usually old. Say "freshest on disk is X (stale); to measure on a current run I need to pull from " rather than silently measuring the stale local copy.

• State the basis in the finding. "Measured against QRVO_2026-05-27 (3 weeks old)"

lets the reader (and you, next turn) catch a stale-basis conclusion before it ships. A finding with no named, dated basis is a guess wearing a measurement's clothes.

If you can't get current data, say the conclusion is provisional on stale input and name what you'd need to confirm it — don't launder a stale-data guess into a verdict. Reusable freshness-gate recipe in references/stale-evidence-gate.md.

The false-FAIL trap (red tests that are your machine, not the code)

The mirror of "green tests prove nothing": a batch of RED tests can be entirely your local environment, not a regression — and reporting them as failures (or worse, "fixing" the source to make them pass) is as wrong as a false PASS. Before you treat a test failure as a real finding, ask: would this fail on a clean CI runner too, or only on THIS box right now?

Seen this session: a broad sweep returned 59 failed, 7277 passed. Every one of the 59 traced to local environment state, not the branch (which didn't even touch the failing files):

• Exported env var flipped behavior. A project env var (e.g.

=true) was exported in the shell, which disabled a safety block the tests assert. 16 url/vision tests "failed." env -u pytest ... -> 200/200 pass.

• macOS tmp path vs Linux assumptions. pytest's tmp_path lives under

/var/folders/...

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: exiao
• Source: exiao/pm-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.