# Verify Feature

> Verify a code change actually works by building/running the app and observing it at its real surface (CLI, API, UI, library, agent), capturing runtime evidence rather than trusting tests. Make sure to use this skill whenever the user has changed code and wants to know it works, is about to merge/push and wants confidence, says "did this actually work", "verify this works", "prove it works", "conf…

- **Type:** Skill
- **Install:** `agentstack add skill-exiao-pm-skills-verify-feature`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [exiao](https://agentstack.voostack.com/s/exiao)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [exiao](https://github.com/exiao)
- **Source:** https://github.com/exiao/pm-skills/tree/main/qa/verify-feature

## Install

```sh
agentstack add skill-exiao-pm-skills-verify-feature
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Verify Feature

Verification is runtime observation. You build the app, run it, drive it to where
the changed code executes, and capture what you see. That capture is your evidence.
Nothing else is.

This is the difference between "I ran the tests" and "I drove the actual changed
code through its real interface and watched it behave." Only the second is
verification.

## What verification is NOT

- **Not running tests.** Running them proves you can run CI, not that the change
  works. Not as a warm-up, not "just to be sure," not as a regression sweep after.
  The time goes to running the app instead.
- **Not typechecking.** Same reason.
- **Not import-and-call.** `from src.foo import bar; print(bar(x))` is a unit test
  you just wrote. Calling a function in isolation only proves the function works in
  isolation. Find the real caller that reaches a user-facing surface.
- **Not re-reading the code or re-running CI.**

## Step 1 — Find the change

The diff is ground truth. Any description of it is a claim. Read both; if they
disagree, that's a finding.

```bash
git diff --stat origin/HEAD...   # committed vs base
git diff HEAD --stat             # uncommitted working tree
gh pr diff                    # PR context
```

State the commit count. If the diff is large, redirect to a file and read it. If
there's no repo diff, say so and use the scope the user named.

## Step 2 — Identify the runtime surface

The surface is where a user or program interacts with the changed behavior. Map the
change to its surface and drive it there:

| Change reaches | Surface | What you do |
|----------------|---------|-------------|
| CLI / TUI | terminal | type the command, capture the pane (ShellExec; `interactive-cli-driving` for prompts) |
| Server / API | socket | send the request, capture the response (curl/httpie) |
| Web / UI (browser) | pixels | drive it and capture with **shot-scraper** — mandatory, see below |
| iOS-native GUI | pixels | drive it (`ios-simulator`), screenshot |

**Single component the full app can't build/reach?** When the change is one frontend
component and the production build OOMs or the surface sits behind auth/a paywall, mount
just that component in a throwaway Vite harness and screenshot the REAL component (not a
mock). Recipe in `references/isolated-component-harness.md`.
| Library | package boundary | exercise the public export — `import pkg`, NOT `import ./src/...` |
| Prompt / agent config | the agent | run the agent, capture its behavior |
| CI workflow | Actions | dispatch it, read the run |

**Internal functions are not surfaces.** Something in the repo calls the changed
function, and that caller ends at one of the rows above. Follow it there. A bash
auth gate is verified by observing the CLI prompt/auto-allow behavior, not by
inspecting the function's return value.

## Step 3 — Drive the changed code

Use the smallest path that makes the changed code execute:
- Changed a flag → run with that flag.
- Changed a handler → hit that route.
- Changed error handling → trigger that error.
- Changed an internal function → find the CLI/request/render path that reaches it.

Read your plan back before running: does this path actually execute the changed
lines? If not, pick a different entry point.

**Destructive path?** If the change touches code that deletes, publishes, sends, or
writes outside the workspace and there's no dry-run or safe target, don't drive it
live. Verify what you can around it and say in the report which path you did NOT
exercise and why.

## Web/UI surface — capture is mandatory, via shot-scraper

When the changed code renders in a browser (a web page, a React/Vue component, any
HTML surface), a PASS is **not valid without an attached shot-scraper capture** of the
changed UI. "It built" and "the console is clean" are not evidence a human can see; a
screenshot is. Screenshot is the floor; record a video when the change is an
interaction (a flow, an animation, a state transition), not just a static render.

```bash
command -v shot-scraper >/dev/null || uv tool install shot-scraper
shot-scraper install   # bundled chromium; no-op if already present

# static render → screenshot
shot-scraper "http://localhost:/" -o /tmp/verify-.png --wait 1500

# interaction → video (drive it with a JS/selector script; shot-scraper
# supports a YAML-config form for multi-step interactions)
shot-scraper video /tmp/verify-.yml --mp4 --timeout 60000
```

Attach the PNG (or MP4) to the report/PR. If the surface is behind auth or a paywall or
the full build won't come up, mount just the changed component in a throwaway Vite
harness (see `references/isolated-component-harness.md`) and shot-scraper THAT — the real
component, never a mock. A UI change reported PASS with no attached capture is not
verified; downgrade it to BLOCKED and say the capture step didn't run.

### An interaction capture must perform the interaction it claims

If the change is a CLICK (a link becoming clickable, a button, a flow), the video/capture
has to actually PERFORM that click and show its RESULT — not reveal, scroll, and stop.
A recording whose script only expands a ``, toggles a "show all", or scrolls to
bring the element into frame is a screenshot in motion; it proves the element is *visible*,
never that it *works*. Before attaching a video, reread its own driver script: does a step
click the exact target the claim is about, and does a later step capture what that click
produced (the new page, the opened file, the changed state)? If not, the capture does not
support a PASS for the interaction — it supports "renders", nothing more.

Burned once (CPE diligence source pills, PR #259): the "click flow" video only clicked the
`show all` toggle and parked the cursor on the one INERT pill, never clicking the three
working evidence links. It was submitted as proof the links open their evidence. It wasn't.

### `target="_blank"` / new-tab links defeat naive browser-click tools

An anchor with `target="_blank"` opens a NEW tab/window. A `browser_click` on it commonly
**hangs and times out** (the tool waits on same-tab navigation that never happens) or errors,
because the result landed in a tab the tool isn't driving. That timeout is a TOOL limitation,
not evidence the link is broken — and it is NOT permission to fall back to `curl` and then
report "the browser click reached the route." curl proves the ROUTE serves; it says nothing
about whether the ANCHOR the user clicks is wired to it. Keep the two claims separate. To
actually observe the click-through, do one of:

- assert the new tab/page: after the click, switch to the newly-opened target and capture
  ITS content (the raw JSON/markdown/page the link points at);
- or drive it same-tab: in the record script, read the anchor's `href` and `page.goto` it (or
  temporarily strip `target="_blank"` in the DOM before clicking) so the navigation is
  observable and recordable;
- or, if you genuinely cannot make the browser follow it, say so explicitly: "link → route
  wiring confirmed by the anchor's href matching a curl-verified endpoint; the in-browser
  click-through was NOT observed (new-tab tooling limitation)." That is an honest partial,
  not a PASS on "clicking the link opens the evidence."

Never merge a curl result and a failed browser click into a single sentence that implies the
click succeeded.

## When you ARE the evaluator in a generator/evaluator loop

If verification is happening inside an agent loop (a separate evaluator judging a
generator's work), the rule sharpens: the evaluator's verification surface is the
RUNNING APP, never the diff. Anthropic's long-running-agent harness earns its keep
exactly here — its two highlighted catches were a FastAPI route-ordering bug that
returned 422 and a delete-key handler with `AND` where it needed `OR`. Both passed
every unit test and would have passed CI; they only surfaced because the evaluator
drove the live app (Playwright / computer-use) and observed the actual behavior. An
evaluator that reads diffs is doing code review, not verification. Drive the app, or
you miss the class of bug the loop exists to catch.

## The inert-feature trap (green tests, zero production effect)

A whole class of "verified" change does nothing in production even though every
test passes. The tests pass *because* they exercise the new code with synthetic
inputs the author constructed; the real corpus or pipeline never feeds it, so it
sits dead. Green CI is not evidence the feature *helps* — only that the code
runs when called. The user's standard is "try it on real data" / "does it
actually help", not "do the tests pass."

Three recurring shapes (all seen shipping as PRs with passing tests, all inert):

1. **Heuristic that never fires on real input.** A markdown-link/relationship
   extractor inferred edge types from keywords on the link's own line. Tests fed
   prose like "Apple competes with Samsung" → typed edges. Real memos put the
   relationship in the `## Competitors` / `## Suppliers` *header* and list bare
   links under it, so inference returned the `mentions` fallback for 100% of real
   edges. Tests green; feature useless.
2. **Validator/gate guarding a marker nothing uses.** A CI gate enforced that a
   sentinel comment appears only in allowlisted files — but the marker was in
   ZERO real files, and a bad actor adding a rogue write won't self-incriminate
   with the comment. It fires only on someone who labels their own violation.
   Pure theater.
3. **Correct engine with no fuel line.** A contradiction-probe eval had real
   logic and a working LLM judge, but nothing built its input objects from actual
   retrieval (zero constructors outside its own tests) and nothing in CI or the
   pipeline called it. An orphan module.

**The probe that catches all three — run it before claiming a feature helps:**

- **Feed it the REAL corpus, not the test fixture.** Run the new function over
  actual production data (real repo files, a real DB row, a live payload) and
  read the OUTPUT. For the extractor above: real output showed every edge came
  back as the `mentions` fallback — that one line was the whole verdict.
- **Count the population the feature acts on.** "Validates pages with a
  `## Timeline`" → `grep -rln '^## Timeline' real/` returned 0 of 58. A feature
  whose trigger population is zero is inert by construction.
- **Trace the wiring both directions.** Does anything PRODUCE the input
  (`grep "TheInputType("` outside tests)? Does anything CONSUME/CALL it
  (`grep` the CI workflows, the pipeline entrypoint, the eval runner)? An orphan
  with no producer and no caller is scaffolding, not a feature.
- **Adversary test for gates/validators.** A gate that only catches a violator
  who voluntarily labels their violation catches nothing. Ask: "what does the
  person doing the wrong thing actually do, and does this fire on THAT?"

If the feature is inert, say so plainly and either wire it or recommend closing
the PR — do not report it as "verified, tests pass." See
`references/inert-feature-probe.md` for the reusable shell recipe.

## The stale-evidence trap (a confident finding measured against old data)

The inert-feature trap is "the code runs but never fires on real input." Its mirror
image: you measure a finding against artifacts on disk (a fixture, a sample
workspace, a data dump, a cached export) and assert a conclusion, but the artifacts
are STALE — so the conclusion is confidently wrong and flips the moment you check
fresh data. This bites hardest when the thing you're measuring (a schema, a
vocabulary, a label set, an output format) is itself regenerated each run, so any
single snapshot is a sample of one, not the rule.

Real shape (seen this session): a "should we add a deterministic check on appendix
H's reliability tiers" question. The conclusion flipped FOUR times, each flip forced
by checking a fresher / more authoritative source:
- #1 "the two tier vocabularies don't align" — measured against a 3-week-old test
  fixture. WRONG.
- #2 "a constrained tier-1 check ships with zero false-fails" — measured against ONE
  real but 3-week-old workspace. WRONG.
- #3 "the taxonomy is LLM-authored fresh every run, no stable schema, BLOCKED" —
  measured across 2-3 older runs whose section headers genuinely drifted. Still WRONG,
  because those runs predated a format change.
- #4 (correct) — only when I read the NEWEST output by mtime AND the CANONICAL SPEC
  (the compiler/evaluator SKILL.md that defines the lint-gated H table + fixed five-tier
  vocabulary) did it resolve: the schema IS pinned, the older "drift" was pre-pinning,
  and the check is shippable. Proven by cross-checking the freshest fixture: 28/30 rows
  matched, 0 inflation.

Four flipped conclusions, every flip because the evidence wasn't current, wasn't
plural, OR wasn't anchored to the source-of-truth spec. The user had to push three
times ("check the latest workspace", "isn't there one from today", "just look for the
latest report.md") to force each correction. Lesson beyond freshness: when the artifact
is generated by a process that has a SPEC (a prompt, a schema definition, a skill that
dictates the format), the spec is the ground truth — a drifting sample means the sample
is old or the spec changed, NOT that there's no rule. Read the spec before concluding
"no stable structure exists."

**Before asserting a finding measured from on-disk artifacts, run the freshness gate:**

- **Date the artifact.** `ls -lt` / check the dir name / read the timestamp. If it's
  a `fixtures/` or `evals/` file, assume it's frozen and possibly months behind prod
  — fixtures are pinned on purpose. Never generalize from a fixture to "what the
  pipeline does now."
- **Find the actually-newest instance.** `find . -name  -newermt ''`,
  or sort the candidate dirs by mtime. "Newest I assumed" != "newest that exists" — check
  by real mtime, not by the path that came to mind first.
- **Check more than one.** If the artifact is regenerated per run, a single snapshot
  can't tell you what's stable vs incidental. Compare 2-3 recent ones; a property that
  differs across them is NOT something you can write a deterministic check against.
- **Know where the real artifacts live.** If runs execute remotely (Modal/CI/cloud)
  and only land in object storage or a DB, the local copies are whatever someone
  happened to pull down — usually old. Say "freshest on disk is X (stale); to measure
  on a current run I need to pull from " rather than silently measuring the stale
  local copy.
- **State the basis in the finding.** "Measured against QRVO_2026-05-27 (3 weeks old)"
  lets the reader (and you, next turn) catch a stale-basis conclusion before it ships.
  A finding with no named, dated basis is a guess wearing a measurement's clothes.

If you can't get current data, say the conclusion is provisional on stale input and
name what you'd need to confirm it — don't launder a stale-data guess into a verdict.
Reusable freshness-gate recipe in `references/stale-evidence-gate.md`.

## The false-FAIL trap (red tests that are your machine, not the code)

The mirror of "green tests prove nothing": a batch of RED tests can be entirely
your local environment, not a regression — and reporting them as failures (or
worse, "fixing" the source to make them pass) is as wrong as a false PASS. Before
you treat a test failure as a real finding, ask: would this fail on a clean CI
runner too, or only on THIS box right now?

Seen this session: a broad sweep returned `59 failed, 7277 passed`. Every one of
the 59 traced to local environment state, not the branch (which didn't even touch
the failing files):

- **Exported env var flipped behavior.** A project env var (e.g.
  `=true`) was exported in the shell, which disabled a safety
  block the tests assert. 16 url/vision tests "failed."
  `env -u  pytest ...` -> 200/200 pass.
- **macOS tmp path vs Linux assumptions.** pytest's `tmp_path` lives under
  `/var/folders/...`

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [exiao](https://github.com/exiao)
- **Source:** [exiao/pm-skills](https://github.com/exiao/pm-skills)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** yes
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-exiao-pm-skills-verify-feature
- Seller: https://agentstack.voostack.com/s/exiao
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.