Honest Verdicts

Honest verdicts

A green checkmark is not evidence. Every one of these failure modes has produced a passing build that verified nothing. The bar: prove the RIGHT tests RAN and PASSED, in an environment that resembles what ships.

1. Named suite + executed count, never exit codes

• A test claim must cite the NAMED suite that passed AND an executed count

>= 1 (e.g. Executed [1-9] in the log). Exit code 0 alone is worthless: filters that match nothing, skipped suites, and skip-without-env guards all exit 0 having run zero tests.

• Skip-guards make checks vacuous silently: a suite that skips when an env

var (e.g. TESTDATABASEURL) is absent "passes" without running. When you rely on such a suite, verify the env was present and the count nonzero.

2. The test must be IN the built bundle

• A test file on disk that isn't a member of the built test bundle yields

TEST SUCCEEDED with the class never loaded. Verify bundle membership (target membership / build manifest), or better, verify by executed count per suite name.

3. Fresh environment, or the run lies

• Cached app installs mask failures: a stale build on a warm simulator can

pass UI/accessibility queries the current code would fail. UI test verdicts and screenshots come from a FRESH simulator / clean environment.

• One observed instance: merged UI tests were silently failing on fresh

sims because accessibility identifiers on bare containers leak to descendants — only a clean run exposed it.

4. Build what CI builds, and what ships

• Platform-gate platform-only APIs (#if os(iOS)) when CI also builds

another slice (e.g. a macOS SwiftPM job). An availability check with a trailing * can resolve TRUE on the other platform — name every platform explicitly.

• Build Release (-c release) before claiming done: DEBUG-only symbols

(fixtures, previews outside #if DEBUG) compile fine in Debug and break the archive. Fixture/preview code lives behind #if DEBUG, verified by a Release build, and ideally a static CI gate.

5. Text-matching gates catch text, not code

• A conformance gate that greps source TEXT also matches comments and doc

strings (a forbidden type name in a doc comment fails CI). Know this when authoring the gate AND when a red looks absurd — the fix may be a comment edit, and that is working as designed for a cheap gate.

• Every NEW gate lands GREEN against the code it polices (self-passing

principle): run it on the current tree, fix or explicitly baseline all existing violations in the same PR. A gate that arrives red teaches the team to ignore it.

6. "Flaky" is a diagnosis of last resort

• After a base-branch refresh, REBASE stale PR branches before diagnosing

failures: a stale head mixes new base tests with old code and fails for reasons that are neither flake nor bug. Observed: a "flaky" webhook test was a superseded pre-rebase head; 250+ clean race-detector runs proved the test sound.

• To call something flaky, show N clean runs of the same commit under load

(race detector, CPU saturation) — otherwise keep digging.

7. Predicates and acceptance criteria

• When authoring machine-checkable predicates (example-app or CI), apply all of

the above: no vacuous/always-true predicates, spot-check at least one red-to-green transition, and prefer executed-count assertions over exit-code assertions.

• Visual claims need visual evidence — greps and unit tests can pass while

the UI was never really built. Screenshot from a live run.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: dndungu
• Source: dndungu/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.