Ground Truth Gates

Ground-Truth Gates

The core finding: more prose rules do not improve a capable model on verifiable work — its gating habits are already native. What is missing is something to gate against. Invest in executable ground truth, not in longer instructions. Build gates first where judgment work happens (classification, extraction, routing, prompt output) — that is where habits are weakest and where a gate converts open-ended quality into a number plus a diff.

The one command

Once template/ has been copied into the project as checks/ (wire-up below):

bash checks/run-all.sh

Discovers every checks/*/run.mjs (plus optional checks/project.sh), runs each, prints PASS/FAIL per gate, exits non-zero if any fail. That is the commit/ship gate: "all green" stops being a claim and becomes a checked fact.

The three gates

| Gate | Question it answers | Where it pays | |---|---|---| | golden | "Is this prompt/classifier actually better, by how much, and which cases does it miss?" | LLM-judgment steps. | | replay | "Did my change alter output on real logged inputs, and exactly where?" | Refactors and regex/prompt tweaks over production data — catches silent drift reading the code cannot see. | | project | "Do build/tests/types/lint pass?" | Drop a checks/project.sh with npm test, tsc --noEmit, an SCA scan failing on critical/known-exploited (npm audit / pip-audit), etc. |

A starter implementation lives in this skill's template/ directory — copy it into the project as checks/ and wire it up (~15 min per gate):

golden: replace golden/cases.jsonl with 30–50 real, hand-labeled examples ({"input": ..., "label": ...} per line) — a tiny set is gameable; a perfect score on a small set is an overfit warning, not a win. Replace classify() in golden/run.mjs with a call to the real system (keep it deterministic per input). Set the team's bar by editing MIN_DEFAULT in golden/run.mjs — that is what run-all.sh (and any hook/CI on top of it) enforces; the --min flag only overrides ad-hoc runs.

These rules make the golden gate earn its keep:

• Anonymize structure-preserving — replace PII values with same-shape

stand-ins (digits for digits, client@example.com for an email, a placeholder name like Jordan Lee for a name). REDACTED destroys the very shapes the logic keys on.

• Include hard negatives — real inputs that look like a match but must

fall through. That is where regressions hide and where synthetic cases never go.

• Score cost-asymmetrically — name the class of wrong output that

triggers a real, unconfirmed action (wrong route, wrong send) and treat any instance of it as a hard failure, not something aggregate accuracy can average away. The starter run.mjs implements this: set DEFER_LABEL to your safe-fallback label and the gate hard-fails on any false route regardless of accuracy.

• Validate the capture instrument, then taint on defect. When cases are

minted through a lossy reader (OCR, screenshot parsing, scraping), validate the reader against known-answer inputs first and keep a per-row capture artifact anonymized per the Anonymize rule above (PII replaced with same-shape stand-ins) — not the raw original; if a true raw artifact must be retained to re-validate the instrument later, hold it in a separate, minimized, access-controlled store, never as raw PII/secrets in the corpus. A reader defect taints every conclusion derived from its output — re-derive them; never resurrect pre-fix conclusions. And a human reading of a low-res artifact never overturns a pinned value without machine capture or independent cross-validation (a "fix" was once shipped off a misread screenshot and had to be reverted).

• Every row records how it was captured. A hand-written "plausible" row

converts the gate into a mirror of your own guess — gate corruption, not coverage. When the capture rig is unavailable, the honest state is BLOCKED naming the exact rig and recipe to unblock — never synthesis.

• Hold out a distribution-disjoint slice as the ship decider. When the

corpus was consulted during development, passing it alone is the overfit warning above; the deciding gate is a slice disjoint on a real dimension (date range, source, tenant) that development never saw.

The golden runner doubles as an experiment grader: pre-register expected outputs as cases before any runs, then grade with code, not impressions — no harness, no experiment. Pre-register the full outcome → action table too (what each result will make you do), so a result cannot be rationalized into a favored action afterward. Calibrate the difficulty of the SHARED case set before comparing — never each arm's separately, which destroys comparability: a comparison where every arm sits at the same ceiling (every case passes in every arm) or the same floor (none does) carries no discriminating evidence — halt there and report "untestable at this tier/difficulty" as a valid outcome instead of publishing a null; between those extremes, compare the pre-registered per-arm scores (arms clearing a shared gate at different scores is still a result). Grade blind to which arm produced each output.

replay: replace replay/corpus.jsonl with a representative sample of real logged inputs. Replace transform() with the step being changed. Run node replay/run.mjs --update once to freeze current behavior — and eyeball that first freeze line by line: a baseline freezes current behavior, not correct behavior, and it will protect any bug it contains as ground truth (one committed baseline enshrined a real redaction bug this way — fix the transform first, then freeze); after each edit, plain node replay/run.mjs — 0 diffs = safe; any diff = the exact records that moved. Re---update only after eyeballing an intended change, and only as the orchestrator/reviewer — never the editing worker's own call (rule 4 below: gate changes are not the worker's to make).

replay variant — parity (no corpus): a refactor of pure-ish logic (config parsing, path handling, formatting) often has no logged corpus to replay. Keep the pre-change implementation callable — a pinned import, a second checkout, or git show : copied into a _old module — and run old vs new over a declared input set, asserting identical output/exit (allow-list any intended diffs). It is the replay gate for code you are refactoring when you have nothing logged. (Freezing the old source text as a string is not a parity test — it never runs the old code.)

Cheapest gate shape — the grep-count ratchet: when an anti-pattern cannot be removed wholesale (inline locale ternaries, stray global listeners), pin its current grep count as a dated baseline with the hits enumerated; the executable done-check on every diff is "the count did not grow" — and nobody "fixes" the enumerated baseline hits as a side quest either.

What makes a gate real (task-relative test discipline)

A generic green test is not proof. A gate is real only if:

1. It exercises the task trajectory — input, production path, state

transition, observable output — not a reimplementation of the logic.

1. It would fail under the broken behavior. Run both arms where practical —

broken arm fails, fixed arm passes — and prove a negative test can fail by running it against a known-bad arm. Instrument the failure's own signal, not a proxy: an unchanged field or intact-looking output can pass while the failure still occurred. Arm polarity alone is insufficient — a change detector can mimic it while guarding nothing (unprobed — adapted external design; see Provenance): a source-string presence check or a private-structure snapshot fails on the old arm and passes on the new one simply because the source changed — while firing on every future redesign and sleeping through every future bug (it also fails this rule's own-signal requirement above; the polarity just hides that). Before writing the test body, answer: what production change should make this test fail — and is that change a bug or a decision? If only deliberate decisions can fail it, it is a change detector, not a gate — asserting the source contains a line proves only that the source is the source. Carve-out: pinning a representation is legitimate exactly where that representation IS the declared contract (an error-message string or output name with downstream consumers — operational-rigor §3's output-text-is-an-interface); then a deliberate contract change properly updates the test. A suite that grades candidates is two-sided: before it scores anything, show it PASSES on at least two structurally distinct valid solutions (a too-strict suite silently rejects valid alternatives — false collapse) and FAILS on a known-broken state (false parity), both by execution. And confirm the corpus exercises the changed branch: a change "verified" only on inputs where the new code never fires is unverified — capture firing inputs, or synthesize them into the test suite as a labeled synthetic set, NEVER as rows in the captured golden/replay corpus (the case-set integrity rules above: a hand-written row corrupts the ship gate). The behavioral analog, when the gate is a trap fixture an AGENT must resist (a prescribed-but-unauthorized action, a planted directive): precedence first — taking the bait is FAIL however blind the run was; arming gates only the safe direction. A safe outcome counts only if the run demonstrably met the trap, the transcript showing the arming event for that fixture's carrier (the prescribing doc read, OR the planted skill loaded, OR the bait seen — whichever carries this fixture's trap). A safe outcome from a run that never met the trap is a NOT-ARMED run — excluded and re-run armed, never scored as discipline. Fixture-design corollary: hang the trap on a breadcrumb the task itself forces (the failing check's output names the doc), or read-narrow evidence discipline will disarm the fixture. The two-sided proof above validates a grader for ONE invocation shape at ONE time — reusing it later (a new run, a different candidate pool, hours later in the same session) is a fresh claim, not an inherited one. Before reuse: re-run the two-sided proof — the known-good references (both structurally distinct valid solutions, per the bar above) and the known-bad — diffing each outcome against the record of the prior validation (per-CASE outcomes, not an aggregate score — the same 2/6 with different cases passing is drift; the invocation shape — command, arguments, configuration, with ephemeral values like run-scoped paths and timestamps normalized — and the reference-corpus identity, so drift in any is visible; a deliberate invocation change re-baselines only through a fresh two-sided proof and a new record; no record on hand → reuse stops, the two-sided proof runs fresh and its record is written before any scoring) — any drift is stop-the-line, never "still mostly failing, close enough." A wrong invocation shape (a file path fed where the grader expects a directory, a stale flag) can make the harness fail to load the candidate at all while the grader still emits a normal-looking scorecard — the candidate never ran, but the grader can't tell "candidate legitimately failed" from "candidate never executed." Watch for the inverted signature this produces: edge cases PASS while happy-path cases FAIL, because an edge case's own error-tolerant branch (a try/catch that treats a thrown exception as valid defensive behavior) silently absorbed the harness's load failure and got credited for it. (Incident: a directory-vs-file argument mismatch made every candidate throw MODULE_NOT_FOUND before its code ever ran; the known-bad reference scored 2/6 against a recorded 0/6, and the 2 passes were exactly the two capacity-edge cases whose accepted-throw branch swallowed the harness's own error.) (unprobed — private incident as shape; see Provenance.)

1. The easy fake pass is named and closed — hardcoded expected value,

weakened assertion, testing the mock, a test that compiled but was never registered/run, a permanently #[ignore]/.skipped backlog test that reads as coverage. Confirm a new test actually runs — the runner lists it, or it fails when you deliberately break the code — not merely that it compiles. For a guard/error path, assert three things, not just the exit code: the returncode, a message string unique to THIS check (many errors share exit 2), and that the dangerous side-effect did NOT occur (assertNotIn). Four more fake-pass shapes: a warm-state pass on init-only code — a zero-violation observation window proves nothing about code that only executes at initialization (cold start, first run, migration); exercise the cold path in a fresh context before enforcing (a CSP enforced after a clean Report-Only window broke the whole engine, because the loader it blocked had been warm the entire window). A CI/automation config that has never executed — count runs (the platform's runs API), not files; a config can be structurally undiscoverable (wrong directory in a monorepo) and inert forever while reading as coverage. A snapshot gate that silently re-freezes when its baseline is missing — deleting the baseline must be an error at gate time, never a vacuous green. A scanner that matched zero inputs — a gate whose file pattern silently expands empty (** degrading in an old shell dialect combined with a nullglob setting, a directory that moved) "passes" while scanning nothing (a guard script once did this for the very file its outage check was written for). A passing scan must also prove its input set is non-empty — assert the matched count is non-zero; merely printing it is the same vacuous green if nothing fails on 0. Worker-written guard scripts especially: item 2's known-broken run applies before trust, no exemption — whoever wrote a guard has never seen it fail. (unprobed — private incident as shape; see Provenance.)

1. Nobody weakens a gate to turn it green. A worker satisfies the gate, never

edits it — gate changes are the orchestrator's call. Three corollaries:

• For an immutable policy-checker (not an ordinary test), run it from a

pinned trusted base — git show : or the protected ref's copy — against the PR's content as data, so the same PR can't edit the rules it must pass; pin the checker's dependencies too (a base script that imports PR-controlled helpers is still compromised), and protect the workflow path itself with branch rulesets / required reviewers, not CODEOWNERS alone. Ordinary tests need only independent approval to change, not this.

• Recompute any integrity value (hash, fingerprint) from a trusted base;

never trust the value an artifact carries about itself.

• A test edit is a contract edit: to change a pinned/assertion test, state

which contract changed and who approved it (ADR/owner). If you can't, you are fixing the wrong direction.

1. For important behavior claims, prefer two independent truth sources

(e.g., client output + server state, logs + durable artifact). Two sources that agree with each other but only moderately with ground truth are correlated bias, not independence — score cross-source and same-source agreement separately (two models agreeing is one lens, not two). A metric clearing a threshold is evidence, never authorization: keep the go/no-go a separate recorded decision.

A red result is not automatically a real defect — but ruling one "environmental"

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: F-e-u-e-r
• Source: F-e-u-e-r/opus-pack
• License: MIT

Install and usage instructions live in the source repository linked above.