# Llm Gold Bound Failure Check

> |

- **Type:** Skill
- **Install:** `agentstack add skill-kennethkhoocy-applied-micro-skills-llm-gold-bound-failure-check`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [kennethkhoocy](https://agentstack.voostack.com/s/kennethkhoocy)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [kennethkhoocy](https://github.com/kennethkhoocy)
- **Source:** https://github.com/kennethkhoocy/applied-micro-skills/tree/main/plugins/applied-micro/skills/llm-gold-bound-failure-check

## Install

```sh
agentstack add skill-kennethkhoocy-applied-micro-skills-llm-gold-bound-failure-check
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# LLM Gold-Bound Failure Check

## Problem

When an LLM scoring pipeline over-predicts one label, the reflex fix is a
prompt clarification ("score positive ONLY when..."). But if the gold standard
itself does not separate the texts you want excluded from the texts it labels
positive, the revision removes true and false positives together. The pilot
fails, the spend is wasted, and — worse — an un-gated adoption would have
silently destroyed recall in production.

## Context / Trigger Conditions

- A domain/label shows precision ≪ recall (e.g. P 0.46 / R 0.96) against gold
- A prompt edit is proposed to exclude a specific text type (boilerplate,
  affirmative-program language, non-risk framing)
- The label's gold council/inter-rater agreement was already the weakest
  (κ below ~0.6 is the warning sign that the construct is contested)

## Solution

**Step 0 — the ~$0 check, BEFORE building anything:** read a sample of gold
POSITIVES for the weak label and ask: do they contain the feature the revision
would exclude? Compare them side-by-side with the false positives.

- Gold positives and false positives are the same kind of text → the failure
  is **gold-bound**. Stop. No prompt passes a gold-scored gate. The levers are:
  (a) re-adjudicate the construct with the gold's owners (changes the gold,
  not the scores), or (b) re-interpret the shipped measure honestly (e.g.
  "discussion salience" instead of "risk exposure") in downstream analyses.
- Gold positives clearly differ from the false positives → a prompt revision
  is plausible; proceed to a gated pilot.

**Gated pilot design (verified):**
1. Split gold into tune/holdout halves, stratified on the weak label's
   positives; fixed seed.
2. Draft ONE surgical edit from tune-half errors only — byte-identical
   elsewhere; verify the diff reverses cleanly.
3. Pre-register the gate on the holdout BEFORE scoring: target-label
   thresholds (e.g. precision ≥ X AND recall ≥ Y) plus a perturbation
   tolerance for untouched labels (e.g. within 0.03 F1 / 0.06 κ of a
   same-serving-rev fresh baseline).
4. Score everything fresh under both prompts (same model revision, same day —
   this doubles as the drift control). Never write through the production
   cache layer.
5. Adopt only on a full pass; a REJECT is a valid, cheap outcome.

## Verification

The pilot report shows: the exact prompt diff, tune-vs-holdout metrics for
old and new prompts, per-label deltas on untouched sections, and spend.
A gold-bound diagnosis is confirmed when the revision moves recall sharply
down while precision stays roughly flat.

## Example

Specialist Directors US, 2026-07-16: DEI over-prediction (P 0.46 / R 0.96,
council κ 0.24–0.59). A risk-framing-only DEI clause was piloted ($1.17,
pre-registered holdout gate). Result: recall 0.895→0.263, precision
0.455 (gate ≥0.60) — REJECT. Reading the tune half showed ~¾ of gold DEI
positives were pure affirmative D&I program text, identical in kind to the
false positives; the failure was predictable at Step 0. Bonus finding: the
DEI-section-only edit left all five other domains within 0.025 F1 / 0.05 κ —
single-section prompt edits isolate cleanly, so the perturbation check is a
cheap add, not paranoia. Same pattern one week earlier: a cyber classifier
pilot gate failure traced to E/D gold contamination (misses were
skills-matrix-checkbox-only positives), not model weakness.

## Notes

- Low inter-rater κ on a label is the leading indicator: contested construct
  → gold-bound failures downstream.
- If the pipeline scores all labels in one completion, any post-campaign
  prompt change forces a full re-score — run this check BEFORE the campaign.
- See also: [llm-campaign-drift-gate] for the companion gate on resume
  boundaries and serving-revision drift (same fresh-baseline discipline).
- See also: [annotator-input-parity-check] — run it FIRST. If the model was
  never shown the document the annotators read, apparent gold-bound failures
  (e.g. the 2026-07-16 E/D "contamination" reading above) are actually input
  mismatch: the 2026-07-21 parity audit showed the specialist-director hand
  labels were pure proxy-statement transcriptions, so checkbox-only positives
  were recoverable from the right input all along.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [kennethkhoocy](https://github.com/kennethkhoocy)
- **Source:** [kennethkhoocy/applied-micro-skills](https://github.com/kennethkhoocy/applied-micro-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-kennethkhoocy-applied-micro-skills-llm-gold-bound-failure-check
- Seller: https://agentstack.voostack.com/s/kennethkhoocy
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.