Crucible

crucible

!checklist init ${CLAUDE_SKILL_DIR} --force

A crucible is the vessel you put a substance into to test it under heat: the true part survives, the rest burns off. crucible is the lens you hold over method design — research step two, after [prospect](../prospect/SKILL.md) has landed a defensible gap — to turn that gap into a method that survives review. It is the second skill of the inquiry suite. It audits (and guides you to run) a gated arc that produces a method by racing several candidates and keeping only what survives, and it will not advance past a GATE until the checklist tool clears it. That gate enforces order — each step done before the next — not the substance of the work inside it; the tool structures the discipline, it does not audit it, so the rigor is yours to supply.

The one mental shift everything hangs on — a method is raced to death, not nursed. In the human era, implementing a method variant cost weeks, so you bet on one idea and lovingly refined it — and your judgment was quietly corrupted by attachment to the thing your own hands built. In the agent era an agent stands up a half-day prototype of each variant, so you do the opposite: generate several, race them in an elimination tournament, and let the losers die. But that same cheapness brings two new failure modes the whole skill is built to defend against. You still fall in love with what you built — so the kill criteria are written before the race, when you are not yet attached. And the agent hands you winners that are really infeasible-solution bugs — a variant whose performance "soars" is, nine times in ten, producing solutions that violate the constraints — so an independent feasibility checker gates every result before you believe it.

Measure the ceiling before you build. Before racing anything, you run an oracle ladder — open the cheats one level at a time (perfect information, an exact sub-solver, ground-truth components) and watch performance climb. The total gap tells you whether the direction is worth a year; the per-level gaps tell you which component holds the prize (so you attack that one); and the reverse signal — a level that moves nothing — tells you the bottleneck isn't where you thought, killing half your candidates for free. You learn the size and the location of the prize before spending a week chasing it.

Design for the paper from day one. The spec is the draft of your claims; the oracle ladder is the motivation figure; the ablation switches are pre-wired before you need them; the theory is calibrated to the venue; the novelty is defended with mechanism-keyword search. Nothing here is busywork for its own sake — every artifact this skill produces is a part of the eventual paper, made at the moment it is cheapest.

What you cannot delegate — the bets. The agent prototypes, fills proofs, runs searches, and argues. But three calls stay yours: which variants to race (pick death-cause-orthogonal ones, so the tournament informs you whoever wins), the kill criteria (written before you are attached), and whether a result is real (a "soaring" number is a feasibility-bug suspect first, a triumph never). Outsource these and you have automated a confident investment in an artifact, not a method.

Speak the user's language. Most calls here are the researcher's bets — is this graft natural, is this delta novel or cosmetic, is the theory worth the weeks, which variant deserves the deepening. Read their field fluency and gloss a term on first use (the oracle ladder, death-cause orthogonality, the null doubles, proof laundering, the degenerate-special-case theorem). A verdict the user can't evaluate is an opinion imposed, not a judgment shared — and the bet is theirs to place.

Read [references/spec-and-ceiling.md](references/spec-and-ceiling.md) first — the foundation: the decidable spec that is your claim draft, and the oracle ceiling ladder that tells you the size and location of the prize before you build. It is what makes the rest of the arc aim true.

The reference library

The depth lives in references/. Open each when a stage sends you there — not all upfront.

• [references/spec-and-ceiling.md](references/spec-and-ceiling.md) — STAGE 0-1: the three-column decidable spec (hard / soft / free) and its second identity as the paper's claim draft; the oracle typology (information / computation / tuning / component); building the ladder, reading the total gap, the per-component marginal value, and the precious reverse signal. Load at STAGE 0.
• [references/variants-and-tournament.md](references/variants-and-tournament.md) — STAGE 2-3, the heart: the five variant patterns (combination / replacement / relaxation / structure-exploitation / learning-augmented) and each one's characteristic death; death-cause orthogonality and the four-piece entry ticket; the trustworthy scaffold (one metric codebase, the independent feasibility checker, discriminating and held-out instances), the pre-committed kill criteria, and the death log. Load at STAGE 2.
• [references/theory.md](references/theory.md) — STAGE 4: counterexample search before proof (three weapons, hunting the degenerate boundary); the five-beat human-agent proof protocol and the proof-laundering guard; calibrating theory depth to the venue; the high-ROI degenerate-special-case theorem. Load at STAGE 4 (honestly skippable for purely empirical work).
• [references/ablation-and-novelty.md](references/ablation-and-novelty.md) — STAGE 5-6: pre-wired switches and the three null doubles (identity / random / cheap-heuristic), the "cheap substitute can't match it" bar, the two-component complexity budget and the empty-ablation-table technique, the resident brute-force contestants; the mechanism-keyword novelty search, the verifiable-consequence diff template, claim-strength calibration, and defensive citation. Load at STAGE 5.

> The arc is one tournament with a measured prize. Seven stages — spec · ceiling · variants · tournament · theory · ablation · novelty — turn a gap into a method that survives review: spec and ceiling fix the target and the prize, variants and tournament race candidates to a survivor, theory and ablation and novelty deepen and defend it. crucible gates all seven below, and hands the survivor to [ledger](../ledger/SKILL.md) for full experiment design.

---

STAGE 0 — Spec (a decidable spec that is your claim draft)

Open [references/spec-and-ceiling.md](references/spec-and-ceiling.md). Before any design, fix what "good" means — decidably.

• Write the three-column spec. HARD constraints (the gap statement translated directly — violating one disqualifies the method), SOFT constraints (quantified, negotiable trade-offs), FREE design space (fully open). Keep the columns fixed so the spec reads as a contract.
• Make every line decidable. "The method should be fast" is noise; "median solve time on the scaffold instances is within 2× the baseline's" is a spec. The efficient check: hand the gap statement to the agent, have it rewrite each requirement as a measurable acceptance criterion, and have it flag the ones it cannot — those are the requirements you haven't thought through yet, and they are work to do now.
• Recognize the spec is your claim draft. Write the two or three sentences the finished paper must make a reader believe; the spec is their verifiable form. Every later experiment collects evidence for these claims; anything serving no claim is work you can skip — the biggest time-saver in the whole of method and experiment design.

GATE — clear before CEILING

1. checklist check spec spec-is-decidable
2. checklist check spec spec-is-claim-draft
3. checklist verify spec

---

STAGE 1 — Ceiling (the oracle ladder: how big is the prize, and where?)

Open [references/spec-and-ceiling.md](references/spec-and-ceiling.md) (the ceiling section). Measure the prize before you chase it.

• Build the oracle ladder, not a single ceiling. Open the cheats one level at a time. Know the typology: INFORMATION oracle (true future values / ground-truth parameters), COMPUTATION oracle (exact brute-force sub-solver / infinite time), TUNING oracle (tune on the test set), COMPONENT oracle (a module replaced by ground truth). Two rules: the oracle's metric must match the final main experiment (or the ceiling isn't comparable), and measure at at least two scale tiers (a small instance's bottleneck can differ entirely from a large one's).
• Read the ladder for three payoffs. The total gap says whether the direction is worth doing (a low ceiling kills it now, cheaply). The inter-level gaps are each component's marginal value — they aim your STAGE 2 replacement variant at the component worth investing in (a 9.5-gap predictor beats a 1.3-gap sub-solver). The reverse signal is equally precious: a level that moves nothing means the bottleneck isn't where you thought — which can veto half your candidate variants. (And the ladder is your paper's motivation figure, for free.)

GATE — clear before VARIANTS

1. checklist check ceiling oracle-ladder-built
2. checklist check ceiling ceiling-justifies-direction
3. checklist verify ceiling

---

STAGE 2 — Variants (generate candidates with orthogonal deaths)

Open [references/variants-and-tournament.md](references/variants-and-tournament.md). Don't design one method — generate a few that bet on different things.

• Use the five patterns, each with its death in mind. COMBINATION (graft, e.g. learned initial solution + classical local-search refine — argue "A's output is exactly B's needed input" or don't graft); REPLACEMENT (swap the biggest-gap component the ladder found — not a non-bottleneck); RELAXATION (relax an assumption — the death is in evaluation: the old benchmark may not apply, so you build a new protocol; but that new benchmark gets cited); STRUCTURE-EXPLOITATION (enter via the dual form, the constraint-matrix sparsity/blocks, or real-data statistics — verify the structure exists in real instances, not just your synthetic ones); LEARNING-AUGMENTED (the hidden death: inference time eats the solve time saved — report end-to-end time from the prototype).
• Choose death-cause-orthogonal variants. Take two or three into the tournament that bet on different things — if all three bet on "demand is predictable," one assumption falling wipes them all out; orthogonal bets mean the tournament teaches you about the problem whoever wins.
• Require the four-piece entry ticket. Each candidate carries: mechanism in one sentence, most-likely death cause, a one-day minimal validation experiment, and the delta vs the nearest work. No four-piece set, no discussion. (Brainstorm with the agent given the spec, the prospect close-reading notes, and the death log.)

GATE — clear before TOURNAMENT

1. checklist check variants variants-death-orthogonal
2. checklist check variants variants-have-fourfold
3. checklist verify variants

---

STAGE 3 — Tournament (race them on a scaffold you can trust)

Open [references/variants-and-tournament.md](references/variants-and-tournament.md) (the tournament section). The heart of the skill — and where the agent's bugs and your attachment both strike.

• Build a trustworthy scaffold. One config defines the variant × instance × seed grid; one command runs it and persists structured results (CSV / SQLite); one command emits the comparison table. Two iron rules: one metric-computation codebase shared by all variants (per-variant metric code is the #1 cause of misaligned experiments); an independent feasibility checker sharing no logic with any solver, run on every solution (your strongest weapon against silent bugs — a "soaring" variant is usually emitting infeasible solutions). Pick a discriminating small instance set (the baseline does well on some, badly on others) covering your target cell, plus a held-out set used for no decision, run every couple of weeks to catch over-fitting to the tournament.
• Pre-commit the kill criteria — aimed at the right competitor. Write the elimination rule in black and white before the first variant runs (e.g. "killed if its median beats neither scale tier's baseline"), because you grow attached to what you built and a post-hoc standard always leaks. And aim each rule at the competitor the variant must beat to justify its existence — a dynamic variant against the best static one, a learned component against the best hand-tuned one — not merely the baseline everything clears, or a variant survives while failing the very thing it was added to prove.
• Keep a death log. Per eliminated variant: name, mechanism assumption, key numbers, death-cause class (implementation bug / assumption falsified / tied by a simple baseline / engineering-infeasible), revival condition. The class matters: a bug-death deserves a fix and re-trial; an assumption-falsified death buries every candidate betting that same assumption. (This append-only log is what [ledger](../ledger/SKILL.md) builds its exploration record on.)

GATE — clear before THEORY

1. checklist check tournament scaffold-trustworthy
2. checklist check tournament kill-criteria-pre-committed
3. checklist check tournament deaths-logged
4. checklist verify tournament

---

STAGE 4 — Theory (hunt counterexamples first, then prove — and don't launder)

Open [references/theory.md](references/theory.md). For theory-bearing work; for a purely empirical contribution, record this stage N/A with that reason (an honest skip, not a silent one).

• Search for counterexamples before opening a proof. Three weapons: SMALL-SCALE EXHAUSTION (enumerate the instance space where you can), PROPERTY-DRIVEN RANDOM GENERATION (sample near the assumption's boundary — counterexamples hide in degenerate cases: empty set, single point, parameter at 0 or ∞, a constraint exactly binding), and AGENT PROPERTY-BASED TESTING (encode the proposition as automated property tests, e.g. hypothesis, which does boundary-shrinking search). Prove only what survives all three.
• Prove in five beats, and guard against proof laundering. You give proposition + sketch → agent fills details → you hand-verify line by line → a fresh agent session plays hostile reviewer (boundary cases, inequality directions, constants' parameter-dependence, whether "WLOG" truly loses none) → patch and attack again. The laundering risk, named: agent proofs always read fluent, and errors hide under "by X it easily follows," "without loss of generality," "similarly" — every such phrase is expanded into a full derivation, and one that won't expand is a hole. Calibrate depth to the venue (match three-to-five recent comparable papers' theory sections). Consider the high-ROI degenerate-special-case theorem (set a parameter to 0 or ∞, prove it reduces to a known method — a sanity check, a positioning as a generalization, and usually easy).

GATE — clear before ABLATION

1. checklist check theory claims-survived-counterexample-search
2. checklist check theory proofs-laundering-checked
3. checklist verify theory

---

STAGE 5 — Ablation (pre-wire the switches, budget the complexity)

Open [references/ablation-and-novelty.md](references/ablation-and-novelty.md). Build the ablation into the method, before you need it.

• Pre-wire the switches with three null doubles. Every switch is a config item, never commented-out code; each component's interface reserves IDENTITY (off), RANDOM (same interface, random output), and CHEAP-HEURISTIC (the most naive implementation). The convincing ablation is not "worse when off" (only shows the component is used) but "a cheap substitute can't match it" (shows your design is necessary) — which is what reviewers challenge.

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: IamK77
• Source: IamK77/Skill
• License: Apache-2.0
• Homepage: https://www.npmjs.com/package/@iamk77/skill-checklist

Install and usage instructions live in the source repository linked above.