Crucible

Crucible

Codebase-level adversarial review. A panel of structurally different models tests each file under pressure, one piece at a time.

The metaphor: a crucible is a vessel that holds material under heat from multiple sources until only what survives the test remains. Crucible puts your codebase under simultaneous pressure from a panel of models drawn from different families, different training corpora, different blind spots — then aggregates the surviving findings into a single severity-ranked report.

Use this when /rival is too narrow (single file or diff) and /raadsmid is too shallow (internal personas, no external models).

---

How to Invoke

/crucible                              # Default: changed files vs main, 4-model sequential chain
/crucible --all                        # Whole repo (with safe excludes)
/crucible --paths "src/api/**/*.ts"    # Glob pattern
/crucible --diff main...HEAD           # Specific git diff range
/crucible --files src/auth.ts src/db.ts  # Explicit file list
/crucible --deep                       # 4-model sequential chain (slower, more thorough)
/crucible --blind                      # Parallel-independent panel (consensus mode)
/crucible --models N                   # Override panel size (1-4, default 4)
/crucible --resume             # Resume an interrupted run
/crucible --include-tests              # Don't skip *.test.* files
/crucible --no-meta                    # Skip cross-file architectural meta-pass
/crucible --deployment-context "..."   # Free-text scoping (e.g. "desktop Tauri sidecar")
/crucible --verify                     # Dynamic verification: write + run repro harnesses for runtime-tagged findings
/crucible --symptoms "audio not captured"  # Feed observed failures to the panel

Combine freely: /crucible --all --deep --blind runs full repo with 3 models per file independently.

--deployment-context "..." is a free-text string that gets spliced into every per-file prompt right before the file content. Use it to keep the panel from generating out-of-scope findings. Examples:

• --deployment-context "Desktop Tauri sidecar bound to 127.0.0.1, single-process. Findings about multi-worker uvicorn or deployed-service auth are out of scope." — for a desktop app sidecar.
• --deployment-context "Public production FastAPI service on Vercel with Edge runtime, multi-region, Auth0 in front." — for a deployed service.
• --deployment-context "CLI tool installed via pip, runs as the invoking user, no network." — for an offline CLI.

The flag is opt-in. Without it, models default to "could be anything" and may flag concerns that don't apply to the actual deployment shape (the most common false-positive class measured in real Crucible runs).

---

Mental Model

For each file in scope:

file.ts ─┬─> Model A (first pass, free to find anything)
         ├─> Model B (sees A's findings, validates + adds)
         └─> Model C (final pass, consolidates with severity ranks)  [--deep only]

Or in --blind mode:

file.ts ─┬─> Model A ─┐
         ├─> Model B ─┼─> Consensus dedup (overlapping findings = high confidence)
         └─> Model C ─┘

Then one final pass:

[all per-file findings] + [project tree] ─> Meta-Reviewer ─> cross-file architectural issues

Sequential is the default because it produces deeper, layered analysis (each model reasons about the prior's gaps). Blind is for catching different blind spots simultaneously and ranking by consensus.

---

8-Phase Workflow

Phase 1 — Resolve Scope

Determine the file list to review. In order of precedence:

| Flag | Source | |---|---| | --files a.ts b.ts | Explicit list | | --paths "" | Glob expansion via git ls-files filtered by pattern | | --diff | git diff --name-only | | --all | git ls-files (or find . -type f if not a git repo) | | (default) | git diff --name-only main...HEAD if branched, else git diff --name-only HEAD |

Default ignore patterns (skipped unless --include-tests or --all-files):

• Lockfiles: *-lock.json, *.lock, Gemfile.lock, go.sum, Cargo.lock, uv.lock
• Generated: dist/, build/, .next/, out/, .cache/, target/, coverage/
• Vendor: node_modules/, vendor/, __pycache__/, .venv/, venv/
• Binary: any file matching common binary extensions (.png, .jpg, .pdf, .zip, .so, .dylib, etc.)
• Tests: *.test.*, *.spec.*, __tests__/, tests/ — unless --include-tests
• Documentation: *.md, *.mdx, *.rst, LICENSE*, CHANGELOG* — unless --include-docs
• Config: package-lock.json, .gitignore, .editorconfig — always skipped

After filtering, sort by review priority:

1. Files in the diff (changed code matters most)
2. Files matching common high-risk patterns: auth*, *security*, *payment*, *db*, api/, routes/, middleware*
3. Smallest first (cheaper to review, faster feedback)
4. Alphabetical

Phase 2 — Snapshot Models + Pre-flight Estimate

Snapshot the model panel BEFORE the run starts. Crucible has its OWN model cache at ~/.crucible/models.json populated by scripts/discover-premium.sh. This cache holds paid SOTA frontier models (DeepSeek-V4, Gemini Pro, Kimi, MiniMax, plus Qwen/GLM fallbacks) — explicitly NOT Rival's free-only roster, which would defeat Crucible's "deep adversarial review" premise.

Steps:

1. If ~/.crucible/models.json is missing or older than 72h, run bash ~/.claude/skills/crucible/scripts/discover-premium.sh to populate/refresh it. The script queries OpenRouter directly, picks the best available paid model from each preferred family (DeepSeek V4 → Google Gemini Pro → Moonshot Kimi → MiniMax → Qwen → GLM in that order), health-pings each, and writes the panel to the cache. Default panel size is 4.
2. Read ~/.crucible/models.json, take the top N model IDs (where N = panel size, default 4 for --all/diff runs and --deep), and pass those literal IDs to the orchestrator via --models. Do NOT use Rival's --auto N rank lookups — those filter for :free only.
3. Fallback: if ~/.crucible/models.json doesn't exist after attempting discovery, fall back to ~/.rival/models.json (free models only) and warn the user that Crucible is running in degraded "free-model" mode.
4. Compute family diversity: count distinct vendor prefixes. The discover-premium script already enforces one model per family, so diversity should be perfect — but warn if for some reason it's not.

Cost note: the SOTA paid panel typically costs $0.30–$0.75 per 1M prompt tokens. A 5-file × 2-model --all run on a small project (~10k tokens of code) is roughly $0.01–$0.05 total. A 50-file --all run is roughly $0.50. Cheap enough that pre-flight should not gate on cost unless > 200 files.

Then count: N files × M models = K total model calls. Print summary and ask confirmation if it crosses ANY of these thresholds:

• More than 10 files (review takes meaningful time)
• More than 30 total model calls (cost / rate limit concern)
• Any single file > 2000 lines (will need chunking, much slower)
• Family diversity warning fired (always confirm)

Format:

Crucible pre-flight
──────────────────────
Scope:    32 files (diff main...HEAD, after ignore patterns)
Models:   minimax/minimax-m2.7, nvidia/nemotron-3-super-120b-a12b:free
Families: 2 distinct (minimax, nvidia) — diversity OK
Calls:    64 total model invocations
Estimate: ~12-18 min wall clock (free tier rate limits)
Cache:    .crucible-cache/2026-04-26-1532/

Proceed? [y/N]

Wait for confirmation. If user says no, do not start.

If under all thresholds AND diversity is healthy, skip the prompt and proceed silently.

Phase 3 — Initialize Cache

Create the run cache directory:

.crucible-cache//
├── manifest.json       # scope, models, options, file list
├── findings/           # per-file JSON, one file per source file (sanitized name)
├── transcripts/        # raw model outputs per call (audit trail)
├── progress.jsonl      # append-only event log (started/finished per file)
└── report.md           # final consolidated report (written in Phase 6)

The cache lives in the repo (or cwd if not a repo). Add a one-line .gitignore entry for .crucible-cache/ if not already ignored.

manifest.json:

{
  "run_id": "2026-04-26-1532",
  "started_at": "2026-04-26T15:32:00Z",
  "scope": "diff main...HEAD",
  "files": ["src/auth.ts", "src/db.ts", "..."],
  "models_requested": 2,
  "mode": "sequential",
  "include_tests": false,
  "include_docs": false
}

If --resume was passed: read existing manifest, scan findings/ for already-completed files, skip those, continue with the rest.

Phase 4 — Per-File Review Loop (run the orchestrator)

The heavy lifting is done by scripts/orchestrate.py — a bundled Python script that calls OpenRouter directly per file, handles reasoning-model output extraction (reasoning_content and `` blocks), drops models that return empty/malformed output twice in a row, and persists findings + transcripts as it goes.

Invoke it once with the full file list and frozen model IDs from Phase 2:

python3 "$(dirname "$(realpath ~/.claude/skills/crucible/skill.md)")/scripts/orchestrate.py" \
  --cache-dir .crucible-cache/ \
  --files   ... \
  --models   \
  --mode sequential \
  --prompt-templates ~/.claude/skills/crucible/review-prompts.md \
  --delay-between-calls 8 \
  --deployment-context "Desktop Tauri sidecar bound to 127.0.0.1, single-process. Multi-worker / deployed-service findings out of scope."  # optional

For --blind mode, pass --mode blind. The orchestrator handles both internally.

The orchestrator emits one progress line per file to stdout:

✓ [3/32] src/auth.ts (4 findings: 1c 2h 1m 0l) — 23.4s

Markers: ✗ if any critical, ⚠ if any high/medium, ✓ if low or none.

File chunking: files under 1500 lines go in whole. The orchestrator currently does NOT auto-chunk larger files — for now, if you have a file over 2000 lines, split it at the LLM-driven scope step (Phase 1) by passing slice ranges in --files (e.g., write a temp file with the relevant region). Auto-chunking is a future improvement.

Resume: the orchestrator skips any file that already has a findings/.json in the cache directory. Re-running with the same --cache-dir after a network blip just continues where it stopped.

Per-file output structure that lands in findings/.json:

{
  "file": "src/auth.ts",
  "duration_s": 23.4,
  "passes": [
    {"model": "minimax/minimax-m2.7", "status": "ok", "findings": [...]},
    {"model": "nvidia/nemotron-3-super-120b-a12b:free", "status": "ok", "validates": [...], "new_findings": [...]}
  ],
  "findings": [
    {"line": 42, "severity": "critical", "category": "security", "title": "...", "explanation": "...", "suggestion": "..."}
  ]
}

The findings array is the final state after the chain. The passes array is the per-model audit trail. Pass status can be ok, empty, malformed, or dropped.

Health summary at the end — after all files, the orchestrator prints which models survived to stderr. If any model was dropped, that gets surfaced in the final report and you should mention it to the user.

Phase 5 — Cross-File Meta-Pass (skip with --no-meta)

After all per-file reviews complete, run one final pass with the top-ranked model. Send:

• Project tree (git ls-files | head -200 or directory listing)
• Aggregated findings (just titles + severities, not full text — keep prompt small)
• The list of file paths reviewed

Prompt template (see review-prompts.md → "Cross-file meta-pass"). Goal: find issues no per-file pass could catch:

• Repeated anti-patterns across files (3+ instances → suggests missing abstraction)
• Inconsistent error handling between files
• Coupling/cycle smells
• Missing layers (e.g., no validation between API and DB)
• Test coverage gaps (which files have logic but no test sibling)

Save meta-findings to findings/_meta.json with "file": null and "category": "architecture".

Phase 5.5 — Dynamic Verification (opt-in --verify)

After the meta-pass and before the report, when --verify is set, run the dynamic-verification stage. For each finding the panel tagged runtime_checkable (stateful / concurrency / timing / ordering / resource-leak / off-by-one / silent-failure), a panel model writes a minimal repro harness and the stage RUNS it in a locked sandbox:

• temp-dir COPY of the target file (never the live working tree)
• NO network (Python socket block via a runpy preamble; node --require shim; bash proxy-blackhole)
• hard wall-clock timeout (default 15s); on macOS the CPU and file-size rlimits also apply, memory rlimit is best-effort (Darwin rejects RLIMIT_AS)
• scrubbed environment

python3 "$(dirname "$(realpath ~/.claude/skills/crucible/skill.md)")/scripts/verify_findings.py" \
  --cache-dir .crucible-cache/ \
  --models   \
  --prompt-templates ~/.claude/skills/crucible/review-prompts.md \
  --symptoms "..."   # optional

Harnesses must print CRUCIBLE_VERDICT: REPRODUCED / NOT_REPRODUCED and exit 0; an inconclusive run is repaired up to --max-repair times. Results land in verification.json (+ harness/output under verification/). The report builder promotes reproduced findings to a VERIFIED (executed repro) tier and demotes failed repros to Unconfirmed Hypotheses.

Cost: $0 on default runs (stage never fires). With --verify, ~1–3 model calls per runtime-tagged finding — a few cents for a handful. Bounded by --verify-limit (default 10; dropped findings are logged, never silently truncated).

Safety note: this stage executes MODEL-WRITTEN code against the target. The sandbox bounds network/CPU/file-size/wall-clock and runs against a copy, but filesystem reads are NOT isolated and the network block is in-process only — do not point --verify at a target whose mere import performs destructive disk operations.

Phase 6 — Aggregate and Write Report (run the report builder)

Run the bundled report builder once the orchestrator is done:

python3 ~/.claude/skills/crucible/scripts/build_report.py --cache-dir .crucible-cache/

It reads every per-file findings JSON in the cache, applies consensus dedup if mode is blind (merges findings on same file within 3 lines that share >70% title-word overlap, keeps higher severity, lists all flagging models), sorts by severity → file → line, and writes report.md in place.

The report uses this template:

# Crucible Report — 

**Scope:** 
**Files reviewed:** 
**Models:** , [, ]
**Mode:** sequential | blind
**Duration:** 
**Total findings:**  ( critical,  high,  medium,  low)

---

## CRITICAL  ()

### `:` — 
**Models:** 
**Category:** 
**Why it matters:** 
**Fix:** 

---

## HIGH  ()
...

## MEDIUM  ()
...

## LOW  ()
...

---

## Architectural / Cross-File  ()

### 
**Files involved:** 
**Why it matters:** 
**Suggested approach:** 

---

## Per-File Summary

| File | Critical | High | Medium | Low | Total |
|---|---|---|---|---|---|
| src/auth.ts | 1 | 2 | 1 | 0 | 4 |
...

## Models Used

- **** —  unique-to-this-model>
- **** — ...

## Skipped Files

 10>

Phase 7 — Verification Pass (Claude / top Anthropic model — REQUIRED)

The OS panel produces findings; Claude (the model driving this skill, presumably the top current Anthropic model) must validate them before declaring the report trustworthy. OS models hallucinate. Three different OS models converging on the same hallucination is still a hallucination.

After build_report.py lands report.md, do this verification pass:

1. Read the full report.md.

1. Spot-check every CRITICAL and HIGH finding against actual source code.

For each one:

• Open the file at the cited line range (read 10 lines before + 10 after)
• Verify the title accurately describes what's there
• Confirm the bug is genuinely present (not a misreading o

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Bambushu
• Source: Bambushu/crucible
• License: MIT
• Homepage: https://github.com/Bambushu/crucible

Install and usage instructions live in the source repository linked above.