# Crucible > Regression CI for Claude Code configs test your skills, subagents, hooks & CLAUDE.md like code. Deterministic + LLM-judge assertions, baselines, transcript diffs. - **Type:** MCP server - **Install:** `agentstack add mcp-klashkaan-cmyk-crucible` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [klashkaan-cmyk](https://agentstack.voostack.com/s/klashkaan-cmyk) - **Installs:** 0 - **Category:** [Developer Tools](https://agentstack.voostack.com/c/developer-tools) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [klashkaan-cmyk](https://github.com/klashkaan-cmyk) - **Source:** https://github.com/klashkaan-cmyk/crucible - **Website:** https://www.npmjs.com/package/crucible-ci ## Install ```sh agentstack add mcp-klashkaan-cmyk-crucible ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About **Regression CI -- and autonomous self-improvement -- for Claude Code configs.** Treat your `.claude/` directory -- skills, subagents, hooks, and `CLAUDE.md` -- as code under test. Run behavioral scenarios N times on every change to **catch regressions**, then let Crucible **improve the config for you** against the same variance-aware, safety-gated oracle. Every win lands as a PR you review -- nothing auto-merges. > You write a text file, hand it to an agent, and hope for the best. Crucible gives you a signal -- then climbs the gradient. [](./LICENSE) [](#requirements) [](#status) --- ## Highlights - **Measure, don't guess.** `pass@k` / `pass^k` across run-to-run variance, **real `$` cost** (`total_cost_usd`), deterministic **and** LLM-judge assertions, baselines, transcript diffs, and `bisect` to the offending commit. - **Gate merges.** Non-zero exit + JUnit when a pass-rate or cost gate fails -- drops into CI like any other test. - **Improve automatically.** `crucible optimize` hill-climbs your config against a `PROGRAM.md` fitness contract; `crucible research` runs open-ended autoresearch -- hypothesis generation, a beam of config lineages, and a self-growing eval. Both are safety-gated and **open a PR, never auto-merge**. - **Safe by construction.** Everything runs in throwaway git worktrees; your real `~/.claude` is never touched. ## Why Claude Code configs are code now. A change to `CLAUDE.md`, a new subagent, or a tweaked hook can silently make your agent *worse* -- and you won't know until it ships. Two facts make this real: - Anthropic shipped a **6-week silent quality regression** in Claude Code that its own evals missed. Teams with workflow-level evals caught it in ~72 hours. ([postmortem](https://www.anthropic.com/engineering/april-23-postmortem)) - The skill/plugin marketplace is exploding, with **no quality bar**. "Working" is undefined until you run it on something real. Existing tools test a skill *in isolation* ("does it trigger?"). Crucible tests the **whole config behaving on a real task** ("did it produce good code, fire the right subagent, and stay under budget?"). ## What it does ``` crucible run --config .claude --suite crucible ``` For each scenario, Crucible: 1. **Isolates** -- copies your fixture into a throwaway workdir, points Claude Code at the config under test via `CLAUDE_CONFIG_DIR` (your real `~/.claude` is never touched). 2. **Captures** -- injects `PostToolUse` + `SubagentStop` hooks that log every tool and subagent that actually fired. 3. **Runs headless** -- `claude -p --output-format json`, capturing the result, turn count, and **real cost** (`total_cost_usd`). 4. **Asserts** -- deterministic checks against the workdir and the capture log. 5. **Repeats k times** -- because headless mode has no seed; reports `pass@k`, `pass^k`, variance, and flags flaky scenarios. 6. **Gates** -- exits non-zero when a scenario's pass-rate or cost gate fails, so it blocks a PR like any other test. Emits JUnit XML. ## Self-improvement: `optimize` & `research` The same oracle that *catches* regressions can *drive* improvement. You write a `PROGRAM.md` -- a short contract stating the objective, the mutable surface (a glob allowlist), and the accept gate (significance, no-regression, safety, cost). Then: ``` crucible optimize --config .claude --program PROGRAM.md # gated hill-climb crucible research --config .claude --program PROGRAM.md # open-ended autoresearch ``` - **optimize** -- a tool-restricted editor proposes one focused change at a time inside a throwaway worktree. Each candidate is screened cheaply, then confirmed at higher `k` with a two-proportion significance test, and accepted only if it beats the current best *and* never regresses a safety scenario. Accepted candidates commit to a branch. - **research** -- a beam of config lineages that explores instead of greedily hill-climbing: an LLM ideator generates hypotheses, the frontier grows its own eval scenarios, and a frozen **canary** suite halts the run if it starts overfitting that grown eval. Both **open a PR for you to review and never merge on their own** -- a self-editing loop should never be able to ship itself. CI wiring lives in [`.github/workflows`](./.github/workflows) (`optimize` on manual dispatch, `research` on a weekly cron), both budget-capped. ## Why not just ask an LLM? "Can't the model already do this -- just ask it whether the config is good?" An LLM gives you an *opinion* in one shot. Crucible gives you a *measurement*: - **Across variance.** Headless mode has no seed; output varies run to run. One answer is a single sample -- it can't tell you the config passes 70% of the time. Crucible runs k trials and reports `pass@k`, `pass^k`, and flakiness. - **Against history.** A model has no memory of last week's behavior. Crucible gates on drift versus a baseline keyed to a git SHA. Anthropic shipped a [6-week silent regression its own evals missed](https://www.anthropic.com/engineering/april-23-postmortem); workflow-level evals caught it in ~72h. That incident *is* the argument here. - **On signals you can't hand a probabilistic model.** `command_not_run`, `no_secrets`, `cost_under` -- and real `total_cost_usd`, turns, and which subagents *actually fired* -- are observed and deterministic. You don't want a model guessing whether a secret leaked, then wiring that into a CI exit code. The LLM *is* used -- as the `judge` assertion -- but deliberately as the **softest, most optional signal**: it never fails a gate unless you opt in with `min_score`. The single-shot opinion is exactly what lets a regression ship silently; Crucible turns it into a measured, baselined, gated one. ## Quick start ```bash npm install -g crucible-ci # or: npx crucible-ci init crucible init # scaffolds crucible/example.scenario.yaml + a GitHub Action crucible run --config .claude --suite crucible --junit results.xml ``` ## A scenario ```yaml name: adds-login-endpoint-safely fixture: ./fixtures/express-api # copied into an isolated workdir per trial prompt: | Add a POST /login route that accepts JSON { email, password }. Use the existing hashing in src/users.js (never plaintext); 200 on success, 401 on bad credentials. Add a test. trials: 2 # non-determinism is expected max_turns: 40 assert: - file_matches: "src/app.js::/login" - command_not_run: "rm -rf*" - cost_under: 1.00 - judge: "Login verifies the password via src/users.js hashing, never plaintext" min_score: 4 # LLM-judge gate (omit min_score = soft signal) gate: min_pass_rate: 0.5 max_cost_usd: 1.00 ``` A runnable `crucible/fixtures/express-api` ships with the repo, so this example works end-to-end out of the box: `git clone`, then `crucible run`. ### Assertion types (v0.1) | Assertion | Passes when | |-----------|-------------| | `file_exists: path` | the file was created in the workdir | | `file_matches: path::regex` | the file exists and matches the regex | | `response_contains: text` | the agent's final message contains this substring | | `response_matches: regex` | the agent's final message matches this regex | | `latency_under: ms` | the run finished within this many milliseconds | | `turns_under: n` | the run used at most `n` agent turns | | `subagent_invoked: name` | that subagent fired during the run | | `tool_invoked: name` | that tool fired during the run | | `command_not_run: glob` | no tool invocation matched the glob | | `command_succeeds: cmd` | the command exits 0 in the workdir | | `cost_under: usd` | the run's `total_cost_usd` stayed under the ceiling | | `judge: rubric` (+ `min_score`) | an LLM scores the output 1-5 vs the rubric | | `file_absent: path` | the file must NOT exist after the run | | `no_secrets: true` | no produced file contains a hardcoded key/token/private key | | `no_known_exposure: catalog` (+ `min_severity`) | the config has no component flagged by the exposure catalog | ### LLM-judge assertions (soft by default) Some quality checks are not mechanical ("did it actually hash the password?", "is the error message helpful?"). A `judge` assertion has a neutral, tool-free LLM score the run's output against a rubric, 1-5: ```yaml - judge: "The error response is helpful and does not leak internal details" - judge: "Password is hashed with bcrypt or argon2, never stored plaintext" min_score: 4 # opt-in gate ``` By design the judge is a **soft signal**: with no `min_score` it is reported but never fails a gate. Add `min_score` to explicitly opt into gating on it. Pick the judge model with `--judge-model` (a cheaper model keeps eval cost down). ## Baselines & regression detection The whole point of *regression* CI: catch a config change that makes the agent quietly worse, even when no single scenario newly fails its own gate. ```bash crucible baseline --config .claude --suite crucible # snapshot known-good -> crucible/baseline.json # ...later, on a PR that edits .claude/ ... crucible run --config .claude --suite crucible --baseline crucible/baseline.json --fail-on-regression ``` A regression is reported when, versus the baseline, a scenario's pass rate drops past a threshold, a previously **stable** scenario becomes **flaky**, or median cost jumps significantly. With `--fail-on-regression` these fail the build. ## Transcript-diff viewer When a scenario regresses, see *why* -- diff what the agent actually did, step by step. Save transcripts on a known-good run and a later run, then diff them: ```bash crucible run --suite crucible --save-transcripts .crucible/good # ...after a config change... crucible run --suite crucible --save-transcripts .crucible/new crucible diff .crucible/good/login.trial0.json .crucible/new/login.trial0.json --html diff.html ``` The terminal shows an aligned, color-coded step diff (tools + subagents, with their inputs) plus turn/cost deltas; `--html` writes a standalone, dependency-free viewer with the two runs side by side and both final messages in full. ## Generate a starter suite Writing the first scenarios is the main thing that stops people adopting eval CI. `crucible generate` reads your existing config and scaffolds them for you: ```bash crucible generate --config .claude --suite crucible ``` It emits one `subagent_invoked` scenario per subagent, a smoke test per skill, and a CLAUDE.md coherence check -- a runnable suite in one command. Generated prompts and assertions are a starting point: review and tighten them. Existing files are skipped unless you pass `--force`. ## Explain a failure A red gate tells you *that* something broke; `crucible explain` tells you *why* and what to change. Point it at a saved transcript: ```bash crucible run --suite crucible --save-transcripts .crucible/run crucible explain .crucible/run/login.trial0.json --scenario crucible/login.scenario.yaml ``` A neutral, tool-free model reads what the agent actually did (steps, final message, cost/turns) and prints a `CAUSE:` / `FIX:` diagnosis -- one concrete config change to try. `--scenario` adds the prompt intent for a sharper read. ## Record / replay (free, deterministic CI) Real `claude` runs cost tokens and flake -- the core objection to LLM-config CI. Record a run once, replay it forever: ```bash crucible run --suite crucible --record .crucible/cassettes # one real run, recorded crucible run --suite crucible --replay .crucible/cassettes # no claude calls at all ``` A cassette captures the run envelope, the tool/subagent invocations, and a snapshot of the files the agent produced. On `--replay`, Crucible materializes those files into a fresh workdir and evaluates assertions against them with no network and no tokens -- so CI is instant and deterministic. Re-record only when you intentionally change the config. Every deterministic assertion replays offline; only `judge` (inherently a model call) still reaches the network. ## Watch mode While authoring a config or scenarios, re-run the suite automatically on every change: ```bash crucible watch --config .claude --suite crucible ``` It watches both the config dir and the scenario dir, debounces a burst of edits into a single run, and prints the gate summary after each pass. VCS and build noise (`.git`, `node_modules`, `dist`) is ignored. Ctrl-C to stop. ## Badges & PR comments Surface results where people look. After a run: ```bash crucible run --badge badge.json # shields.io endpoint JSON -> live README badge crucible run --pr-comment # sticky PR results comment (in CI) ``` The badge renders as `crucible | N/N passing` (green when all gates pass, red otherwise) via a shields.io endpoint URL pointed at the raw `badge.json`. The PR comment posts a results table and updates itself in place on every run, so a PR shows one always-current Crucible comment rather than a pile of them. It needs the GitHub Actions environment (`GITHUB_TOKEN`, a `pull_request` event); outside CI it warns and skips. ## CI `crucible init` drops a ready GitHub Action that runs on changes to `.claude/**`. It installs Claude Code + Crucible, runs the suite, and uploads the JUnit report. Set `ANTHROPIC_API_KEY` as a repo secret. ## Requirements - Node >= 20 - [Claude Code](https://www.anthropic.com/claude-code) on `PATH` (`claude`), authenticated (API key or subscription) ## Status **v0.1 -- alpha.** The headless runner, hook-based capture, deterministic assertions, k-trial stats, JUnit output, and the GitHub Action are implemented. Live end-to-end runs depend on your local `claude` binary and an authenticated account. On the roadmap: - **LLM-judge assertions** -- grade output against a rubric (soft signal, not a hard gate). - **Transcript-diff viewer** -- see *what changed* between a passing and failing config, turn by turn. - **Baselines + regression diffing** -- store results keyed by config git SHA; gate on deltas. - **Hosted parallel runner** -- trials are slow and token-costly; offload them and get history + dashboards. ## How it isolates your config Crucible never edits your real config. Each trial runs in `mkdtemp()` with `CLAUDE_CONFIG_DIR` pointed at the config under test and a generated `--settings` file that adds only the capture hooks. Delete-on-exit by default; pass `--keep-workdirs` to inspect a run. ## Terms & consent On first run, Crucible shows its [Terms & Conditions](./TERMS.md) and asks you to accept. Accepting is one-time and stored locally. In CI / non-interactive use, continued use constitutes acceptance (set `CRUCIBLE_AGREE=1` to record it explicitly, or run `crucible agree`). The Terms cover the anonymous telemetry below and the MIT license; if you do not agree, do not use the tool. ## Find the offending commit -- `crucible bisect` A scenario regressed but you don't know which `.claude/` change did it? Crucible binary-searches your git history for you: ```bash crucible bisect --good v1.2.0 --suite crucible # or target one scenario, or use a baseline as the "bad" signal: crucible bisect --good HEAD~20 --scenario adds-login-endpoint-safely crucible bisect --good HEAD~20 --baseline crucible/baseline.json ``` It only tests commits that **touched the config**, each materialized in a throwaway `git worktree` (your working tree is never touched), and runs ~log2(n) suites to pinpoint the first bad commit: ``` First bad config commit: 51e0a19 refactor: tighten the security-reviewer description alice, 2026-05-28 (2 run(s)) inspect: git show 51e0a19 -- .claude ``` ## Lint your config (instant, no model calls) Catch config mistakes statically -- free, offline, no API key needed: ```bash crucible lint --config .claude # add --json for machine-readable output ``` It flags: invalid `settings.json`, hooks pointing at scripts that don't exist, subagents with no `name`/`description` or **duplicate names**, skills with no `description` (which silently never au … ## Source & license This open-source MCP server is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [klashkaan-cmyk](https://github.com/klashkaan-cmyk) - **Source:** [klashkaan-cmyk/crucible](https://github.com/klashkaan-cmyk/crucible) - **License:** MIT - **Homepage:** https://www.npmjs.com/package/crucible-ci Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/mcp-klashkaan-cmyk-crucible - Seller: https://agentstack.voostack.com/s/klashkaan-cmyk - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.