Critic Gauntlet

Critic Gauntlet

For high-stakes work where the cost of being wrong is high. Forces independent adversarial critics against a target, surfaces raw output verbatim before synthesis, and iterates until convergence or specification-level findings emerge.

The value is cross-model disagreement: different model families share different blind spots, so a hole one misses another tends to catch. Convergence across independent models is a strong signal; a lone-critic finding is sometimes the most valuable thing in the round.

The harness is the same in every mode: parallel spawn, liveness gate, raw output verbatim, fresh-agent synthesis, convergence math. What changes per mode is only the rubric (the brief the critics answer) and a small amount of per-mode policy. Modes are selected by a --mode flag; see the Modes section.

Roster

The full roster is five critics:

1. Claude general-purpose subagent (baseline, always available, no API key, included with Claude Code)
2. Codex CLI (requires the codex CLI installed and authenticated)
3. Grok via the xAI API (requires XAI_API_KEY)
4. Gemini via the Google AI Studio API (requires GEMINI_API_KEY)
5. DeepSeek via an OpenAI-compatible endpoint (requires DEEPSEEK_API_KEY; US-hosted Fireworks by default, DEEPSEEK_BASE_URL switches to the PRC-hosted first-party API or a self-hosted vLLM)

Only critic 1 is required. The other three are bolt-ons you enable when you have the credentials. Running with just the Claude subagent is valid but weak: you lose the cross-model diversity that is the whole point. Two or three models arguing is materially better; four is the recommended bar for a decision you cannot cheaply reverse.

If you have all five configured, run all five by default. Drop a critic on a given round only when speed matters more than coverage. Grok, Gemini, and DeepSeek carry higher noise floors than Claude and Codex, so if you are trimming, drop from that tier first. A critic newly added to the roster is UNCALIBRATED (DeepSeek as of 2026-07): run it and read it, but it does not count toward binding thresholds until the promotion criteria in the synthesis section hold.

Modes

Select the rubric with --mode architecture|science|editorial (default architecture). Each mode has a brief template in modes/.brief-template.md and an API-critic system prompt in modes/.system.txt. You author the per-run brief from the template; the helper scripts read the system prompt automatically.

| Mode | Use for | Rubric sections | | --- | --- | --- | | architecture | ADR-grade decisions: system shape, delivery mechanism, multi-tenancy model, new critical-path dependency, repo structure | boring baseline, blank-sheet design, three biggest holes, steel-manned 80% and 110% alternatives, unstated assumptions, consequences for ADR, recommendation | | science | Working papers before submission (SSRN / Zenodo tier) | claim-vs-evidence, identification and confounds, method-question fit, data provenance and reproducibility, re-identification exposure, limitations honesty, recommendation | | editorial | Published-grade articles before release | five lenses: journalistic discipline, defamation and regulatory risk, reader engagement, ai-slop-ness, CTA conversion; scored 0-10 with quoted evidence |

Posture is identical across modes: no sympathetic openers, lead with the strongest objection, no balanced view, every deduction cites the exact passage, no em-dashes or double-dashes.

Per-mode policy that differs from the architecture default:

• science data-sovereignty. The external API critics (Grok, Gemini, DeepSeek) are third-party vendors, and DeepSeek's default endpoint is additionally PRC-hosted (strictest tier; prefer a US-hosted endpoint or drop the critic when in doubt). In science mode they may read ONLY the anonymized paper and its stated public sources, NEVER the raw dataset or any file carrying subject identity. A properly anonymized paper is safe to send; the underlying data and any identity key are not. This is a policy about egress of identified data to outside APIs, not a ban on running multiple critics: an anonymized artifact runs the full roster. The brief template restates this rule in its header.
• editorial calibration. Model families differ in how readily they flag editorial risk; some run lenient on prose and strict on architecture, or the reverse. Do not assume a critic's architecture-mode temperament carries into editorial. Weight by which critics actually converge on quoted evidence. The API helper scripts run a two-call protocol in editorial mode: call 1 sees the article alone and returns the cold-read log, call 2 gets the brief and materials plus those notes, so the cold first pass is real rather than reconstructed.

When to invoke

--mode architecture (default): system shape, delivery mechanism, multi-tenancy model, a new critical-path dependency, repo structure. Anything where the shape, once shipped, is expensive to change. Non-architectural changes (bug fixes, refactors, feature additions inside an existing shape) do NOT need this. The gauntlet is expensive in attention and time; do not run it on small decisions.

--mode science: before a working paper is submitted (SSRN / Zenodo tier), or when a paper's conclusions are load-bearing enough that a desk-reject would be costly. Not for early drafts still finding their claim.

--mode editorial: before releasing a flagship, litigious-tier, or template-defining article (the first specimen of a new format, a piece on a named subject that could prompt a legal response). Not per-issue: rerun on template changes or periodically. Your own publish gate stays the gate; this is a complementary layer.

What you need before invoking

1. The material under review at a known file path, saved as /proposal-v.md. For architecture that is the ADR proposal (status, context, proposal, what it trades, open questions). For science it is the anonymized paper (plus its sources, subject to the science data-sovereignty rule). For editorial it is the article draft plus its source material and house references.
2. A work folder to hold artifacts. Suggested convention: /decisions/ADR-NNN-/ for architecture, /critic-runs// for science and editorial. The brief, proposal, critiques, and synthesis all live together.
3. Optional context files the critics should read (an existing ARCHITECTURE.md, contributor/AI guidelines, voice/policy docs, prior critiques if this is round 2+), subject to per-mode policy (science withholds identified data from the external API critics).

The flow

Step 1: Probe the roster

Before doing anything else, determine which critics are available on this machine and announce the roster to the user.

# Probe each critic. Print "available" or "skipped: ".
echo "Critic roster:"

# Claude (always available)
echo "  Claude general-purpose subagent: available"

# Codex CLI
if command -v codex &>/dev/null; then
    echo "  Codex CLI: available"
else
    echo "  Codex CLI: skipped (codex not on PATH)"
fi

# Grok via xAI
if [ -n "${XAI_API_KEY:-}" ]; then
    echo "  Grok (xAI): available"
else
    echo "  Grok (xAI): skipped (XAI_API_KEY not set)"
fi

# Gemini via Google AI Studio
if [ -n "${GEMINI_API_KEY:-}" ]; then
    echo "  Gemini (Google AI Studio): available"
else
    echo "  Gemini (Google AI Studio): skipped (GEMINI_API_KEY not set)"
fi

# DeepSeek (endpoint configurable; default is US-hosted Fireworks)
if [ -n "${DEEPSEEK_API_KEY:-}" ]; then
    echo "  DeepSeek (${DEEPSEEK_BASE_URL:-https://api.fireworks.ai/inference/v1}): available"
else
    echo "  DeepSeek: skipped (DEEPSEEK_API_KEY not set)"
fi

If only Claude is available, warn the user that this is a degenerate run and ask whether to proceed. Otherwise proceed without prompting.

Step 2: Write the brief

Pick the mode. Copy modes/.brief-template.md and fill in the placeholders (title, required reading, prior-round summary, and for editorial/science the product/paper context). Save the filled brief to /brief-v.md where N is the round number.

Each mode's rubric sections are listed in the Modes table above. Do not hand-write the rubric; use the template. The API-critic system prompt for the mode is applied automatically by the helper scripts from modes/.system.txt; you do not paste it anywhere.

The shared posture in every brief: no sympathetic openers, lead with the strongest objection, no balanced view, every deduction cites the exact passage, no em-dashes or double-dashes, markdown, no closing pleasantry.

Step 3: Spawn the critics in parallel

In a single message, fire all enabled critic tool calls.

All critics answer the same brief and follow ITS output format (which differs by mode: the architecture brief has 5 sections, science has 7, editorial has the five-lens format). Do not hardcode "5 sections" in any critic prompt; say "follow the brief's output format." The helper scripts take --mode and load the matching system prompt; the Claude subagent and Codex read the format from the brief itself.

Critic 1: Claude general-purpose subagent. Use the Agent tool with subagent_type: "general-purpose". Tell the agent to read the brief, read the required files, follow the brief's output format, write its critique to /critique-v-claude.md, and confirm with a one-line output. Run in background.

Critic 2: Codex CLI. Use Bash with codex exec --sandbox workspace-write --skip-git-repo-check --cd "". The prompt tells Codex to follow the brief's output format. Pipe --mode from this skill folder. It reads XAI_API_KEY from env (or a .env / shell rc fallback), loads the mode system prompt, concatenates brief + proposal + prior critiques, calls the xAI API, and writes critique-v-grok.md. Run via Bash in background.

Critic 4: Gemini via the Google AI Studio API. Use the helper script gemini-critic.sh --mode from this skill folder. Same shape as Grok. Reads GEMINI_API_KEY, loads the mode system prompt, calls the Google AI Studio API, writes critique-v-gemini.md. Run via Bash in background. In science mode, confirm the proposal file handed to the external critics is the anonymized artifact only.

Critic 5: DeepSeek via an OpenAI-compatible endpoint. Use the helper script deepseek-critic.sh --mode from this skill folder. Same shape as Grok; reads DEEPSEEK_API_KEY. The endpoint defaults to Fireworks (US-hosted, serving the MIT open weights; model id accounts/fireworks/models/deepseek-v4-pro). DeepSeek's first-party API is PRC-hosted; opt into it deliberately via DEEPSEEK_BASE_URL=https://api.deepseek.com/v1 + DEEPSEEK_MODEL=deepseek-v4-pro, and never for material that must not egress to a PRC vendor. Self-hosted vLLM works the same way. The critique header records the serving endpoint. Writes critique-v-deepseek.md. Run via Bash in background.

The helper scripts take the work folder path and the round number, plus an optional --mode (default architecture). Both auto-pick up prior-round critiques when N > 1.

Step 4: Verify every critic returned a real critique (liveness gate)

Do not synthesize until every enabled critic has either produced a valid critique or been explicitly dropped by the user. A missing critic must never be silently absorbed: a degraded roster is a decision, not a default. The most common way a gauntlet quietly loses signal is a critic that erred without anyone noticing, and the synthesis treating three-of-an-intended-four as if four had agreed.

Typical timing: Claude subagent 2-3 min, Codex CLI 5-10 min, Grok and Gemini under a minute. DeepSeek is not yet timed; expect thinking-mode latency in minutes, not seconds.

Success condition (same gate for all critics). A critic passed only if ALL hold:

1. Its critique-v-.md file exists and is at least ~500 bytes. Real critiques run 3 KB and up; anything smaller is a stub or error.
2. The file is the actual multi-section critique in the brief's format, not an error payload. Reject if it leads with ERROR: or contains raw API error JSON.
3. For the Bash critics (Codex, Grok, Gemini): the process exit code was 0. The helper scripts set -euo pipefail and exit non-zero on any missing-key / API / empty-response failure. Capture the last stderr line as the failure reason.
4. For the Claude subagent: the Agent tool returned success and the file was written. A subagent that erred without writing the file is a fail even if it returned some text.

On any critic failing: retry once, then halt.

1. Retry the failed critic exactly once. Transient infrastructure, rate-limit, and spawn errors usually clear on the second attempt.
2. If it fails the second time, STOP. Do not synthesize. Tell the user, in plain language: which critic is down, the captured failure reason, the role lost (roster below), the surviving roster, and the choice: proceed with the reduced roster / pause and fix / abort the round.
3. The default recommendation depends on WHICH critic died:

• Lost an ANCHOR (Claude or Codex, the two low-noise models): default to PAUSE. The synthesis leans on anchor agreement; losing one materially degrades signal.
• Lost a NOISE-FLOOR critic (Grok, Gemini, or DeepSeek): a reduced run is acceptable. Default to proceed; note the loss in the synthesis roster line.

Billing note for the Claude critic. The Claude critic runs as a Claude Code subagent on your existing subscription, not as a metered API call, and it requires no ANTHROPIC_API_KEY. Do not "fix" a Claude failure by setting one: in Claude Code, setting ANTHROPIC_API_KEY anywhere in the environment routes the entire app to API billing instead of your subscription. If the subagent fails, retry it (above); never reach for a key.

Roster roles (so a loss is costed correctly):

• Claude subagent: operational-nuance anchor, low noise, no API key.
• Codex CLI: specification-bug anchor, low noise.
• Grok: privacy / policy / jurisdictional angles. Higher noise.
• Gemini: catches blind spots shaped by Google's training distribution. Highest calibrated noise.
• DeepSeek: a training distribution and RLHF lineage unlike the other four; strong quantitative bent. UNCALIBRATED as of 2026-07: noise tier by default until proven.

Whatever the outcome, the synthesis roster line must name every critic that was attempted and its status (returned / dropped-after-2-fails / user-skipped), so the degraded-roster fact survives into the record.

Step 5: Surface raw critic outputs verbatim

Read each critique file and present its content to the user verbatim. Do not summarize before the user has seen the raw text. A one-line preamble per critique is fine ("Claude returned. Recommends X."), but the actual critique content goes through verbatim. The point of independent critics is defeated if a synthesizer filters them before the human sees them.

Step 5.5: Decision discipline (non-negotiable; this is where ADR-002 failed)

The critique stage is not the weak point of this gauntlet; synthesis is. The classic failure: critics unanimously reject an approach and converge on a simpler alternative, then the synthesis (written by the same agent that wrote the proposal) overrides all of them with one persuasive sentence and ships the exact architecture the gauntlet was run to prevent. These rules bind the synthesis so the producer can no longer grade the critics:

1. Convergence BINDS the decision, not just the next iteration. If critics converge against the proposed approach (5-of-5, 4-of-5, 4-of-4, 3-of-4, 3-of-3, or 2-of-3, counting calibrated critics only), you may NOT adopt the rejected approach in decision.md. The convergent alternative is the default outcome.
2. Overriding convergence requires explicit escalation, never prose. If you believe the converged critics are wrong, you may not bury the reversal in the synthesis or decision. Stop and put it to the user in pl

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: alexmakarski
• Source: alexmakarski/critic-gauntlet
• License: MIT

Install and usage instructions live in the source repository linked above.