Auto Review Loop

Auto Review Loop: Autonomous Research Improvement

> 🔒 Do not wrap this skill in /loop, /schedule, or CronCreate. It > already loops internally (review → fix → re-review) and the reviewer carries > round-to-round memory in one threadId (codex-reply). An external timer > re-enters from the top each tick — fresh threadId, reviewer memory reset — > firing the verdict on wall-clock time instead of on artifact change: zero new > signal, full token cost. If you want to schedule something, schedule the > external wait that precedes it (experiments done → then run this once). See > [shared-references/external-cadence.md](../shared-references/external-cadence.md).

Autonomously iterate: review → implement fixes → re-review, until the external reviewer gives a positive assessment or MAX_ROUNDS is reached.

Context: $ARGUMENTS

Constants

• MAX_ROUNDS = 4
• POSITIVE_THRESHOLD: score >= 6/10 AND verdict ∈ {"ready", "almost"} — both must hold. This matches the operative Phase-E STOP CONDITION exactly; the verdict vocabulary is {"ready", "almost", "not ready"} (a high score with a "not ready" verdict does NOT stop the loop). Earlier wording here used or and a stale verdict set ("accept"/"sufficient"/"ready for submission") — that was an internal inconsistency; the AND form is authoritative.
• REVIEW_DOC: review-stage/AUTO_REVIEW.md (cumulative log) (fall back to ./AUTO_REVIEW.md for legacy projects)
• REVIEWER_MODEL = gpt-5.5 — Default model for the Codex backend. Must be an OpenAI model (e.g., gpt-5.5, o3, gpt-4o). Manual backend uses whatever model the user chooses.
• REVIEWER_BACKEND = codex — Default: Codex MCP (xhigh). Override with — reviewer: oracle-pro for Oracle MCP, or — reviewer: manual for Manual Review MCP. If manual-review MCP is unavailable, stop and print the install command; do not fall back to Codex. See shared-references/reviewer-routing.md.
• OUTPUT_DIR = review-stage/ — All review-stage outputs go here. Create the directory if it doesn't exist.
• HUMAN_CHECKPOINT = false — When true, pause after each round's review (Phase B) and present the score + weaknesses to the user. Wait for user input before proceeding to Phase C. The user can: approve the suggested fixes, provide custom modification instructions, skip specific fixes, or stop the loop early. When false (default), the loop runs fully autonomously.
• COMPACT = false — When true, (1) read EXPERIMENT_LOG.md and findings.md instead of parsing full logs on session recovery, (2) append key findings to findings.md after each round.
• REVIEWER_DIFFICULTY = medium — Controls how adversarial the reviewer is. Three levels:
• medium (default): Current behavior — MCP-based review, the executor controls what context the reviewer sees.
• hard: Adds Reviewer Memory (the reviewer tracks its own suspicions across rounds) + Debate Protocol (the executor can rebut, the reviewer rules).
• nightmare: Everything in hard + Codex exec reviewer reads the repo directly via codex exec (the executor cannot filter what the reviewer sees) + Adversarial Verification (the reviewer independently checks if code matches claims).
• RENDER_HTML = true — When true (default), auto-render review-stage/AUTO_REVIEW.md to HTML on loop termination via /render-html. Uses --no-review (the loop itself IS the cross-model review; the HTML is a structural conversion). Set false to skip, or pass — render html: false.

> ⚠️ Nightmare + Manual incompatibility: If REVIEWER_BACKEND = manual and REVIEWER_DIFFICULTY = nightmare, STOP with: > "difficulty: nightmare requires Codex CLI / codex exec and is not compatible with --reviewer: manual. Use difficulty: hard, or switch reviewer to codex."

> 💡 Override: /auto-review-loop "topic" — compact: true, human checkpoint: true, difficulty: hard

Reviewer Calling Convention

When calling the reviewer, branch on REVIEWER_BACKEND:

If REVIEWER_BACKEND = codex: Use mcp__codex__codex for new review threads. Use mcp__codex__codex-reply for follow-up rounds (reuse threadId).

If REVIEWERBACKEND = manual: Use mcp__manual_review__review for new review threads with: prompt: [exact same prompt that would go to Codex] config: {"modelreasoningeffort": "xhigh"} Save the returned threadId. Use mcp__manual_review__review_reply for follow-up rounds with: threadId: [saved manual-review threadId] prompt: [follow-up prompt] config: {"modelreasoning_effort": "xhigh"}

Prompt fidelity: the manual prompt must be exactly the same text that Codex would receive. Review tracing applies equally to both backends.

State Persistence (Compact Recovery)

Long-running loops may hit the context window limit, triggering automatic compaction. To survive this, persist state to review-stage/REVIEW_STATE.json after each round:

{
  "round": 2,
  "threadId": "019cd392-...",
  "status": "in_progress",
  "difficulty": "medium",
  "last_score": 5.0,
  "last_verdict": "not ready",
  "pending_experiments": ["screen_name_1"],
  "timestamp": "2026-03-13T21:00:00"
}

Write this file at the end of every Phase E (after documenting the round). Overwrite each time — only the latest state matters.

On completion (positive assessment or max rounds), set "status": "completed" so future invocations don't accidentally resume a finished loop.

Output Protocols

> Follow these shared protocols for all output files: > - [Output Versioning Protocol](../shared-references/output-versioning.md) — write timestamped file first, then copy to fixed name > - [Output Manifest Protocol](../shared-references/output-manifest.md) — log every output to MANIFEST.md > - [Output Language Protocol](../shared-references/output-language.md) — respect the project's language setting

Workflow

Initialization

1. Check for review-stage/REVIEW_STATE.json (fall back to ./REVIEW_STATE.json if not found — legacy path):

• If neither path exists: fresh start (normal case, identical to behavior before this feature existed)
• If it exists AND status is "completed": fresh start (previous loop finished normally)
• If it exists AND status is "in_progress" AND timestamp is older than 24 hours: fresh start (stale state from a killed/abandoned run — delete the file and start over)
• If it exists AND status is "in_progress" AND timestamp is within 24 hours: resume
• Read the state file to recover round, threadId, last_score, pending_experiments
• Read review-stage/AUTO_REVIEW.md to restore full context of prior rounds (fall back to ./AUTO_REVIEW.md)
• If pending_experiments is non-empty, check if they have completed (e.g., check screen sessions)
• Resume from the next round (round = saved round + 1)
• Log: "Recovered from context compaction. Resuming at Round N."

1. Read project narrative documents, memory files, and any prior review documents. When COMPACT = true and compact files exist: read findings.md + EXPERIMENT_LOG.md instead of full review-stage/AUTO_REVIEW.md and raw logs — saves context window.
2. Read recent experiment results (check output directories, logs)
3. Identify current weaknesses and open TODOs from prior reviews
4. Initialize round counter = 1 (unless recovered from state file)
5. Create/update review-stage/AUTO_REVIEW.md with header and timestamp

Loop (repeat up to MAX_ROUNDS)

Phase A: Review

Route by REVIEWER_DIFFICULTY:

Medium (default) — MCP Review

Send comprehensive context to the external reviewer using the selected backend.

For codex backend:

mcp__codex__codex:
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    [Round N/MAX_ROUNDS of autonomous review loop]

    Review the work directly from its artifacts — executor notes are not
    evidence, so read the files yourself rather than trusting my framing:
    - Claims / paper draft: 
    - Methods / code under review: 
    - Raw results (verbatim files, not a summary): 
    - Changed since last round:  — read the diff, not my description

    Please act as a senior ML reviewer (NeurIPS/ICML level).

    1. Score this work 1-10 for a top venue
    2. List remaining critical weaknesses (ranked by severity)
    3. For each weakness, specify the MINIMUM fix (experiment, analysis, or reframing)
    4. State clearly: is this READY for submission? Yes/No/Almost

    Be brutally honest. If the work is ready, say so clearly.

For manual backend: use mcp__manual_review__review with the prompt text above and config: {"model_reasoning_effort": "xhigh"}. Save the returned threadId.

If this is round 2+, use mcp__codex__codex-reply (codex) or mcp__manual_review__review_reply (manual) with the saved threadId.

Hard — MCP Review + Reviewer Memory

Same as medium, but prepend Reviewer Memory to the prompt. Use the selected backend.

For codex backend:

mcp__codex__codex:
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    [Round N/MAX_ROUNDS of autonomous review loop]

    ## Your Reviewer Memory (persistent across rounds)
    [Paste full contents of REVIEWER_MEMORY.md here]

    IMPORTANT: You have memory from prior rounds. Check whether your
    previous suspicions were genuinely addressed or merely sidestepped.
    The author (Claude) controls what context you see — be skeptical
    of convenient omissions.

    Review directly from the artifacts (paths below) — read the files yourself:
    - Claims / methods / code: 
    - Raw results: 
    - Changed since last round:  (read the raw diff)

    Please act as a senior ML reviewer (NeurIPS/ICML level).
    1. Score this work 1-10 for a top venue
    2. List remaining critical weaknesses (ranked by severity)
    3. For each weakness, specify the MINIMUM fix
    4. State clearly: is this READY for submission? Yes/No/Almost
    5. **Memory update**: List any new suspicions, unresolved concerns,
       or patterns you want to track in future rounds.

    Be brutally honest. Actively look for things the author might be hiding.

Nightmare — Codex Exec (GPT reads repo directly)

Do NOT use MCP. Instead, let GPT access the repo autonomously via codex exec:

codex exec "$(cat &1

Key difference: In nightmare mode, GPT independently reads code, result files, and logs. Claude cannot filter or curate what GPT sees. This is the closest analog to a real hostile reviewer who reads your actual paper + supplementary materials.

Phase B: Parse Assessment

CRITICAL: Save the FULL raw response from the external reviewer verbatim (store in a variable for Phase E). Do NOT discard or summarize — the raw text is the primary record.

Then extract structured fields:

• Score (numeric 1-10)
• Verdict ("ready" / "almost" / "not ready")
• Action items (ranked list of fixes)

STOP CONDITION: If score >= 6 AND verdict ∈ {"ready", "almost"} (exact match — "not ready" does NOT qualify) → stop loop, document final state.

Phase B.5: Reviewer Memory Update (hard + nightmare only)

Skip entirely if REVIEWER_DIFFICULTY = medium.

After parsing the assessment, update REVIEWER_MEMORY.md in the project root:

# Reviewer Memory

## Round 1 — Score: X/10
- **Suspicion**: [what the reviewer flagged]
- **Unresolved**: [concerns not yet addressed]
- **Patterns**: [recurring issues the reviewer noticed]

## Round 2 — Score: X/10
- **Previous suspicions addressed?**: [yes/no for each, with reviewer's judgment]
- **New suspicions**: [...]
- **Unresolved**: [carried forward + new]

Rules:

• Append each round, never delete prior rounds (audit trail)
• If the reviewer's response includes a "Memory update" section, copy it verbatim
• This file is passed back to the reviewer in the next round's Phase A — it is the reviewer's persistent memory

Phase B.6: Debate Protocol (hard + nightmare only)

Skip entirely if REVIEWER_DIFFICULTY = medium.

After parsing the review, the executor gets a chance to rebut:

Step 1 — Executor Rebuttal:

For each weakness the reviewer identified, the executor writes a structured response:

### Rebuttal to Weakness #1: [title]
- **Accept / Partially Accept / Reject**
- **Argument**: [why this criticism is invalid, already addressed, or based on a misunderstanding]
- **Evidence**: [point to specific code, results, or prior round fixes]

Rules for the executor's rebuttal:

• Must be honest — do NOT fabricate evidence or misrepresent results
• Can point out factual errors in the review (reviewer misread code, wrong metric, etc.)
• Can argue a weakness is out of scope or would require unreasonable effort
• Maximum 3 rebuttals per round (pick the most impactful to contest)

Step 2 — Reviewer Rules on Rebuttal:

Send the executor's rebuttal back to the reviewer for a ruling:

Hard mode — use the selected backend for the rebuttal step:

For codex:

mcp__codex__codex-reply:
  threadId: [saved]
  config: {"model_reasoning_effort": "xhigh"}
  prompt: |
    The author rebuts your review:

For manual: use mcp__manual_review__review_reply with the same threadId and prompt.

The prompt content:

    The author rebuts your review:

    [paste executor's rebuttal]

    For each rebuttal, rule:
    - SUSTAINED (author's argument is valid, withdraw this weakness)
    - OVERRULED (your original criticism stands, explain why)
    - PARTIALLY SUSTAINED (revise the weakness to a narrower scope)

    Then update your score if any weaknesses were withdrawn.

Nightmare mode (codex exec):

codex exec "$(cat &1

Step 3 — Update score and action items based on the ruling:

• SUSTAINED weaknesses: remove from action items
• OVERRULED: keep as-is
• PARTIALLY SUSTAINED: revise scope

Append the full debate transcript to review-stage/AUTO_REVIEW.md under the round's entry.

Human Checkpoint (if enabled)

Skip this step entirely if HUMAN_CHECKPOINT = false.

When HUMAN_CHECKPOINT = true, present the review results and wait for user input:

📋 Round N/MAX_ROUNDS review complete.

Score: X/10 — [verdict]
Top weaknesses:
1. [weakness 1]
2. [weakness 2]
3. [weakness 3]

Suggested fixes:
1. [fix 1]
2. [fix 2]
3. [fix 3]

Options:
- Reply "go" or "continue" → implement all suggested fixes
- Reply with custom instructions → implement your modifications instead
- Reply "skip 2" → skip fix #2, implement the rest
- Reply "stop" → end the loop, document current state

Wait for the user's response. Parse their input:

• Approval ("go", "continue", "ok", "proceed"): proceed to Phase C with all suggested fixes
• Custom instructions (any other text): treat as additional/replacement guidance for Phase C. Merge with reviewer suggestions where appropriate
• Skip specific fixes ("skip 1,3"): remove those fixes from the action list
• Stop ("stop", "enough", "done"): terminate the loop, jump to Termination

Feishu Notification (if configured)

After parsing the score, check if ~/.claude/feishu.json exists and mode is not "off":

• Send a review_scored notification: "Round N: X/10 — [verdict]" with top 3 weaknesses
• If interactive mode and verdict is "almost": send as checkpoint, wait for user reply on whether to continue or stop
• If config absent or mode off: skip entirely (no-op)

Phase C: Implement Fixes (if not stopping)

For each action item (highest priority first):

1. Code changes: Write/modify experiment scripts, model code, analysis scripts
2. Run experiments: Deploy to GPU server via SSH + screen/tmux
3. Analysis: Run evaluation, collect results, update figures/tables
4. Documentation: Update project notes and review document

Prioritization rules:

• Skip fixes requiring excessive compute (flag for manual

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: wanshuiyin
• Source: wanshuiyin/Auto-claude-code-research-in-sleep
• License: MIT

Install and usage instructions live in the source repository linked above.