Panel Review

/panel-review — multi-agent code review with a dev approval gate

When to use

Open a PR. Run /panel-review from inside the consumer repo to fan three model agents over the diff and produce a structured, dev-approved review comment plus a durable audit-trail build log.

No-arg invocation (/panel-review) reviews the current branch's diff vs main. Useful when you want a panel pass before opening a PR.

Skip for: trivial doc-only PRs (overhead exceeds value), out-of-band reviews where you just want a quick scan rather than a posted review.

What it produces

Two artifacts:

1. /docs/build-logs/panel-review-pr-{N}.md — durable build log. PR metadata, all three agents' raw findings, the Sonnet confidence recheck adjustments, the polished comment, the dev decision, and a Retro section. Committed to the PR's head branch so it merges with the PR. If a review is rerun on the same PR, append -{ISO-date} to the filename rather than overwriting (e.g. panel-review-pr-5-2026-05-10.md).

1. A posted PR review comment via gh pr review, gated on explicit dev approval. One of two shapes only: an approve-with-summary (no blockers) or a request-changes-with-blockers. Questions and nits stay in the build log; they are not posted publicly by default.

Hard rules

1. Worktree on entry. The skill creates an isolated worktree on a new branch off the PR's head, then runs gh pr checkout {N} inside it. Build log accumulates inside the worktree during the run and is committed to that branch before exit.

1. 3-agent fan-out, parallel. Spawn Opus, Sonnet, Haiku in parallel via the Task / Agent tool. Wait for all three before synthesizing.

1. Different prompts for the cheap model. Opus and Sonnet share a prompt focused on regression risk, security issues, and bugs. Haiku gets a different prompt focused on convention drift, naming, dead code, comment quality, and "does the diff match the PR description." This split keeps Opus and Sonnet's context free of style nits and gives Haiku a scope it handles well.

1. Discrete severity levels. Each finding is nit | question | blocker. No numeric scoring; LLMs cluster at round numbers and anchor differently across calls. Severity is "how bad if true," not confidence.

1. Auto-merge by max severity. Dedupe findings by location and topic; the deduped finding's severity is the highest level any agent assigned it.

1. Confidence and polish merged into one Sonnet pass. After fan-out merge, a single fresh-context Sonnet does both:

• For each blocker-level finding, verify it's real. False positives demote to question, with the demotion and Sonnet's reasoning recorded in the build log. Never silently dropped.
• Format the final review comment for clarity, tone, contradictions, and mis-categorization. One agent, one pass, no authority conflict between confidence and polish.

1. Dev approval gate before posting. The polished comment is shown to the dev with three labeled sections (Confirmed blockers, Questions for you/author, Low priority / nits). Dev approves, edits, or rejects. Only on explicit approval does the skill invoke gh pr review.

1. Never auto-posts. No path through this skill posts to a PR without explicit dev approval. If the gate is unclear, default to not posting.

1. Build log lives on the PR's branch. Committed before the worktree is removed so the audit trail merges with the PR. Reruns get an ISO-date suffix.

1. Retro items live in the build log. After dev decisions are recorded, the skill writes a "Retro" section listing tests, doc notes, or lint rules the dev might want to add. After worktree cleanup, the skill prompts the dev: "Want to tackle these now?" Dev decides whether to act now or later. The skill ends here either way.

1. Token usage is captured. Every Agent call's total_tokens is recorded. The build log frontmatter has total_tokens (sum across all calls); the body has a ## Token usage section with per-agent and per-phase breakdown. The retro prompt includes the total so the dev has cost context when deciding whether the review was worth it.

1. Braintrust trace logging is env-var opt-in. If BRAINTRUST_API_KEY is set, the skill logs the PR diff, agent prompts and responses, dev decisions, and the posted comment to the fdet-panel-review braintrust project via the SDK at skills/panel-review/traces.ts. Without the env var, all trace calls are silent no-ops. Logging failures degrade silently after a one-time stderr warning; they never block the skill's primary work. Nothing braintrust-related (API keys, traces, dataset exports) is committed to the public toolkit repo.

1. Fork PRs may fail at post time. gh pr review requires write access to the base repo. If the skill is run on a PR from a fork without write access, the post step will fail; the comment is preserved in the build log so it isn't lost.

1. North-star pre-flight is mandatory but non-blocking. Phase 0 always runs. Absence of docs/00-north-star.md produces a one-line dev warning and a flag in the build log header; presence loads the locked paragraph and forwards it to the fan-out agents as project-identity context. The skill never refuses to run on absence, since toolkits and libraries are legitimate consumers without a north-star.

1. Privacy default: questions and nits go to chat and build log, not the public PR. Public comments are one of two shapes only: an approve-with-summary (Shape A) or a request-changes-with-blockers (Shape B). Questions and nits ARE surfaced to the dev in chat for them to act on; they are simply not posted publicly by default. The dev may opt in to surfacing specific items in the public comment item by item. Rationale: most questions and nits surface as refactor preferences or "would be nicer if" observations whose public-PR placement adds noise without proportionate signal, but they remain useful internal working items for the dev.

1. Phase 4 cull is the noise filter the run depends on. Each agent under-grades fewer findings than it over-grades; without aggressive culling, the chat surface drowns the dev. The recheck Sonnet has explicit authority to drop questions and nits with a one-line reason. Dropped items live in the build log raw findings; they do not surface to chat. If the cull repeatedly drops items the dev would have wanted to see, that is a signal to refine the cull prompt, not to disable it.

1. CI scripts must not echo trace content. Any script under scripts/ or workflow under .github/workflows/ that touches braintrust data may print only aggregate numbers. Never PR diff content, agent responses, finding text, or dev decisions beyond aggregate counts. Repo is public, GitHub Actions logs are public, braintrust traces are private; the boundary between them is preserved by this rule. Canonical statement of the rule lives in CLAUDE.md; repeated here so future SKILL.md edits surface it to the agent during a run.

The 3-agent panel

Spawned in parallel via the Task / Agent tool with explicit model="".

Opus and Sonnet (same prompt):

> You are reviewing the diff below for regression risk, security issues, and bugs. For each finding, return JSON: { severity, location, finding, suggestion }. > > - severity: one of nit | question | blocker > - location: file:line or file:line-range > - finding: one sentence stating the problem > - suggestion: one sentence stating the fix > > Severity rubric (apply the bar strictly; default to a lower tier when in doubt): > - blocker: an ACTUAL bad bug. Will cause a crash, vulnerability, data loss, regression, or wrong production behavior. "Would page someone at 2am" severity. > - question: you are genuinely unsure this is correct AND being wrong would change runtime behavior. > - nit: small style, polish, or refactor suggestion. Doesn't change correctness or behavior. > > Do not flag stylistic preferences (em-dashes, hyphenated compounds, naming conventions, etc.). > Do not flag hypothetical library or SDK behavior. Only flag if you can verify the issue from the diff content alone or from authoritative source you can cite. "If the SDK does X then this is wrong" is not a finding. > > Be concise. Do not include praise. Return the findings list and nothing else.

Haiku (different prompt):

> You are reviewing the diff below for: (a) convention drift from existing patterns in the repo, (b) naming clarity, (c) dead code, (d) comment quality, and (e) whether the diff matches the PR description. > > Return findings in the same JSON shape as the panel: { severity, location, finding, suggestion }. Apply the same strict severity bar: blocker only if it would break a tool, check, or contract; question only if you are genuinely unsure AND wrong-ness would change behavior; nit for everything else, including refactor preferences and "would be nicer if" suggestions. > > Do not flag stylistic preferences (em-dashes, hyphenated compounds, naming conventions, etc.). > Do not flag hypothetical library or SDK behavior. Only flag if you can verify the issue from the diff content alone or from authoritative source you can cite. "If the SDK does X then this is wrong" is not a finding. > > Be concise. Do not duplicate findings about regression risk, security, or bugs (those are covered by the panel). Return the findings list and nothing else.

Phases

Seven phases. Build log populated throughout.

1. North-star pre-flight. Check for /docs/00-north-star.md. If present, parse it and load the locked paragraph for use as project-identity context in the fan-out prompts (agents can then flag drift from the project's stated identity). If absent, warn the dev once but proceed; the run is degraded but not blocked, since toolkits and libraries legitimately won't have one. Either outcome is recorded in the build log's pre-flight section.

1. Setup. Resolve target: PR number from arg, or current branch's diff vs main. Enter worktree, run gh pr checkout {N} inside it. Open the build log at /docs/build-logs/panel-review-pr-{N}.md and write the header (PR number, repo, head SHA, reviewed-at date, model panel, north-star presence).

1. Fan-out. Spawn three Agent calls in parallel. Pass each agent: the unified diff (gh pr diff {N}), the PR title and description, the locked north-star paragraph from Phase 0 (or a note that none was found), and the agent-specific prompt. Wait for all three. Append each agent's raw findings verbatim to the build log under labeled subsections.

Trace each agent — REQUIRED. Immediately after each agent returns, run this single command (do NOT construct JSON inline; the wrapper validates the shape and catches missing fields before the run continues): ``bash npm run append-trace -- \ --jsonl /tmp/claude/panel-review-traces-pr-{N}.jsonl \ --name panel-review/fan-out/ \ --input-file \ --output-file \ --metadata '{"phase":"fan-out","model":"","pr_number":,"repo":"","head_sha":"","total_tokens":,"skill_version":""}' `` Write the agent's input and output to temp JSON files first (the wrapper reads from disk to avoid CLI quoting issues on large payloads). Three fan-out agents = three append-trace calls. Skipping any of them means Phase 6 verification will fail.

1. Aggregate. Dedupe findings across the three agents by (location, finding-keyword). For each deduped finding, record: which agents flagged it, the max severity, and the merged location/finding/suggestion. Append the merged list to the build log as a table.

1. Confidence, cull, and polish (single Sonnet pass). Spawn one fresh-context Sonnet via Agent (model="sonnet"). Pass it the merged findings list with severities and contributing-agent attribution. Its mandate has THREE parts:
2. Verify blockers. For each blocker-severity finding, decide confirmed or false-positive. False positives are demoted to question and the demotion is recorded with reasoning. Never silently dropped.
3. Cull questions and nits. This is the noise filter the run lives or dies on. Default toward dropping. Keep a question only if it asks a behavior-changing yes/no the dev can actually answer (not "is this style choice good?" but "is this code path correct?"). Keep a nit only if it suggests a specific actionable change worth doing on this PR (not "would be nicer if" speculation, not generic refactor advice). Dropped items stay in the build log raw findings but are marked as culled with a one-line reason; they do not surface to chat or to the public comment.
4. Write the polished comment in Shape A or Shape B (see "Posted comment format"). Specific about what the agents actually probed in this run, not generic.

Append the Sonnet output verbatim to the build log: the recheck decisions, the cull decisions with reasons, and the comment.

The cull is aggressive on purpose. In the dogfood run that produced v0.1, agents under a looser rubric produced 11 non-blocker items; manual culling reduced this to ~5 real items. The recheck Sonnet's pre-cull pass is what makes the chat surface signal-dense rather than noise.

Trace the recheck — REQUIRED. Run the append-trace wrapper after the recheck Sonnet returns: ``bash npm run append-trace -- \ --jsonl /tmp/claude/panel-review-traces-pr-{N}.jsonl \ --name panel-review/recheck-and-polish \ --input-file \ --output-file \ --metadata '{"phase":"recheck","model":"sonnet","pr_number":,"repo":"","head_sha":"","total_tokens":,"skill_version":""}' ``

1. Dev gate and post. Show the dev:

• The polished comment in either Shape A (approve) or Shape B (request changes), determined by whether any confirmed blockers remain after the recheck.
• The recheck summary: how many blockers were confirmed vs demoted to questions.
• The culled question and nit list (post-Phase-4 filtering).
• The cull summary: how many items were dropped during Phase 4, with one-line reasons.

Per-question triage. For each culled question, ask the dev: do you already know the answer (then resolve internally with a code change or doc note), do you want to ask the PR author publicly (then opt in to surfacing it in the public comment), or drop it? Questions are not just "things to surface"; they are decisions the dev makes about who should answer them.

Per-nit triage. For each culled nit, ask the dev: fix now (apply in this PR before merge), defer (add to the build log as a [ ] todo for a follow-up PR), or drop?

Dev approves, edits, or rejects the public comment after triage.

• On approval with confirmed blockers (Shape B): gh pr review {N} --request-changes --body-file
• On approval with no confirmed blockers (Shape A): gh pr review {N} --comment --body-file
• On rejection: do not post; record the rejection in the build log with the dev's reason.

Append the dev decision and the posted-or-rejected outcome to the build log. Add a "Retro" section listing candidate tests, doc notes, or lint rules. Add a "Token usage" section with the per-agent breakdown captured from each Agent call's total_tokens, plus a row for the recheck-and-polish pass and a Total row. Update the frontmatter total_tokens to the sum. Commit the build log to the PR's branch and push.

Trace the dev gate — REQUIRED. This is the labeled trace; the dev_decision metadata in the trace is what makes it usable for evals. Critical: the input.post_cull_findings field must include the full post-cull findings list (each with id, severity, finding, suggestion, location). The eval scorers in skills/panel-review/evals/precision.eval.ts use this to look up each decision's severity. Build the input and metadata JSON files first, then run: ```bash npm run append-trace -- \ --jsonl /tmp/claude/panel-review-traces-pr-{N}.jsonl \ --

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: mollyretter
• Source: mollyretter/forward-deployed-engineer-toolkit
• License: MIT

Install and usage instructions live in the source repository linked above.