# Judge Panel

> Build and calibrate a multi-judge evaluator, then prove it agrees with held-out human labels before trusting it as a JUDGE. Use when an optimization target needs human-like judgment, when promoting an amber candidate toward a loop, or when another skill needs a trusted scoring panel.

- **Type:** Skill
- **Install:** `agentstack add skill-tylerjnewman-skills-judge-panel`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [TylerJNewman](https://agentstack.voostack.com/s/tylerjnewman)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [TylerJNewman](https://github.com/TylerJNewman)
- **Source:** https://github.com/TylerJNewman/skills/tree/main/plugins/autoresearch-targets/skills/judge-panel

## Install

```sh
agentstack add skill-tylerjnewman-skills-judge-panel
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

A **panel** is a JUDGE only after it agrees with human labels above a stated threshold. Until then it is a vibe with a quorum — three judges sharing one blind spot still vote wrong together. The conserved quantity is **agreement**: a measured false-negative and false-positive rate against ground truth, not the confidence of the votes.

The payoff: a calibrated panel converts an **amber** target — one blocked only because scoring needs judgment — into a loop-able one. This is the promotion mechanism, and it is reusable for any eval, not just autoresearch.

## Procedure

1. **Write the rubric.** Decompose the judgment into binary or low-cardinality sub-criteria, each with a pass/fail rule a stranger could apply. Binary failure modes beat a 1–10 score; a number that means something different to each judge cannot be aggregated.
   **Done when:** every sub-criterion has an explicit pass/fail rule.

2. **Assemble the panel.** At least three judges scoring independently, varied by prompt or model so their errors are not identical. State the aggregation rule: majority for throughput, **minimum** when a false pass is the costly error (anything safety- or compliance-adjacent), a quantile in between.
   **Done when:** panel size, judge diversity, and aggregation rule are named.

3. **Calibrate against held-out human labels.** Score a labeled held-out set the panel has never seen. Measure agreement: the false-negative rate (panel fails what humans pass) starves the loop; the false-positive rate (panel passes what humans fail) feeds reward hacking. The panel passes only if both clear threshold.
   **Done when:** FN and FP rates are measured against ground truth and compared to a stated threshold.

4. **Seal the set and freeze the panel.** Hold the calibration set out of every optimizer forever. Then freeze what was measured: model, prompt, rubric, temperature, fixtures, parser. An agreement receipt is only valid for the exact panel that earned it — change any of these and the receipt expires; recalibrate. State the correlated-error risk — where all judges likely share a weakness — and route it to the contract's guard metric.
   **Done when:** the held-out set is sealed, the panel is frozen, and the shared blind spot is named.

5. **Return the verdict.** ADMIT — the panel is a trusted JUDGE; hand its scoring command to `frame-target`. Or REJECT — agreement is below threshold; keep human and do not loop.
   **Done when:** the verdict carries its agreement receipt.

## Output contract

```text
Rubric:        
Panel:         
Aggregation:   majority | minimum | quantile-
Agreement:     FN  / FP  vs human held-out
Threshold:     
Frozen:        
Blind spot:    
Verdict:       ADMIT (scoring command:  → frame-loop JUDGE) | REJECT (keep human)
```

A rejected panel is a successful result: it found that the target is not yet loop-able and stopped you from optimizing a number no one validated.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [TylerJNewman](https://github.com/TylerJNewman)
- **Source:** [TylerJNewman/skills](https://github.com/TylerJNewman/skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-tylerjnewman-skills-judge-panel
- Seller: https://agentstack.voostack.com/s/tylerjnewman
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.