# Reward Judge Operational

> >-

- **Type:** Skill
- **Install:** `agentstack add skill-qwen-applications-skill-rm-reward-judge-operational`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Qwen-Applications](https://agentstack.voostack.com/s/qwen-applications)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** Apache-2.0
- **Upstream author:** [Qwen-Applications](https://github.com/Qwen-Applications)
- **Source:** https://github.com/Qwen-Applications/Skill-RM/tree/main/skills/reward_judge_operational

## Install

```sh
agentstack add skill-qwen-applications-skill-rm-reward-judge-operational
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Reward Judge

Use this skill to organize and access reward-judging resources. The skill is a controller and resource interface, not a per-sample prompt template.

This skill has one clean operational path: inspect the visible resource index, use only resources listed for the current sample, and produce the required verdict JSON.

## Inputs

The host message provides the visible judging task:

- benchmark and task metadata, when the operational setting allows it;
- user prompt or instruction;
- candidate responses and their current labels;
- required final output format.

Hidden chosen/rejected/gold/test labels are not available and must not be inferred.

## Resource Interface

After this skill is loaded, inspect the resource index. Use only resources that can materially affect the verdict.

Common resource types:

- `rubric`: benchmark or task criteria.
- `principle`: high-level judging principles.
- `metadata`: visible benchmark/task metadata.
- `reference`: visible reference answer or ground truth.
- `checklist`: visible constraints, criteria, or checklist items.
- `verifier`: runnable or documented verifier signal over visible content.
- `tool`: runnable tool over visible content, such as `python_sandbox`.
- `tool_protocol`: code, math, or factuality verification protocol.
- `external_pipeline`: same-backbone judging pipeline evidence exposed by the current run.
- `calibration`: position, verbosity, style, and confidence-bias controls.
- `aggregation`: policy for combining conflicting evidence.

Prefer resource reads in this order when relevant:

1. Visible hard evidence: reference, ground truth, checklist, executable verifier, or test-case evidence.
2. Reliable same-backbone external pipeline output.
3. Benchmark/task-specific rubric and principles.
4. Bias-control and aggregation resources for close calls.

Do not read every resource. For ordinary clear cases, direct judgment is acceptable. For close, exact, or correctness-sensitive cases, load only the few resources needed.

## Tool Use

Use `list_resources` only when the index returned at skill load is insufficient.

Use `view_resource` to read reference resources such as rubrics, principles, visible references, checklists, or external outputs.

Use `run_resource` only for resources whose index entry has `implementation_kind` like `runtime_verifier` or `runtime_llm_pipeline`. Do not run `reference` or `tool_protocol` resources; read them with `view_resource` instead. Treat runtime output as evidence, not as a hidden label.

Use `python_sandbox` when exact deterministic checking can change the verdict. It runs short Python over only:

- `prompt`: the visible user prompt;
- `candidates`: the current visible candidate responses keyed by label;
- `sample`: `{"prompt": prompt, "candidates": candidates}`.

It does not receive hidden labels, chosen/rejected origin, benchmark gold fields, files, network, or oracle metadata. Use it for counts, regex/format checks, JSON/list structure, simple arithmetic, supplied examples, small code-behavior checks, and answer extraction. Print compact JSON evidence or assign a JSON-serializable `result`, then use the evidence in `final_answer`.

For RM-Bench / scaled pairwise samples, if `external.openrs_pairwise_qwen` is available, it is a high-value external pipeline signal using the same base model with a preserved OpenRS-style pairwise prompt. Run it when the direct comparison is not already obvious, then reconcile its result with visible hard evidence and the rubric.

For JudgeBench, treat the task as forced choice. If this skill is loaded, use at least one relevant recommended resource before finalizing. For objective-answer, LiveBench, and LiveCodeBench samples, first check visible reference or verifier evidence when available; if that does not decide the winner and `external.openrs_pairwise_qwen` is listed, run it as the normal external-pipeline signal. Use `Tie` only when both responses are genuinely equivalent after resource checks.

For JudgeBench / MMLU-Pro / LiveBench / LiveCodeBench samples, verdict labels `A` and `B` always mean Response A and Response B, not the answer option inside the question. Use `sample.reference_or_ground_truth` and consider `verifier.reference_match` to extract each response's final answer before deciding. If the verifier is inconclusive, both responses match the reference, no reference is available, or a tie is tempting, use the listed external pipeline resources as soft evidence and reconcile them with the visible prompt/reference evidence.

If `verifier.ground_truth_score_pair` is available on an objective-answer sample, run it when the candidates' final answers differ and a visible ground truth or reference can decide correctness. Treat its scores as same-base-model verifier evidence over visible reference content; reconcile it with exact reference matches and the candidate text before finalizing.

OpenRS-style operational runs may expose `sample.task_metadata`, `sample.reference_or_ground_truth`, and `sample.checklist_or_constraints`. Treat `ground_truth`, `constraints`, `check_list`, and sanitized `additional_metadata` as resource evidence only when they are present in the runtime resource index.

## Decision Procedure

1. Identify the user’s actual task and mandatory constraints.
2. Build one shared criterion for the current candidate set.
3. Check visible hard failures before style: wrong final answer, violated required format, unsafe refusal boundary, missing requested deliverable, unsupported factual claim, or failed checklist item.
4. If the resource index contains a visible reference, ground truth, checklist, verifier signal, or reliable external pipeline output that is relevant to the current task, use it before general style preference.
5. Use benchmark/task-specific resources only when they improve the current judgment.
6. Apply bias controls before finalizing: do not prefer position, length, markdown polish, confident tone, or fluent rationale unless it improves task success.
7. Aggregate evidence and return the required JSON.

## Leakage Rules

- Do not use hidden chosen/rejected/gold/label information.
- Do not infer the answer from dataset construction artifacts.
- Do not request or use oracle-only resources.
- Do not treat external pipeline signals as gold labels.
- Record resources actually used in `used_resources`.

## Output

Return JSON only:

```json
{
  "verdict": "A|B|Tie",
  "confidence": 0.0,
  "used_resources": [],
  "reason": "short reason"
}
```

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Qwen-Applications](https://github.com/Qwen-Applications)
- **Source:** [Qwen-Applications/Skill-RM](https://github.com/Qwen-Applications/Skill-RM)
- **License:** Apache-2.0

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-qwen-applications-skill-rm-reward-judge-operational
- Seller: https://agentstack.voostack.com/s/qwen-applications
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.