Review And Promote Traces

Review and Promote Traces

Use this skill for the judgment and promotion loop after trace evaluation.

Interactive Q&A protocol (mandatory)

BEFORE the first scoping question, search for a structured question tool (e.g., AskUserQuestion or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment.

If context is unclear, ask one question at a time using the structured question tool (loaded per the HARD-GATE above).

Example question structure:

What do you want to review first?
A) One specific run
B) Pending queue triage across runs

Rules:

• Ask one question per message.
• Use the structured question tool for every question. Structure each with a short header, 2-4 options with labels and descriptions, and place the recommended option first. Do not add "(Recommended)" or similar annotations to option labels.
• Ask one follow-up when needed, then continue.

Workflow

1. If needed, flag evaluated run:

• Use flag_review_item with run_id.

1. Retrieve review items:

• Use list_review_items with pagination.

1. Collect human judgments:

• Present review items to the user and ask for judgment one item at a time when needed.
• Capture judgment_value and optional notes for each item.
• Use the structured question tool (loaded per the HARD-GATE above) to present options that match the live evaluation setup:
• Binary example:
• A) Pass
• B) Fail
• Categorical example:
• A)
• B)
• C)
• Continuous example:
• A) Enter numeric value (within configured range)
• B) Skip this item for now
• If valid options are unclear, look them up before asking:

1) Inspect list_review_items payload for item result details and expected value hints. 2) Use get_result(run_id) for run-level context and chain outputs. 3) If evaluation id is available in run metadata, call get_evaluation(evaluation_id) and read config/judgment criteria to derive valid judgment options.

• When options remain ambiguous after lookup, ask one clarification question using the structured question tool before proceeding.

1. Submit judgments:

• For each target item, call judge_review_item.
• Pass the user-provided judgment_value and optional notes into judge_review_item.
• Include notes when judgment context matters.

1. Promote judged outputs:

• Use add_reviewed_items_to_dataset with run_id.

1. Report result:

• number of items judged
• promotion status and row counts
• any skipped or blocked items

Queue-wide triage guidance

• Group by run and status first.
• Prioritize high-impact runs or oldest pending runs.
• Keep an audit trail of judgment rationale in notes.

Scopes reference

• list_review_items requires review:read
• flag_review_item, judge_review_item, and add_reviewed_items_to_dataset require review:write

If a scope error occurs, ask the user to create a key with the missing scope in Truesight Settings.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Goodeye-Labs
• Source: Goodeye-Labs/truesight-mcp-skills
• License: MIT
• Homepage: https://truesight.goodeyelabs.com/docs/mcp-integration

Install and usage instructions live in the source repository linked above.