Eval Result Interpreter

Purpose

This skill takes eval results - a Copilot Studio evaluation CSV file (the primary worked example), an export from your own evaluator/harness, a pasted summary, or a plain-English description of results - and produces a structured triage report. It is the final step in the eval lifecycle: plan → generate → run → interpret. The output tells you whether to ship, what broke, why it broke, and what to fix first.

> Platform context. All the operational examples below use Microsoft Copilot Studio because its evaluation CSV format is well-documented and concrete. The triage framework itself (4 layers, 3 root cause types, SHIP/ITERATE/BLOCK verdict) is platform-agnostic - point it at any evaluator output with per-case results and the same analysis applies.

This skill serves Stages 2-4 of the MS Learn 4-stage evaluation framework. In Stage 2 (Set Baseline & Iterate), it interprets your first eval results and guides fixes. In Stage 3 (Systematic Expansion), it identifies coverage gaps worth expanding into. In Stage 4 (Operationalize), it triages regression failures after agent updates. Use the evaluation checklist template to track which stage you are in and what to interpret next.

Knowledge source: This skill's analysis framework is grounded in Microsoft's Triage & Improvement Playbook (github.com/microsoft/triage-and-improvement-playbook) - the 4-layer triage system, SHIP/ITERATE/BLOCK decision tree, 3 root cause types, 26 diagnostic questions, and remediation mapping.

When to use this skill vs. eval-triage-and-improvement

These two skills share the same triage framework but serve different modes of work:

| Use eval-result-interpreter when… | Use eval-triage-and-improvement when… | |---|---| | You have a CSV file or concrete results and want a one-shot structured report | You want interactive guidance walking through diagnosis step by step | | This is your first look at results - you need a verdict and top actions fast | You are in an ongoing improvement loop - fixing, re-running, and re-triaging | | You want a customer-deliverable artifact (the .docx triage report) | You need detailed remediation help for specific quality signals (e.g., "wrong tool fires - now what?") | | The eval run is relatively straightforward (`, parse the input and produce the output below. Accept any of these input formats:

Format 1 - Copilot Studio CSV file (primary worked example; use as the template for other platforms)

The user provides a file path to a CSV exported from Copilot Studio agent evaluation. The CSV has these columns:

| Column | Description | |---|---| | question | The test case input sent to the agent | | expectedResponse | The expected answer (may be empty for General Quality tests) | | actualResponse | The agent's full response | | testMethodType_1 | The test method used (e.g., GeneralQuality, CompareMeaning, KeywordMatch, ToolUse, ExactMatch, Custom) | | result_1 | Pass or Fail | | passingScore_1 | The threshold score (may be empty) | | explanation_1 | The grader's reasoning for the verdict |

A single row may have multiple test methods: testMethodType_2, result_2, passingScore_2, explanation_2, etc.

When the user provides a file path, read the CSV and parse it. Count Pass/Fail totals and per test method.

Format 2 - Plain-text summary

A pasted pass/fail count, list of failures, or verbal description of results.

Format 3 - Scenario plan reference (optional, improves accuracy)

If the user also provides the scenario plan table from /eval-suite-planner, use it to map each CSV row to its original category (core business, capability, safety, edge case) and Scenario ID. This is more accurate than inferring categories from question content alone. Say: "Using your scenario plan for category mapping."

Work with whatever detail is available. If input is sparse, state what you assumed. Do not ask for more - give the best triage possible with what is provided.

---

Output structure

0. Pre-triage infrastructure check (per the Triage Playbook)

Before analyzing failures, verify infrastructure was healthy during the eval run. If any of these were unhealthy, mark affected cases as infrastructure-blocked, not agent-failed:

• Were all knowledge sources accessible and fully indexed?
• Did any API backends return errors, timeouts, or rate-limiting?
• Were authentication tokens valid throughout the run?
• Did the eval environment match the intended configuration?

If you cannot determine infrastructure health from the input, state: "Infrastructure health not verifiable from this input - proceeding with analysis. If failures seem inconsistent, re-run after verifying all knowledge sources and APIs are accessible."

1. Score summary

Parse the results and produce:

| Metric | Value | |---|---| | Total test cases | X | | Passed | X | | Failed | X | | Pass rate | X% | | Test methods used | GeneralQuality, CompareMeaning, etc. |

If the CSV has multiple test methods per row, also report pass rate per method.

2. Verdict - per the Triage Playbook's SHIP/ITERATE/BLOCK decision tree

Apply this decision tree from the Playbook:

ALL safety/compliance test cases above blocking threshold (>=95%)?
    NO  -> BLOCK: Fix safety issues before anything else.
    YES ->
        ALL core business test cases above threshold (>=80%)?
            NO  -> ITERATE: Focus on the lowest-scoring area.
            YES ->
                Capability test cases above threshold?
                    NO  -> SHIP WITH KNOWN GAPS: Document gaps, monitor.
                    YES -> SHIP.

Use risk-based thresholds (from the Playbook's Layer 1). Adjust for context:

| Risk Profile | Safety/Compliance | Core Business | Capabilities | |---|---|---|---| | Low-risk internal tool | 90%+ | 75%+ | 65%+ | | Medium-risk customer-facing | 95%+ | 85%+ | 75%+ | | High-risk regulated | 98%+ | 92%+ | 85%+ | | Safety-critical | 99%+ | 95%+ | 90%+ |

If the CSV does not include tags or categories, infer from the question content whether each case is core business, capability, or safety. State your inference.

State the verdict prominently:

• "Verdict: SHIP." - All signals above thresholds.
• "Verdict: SHIP WITH KNOWN GAPS." - Core passing, some capability gaps documented.
• "Verdict: ITERATE." - Core business or important signals below threshold.
• "Verdict: BLOCK." - Safety failures OR overall pass rate root cause |

|---|---|---| | 1 | Is the agent's actual response acceptable (would a real user be satisfied)? | Eval Setup Issue - grader or expected value is wrong | | 2 | Is the expected answer still current and accurate? | If NO -> Eval Setup Issue - outdated expected answer | | 3 | Does the test case represent a realistic user input? | If NO -> Eval Setup Issue - unrealistic test case | | 4 | Could a reasonable alternative response also be correct but the grader rejects it? | Eval Setup Issue - grader too rigid | | 5 | Is the test method appropriate for what's being tested? | If NO -> Eval Setup Issue - wrong method |

If the eval passes all 5 checks, classify using the Playbook's 3 root cause types:

• Eval Setup Issue - the test case, expected answer, or test method is wrong. The agent may be performing correctly. Per the Playbook: at least 20% of failures in a new eval are eval setup issues, not agent issues. Sub-types: outdated expected answer, overly rigid grader, unrealistic test case, wrong eval method, grader factual error, grader systematic bias, ambiguous acceptance criteria.
• Agent Configuration Issue - the agent genuinely produced a bad response. Fix via system prompt, knowledge sources, tool config, or topic routing.
• Platform Limitation - caused by underlying platform behavior you cannot fix through configuration. Indicators: same failure persists across multiple prompt/config variations; retrieval consistently returns wrong documents despite correct config. Document and design a workaround.

Group failures that share a root cause. For example: "Cases 3, 5, and 7 all fail with 'Question not answered' - this is likely a single agent configuration issue (missing knowledge source or scope gap), not three independent problems."

3b. Platform diagnostic tools (recommend when applicable)

Copilot Studio provides built-in tools that accelerate triage. Reference these when they would help the customer investigate further:

| Tool | What it does | When to recommend | |---|---|---| | Activity map | Shows the agent's decision process for a test case - which topics triggered, which knowledge sources were retrieved, which actions were called. Available by clicking into any test case result in the UI. | Recommend for any failure where the root cause is unclear from the CSV alone. Say: "Open the activity map for case X to see whether the agent retrieved the right knowledge source or routed to the wrong topic." | | Result comparison | Compares two evaluation runs side by side, showing which cases flipped pass→fail or fail→pass. Available when you have multiple runs of the same test set. | Recommend in the next-run section (section 8) when the customer is about to re-run after changes. Say: "After re-running, use Result comparison to verify your changes fixed the target failures without breaking passing cases." | | Set-level grading | Evaluates quality across the entire test set as a whole (not just individual case pass/fail). Provides an aggregate quality assessment. | Recommend when the customer has borderline results (pass rate near a threshold) or when individual case results are inconsistent. The set-level view can reveal whether the agent is generally competent despite a few failures, or whether failures indicate a systemic problem. |

When triaging failures, always suggest the activity map for cases where you cannot determine root cause from the CSV explanation alone. The activity map is the single most useful diagnostic tool - it shows you exactly what the agent "thought," not just what it said.

Supplementary signal: User reactions (thumbs up/down)

If the agent is already deployed (even in preview), Copilot Studio captures user reactions - thumbs up/down on agent responses. These are not part of the eval CSV, but they complement eval results:

• Eval says PASS but users give thumbs down: The eval may be too lenient, or the test cases may not represent real user expectations. Investigate the gap between what the grader accepts and what users actually want.
• Eval says FAIL but users give thumbs up: The eval may be too strict (grader rigidity), or users have lower standards than the eval. Revisit the expected responses for these scenarios.
• Cluster of thumbs-down on a topic not covered by eval: Coverage gap - add test cases for that topic area in the next eval iteration.

If user reaction data is available, mention it in the pattern analysis (section 6) to cross-reference eval results with real-world satisfaction. Do not treat reactions as a replacement for structured eval - they are noisy, biased toward users who bother to click, and cannot diagnose root causes. They are a signal, not a verdict.

4. Explanation analysis

4a. General Quality scoring criteria

When the test method is GeneralQuality, Copilot Studio scores the response on 4 distinct criteria. A low General Quality score means one or more of these failed - the customer needs to know WHICH one to fix the right thing:

| Criterion | What it evaluates | Low score means | Remediation direction | |---|---|---|---| | Relevance | Does the response address the user’s question? | The agent ignored the question, answered a different question, or said “I don’t know” when it shouldn’t have. | Check knowledge source coverage - is the topic in scope? Check topic routing - is the right topic triggering? Open the activity map to see what the agent retrieved. | | Groundedness | Is the response based on the agent’s configured knowledge sources (not hallucinated)? | The agent made up information or stated facts not in its knowledge sources. This is the hallucination detector. | Review which knowledge sources were retrieved (activity map). If the right source exists but wasn’t retrieved, check indexing and chunking. If no source covers this topic, add one - or instruct the agent to say “I don’t have that information.” | | Completeness | Does the response fully answer the question without missing key parts? | The agent gave a partial answer - it addressed the topic but left out important details. | Check whether the knowledge source contains the full answer. If it does, the agent may be truncating or summarizing too aggressively - adjust system instructions. If the source is also incomplete, update the source. | | Abstention | Does the agent appropriately decline when it should? (Not over-answering, not under-answering.) | The agent either answered when it should have declined (e.g., out-of-scope question, unsafe request) OR declined when it should have answered (over-constrained). | Review system instructions for scope boundaries. Low abstention + low relevance = agent answering everything poorly. Low abstention + high relevance = agent answering things it shouldn’t be (scope leak). |

How the 4 criteria interact: A passing General Quality score means all 4 criteria passed. A failing score means at least one failed - check the explanation field to determine which. The most common failure pattern is Relevance failing alone (knowledge gap), followed by Groundedness failing alone (hallucination). When both Relevance and Groundedness fail together, the agent is likely retrieving the wrong knowledge source entirely.

When NOT to rely on General Quality alone: General Quality checks response quality holistically but cannot verify specific factual values, check tool invocation correctness, or validate structured output formats. Use it alongside targeted methods (CompareMeaning for factual accuracy, ToolUse for action verification, KeywordMatch for required terms).

4b. Explanation pattern mapping

Parse the explanation fields from the CSV. Copilot Studio’s General Quality explanations use these patterns - map each to the criteria above and the Playbook’s diagnostic questions:

| Explanation pattern | Quality signal | Playbook diagnostic area | |---|---|---| | "Seems relevant; Seems complete; Based on knowledge sources" | All passing | - | | "Question not answered; Further checks skipped because relevance failed" | Relevance failure | Diagnostics 2.1-2.5 (factual accuracy / knowledge grounding) | | "Seems relevant; Seems incomplete" | Completeness failure | Diagnostics 2.15-2.18 (response quality) | | "Knowledge sources not cited" | Source attribution failure | Knowledge grounding diagnostics | | "Seems relevant; Seems complete" (no "Based on knowledge sources") | Groundedness concern | Diagnostics 2.4-2.5 (hallucination risk) |

For each explanation pattern found in the failures, name the diagnostic area and suggest the specific Playbook question to investigate.

4c. Conversation (multi-turn) result interpretation

When interpreting results from conversation test sets (multi-turn evaluations), the failure patterns differ from single-response tests. Apply these additional diagnostic lenses:

Turn-level diagnosis: A conversation test case fails as a whole, but the root cause is usually in a specific turn. Read the agent's responses turn by turn to locate the first turn where quality degrades. Common patterns:

| Pattern | What it means | Fix direction | |---|---|---| | Turn 1 passes, Turn 3+ fails | Context loss - the agent forgot earlier context. Check whether the agent's orches

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: varunk130
• Source: varunk130/AI-Eval-Skills
• License: MIT

Install and usage instructions live in the source repository linked above.