Eval Suite Planner

Purpose

This skill takes a plain-English description of an agent and produces a structured eval suite plan. It is the first step in the eval lifecycle - use it before generating test cases or running any evals. The output tells you exactly what scenarios to build, which evaluation methods to use, and how to know when you're done.

> Platform context. The plan is platform-agnostic. Microsoft Copilot Studio is referenced throughout as the primary worked example because its named test methods (GeneralQuality, CompareMeaning, KeywordMatch, ToolUse, ExactMatch, Custom) and analytics surface make the recommendations concrete. The same scenario routing, quality signals, and thresholds apply to any agent platform (custom LLM apps, LangChain/LangGraph, AutoGen, Semantic Kernel, OpenAI Assistants, etc.) - map the recommended methods to your evaluator's equivalents.

This skill covers Stage 1 (Define) of the MS Learn 4-stage evaluation framework. After planning, use /eval-generator for Stage 2 (Set Baseline & Iterate), then expand coverage (Stage 3) and operationalize into CI/CD (Stage 4).

Knowledge sources: This skill's guidance is grounded in three Microsoft sources:

• Eval Scenario Library (github.com/microsoft/ai-agent-eval-scenario-library) - 5 business-problem scenario types with 29 sub-scenarios, 9 capability scenario types with 49 sub-scenarios, quality signals, and evaluation method selection
• MS Learn agent evaluation documentation - the 4-stage iterative evaluation framework (Define, Set Baseline & Iterate, Systematic Expansion, Operationalize), 7 test methods, acceptance criteria design, and evaluation categories
• MS Learn evaluation checklist (guidance/evaluation-checklist) - a 4-stage checklist template with a downloadable editable version. The checklist defines Stage 3 expansion categories (Foundational core, Agent robustness, Architecture test, Edge cases) and introduces acceptance criteria design

Instructions

When invoked as /eval-suite-planner , read the description, infer the agent's primary task, key capabilities, and failure modes, then produce the following output in this exact order. Do not ask clarifying questions, do not pad responses, do not hedge.

---

Step 0 - Match the agent to scenario types

Use this routing table (from the Eval Scenario Library's Entry Path A) to identify which business-problem and capability scenario types apply to the described agent:

| If the agent... | Business-problem scenarios | Capability scenarios | |---|---|---| | Answers questions from knowledge sources | Information Retrieval (6 sub-scenarios) | Knowledge Grounding + Compliance | | Executes tasks via APIs/connectors | Request Submission (6 sub-scenarios) | Tool Invocations + Safety | | Walks users through troubleshooting | Troubleshooting (6 sub-scenarios) | Knowledge Grounding + Graceful Failure | | Guides through multi-step processes | Process Navigation (6 sub-scenarios) | Trigger Routing + Tone & Quality | | Routes conversations to teams/departments | Triage & Routing (5 sub-scenarios) | Trigger Routing + Graceful Failure | | Handles sensitive data (PII, financial, health) | (add to whichever applies) | Safety + Compliance | | Serves external customers | (add to whichever applies) | Tone & Quality + Safety | | Is about to be updated or republished | (add to whichever applies) | Regression - re-run existing tests after changes | | All agents (always include) | - | Red-Teaming - adversarial robustness testing |

Most agents match 1-2 business-problem types and 3-4 capability types. Select the ones that fit and name them explicitly.

About the Regression row: A regression set is not a separate scenario type - it is your existing suite of passing tests, re-run after any agent change to verify nothing broke. Include the regression row when the customer mentions upcoming changes (prompt edits, knowledge source updates, connector/plugin changes, republishing). When it applies:

• Flag that the customer's current passing test cases become their regression baseline
• Recommend re-running the full eval suite (or at minimum the core business + safety subsets) after every change
• This maps to Stage 4 (Operationalize) of the MS Learn framework - embedding evals into the agent's update workflow so regressions are caught before they reach users
• The regression set grows over time: every bug found and fixed should add a test case that catches that specific failure, preventing it from recurring

Step 0b - Incorporate production data (for live agents)

If the customer already has an agent in production, recommend supplementing the synthetic eval plan with real user data. Copilot Studio offers two production-data features that create higher-fidelity test sets:

Themes-based test sets (recommended for production agents): Copilot Studio's Analytics page groups real user questions into themes - clusters of related questions that triggered generative answers (e.g., "billing and payments", "password reset", "shipping status"). Customers can create a test set directly from any theme:

1. Go to the agent's Analytics page > Themes list
2. Hover over a theme > select Evaluate
3. Select Create and open to generate a test set from real user questions in that theme

When to recommend themes-based test sets:

• The agent is already in production with real user traffic
• The customer wants to track quality per topic area (e.g., "How well does the agent handle billing questions specifically?")
• The synthetic plan may miss question phrasings real users actually use
• The customer is investigating a quality drop in a specific area flagged by analytics

How themes complement the eval plan: The scenario plan (Step 0) defines WHAT to test based on the agent's design. Themes-based test sets validate HOW WELL the agent handles what users actually ask. Use both:

• Synthetic test cases (from /eval-generator) = structured coverage of planned scenarios, edge cases, and adversarial tests
• Themes-based test sets (from production) = real-world phrasing, actual user patterns, production-frequency weighting

Tell the customer: "Your eval plan covers what the agent SHOULD handle. Themes-based test sets cover what users ACTUALLY ask. The gap between these two is where surprises live."

Prerequisite: Themes require the themes (preview) feature to be available in the customer's environment. Check prerequisites before recommending.

Production data import: Customers can also import real user conversations as test cases directly. This is useful for reproducing specific reported issues or building regression tests from support tickets.

Step 0c - Plan test set creation strategy

Copilot Studio offers multiple ways to create single-response test sets beyond CSV import. During planning, make the customer aware of all their options so they can combine approaches for best coverage:

| Creation method | What it does | When to recommend | |---|---|---| | CSV import (from /eval-generator) | Import structured test cases with Question, Expected response, and Testing method columns | Always - this is the primary method for structured, scenario-driven coverage | | Quick question set | Auto-generates a small set of questions from the agent's knowledge sources | Early exploration - quickly see what questions the agent's content can answer before writing detailed cases | | Full question set | Auto-generates a comprehensive set of questions from knowledge sources | Broader coverage check - use after the initial eval to find gaps the structured plan missed | | Test chat → test set | Converts a manual test chat session into a reusable test set | When someone has already tested the agent manually and wants to make those tests repeatable | | Themes-based (see Step 0b) | Creates test sets from real user question clusters in production analytics | Production agents - captures actual user phrasing and topic distribution | | Manual entry | Create individual test cases directly in the Copilot Studio UI | One-off additions, Custom method cases (which cannot be CSV-imported), and edge cases discovered during testing |

Recommended strategy: Start with CSV import from /eval-generator for structured scenario coverage, then supplement with Quick or Full question sets to catch blind spots the plan didn't anticipate. For production agents, add themes-based test sets for real-world validation. Use manual entry for Custom-method cases that require rubric definitions.

Tell the customer: "CSV import gives you precision - every case tests a specific scenario you designed. Auto-generation gives you breadth - it finds questions you didn't think to ask. Use both."

Step 0d - Choose test data generation approach

Per the MS Learn Common evaluation approaches, there are three strategies for generating request-response pairs. The choice affects multi-turn fidelity, cost, and what kinds of failures you can detect:

| Approach | How it works | Strengths | Weaknesses | Best for | |---|---|---|---|---| | Echo | Replay a static list of prompts word-for-word | Low cost; fair A/B comparisons when changing one variable (model upgrade, single tool change) | Can’t adapt to different responses - later turns may not match conversation context | Single-turn scenarios, deterministic checks (citation display, tool trigger, simple Q&A) | | Historical replay | Replay each turn in context of prior prompts and responses | Detects where and how much each turn diverges from the ideal path | Still can’t handle truly dynamic conversations (learning, real-time web search) | Model change comparisons, understanding per-turn divergence from baseline behavior | | Synthesized personas | A human or agentic actor generates conversation in real time based on a scenario and persona | Dynamically assesses complex scenarios (tutoring, negotiation, multi-step troubleshooting) | Grading requires nuance; higher cost (LLM or human tester per conversation) | Multi-turn agents, complex workflows, persona-dependent behavior |

How to recommend:

• Simple FAQ / knowledge agents → Echo is sufficient. Most test cases are single-turn with deterministic expected answers.
• Task agents being upgraded (model change, tool swap) → Historical replay to compare before/after at each turn.
• Complex multi-step agents (process navigation, troubleshooting, triage) → Synthesized personas for realistic coverage. Echo won’t catch context-dependent failures.
• Hybrid is common: use Echo for the core regression set (fast, cheap, repeatable) and Synthesized personas for the exploratory/edge-case set (realistic, expensive, high-signal).

Tell the customer: “Echo tells you if the same questions still get the same answers. Synthesized personas tell you if the agent can actually handle a real conversation. You need both - Echo for speed, personas for truth.”

Output structure

1. One-line summary

Restate the agent's task in one sentence, starting with "Agent task:". Name the matched business-problem and capability scenario types by their plain names (e.g., "Information Retrieval", "Knowledge Grounding").

2. Scenario plan table

This table is the primary handoff artifact to /eval-generator - the generator will produce one test case per row. Make it complete enough that the generator needs no additional context.

Produce a table with these columns:

| # | Scenario Name | Category | Tag | Evaluation Methods | |---|---|---|---|---|

Be specific: name the actual scenario based on the agent description, not just the category.

Use this category distribution (from the Eval Scenario Library's eval-set-template):

• Core business scenarios: 30-40% of test cases
• Capability scenarios: 20-30%
• Edge cases & safety: 10-20%
• Variations (different phrasings of core): 10-20%

For evaluation methods, use the Scenario Library's quality-signal-to-method mapping:

| What you're testing | Primary method | Secondary method | |---|---|---| | Factual accuracy (specific facts, numbers) | Keyword Match (All) | Compare Meaning | | Factual accuracy (flexible phrasing) | Compare Meaning | Keyword Match (Any) | | Policy compliance (mandatory language) | Custom | Keyword Match (All) | | Policy compliance (nuanced judgment) | Custom | General Quality | | Tool invocation correctness | Capability Use | Keyword Match (Any) | | Knowledge source selection | Capability Use | Compare Meaning | | Topic routing accuracy | Capability Use | - | | Response quality, tone, empathy | General Quality | Compare Meaning | | Tone/brand voice adherence | Custom | General Quality | | Hallucination prevention | Compare Meaning | General Quality | | Regulatory/HR/legal compliance | Custom | Keyword Match (All) | | Edge case handling | Keyword Match (Any) | General Quality | | Negative tests (must NOT do X) | Keyword Match - negative | Capability Use - negative |

When to use Custom: The Custom test method lets you define evaluation instructions (a rubric) with labeled outcomes (e.g., "Compliant" / "Non-compliant") and assign pass/fail to each label. Use it when:

• The pass/fail criteria require judgment, not just keyword presence - e.g., "Is this response empathetic?" or "Does this follow our escalation policy?"
• You need domain-specific rubrics - e.g., HR compliance, medical disclaimers, financial suitability
• Standard methods (Keyword Match, Compare Meaning) cannot capture the quality signal - the answer is not about specific words or semantic similarity, it is about whether the response meets a policy or standard
• You want to test tone, style, or brand voice beyond what General Quality covers

Custom is not available for CSV import - test cases using Custom must be created directly in Copilot Studio's evaluation UI.

Beyond Custom - rubric-based grading: For customers who need more granular quality scoring than pass/fail, the Copilot Studio Kit supports rubric-based grading on a 1-5 scale. Rubrics replace the standard validation logic with a custom AI grader aligned to domain-specific criteria. Two modes: Refinement (grade + rationale - use first to calibrate the rubric against human judgment) and Testing (grade only - use for routine QA after the rubric is trusted). If the plan includes Custom methods for compliance, tone, or brand voice, note in the plan rationale that rubric-based grading is an advanced option for ongoing calibrated quality assurance.

Planning for rubric calibration effort: Rubric refinement is iterative - expect 3-5 calibration rounds before AI-human alignment is acceptable. Each round involves: running the test set, human-grading every case with written reasoning, comparing alignment scores, and refining the rubric. Plan for this when the eval plan includes rubric-graded scenarios:

• Alignment target: 75-90% average alignment (formula: 100% × (1 − |AI grade − Human grade| / 4)). Don’t plan for 100% - some subjectivity is inherent and diminishing returns start around 85%.
• Time investment: Each calibration round requires a domain expert to grade and write reasoning for every test case. Budget 1-2 hours per round for a 10-15 case set.
• When rubrics are worth the investment: Compliance-heavy domains, regulated industries, brand-sensitive customer-facing agents, or any scenario where "pass/fail" is too coarse and you need calibrated quality scores over time.
• **When to skip

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: varunk130
• Source: varunk130/AI-Eval-Skills
• License: MIT

Install and usage instructions live in the source repository linked above.