Eval Generator

Purpose

This skill generates concrete eval test cases - with realistic inputs, expected outputs, and evaluation method configurations. It is the second step in the eval lifecycle: plan → generate → run → interpret.

> Platform context. This skill is platform-agnostic. Microsoft Copilot Studio is used throughout as the primary worked example - its CSV import format, named test methods (GeneralQuality, CompareMeaning, KeywordMatch, ToolUse, ExactMatch, Custom), and Copilot Studio Kit rubrics give us concrete, well-documented outputs. The same plan-to-test-case transformation works against any agent platform (custom LLM apps, LangChain/LangGraph, AutoGen, Semantic Kernel, OpenAI Assistants, etc.) - re-shape the CSV columns or feed the test cases directly to your own evaluator.

This skill covers Stage 2 (Set Baseline & Iterate) of the MS Learn 4-stage evaluation framework. Use /eval-suite-planner first for Stage 1 (Define), then generate test cases here, run them, and interpret results with /eval-result-interpreter. Stage 3 (Systematic Expansion) means repeating this cycle with broader coverage - the checklist defines four expansion categories: Foundational core, Agent robustness, Architecture test, and Edge cases. Stage 4 (Operationalize) means embedding these evals into your agent's CI/CD pipeline. Point customers to the editable checklist template to track their progress across all four stages.

Primary mode: If the conversation already contains output from /eval-suite-planner, use that plan's scenario table, evaluation methods, quality signals, and tags as the blueprint. Generate one test case per row in the plan.

Fallback mode: If no plan exists in the conversation, accept a plain-English agent description and generate test cases from scratch (6-8 cases minimum).

Instructions

When invoked as /eval-generator (with or without additional input):

Step 1 - Detect input mode

Check the conversation history for output from /eval-suite-planner. Look for the scenario plan table (a markdown table with columns: #, Scenario Name, Category, Tag, Evaluation Methods).

• Plan found: Use it as the blueprint. Say: "Generating test cases from your eval suite plan (X scenarios)." Generate one test case per row.
• No plan, but user provides an agent description: Generate from scratch. Say: "Generating eval scenarios for: [agent task in your own words]." If the description is fewer than two sentences or doesn't mention success criteria, ask exactly one clarifying question, then wait.
• No plan and no description: Say: "I need either an agent description or a plan from /eval-suite-planner. Run /eval-suite-planner first for the best results, or give me a description and I'll generate directly."

Step 1b - Determine evaluation mode (Single Response vs. Conversation)

Before generating test cases, determine which evaluation mode fits the agent. This affects the output format, available test methods, and import options.

Choose Conversation mode when the agent:

• Handles multi-step tasks that require context across turns (e.g., booking a trip with departure, return, and seat selection)
• Needs to ask clarifying questions before completing a request
• Must maintain state (e.g., remembering a customer's account after initial identification)
• Has handoff or escalation flows that depend on prior turns

Choose Single Response mode when the agent:

• Answers standalone questions (FAQ, policy lookup, factual retrieval)
• Routes to a single tool per request
• Produces a self-contained output per input (e.g., a summary, a classification)

Default: If the plan or agent description does not indicate multi-turn behavior, default to Single Response.

If Conversation mode is selected, say: "This agent benefits from conversational (multi-turn) evaluation. I will generate conversation test cases - each is a multi-turn dialogue, not a single question."

Then skip to Step 2b below for conversation-specific generation.

Step 2a - Generate single-response test cases

When generating from a plan: Match each scenario row exactly - use its Scenario Name, Tag, and Evaluation Methods. Translate the evaluation methods into the right expected fields and test configurations.

When generating from scratch (no plan):

• 6-8 total scenarios
• At least 2 happy-path / core business cases
• At least 2 edge cases (empty input, long input, ambiguous input, malformed input)
• At least 1 adversarial case (prompt injection, out-of-scope request, policy violation attempt)
• Fill remaining with whatever gives the most signal for this agent

Step 2b - Generate conversation (multi-turn) test cases

Use this step instead of Step 2a when the agent requires multi-turn evaluation.

Conversation test set constraints (Copilot Studio limits):

• Up to 20 test cases per test set
• Each test case supports up to 12 total messages (6 user-agent Q&A pairs)
• Supported test methods: General Quality, Keyword Match, Capability Use (called "Capabilities match" in conversation UI), Custom (Classification)
• NOT supported in conversation mode: Compare Meaning, Text Similarity, Exact Match

When generating from a plan: Group related scenarios into multi-turn conversations. A single conversation test case can cover 2-3 related scenarios that would naturally occur in sequence (e.g., "check order status" then "request refund on that order" then "confirm refund policy").

When generating from scratch (no plan):

• 4-6 conversation test cases
• At least 2 happy-path multi-step task completions
• At least 1 conversation with a clarification loop (user gives vague input, agent asks, user clarifies)
• At least 1 conversation with a mid-conversation topic switch or escalation
• At least 1 adversarial turn mid-conversation (e.g., user attempts prompt injection after establishing context)

Conversation test case format (displayed in conversation):

For each test case, produce a structured conversation block:

> Conversation Test Case #N: [Scenario Name] > > Turn 1 - User: [realistic user message] > Turn 1 - Agent (expected): [expected agent response or behavior description] > > Turn 2 - User: [follow-up that depends on Turn 1 context] > Turn 2 - Agent (expected): [expected response maintaining context] > > Turn 3 - User: [further follow-up or new request within same session] > Turn 3 - Agent (expected): [expected response] > > Test method: General Quality / Keyword Match / Capability Use / Custom (Classification) > Keywords (if Keyword Match): [comma-separated keywords expected in any agent response] > What this tests: [1 sentence: what multi-turn capability is being evaluated] > Critical turn: [which turn is most likely to fail and why]

Rules for conversation test cases:

• Each turn must build naturally on the previous turn - do not write turns that could stand alone
• Agent expected responses should describe the behavior, not the exact wording (these are reference responses for the LLM judge)
• Include at least one conversation where the user's intent shifts or expands across turns
• Flag which turn is the "critical" turn - the one most likely to fail (e.g., Turn 3 where context from Turn 1 must be retained)

Important - no CSV import for conversation test sets: Unlike single-response test cases, conversation test sets cannot be imported via CSV. The customer must create them in Copilot Studio using one of these methods:

1. Quick conversation set - auto-generates 10 short conversations from agent description and instructions
2. Full conversation set - generates conversations from agent knowledge sources or topics (short or long)
3. Use test chat - converts the latest test chat session into a conversation test case
4. Manual entry - add user questions and agent reference responses turn by turn in the UI

The conversation test cases generated by this skill serve as a planning blueprint - the customer uses them to:

• Guide what they enter manually in the Copilot Studio conversation editor
• Compare against AI-generated conversation sets to check coverage
• Document the multi-turn scenarios that matter before creating test sets in the UI

Output - Single Response

Copilot Studio Test Set Table (displayed in conversation)

This is the primary output for single-response mode. Produce a markdown table matching the Copilot Studio test set format:

| # | Question | Expected Response | Test Method | Pass Score | |---|---|---|---|---| | 1 | [realistic user input] | [expected answer, or leave blank for General Quality] | General Quality | - | | 2 | [realistic user input] | [expected answer for comparison] | Compare Meaning | 50 | | 3 | [realistic user input] | [keywords to check] | Keyword Match (All) | - |

Map the evaluation methods from the plan (or from your analysis) to Copilot Studio's 7 test methods using this selection guide:

Test Method Selection Guide

Use this table to select the correct test method based on what you need to verify about the response. When in doubt, match to the "What you're testing" column first, not the method name.

| What you're testing | Test Method | Expected Response format | Pass Score | |---|---|---|---| | Overall response quality, tone, helpfulness (LLM judge checks: relevance, groundedness, completeness, abstention) | General Quality | Leave blank - no expected response needed | - | | Factual accuracy with flexible phrasing (meaning matters, wording doesn't) | Compare Meaning | Write the ideal answer in natural language | 50 (default) | | Factual accuracy with specific required facts, numbers, or policy language | Keyword Match (All) | Comma-separated list of keywords/phrases that MUST all appear | - | | Partial keyword coverage (at least one key term should appear) | Keyword Match (Any) | Comma-separated list of keywords/phrases where ANY match is a pass | - | | Correct tool/topic/connector invocation | Capability Use (shown as "Tool use" in UI) | Comma-separated list of expected tool or topic names | - | | Classification, labeling, or structured output with exact expected values | Exact Match | The exact expected string (case-sensitive) | - | | Response phrasing matters (not just meaning - specific wording preferred) | Text Similarity | Write the expected phrasing | 0.70 (default; scale 0-1) | | Domain-specific criteria that don't fit above methods | Custom | Write evaluation instructions and pass/fail label definitions | - |

How General Quality scoring works

General Quality is the default test method and the most commonly used. Understanding its 4 scoring criteria helps customers write better test cases and interpret results correctly. General Quality uses an LLM judge that scores each response on:

| Criterion | What it checks | A low score means | Tip for the customer | |---|---|---|---| | Relevance | Does the response directly address the user's question? Does it stay on topic? | The agent went off-topic, added irrelevant information, or answered a different question than what was asked | If relevance scores are low, check the agent's topic routing | | Groundedness | Is the response based on the agent's knowledge sources? Does it avoid hallucinating unsupported claims? | The agent invented facts, cited sources it doesn't have, or generated plausible-sounding but unverified information | Low groundedness is the most dangerous failure. Check knowledge source coverage | | Completeness | Does the response cover all aspects of the question with sufficient detail? | The agent gave a partial answer, missed key details, or was too brief to be useful | If completeness is low but relevance is high, the agent understands the question but its knowledge sources may have gaps | | Abstention | Did the agent attempt to answer at all? (Abstaining on an answerable question is bad; abstaining on an unanswerable question is good) | The agent refused to answer a question it should have answered, OR answered a question it should have declined | Abstention failures often indicate overly restrictive or overly permissive system instructions |

All 4 criteria must be met for a response to score high. A response that is relevant, grounded, and complete but fails abstention (e.g., answering a question outside its scope) still gets a low score. This prevents agents from "helpfully" answering questions they shouldn't.

When NOT to rely on General Quality alone: If your scenario has a specific correct answer (a fact, a number, a policy), pair General Quality with Compare Meaning or Keyword Match. General Quality evaluates how well the agent responds, but without an expected answer it cannot catch factual errors where the agent sounds great but says the wrong thing.

How Custom test methods work

Custom is the newest test method in Copilot Studio. It lets you define your own evaluation criteria when none of the built-in methods fit - for example, compliance checks, tone audits, or domain-specific quality bars. Custom works for both single-response and conversation test sets.

A Custom test has two components you configure:

| Component | What it is | How to write it | |---|---|---| | Evaluation instructions | A prompt that tells the LLM judge what to look for in the agent's response | Be goal-oriented. Use bullet points and headings. Describe the specific qualities you want to assess. | | Labels | Named categories the judge assigns to each response, each mapped to Pass or Fail | Use 2-3 labels. Each label has a name and a description of what qualifies. One label = Pass, the other = Fail. |

Example - HR Policy Compliance custom test:

• Evaluation instructions: "Evaluate the agent's response for HR policy compliance. Check: protects privacy, avoids discrimination or bias, provides safe HR-aligned guidance, does not give legal advice."
• Labels: Compliant (Pass) - "Response follows all HR policies and guidelines." / Non-Compliant (Fail) - "Response violates one or more HR policy requirements."

When to use Custom:

• Domain-specific compliance or regulatory requirements the built-in methods cannot express
• Tone or style evaluation (e.g., "Is the response empathetic?" or "Does it match our brand voice?")
• Safety or guardrail checks beyond what General Quality's abstention criterion covers
• Classification accuracy where labels are complex or context-dependent (not a simple exact match)

Important - Custom is NOT available via CSV import. If your test plan includes Custom test cases, import the other test cases via CSV first, then add Custom test cases manually in the Copilot Studio UI. Note this in the docx report.

Advanced - Rubric-based grading (Copilot Studio Kit)

For customers using the Copilot Studio Kit, rubrics provide a structured 1-5 grading scale for Generative Answers test cases. Rubrics are more granular than the built-in test methods - they replace the standard validation logic with a custom AI grader that scores responses against domain-specific criteria.

Rubrics operate in two modes:

| Mode | Assignment level | AI output | Cost | Use when | |---|---|---|---|---| | Testing mode | Individual test case | Grade only (1-5) | Lower | Rubric is refined and trusted; routine QA and regression testing | | Refinement mode | Entire test run | Grade + detailed rationale | Higher | Creating or improving a rubric; comparing AI vs. human grading to minimize misalignment |

Key workflow - the rubric refinement cycle: Rubric refinement is iterative, not a one-time configuration. The cycle is: **Run → Review → Grade → Refine → S

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: varunk130
• Source: varunk130/AI-Eval-Skills
• License: MIT

Install and usage instructions live in the source repository linked above.