Rasa Simulating Conversations

Goal-Driven Simulation for Rasa

Simulate realistic user conversations against a live Rasa assistant to catch flow failures, missing slot handling, and poor bot responses — before shipping.

---

How this works

1. You write (or generate) a scenario YAML in eval/scenarios/
2. This skill calls the evaluate_agent MCP tool
3. The tool uses an LLM to play the user role, driving the conversation via the REST API
4. After each run it checks assertions (deterministic) and success criteria (LLM judge)
5. Results land in eval/results///run_N.txt

---

Scenario YAML format

Currently only the text-based simulation is supported.

scenario:
  name: User successfully completes 

  simulation_context: >
    

  setup:
    initial_slots:                  # injected before conversation starts
      authenticated: true           # set any slots that should be pre-filled

  goals:
    criteria:                       # evaluated by LLM judge
      - Agent collects all required information without confusion
      - Agent confirms the task was completed successfully
      - Conversation ends naturally
    assertions:                       # deterministic checks against the tracker
      - flow_started:
          flow_ids: []       # one or more flow ids
          operator: any               # any | all
      - flow_completed:
          flow_id:           # singular; map; optional flow_step_id
          flow_step_id:      # optional
      - flow_cancelled:
          flow_id: 
      - action_executed: 
      - slot_was_set:
          - name:          # set to any non-null value
          - name: 
            value: "" # set to an exact value
      - slot_was_not_set:
          - name: 
      - bot_uttered:
          utter_name:    # any of utter_name / text_matches / buttons
          text_matches: "regex pattern"
      - bot_did_not_utter:
          text_matches: "I don't know"
      - sequencing:                    # ordered: each step matches an event strictly after the previous step's match
          - flow_started: transfer_money
          - slot_was_set: recipient    # slot name only (string); no value match
          - action_executed: action_submit_transfer
          - flow_completed:   # NOTE: plain string inside sequencing
                                       # also available as steps: flow_cancelled, flow_interrupted

Supported assertion types

Every assertion is a single-key item. These are the ONLY supported types (validated by validate_scenario); anything else fails schema validation.

| Type | Example | What it checks | | --------------------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------------------ | | flow_started | flow_started: {flow_ids: [transfer_money], operator: any} | one/all of these flows started | | flow_completed | flow_completed: {flow_id: transfer_money} | flow completed (optionally at flow_step_id) | | flow_cancelled | flow_cancelled: {flow_id: transfer_money} | flow was cancelled | | action_executed | action_executed: action_submit_transfer | this action or utter_* ran | | slot_was_set | slot_was_set: [{name: recipient, value: "John"}] | slot set (to value if given) at any point — checked over event history | | slot_was_not_set | slot_was_not_set: [{name: recipient}] | slot was never set (to value if given) | | bot_uttered | bot_uttered: {text_matches: "balance"} | bot sent a response matching utter_name / text_matches (regex) / buttons | | bot_did_not_utter | bot_did_not_utter: {text_matches: "I don't know"} | bot did not send such a response | | generative_response_is_relevant | generative_response_is_relevant: {threshold: 0.7} | LLM-judged relevance of the generated reply ≥ threshold | | generative_response_is_grounded | generative_response_is_grounded: {threshold: 0.7} | LLM-judged grounding of the generated reply ≥ threshold | | pattern_clarification_contains | pattern_clarification_contains: [transfer_money, check_balance] | clarification offered these flows | | sequencing | see example above | the listed steps occurred in order (values are plain strings) |

Notes:

• flow_started uses plural flow_ids + operator; flow_completed /

flow_cancelled use singular flow_id in a map. The terse flow_started: string form still works but is deprecated.

• sequencing supports exactly these six step types, each a single-key item

with a plain string value: action_executed, slot_was_set, flow_started, flow_completed, flow_cancelled, flow_interrupted.

• Inside sequencing the value is always a bare string — even

flow_completed/flow_cancelled (top-level these need a {flow_id: ...} map), and slot_was_set (top-level this needs a [{name, value}] list; here it is just the slot name and does not check the value).

• flow_interrupted is available only inside sequencing.
• It checks order only (each step after the previous, not necessarily

adjacent), not slot/response values.

• generative_response_is_relevant assertions are only relevant for single-turn knowledge-based questions.
• pattern_clarification_contains scans the full event list for clarification offers and works in multi-turn scenarios too.
• **Prefer structural assertions over bot_uttered / bot_did_not_utter with

text_matches.** Assertions should verify business logic, not wording — the bot rephrases every run, so regexes over response text are both brittle (false failures on wording drift) and weak (they don't prove the underlying behavior). Express the intent with structural checks instead: a flow result (flow_completed / flow_cancelled), an action (action_executed, including utter_*), a slot (slot_was_set / slot_was_not_set), or ordering (sequencing). For "the bot must not reveal X", prefer bot_did_not_utter: {utter_name: ...} or assert the relevant flow never completed, rather than a regex over the wording.

• Use text_matches **only when a specific literal token must appear verbatim

and that token is the business logic** — e.g. a product or subscription-plan name, an account/reference number, a URL, or a code. Do not use it to check generic phrasing like "not found", "sorry", or "your bill is". ---

Step-by-step workflow

Step 1 — Check eval/conftest.yml

Before doing anything else, check whether eval/conftest.yml exists:

Use Glob("eval/conftest.yml") to check.

If it exists — read it and confirm the required fields are present (see below). Proceed to Step 2.

If it does not exist — create it with the exact content below, including every comment line — do not omit or paraphrase them. Then tell the user to verify or fill in the details before continuing:

# eval/conftest.yml — simulation configuration

# LLM used to drive the user simulator (generates user turns)
simulation:
  llm:
    provider: openai       # openai | anthropic | azure | ...
    model: gpt-5.1
    timeout: 30
    reasoning_effort: "none"
    cache:
      no-cache: true
  # Prompt template override — path is relative to the project root.
  # simulated_user_prompt: prompt_templates/my_simulated_user_prompt.jinja2    # (variable: simulation_context)

# LLM used to evaluate success criteria and quality metrics (LLM judge)
evaluation:
  llm:
    provider: openai
    model: gpt-5.1
    timeout: 30
    reasoning_effort: "none"
    cache:
      no-cache: true
  # Prompt templates overrides — paths are relative to the project root.
  # Use these to customise how the judge scores conversations.
  # criteria_judge_prompt: prompt_templates/my_criteria_judge_prompt.jinja2    # (variables: criteria_text, transcript, event_ledger)
  # metrics_judge_prompt: prompt_templates/metrics_judge.jinja2      # (variable: transcript)

Stop and tell the user:

eval/conftest.yml has been created. Fill in the provider and model details,
then say "continue" and I'll pick up from here.

Do not proceed until the user confirms.

Step 2 — Ensure the bot is trained and Rasa server is running

Before running simulation, verify credentials.yml and that the assistant is live.

Check credentials.yml first. Read the file and confirm it contains both:

• rest: — enables the REST input channel used by simulation and talk_to_assistant
• inspector: — required when starting the server with --inspect (Inspector URLs in run reports)

If either block is missing, tell the user which one(s) are absent and that they need to add them and restart the Rasa server — do not edit credentials.yml yourself unless the user explicitly asks you to. Stop and wait for confirmation before continuing.

Then verify the bot responds:

Use talk_to_assistant with message ["hello"] to verify the bot responds.

If the endpoint is not reachable (connection refused, timeout, or similar) — the server is probably not running. Tell the user to start it:

The Rasa server doesn't appear to be running. Start it with:
  rasa run --inspect --credentials credentials.yml
Then say "continue" and I'll pick up from here.

Stop and wait — do not proceed until the user confirms the server is up.

If the call returns 404 — even with rest: present, the REST webhook may not be active yet (server started without --credentials, or not restarted after editing credentials.yml). Confirm rest: is in credentials.yml, then ask the user to restart with --credentials credentials.yml if needed.

The --inspect flag loads the Inspector in the same process, so a single server is all that's needed — no separate rasa inspect command.

After retraining: always check agent_reloaded in the train_rasa_assistant response.

• If agent_reloaded: true — the server picked up the new model, proceed to simulate.
• If agent_reloaded: false — the server is not running. Do NOT attempt workarounds (e.g. SlotSet in custom actions).

Tell the user: `` The model was trained but the server is not running. Start it with: rasa run --inspect Then say "re-run the simulation" and I'll continue from here. `` Stop and wait — do not re-run the simulation until the user confirms the server is up.

Step 3 - Ensure MCP server is running

Verify that you have access to the following tools: list_project_flow_definitions, validate_scenario and evaluate_agent. If one of the tools is not available, report which tool is missing to the user and stop: `` The following required MCP tools are not available: (list unavailable tools here) Please make sure that Rasa MCP tools are installed and running. Run the setup wizard from your project root: rasa tools init. See the following page for more details: https://rasa.com/docs/pro/installation/rasa-mcp-tools/ Then restart your agentic IDE (e.g. Cursor or Claude Code) so it picks up the new MCP configuration, and say "continue" when you're ready. ``

Step 4 — Find or create scenario files

Scenarios live in a flat directory:

Use Glob("eval/scenarios/*.yml") to list available scenarios.

Generate scenario(s) from the flow definition:

1. Read the flow with list_project_flow_definitions, then open the flow file.
2. Identify the flow's own structure: the goal it serves, the information it

collects (each slot + its description/validation), every branch/condition, and every decision or exit point.

1. Write each scenario to eval/scenarios/.yml, named

after the situation (e.g. transfer_wrong_account_corrected.yml).

Cover the happy path first, then add scenarios for realistic ways a real user might deviate. Derive the deviations from this flow rather than a fixed list — each collected input, branch, and exit point is a place where a real conversation can diverge. Use these as thinking prompts, not a checklist to fill mechanically:

• The user gives information that is missing, invalid, or out of range — and may

then correct it.

• The user is indirect or vague and reveals their need gradually.
• The user changes their mind, hesitates, or wants to stop partway.
• The user goes off-topic, asks something unrelated, or asks for a human.
• The user asks for something this flow (or the bot) does not handle.
• A meaningful branch/condition in the flow is exercised (take each path).

Aim for a handful of high-value scenarios per flow — enough to exercise the happy path and the most likely real deviations — not one of every possible kind. At a minimum, include: the happy path, one where the user supplies a bad value (and corrects it), and one where the user changes their mind or stops partway.

Before writing simulation_context: read config.yml and check whether it defines a language property. If it does, that value is the bot's main language — add one short instruction to every simulation_context, e.g. "Speak in German." or "Speak in Spanish.", so the user simulator knows which language to use for its messages. Without that instruction, the simulator defaults to English even when the bot expects another language.

Write the rest of simulation_context in English by default — persona, goal, and test data stay in English. Do not translate the whole briefing into the bot language; only the simulator's spoken turns should follow the "Speak in …" instruction. Do not use additional_languages from config.yml unless the user explicitly asks for that. Only write simulation_context itself in another language if the user explicitly requests it.

Writing simulation_context: write a short, natural briefing of who the user is and what they want, grounded in this flow and the plausible data they'd have. Give the simulator a persona, a goal, and the facts it needs (e.g. which credential to use, which detail to get wrong) — then let its behavior emerge. Do not script turn-by-turn what the user should say, and vary the persona (cooperative / impatient / uncertain, terse / chatty) across scenarios so the set isn't formulaic. The simulator only ever plays the user — never describe what the bot should do here (that belongs in criteria / assertions).

Backend-dependent values: some facts the simulator provides must match the bot's connected backend / mock data for even the happy path to succeed — e.g. a valid password or PIN, an existing order number, a known account or phone number. Do not invent these blindly. If a scenario is meant to succeed and depends on a value you cannot confirm is valid test data:

1. First check the existing scenarios in eval/scenarios/*.yml — a working

happy-path scenario usually already embeds valid test data in its simulation_context or initial_slots (e.g. a phone number + password that authenticate). Reuse those known-good values instead of guessing.

1. Only if no existing scenario provides them, ask the user

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: RasaHQ
• Source: RasaHQ/rasa-agent-skills
• License: Apache-2.0

Install and usage instructions live in the source repository linked above.