Generate Synthetic Data

Generate Synthetic Data

Generate realistic synthetic traces for LLM evaluation datasets using dimension-based variation.

When to use

• User needs test data for an evaluation but has no production traces.
• create-evaluation delegates here when real traces are unavailable.
• User wants to augment sparse real data with targeted synthetic examples.

Do NOT generate any synthetic data until scoping is complete and the user has approved the generation plan (dimensions, tuple count, trace structure, output destination).

BEFORE the first scoping question, search for a structured question tool (e.g., AskUserQuestion or similar interactive widget) and load it. Use that tool for EVERY scoping question. Fall back to plain-text lettered options ONLY if no such tool exists in the environment.

Scoping protocol

Ask these five questions. Skip any already answered in the conversation (e.g., if create-evaluation already established the system type, do not re-ask).

1. System type. What kind of AI system produces the traces?

• Simple RAG, tool-calling agent, multi-turn chat, support bot, classification pipeline, other

1. Trace structure. What columns does each trace contain?

• Offer common patterns based on system type:
• Simple RAG: user_query, retrieved_context, response
• Tool-calling agent: user_request, tool_calls, final_answer
• Multi-turn chat: conversation_history, assistant_response
• Support bot: customer_message, kb_lookup, agent_reply
• Let the user rename, add, or remove columns

1. Dimensions of variation. What axes should drive diversity?

• Propose 3-5 starter dimensions based on system type and known failure modes
• Each dimension needs 3-6 discrete values
• Example for RAG: query complexity (simple factual, multi-hop, ambiguous, comparative), domain coverage (billing, technical support, account management), context quality (perfect match, partial match, irrelevant, missing)

1. Dataset size. How many final traces?

• Default recommendation: 50-100 for initial eval scoping, 200+ for statistical significance

1. Output destination. Where should the data go?

• Default: MCP upload when called from create-evaluation, file when standalone
• Options: Truesight dataset (via MCP), JSONL file, CSV file, both

Use the structured question tool (loaded per the HARD-GATE above) for every question. One question per message.

Core methodology

Follow this sequence exactly. Do not skip steps or combine them.

Step 1: Draft seed tuples with the user

Generate ~20 tuples as dimension-value combinations. Each tuple is a row of dimension values that will become one trace.

Format tuples as a table:

| # | querycomplexity | domain | contextquality | edge_case | |---|-----------------|--------|-----------------|-----------| | 1 | simple factual | billing | perfect match | none | | 2 | multi-hop | technical support | partial match | none | | 3 | ambiguous | compliance | irrelevant | non-English |

Rules:

• Cover every dimension value at least once across the ~20 tuples.
• Avoid uniform distribution. Weight toward failure-prone combinations.
• Present the table to the user and ask: "Do these combinations represent realistic scenarios your system encounters? Any to add, remove, or adjust?"

Do not proceed until the user validates the tuples.

Step 2: LLM-expand tuples

After user approval of seed tuples:

• Generate 10+ additional tuples using the same dimensions.
• No duplicate dimension-value combinations allowed.
• Prioritize underrepresented dimension values and novel cross-dimension pairings.
• Present the expanded set for optional user review (do not block on this).

Step 3: Convert tuples to natural language (two-step)

This is the key quality technique. Do NOT generate traces in a single step.

Step 3a: Tuple to scenario sketch. For each tuple, write a 1-2 sentence scenario description that captures the dimension values in natural terms.

Example tuple: (multi-hop, billing, partial match, none) Scenario: "A customer asks whether upgrading their plan mid-cycle affects their next invoice and any unused credits. The knowledge base has pricing docs but nothing about proration."

Step 3b: Scenario to full trace. Convert each scenario sketch into the full trace structure (matching the columns from scoping).

Example trace for the above scenario:

{
  "user_query": "If I upgrade from Basic to Pro halfway through my billing cycle, will my next invoice be higher? And what happens to the unused days on Basic?",
  "retrieved_context": "Pro plan costs $49/month. Basic plan costs $19/month. Upgrades take effect immediately.",
  "response": "When you upgrade mid-cycle, your next invoice will reflect the Pro plan price of $49/month. The remaining days on your Basic plan will be prorated as a credit on your next invoice."
}

Why two steps: single-step generation produces repetitive phrasing and shallow variation. The scenario sketch forces diverse framing before trace generation locks in wording.

Step 4: Filter for quality

Review generated traces and remove:

• Awkward or unnatural phrasing
• Dimension-value mismatches (trace doesn't reflect its tuple)
• Near-duplicate traces (high textual similarity despite different tuples)
• Traces that could not plausibly come from the target system

Report how many traces survived filtering and the final count.

Output

MCP-first path (default when called from create-evaluation)

Invoke the upload_dataset tool with:

• name set to a descriptive dataset name
• columns set to an array of all column names (input columns + any judgment/notes columns if provided by the caller)
• input_columns set to the trace structure columns
• rows set to the generated trace data
• idempotency_key set to a unique string for safe retries

If the caller provided judgment_configs, pass them through to upload_dataset.

File-first path (default when standalone)

Write traces to the user's chosen format:

• JSONL (default): one JSON object per line. Filename: synthetic-traces-YYYY-MM-DD.jsonl
• CSV: standard CSV with headers matching trace columns. Filename: synthetic-traces-YYYY-MM-DD.csv

After writing, offer to upload to Truesight via the upload_dataset tool.

Optional: pipeline execution

If the user has a live system available, recommend running the synthetic inputs through it to get real outputs. Synthetic inputs with real outputs are more valuable than fully synthetic traces for evaluation scoping.

Anti-patterns

• Generating traces without dimension-based variation. This produces clustered, non-diverse data.
• Single-step tuple-to-trace generation. This reduces phrasing diversity compared to two-step.
• Dimensions disconnected from actual failure modes. This generates variety without evaluation value.
• Skipping user validation of seed tuples. This risks generating unrealistic scenarios.
• Using synthetic data where real traces are available. Synthetic is a fallback, not a preference.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Goodeye-Labs
• Source: Goodeye-Labs/truesight-mcp-skills
• License: MIT
• Homepage: https://truesight.goodeyelabs.com/docs/mcp-integration

Install and usage instructions live in the source repository linked above.