Magic Data Synthesis

Natural Language Triggers

Activate this skill when the user says things like:

• "synthesize data" / "generate missing values" / "fill in the blanks with LLM"
• "translate this column" / "convert HTML to markdown"
• "annotate these records" / "label this data"
• "enrich this dataset" / "generate descriptions"
• "fill the TBD placeholders" / "replace sentinels with real content"
• "use DataDesigner" / "run data-designer"

These produce the SAME behavior as the synthesis workflow. Natural language works equally well.

When to Use

• Columns have missing values, sentinels ("X", "N/A", "TBD"), or placeholders needing contextual generation
• Format conversion (HTML→markdown), translation, annotation, labeling, summarization
• Structured field extraction from unstructured text into multiple columns
• Reference join leaves gaps → LLM fills remaining (use enrich_from_reference.py first)

When NOT to Use: Rule-based fixes (regex, type casting, dedup) → magic-data-cleaning. Reshaping, joins, aggregation → magic-data-transformation. Schema enforcement → magic-data-validation. If a Python function can produce correct output for every case, use programmatic generation instead of DataDesigner.

Data Processing Expertise

Domain Knowledge

> Universal — consumed by interactive agents (Way 1), programmatic extractors (Way 2), and agents writing custom pipeline code (Way 3).

engine: DataDesigner (primary), batch_synthesize.py (legacy reference)
config_format: Python file with load_config_builder() function
preview_gate: HARD GATE — always before full generation, even in autonomous mode
pipeline_mode: agent generates synthesize_column() function, calls inline
model_config: MAGIC_LLM_* env vars (MODEL, BASE_URL, API_KEY, MAX_TOKENS); provider="local" targets LM Studio at localhost:1234; MCP templates also read MAGIC_TOOL_MODEL (tool-capable model)
seed_types: LocalFileSeedSource (workspace files), DataFrameSeedSource (checkpoints)
quality_control: LLMJudgeColumnConfig with multi-rubric scoring (one call per row)
quality_threshold: mean_100 >= 70% good, >= 60% acceptable, , , , ) for local models
prompt_sanitization: injection patterns filtered, whitespace normalized, values truncated
dd_install_check: importlib.util.find_spec("data_designer") — guide install if absent
dd_version: ">=0.6 — async DAG engine + per-model AIMD adaptive concurrency are the v0.6 default"
capability_reference: references/datadesigner_capabilities.md — full engine surface (11 columns, 14 samplers, 6 seed sources, processors, workflow chaining, message traces, MCP tool-use, agent-rollout ingestion, person sampling). Read it when a task needs a primitive the recipe templates don't already wire up.

Feasibility decision per column: | Task | Use LLM (this skill) | Use Code (other skills) | |------|---------------------|------------------------| | Fill missing descriptions, translate, annotate | Yes | No | | Date normalization, type casting, regex replace | No | Yes — cleaning/transformation | | Value exists in reference table | No | Yes — enrich_from_reference.py first | | Entity extraction, summarization | Yes | No |

Thinking

Before any synthesis, ask:

• Can code handle this? — Deterministic transforms (regex, type cast) → use cleaning/transformation. LLM only when contextual understanding is required.
• Does a reference dataset have the answer? — Join first, synthesize only the gaps.
• What's the cost at scale? — Preview on 5-20 rows. Extrapolate. A bad prompt on 50K rows = $100+ wasted.
• Which model fits this task? — Larger models produce higher quality but may be 10–50× slower for local inference. Match model size to output complexity:
• Short outputs (glosses, labels, single sentences, yes/no): Small models (1–8B params) usually suffice. Speed: 10–50 tok/s locally.
• Medium outputs (definitions, short paragraphs, translations): Mid-range models (8–30B) balance quality and speed. Speed: 3–15 tok/s locally.
• Complex outputs (multi-paragraph, nuanced reasoning, creative text): Large models (30B+) needed. Speed: 1–5 tok/s locally.

Always test with the 5-row preview — measure BOTH quality AND speed. If preview takes >30s per row for short-output tasks, switch to a smaller model.

• Thinking vs non-thinking models — Reasoning models (Qwen 3.5, DeepSeek R1) use 3000+ internal tokens before producing output. For simple generation tasks (definitions, translations), this overhead is wasted — prefer non-thinking models (Gemma, Llama, Phi) which produce output directly. Reserve thinking models for tasks requiring multi-step reasoning (complex annotation, code generation, mathematical derivation).

Rules

• Feasibility assessment per column: Every column proposed for synthesis must pass a feasibility check — can code handle this? (→ cleaning/transformation). Does a reference dataset have the answer? (→ enrich first). Only synthesize when contextual understanding is required.
• Preview gate is non-negotiable: Always run data-designer preview before data-designer create, even in autonomous mode. A bad prompt on 50K rows = $100+ wasted, producing content harder to clean than the original sentinels.
• Pre-filter seed data: Remove rows with sentinel values or nulls in the target column before synthesis. DataDesigner generates for ALL rows in the seed; unfiltered sentinels get "translated" literally (TBD→"À déterminer").
• Validate config before preview: Run data-designer validate before data-designer preview — catches config errors before any API calls or cost.
• Mutation ordering: Never mix deterministic cleaning and LLM synthesis in the same pass. Clean first (types, encoding, whitespace), checkpoint, then synthesize on the cleaned data.
• DataDesigner vs. programmatic generation: Not all synthesis tasks require the DataDesigner LLM engine. Choose the right approach:

| Task | Use DataDesigner (LLM) | Use Programmatic (Code) | |------|----------------------|------------------------| | Fill sentinels/placeholders with contextual text | Yes | No | | Translate text columns | Yes | No | | Annotate, label, or classify free text | Yes | No | | Extract entities from unstructured text | Yes | No | | Generate data following known mathematical/logical rules | No | Yes | | Generate data matching a mechanical template with variable substitution | No | Yes | | Generate synthetic examples requiring natural language variation | Yes | No | | Augment training data where diversity of phrasing matters | Yes | No | | Format conversion with deterministic rules (date formats, unit conversions) | No | Yes — use cleaning/transformation |

Decision heuristic: Ask "Can I write a Python function that produces correct output for every input?" If yes, use programmatic generation — it's faster, cheaper, and provably correct. If output quality depends on semantic understanding, creativity, or contextual reasoning, use DataDesigner.

Document the routing decision: When choosing programmatic over DataDesigner (or vice versa), state the reason in the output. Example: "Using programmatic generation because bit manipulation rules are deterministic — LLM arithmetic is unreliable (verified: 1/10 correct in prior test)." This makes the decision reviewable and helps users understand when LLM cost is being incurred vs. avoided.

• Parallel request tuning: The max_parallel_requests parameter in ModelConfig controls how many LLM calls DataDesigner makes simultaneously. Tune it based on the endpoint:

| Endpoint Type | Suggested Parallelism | Why | |--------------|----------------------|-----| | Local small model (10B params, e.g. Qwen 35B) | 1–2 | Model weights consume most GPU memory; minimal room for batch parallelism | | Cloud API (Gemini, OpenAI) | 4–8 | Rate limits, not memory, are the constraint; check provider docs for limits | | Cloud batch API (OpenAI batch, Vertex batch) | 10–20+ | Batch APIs are designed for throughput; limited by cost budget, not infra |

When in doubt, start with max_parallel_requests=2, measure throughput on preview, then scale up. Doubling parallelism on a local endpoint and seeing no speedup means the GPU is saturated.

DD ≥0.6 reconciliation: From v0.6 the async DAG engine is the default and applies per-model AIMD adaptive concurrency — it drops ~75% on a 429 and adds a slot after a run of successes, so it auto-tunes the live concurrency level. On DD ≥0.6 treat the table values as a ceiling, not a fixed level: AIMD starts conservative and ramps within max_parallel_requests. The knob that still matters for slow self-hosted endpoints is timeout. Set DATA_DESIGNER_ASYNC_ENGINE=0 only if you need deterministic sync ordering. (Full detail: references/datadesigner_capabilities.md §8.)

• Quality verification at scale: Preview validates a small sample, but full runs need post-generation quality checks:

| Dataset Size | Verification Strategy | |-------------|---------------------| | 500 rows | Automated checks first, then manual review of flagged rows |

Automated quality signals (check these before presenting results):

• Empty or whitespace-only outputs → generation failed silently
• Outputs that echo the input verbatim → prompt may be malformed
• Outputs in the wrong language (when generating non-English content) → verify first 5 rows
• Outputs significantly shorter or longer than the preview sample → model may be truncating or hallucinating
• Duplicate outputs across different input rows → model may be ignoring seed context
• Mode collapse: many rows producing near-identical phrasing → increase temperature or add diversity samplers

Domain-specific spot checks: For tasks with ambiguous inputs (e.g., terms with multiple meanings), flag entries where the generated output may reflect the wrong sense. Example: "bank" means both a financial institution and a river bank — a model may pick the wrong sense depending on context.

• Verification scope (what the test suite actually exercises vs. documents): The engine integration is verified end-to-end through validate → preview → create: a create() run is asserted to persist a well-formed dataset on disk with the expected row count. Documented but NOT exercised by tests: partitioned/sharded scale runs, checkpoint/resume, multi-batch merge beyond a small smoke run, and cloud-cost controls. Treat those as design guidance to verify against your own endpoint before a large run — they are not covered by an automated test.
• When to use LLM: Deterministic text fixes (whitespace, encoding, casing) are always code via cleaning. Semantic operations (filling sentinels with meaningful content, translation, annotation) require LLM via this skill.

Constraints

• MUST pre-filter seed data before synthesis — remove rows with sentinel values (TBD, N/A, X) or nulls in the target column from the seed dataset. DataDesigner generates for ALL rows in the seed; if sentinels are included, they get "translated" literally (e.g., TBD→"À déterminer") or nulls produce garbage. Filter first, generate, then merge back.
• MUST run preview/dry-run before full-scale synthesis — present cost estimate to user
• MUST validate output with quality checks before checkpointing as complete
• MUST assess feasibility per column — LLM is not the default; it's chosen when code cannot solve it
• MUST run data-designer validate before preview — catches config errors before any API calls
• SHOULD use a SEPARATE, ideally stronger, judge model for rejection sampling. WHY: a model judging its own output (same model as the generator) is a known-weak placeholder, not a calibrated gate — it tends to rate its own work favourably, so the filter passes low-quality rows. The bundled templates expose a distinct judge/judge-model alias (set MAGIC_*_JUDGE_MODEL + _PROVIDER to a stronger/cloud judge); when you leave it pointed at the generator model, treat the judge scores as a smoke signal, not a quality guarantee, and spot-check the kept rows manually.
• NEVER skip the preview gate, even in autonomous/pipeline mode. WHY: a bad prompt on 50K rows = $100+ wasted, and produces content HARDER to clean than the original sentinels
• NEVER synthesize binary data — text-representable only
• NEVER use a local reasoning model (Qwen, DeepSeek) without extract_reasoning_content=True. WHY: these models put ALL output in the reasoning/thinking field — the content field returns empty, giving you zero results with no error. SET IT ON the LLM column (LLMTextColumnConfig/LLMCodeColumnConfig), NOT on ChatCompletionInferenceParams (that raises a Pydantic "Extra inputs are not permitted"). NOTE: this flag is dual-purpose — beyond the local-model workaround it is also the native CoT-capture feature (emits a {column}__reasoning_content column) for reasoning-stage data; pair it with with_trace to keep both the full trace and the isolated chain-of-thought (see references/datadesigner_capabilities.md §6).
• NEVER set max_tokens below 2048 with local reasoning models. WHY: they use 3000+ tokens for internal reasoning before producing the answer; low limits truncate mid-thinking and return empty output
• NEVER assume enable_thinking: false works for all models. WHY: LM Studio Qwen ignores this flag; use extract_reasoning_content=True instead as the reliable approach
• NEVER skip output language verification on preview when generating non-English content. WHY: models default to English even when explicitly asked for another language — always visually verify the first 5 rows
• NEVER retry a failing LLM endpoint more than once without diagnosing the failure. WHY: blindly retrying a dead endpoint wastes time; instead health-check the endpoint, save partial results, and offer a fallback provider
• MUST save partial results before switching providers or stopping. WHY: DataDesigner writes completed batches to tmp-partial-parquet-files/ — these contain already-generated rows that should not be re-generated
• MUST offer a fallback provider when the primary fails: local model failed → offer Gemini (if GOOGLEAPIKEY set); cloud API failed → offer local model (if LM Studio running)

Seed Patterns

> Adaptable snippets — agents copy and adapt these for custom pipelines.

Pipeline synthesis function (DataDesigner CLI)

import subprocess, json, pandas as pd

def synthesize_column(seed_path: str, config_path: str, num_records: int, artifact_path: str) -> pd.DataFrame:
    """Run DD synthesis via CLI. Informed by magic-data-synthesis SKILL.md."""
    # Validate config first
    subprocess.run(["data-designer", "validate", config_path], check=True)
    # Preview for cost estimation (5 rows)
    subprocess.run([
        "data-designer", "preview", config_path,
        "--num-records", "5", "--save-results",
        "--artifact-path", f"{artifact_path}/preview",
    ], check=True)
    # Full generation (after PAUSE gate approval)
    subprocess.run([
        "data-designer", "create", config_path,
        "--num-records", str(num_records),
        "--artifact-path", artifact_path,
    ], check=True)
    # Load result
    from pathlib import Path
    parquet = next(Path(artifact_path).rglob("*.parquet"))
    return pd.read_parquet(parquet)

Minimal single-column config (loadconfigbuilder pattern)

# File: workspace/configs/my_recipe.py
from data_designer.config.config_builder import DataDesignerConfigBuilder, ModelConfig
from data_designer.config.models import ChatCompletionInferenceParams
from data_designer.config.seed_source_types import LocalFileSeedSource

def load_config_builder() -> DataDesignerConfigBuilder:
    # ADAPT: model config — change provider/model for your endpoint
    model = ModelConfig(
        alias="my-model", model="gemini-3.1-flash-lite-preview",
        inference_para

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Votee-AI](https://github.com/Votee-AI)
- **Source:** [Votee-AI/magic-agent-skills](https://github.com/Votee-AI/magic-agent-skills)
- **License:** Apache-2.0
- **Homepage:** https://docs.votee.ai/magic-agent-skills

Install and usage instructions live in the source repository linked above.