Cmd Golden Tests

Golden Tests

Golden (snapshot) tests capture the exact output of a pipeline or subsystem as a reference file, then fail any run that deviates from it. They are the highest-fidelity regression check: if anything in the pipeline changes — parsing, mapping, migration, export — the diff tells you exactly what shifted.

Reference Implementation

The paradigm for this skill comes from boredm/gint_to_boredm. Read it before adapting anything:

• Makefile targets: makefiles/golden.mk
• Snapshot scripts: scripts/golden_thompson.py
• Fixtures: server/tests/fixtures/
• Workflow golden: golden.md

---

The Five Setup Questions

Answer these before writing any code or Makefile targets.

1. What is the fixture / on-device schema?

The input data the pipeline runs against. Must be:

• Checked in to the repo (or clearly documented how to obtain it)
• Small enough to run in CI without special infra
• Representative of real edge cases, not just happy-path data

In boredm: real .gpj files in thompson_sample/ (binary, not committed to the main repo — seeded via make seed-thompson).

2. What is the API under test?

How the system is invoked in the golden test. Three patterns:

| Pattern | Use when | Example | |---------|----------|---------| | Direct function call | Testing a pure transformation | schema_grouping.build_grouping_outputs(schemas) | | HTTP endpoint via TestClient | Testing a route end-to-end | POST /api/units/generate-migrations/{schema} | | Full workflow via manifest | Testing the entire pipeline | Run wf1–wf7, read manifest from disk |

Avoid mixing patterns in the same golden — pick the right scope.

3. What is the golden truth dataset?

The reference snapshot file. Key design choices:

Semantic golden (single compact file):

• Extracts a stable subset of the output (row counts, field counts, match types, transformation types)
• Intentionally excludes volatile fields: timestamps, file paths, binary hashes, job IDs
• Sorts all dict keys for deterministic JSON output
• Good for: full-pipeline end-to-end tests

Phase-level goldens (per-schema per-phase files):

• One file per {phase}_{schema}.json — e.g., wf1_grouping_file_group_1.json
• Captures the intermediate state after each workflow phase
• Good for: pinpointing which phase introduced a regression

Unit-test-style fixture (step-by-step JSON):

• Inline expected values alongside inputs in the same file
• Steps: baseline → mutation → restored
• Good for: idempotency tests, lifecycle tests

4. When and how do you update it?

Never silently. The update workflow must be:

1. Run the full pipeline
2. Inspect the diff (unified diff, colorized)
3. Decide: expected change or regression?
4. Only then run the explicit update command

Makefile targets enforce this:

golden-{dataset}-verify   # compare without running pipeline
golden-{dataset}-update   # rewrite golden from latest output
golden-{dataset}-wf       # run pipeline + compare (fails on diff)

5. How do you handle failures, updates, false positives, and false negatives?

| Scenario | Action | |----------|--------| | Expected change (feature added, behavior improved) | Inspect diff → make golden-{dataset}-update → commit both code and golden | | Regression (pipeline broke something) | Fix root cause, never update golden to hide it | | False positive (volatile field leaked into snapshot) | Remove volatile field from snapshot extractor, not from golden | | False negative (golden too coarse, misses real change) | Add a phase-level golden or tighten the snapshot to cover the missed surface |

---

Makefile Target Naming Convention

Follow this naming pattern. Replace {dataset} with the fixture dataset name (e.g., thompson).

## Golden Tests
golden-{dataset}-test          ## Unit-test-like suite; pytest against JSON fixtures; no server needed
golden-{dataset}-wf            ## Full end-to-end workflow + semantic manifest comparison; server required
golden-{dataset}-verify        ## Compare latest manifest to golden without re-running workflow
golden-{dataset}-update        ## Rewrite semantic golden from latest manifest output
golden-{dataset}-phase-verify  ## Compare all phase-level goldens ({phase}_{schema}.json files)
golden-{dataset}-phase-update  ## Update all phase-level goldens from latest manifest

Separate into two modes in the help output:

[Golden — unit-like]
  golden-{dataset}-test         Fast; no server; pytest fixtures

[Golden — full workflow]
  golden-{dataset}-wf           Slow; server required; full e2e
  golden-{dataset}-verify       Compare only; no re-run
  golden-{dataset}-update       Rewrite golden (inspect diff first!)

---

Snapshot Storage Layout

server/tests/fixtures/
  {dataset}_semantic_wf_verification_golden.json   # single compact semantic golden
  {dataset}_schema_grouping_golden.json            # unit-test-style fixture
  {dataset}_unit_migration_golden.json             # lifecycle fixture with steps
  {dataset}_phase_goldens/
    wf1_grouping_{schema}.json
    wf2_mapping_{schema}.json
    wf3_units_{schema}.json
    wf3b_migrations_{schema}.json
    wf4_normalized_{schema}.json
    wf5_qc_{schema}.json
    wf6_exports_{schema}.json
    wf7_insights_{schema}.json

---

Snapshot Script Pattern

A standalone script (not pytest) manages semantic and phase goldens. Key functions:

def build_snapshot(manifest: dict) -> dict:
    """Extract stable, deterministic subset from full pipeline manifest."""
    # 1. Pull only the fields you care about (row counts, match types, etc.)
    # 2. Exclude volatile fields: timestamps, paths, binary hashes, job IDs
    # 3. Sort all nested dicts for deterministic output

def compare_to_golden(snapshot: dict, golden_path: Path) -> bool:
    """Unified diff, colorized. Returns True if match."""
    # Uses difflib.unified_diff with red/green color codes

def write_golden(golden_path: Path, snapshot: dict) -> None:
    """Rewrite golden file. Sort keys, trailing newline."""
    # json.dumps(data, indent=2, sort_keys=True) + "\n"

CLI flags:

--verify           compare latest manifest to golden (default)
--update-golden    rewrite semantic golden
--phase-goldens    verify per-phase files
--update-phase-goldens  rewrite per-phase files
--schema           target a specific schema only
--all-schemas      run across all schemas

---

What NOT to Put in a Golden

These leak into diffs and cause false positives:

• Timestamps (created_at, updated_at, dataset directory names with dates)
• File system paths (absolute or dataset-relative)
• Binary content or checksums
• Cache fingerprints, job IDs, run UUIDs
• Full JSON payloads when a count or summary suffices

---

Adding Golden Tests to a New Repo

1. Answer the five setup questions above
2. Seed fixture data and document the seed command (make seed-{dataset})
3. Write the snapshot extractor script (scripts/golden_{dataset}.py)
4. Write pytest fixtures for unit-test-like checks (server/tests/test_*_golden.py)
5. Add Makefile targets following the naming convention above
6. Run once, capture baseline, commit golden files alongside code
7. Add to CI: fail build on golden diff, never auto-update in CI

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Olshansk
• Source: Olshansk/agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.