Ml Experiment Design

ML Experiment Design

Overview

An ML idea is not runnable until it is a CARD. This skill turns a rough idea into a compact, one-screen experiment card — rung R0, the precondition for planning or launching anything. Fill it WITH your human partner; do not write training code until the card is settled.

Core principle: One change under test, one locked metric, one decision rule. If you cannot state in advance what result would change your mind, you are not ready to run.

Upstream: the idea comes from superpowers-ml:brainstorming. Downstream: the card drives superpowers-ml:ml-feedback-ladder (how to verify it cheaply) and superpowers-ml:writing-plans (the tasks).

The Experiment Card

Settle every field. Keep each to a line or two — a card, not a document.

• Research question — the one question this run answers.
• Hypothesis — specific and falsifiable: what changes, and the direction you expect.
• Baseline — the exact, runnable comparison (config / commit, not "the usual setup").
• Variant(s) — the SINGLE change under test. Anything else is a separate experiment.
• Primary metric — one metric, locked now, that decides the outcome.
• Guardrail metrics — what must NOT regress while the primary metric moves (cost, latency, memory, a quality/safety metric).
• Dataset / split — train / val / test, and the leakage risk you checked for.
• Seed policy — how many seeds, fixed or swept; how you tell seed noise from a real effect.
• Budget — the compute / wall-clock ceiling you will spend before stopping to decide.
• Cheapest useful rung — the smallest rung that tells you something real, to start at (see superpowers-ml:ml-feedback-ladder).
• Success criterion — the threshold on the primary metric that CONFIRMS the hypothesis, set before running.
• Exploratory-only — what this run may suggest but can never confirm (anything the locked primary metric does not measure).

One Change At A Time

A variant is ONE change, or you cannot attribute the result — two changes is two experiments. If your human partner wants to vary many things, that is a study: write a card per change, then design a ladder for each.

Template

Question:         ...
Hypothesis:       changing X will   because ...
Baseline:         
Variant:          
Primary metric:        Success: 
Guardrails:       
Data / split:          Leakage check: 
Seeds:            
Budget:           
Start rung:       
Exploratory-only: 

Red Flags - STOP

• No locked primary metric, or the metric is chosen after seeing results.
• Success criterion not written down before the run.
• More than one change under test in a single variant.
• "We'll know it when we see it" — no decision rule.

A card you can fill in five minutes saves a burned cluster job. Write it first.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: pengzhangzhi
• Source: pengzhangzhi/superpowers-ml
• License: MIT

Install and usage instructions live in the source repository linked above.