Ab Test Design

A/B Test Design

The Law

A TEST WITHOUT A STOPPING RULE RUNS UNTIL THE RESULT LOOKS GOOD.
"We'll stop when we see something significant" is peeking — it inflates false positive rates until the next meaningless spike ships to production.
Hypothesis + metric + sample size + duration + stopping rule + decision rule IS an experiment.

When to Use

Trigger before:

• Running any A/B or multivariate test that will produce a ship/no-ship decision
• Adding feature flag logic that will split users into groups
• Launching any "let's try this and see what happens" experiment

When NOT to Use

• Observational studies with no control group (not an A/B test — use a different analysis method)
• Rollouts to -design-.md` before any traffic is split.

Rationalization Red Flags

These thoughts mean the test is not designed — stop:

• "We'll stop when we see something significant" — this is peeking; each check at α=0.05 adds ~20% false positive inflation; at 5 checks, your real alpha is ~0.23, not 0.05
• "The effect will be obvious" — "obvious" effects are usually noise that survived confirmation bias; calculate the MDE required to be obvious and check whether your traffic supports detecting it
• "We don't need a hypothesis, we're just testing" — tests without hypotheses cannot produce learning; you will see a result and confabulate a reason for it
• "We'll decide what to do based on the results" — the decision rule must be defined before the test; post-hoc decision criteria are chosen to match the outcome
• "A week is enough" — a week may be enough duration; whether it is enough sample depends on the calculation, not the calendar

Completion Statement Format

When ab-test-design is satisfied, state it like this:

Experiment designed.
File: product/experiments/-design-.md ✓

Hypothesis: If [change], then [metric] will [direction] by [MDE] because [mechanism] ✓
Primary metric: [name] — baseline: X% (from [data source], last [N] days) ✓
Guardrails: 

Sample size:  (power: 80%/90%, α: 0.05, MDE: Y%) ✓
Duration:  =  ÷  ✓
  [Minimum 7 days / within 6-week maximum ✓]

Stopping rule: fixed at ; early stop if p < 0.001 at ≥ 75% sample ✓
Decision rule:
  Ship condition: ≥ MDE at p < 0.05, no guardrail breached ✓
  No-ship condition: flat or negative at full duration ✓
  Guardrail breach: investigate before decision ✓
  Insufficient power: extend or redesign ✓

Sample size and duration must be calculated, not estimated. A decision rule must exist for all four outcomes.

Why This Matters

Underpowered tests ship features based on noise. Peeked tests ship features based on chance peaks. Tests without decision rules ship features based on whoever argues most convincingly after the results come in. A properly designed experiment is the only way to distinguish a real 3% improvement from a random fluctuation at that magnitude — and 3% improvements compound into meaningful product quality over time.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: RBraga01
• Source: RBraga01/builder-product
• License: MIT
• Homepage: https://rbraga01.github.io/builder-product/

Install and usage instructions live in the source repository linked above.