Experiment Design

Experiment Design

If you change your list, your copy, and your offer at the same time, you learn nothing. This skill forces you to isolate one variable per experiment so you actually learn what's working.

Why this exists

Most cold email operators run "throw-everything" experiments. Campaign 1 gets a new list, new copy, and a new offer. It works better. They declare victory. But they can't tell you WHY — was it the list? The copy? The offer?

Then campaign 2 changes all three again. Regression. Nobody knows why.

This skill is the antidote: plan each experiment around ONE variable, keep everything else constant, and confidence-weight the results.

The three experiment types

A. List-only experiment

• What varies: the list (targeting criteria)
• What stays fixed: copy, offer, sending infrastructure, sequence timing
• What you learn: whether this segment is a better fit than the baseline
• Confidence on learnings: HIGH on targeting, LOW on copy (because copy wasn't tested)

B. Copy-only experiment

• What varies: the copy (subject, body, sequence, or A/B variant)
• What stays fixed: list, offer, infrastructure
• What you learn: whether this copy resonates with this audience
• Confidence on learnings: HIGH on copy, LOW on targeting

C. Combined experiment (use sparingly)

• What varies: list AND copy (and sometimes offer)
• What stays fixed: only infrastructure
• When to use: launching a whole new campaign for a new ICP. You can't isolate because everything is new.
• Confidence on learnings: MEDIUM on everything. Use as hypothesis-generation, not conclusion.

The Framework

Step 1: Name your hypothesis

Every experiment starts with a one-sentence hypothesis:

> "Targeting Heads of Marketing at 50-200 person B2B SaaS companies will get a higher positive reply rate than our current VP Sales baseline, because [reason]."

Or:

> "Leading with a question about their recent product launch will get a higher reply rate than our current benefit-focused opener, because [reason]."

If you can't write the hypothesis in one sentence, you don't understand the experiment yet. Go back.

Step 2: Identify the single variable

Write down exactly what changes and what stays the same.

Variable: Target job title
Change: "VP Sales" → "Head of Marketing"

Constants:
- Industry filter: unchanged
- Headcount: unchanged
- Geography: unchanged
- Copy: unchanged (same 4-step sequence)
- Offer: unchanged (same lead magnet)
- Sending infrastructure: unchanged (same 20 domains, 40 inboxes)
- Send schedule: unchanged

If ANY constant is actually changing, stop. Either lock it down, or reclassify as a combined experiment.

Baseline sanity check — the 1% rule

Before running any experiment, confirm your baseline is healthy: overall reply rate ≥1% after 200+ sends. If your baseline is below 1% after 200 sends, the problem isn't your experiment — your infrastructure or copy is already broken. Run /email-deliverability-audit first.

Running an experiment on a broken baseline is wasted effort: you'll learn that "both arms are bad," not "which arm wins."

Step 3: Calculate minimum sample size

The smaller your effect, the more leads you need. Use these rough rules for cold email:

| Current baseline | Expected lift | Minimum sends per arm | |---|---|---| | 1% positive reply rate | 2x (1% → 2%) | ~500 | | 1% positive reply rate | 1.5x (1% → 1.5%) | ~2,000 | | 1% positive reply rate | 1.2x (1% → 1.2%) | ~10,000 | | 2% positive reply rate | 2x (2% → 4%) | ~250 | | 2% positive reply rate | 1.5x (2% → 3%) | ~1,000 |

Rule of thumb: if your test has fewer than 500 sends per arm, you can't tell signal from noise.

For most beginners, 2,000 sends per arm is the right default.

Step 4: Build the success criteria up front

Before launching, write:

Success = positive reply rate > X% (our current baseline is Y%)
Failure = positive reply rate 
Type: List-only | Copy-only | Combined
Variable: 

Result:  at % vs %
Confidence: HIGH | MEDIUM | LOW (based on experiment type + sample size)

Learnings (by confidence):
  HIGH confidence:
    - 
  MEDIUM confidence:
    - 
  LOW confidence:
    - 

HIGH only if: experiment type isolates the variable AND sample size meets the minimum.

Step 8: Decide what to do with the result

• Winner by ≥20% lift, HIGH confidence: adopt as new baseline. Document. Move to next experiment.
• Winner by 10-20% lift, HIGH confidence: run a replication experiment with fresh leads. If it wins again, adopt.
• **Winner by /experiments/YYYY-MM-DD-.yaml

Schema:

```yaml
experiment:
  name: 
  hypothesis: 
  type: list-only | copy-only | combined
  variable: 
  constants: 

success_criteria:
  positive_reply_rate_target: 
  baseline: 
  minimum_sends_per_arm: 
  measurement_date: 

arms:
  control:
    smartlead_campaign_id: 
    description: 
  variant:
    smartlead_campaign_id: 
    description: 

results: 
  control_positive_reply_rate: null
  variant_positive_reply_rate: null
  winner: null
  confidence: null
  decision: null

References

• references/sample-size-calculator.md — longer math for power calculations
• references/example-experiments/ — 3 worked examples (list, copy, combined)
• /positive-reply-scoring skill — how to actually measure the outcome

---

What to do next

Launch the planned experiment via /smartlead-campaign-upload-public (manual) or /auto-research-public (automated). Use the variants.yaml from /campaign-copywriting.

Then wait 21 days before evaluating — reply rate needs that long to stabilize. After 21 days, /positive-reply-scoring on each arm.

Or wait: if you don't have 2,000+ leads per experiment arm, you can't detect normal-sized effects. Build a bigger list (/prospeo-full-export, /disco-like) first.

Related skills

• /campaign-copywriting — produces the copy variants this experiment tests
• /smartlead-campaign-upload-public — launches each arm
• /positive-reply-scoring — measures the outcome after 21 days

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: growthenginenowoslawski
• Source: growthenginenowoslawski/coldoutboundskills
• License: MIT

Install and usage instructions live in the source repository linked above.