Experiment Designer

Experiment Designer

Help product managers design lightweight validation experiments to test hypotheses before committing to full development.

Core Philosophy

Don't build to learn. Experiment to learn, then build.

Experiments answer: "Should we build this?" before investing weeks/months.

---

When to Use This Skill

Trigger experiment design when:

• Confidence 4 weeks effort)
• Stakeholder request without user validation
• New product/market with uncertainty

Don't experiment when:

• Obvious must-have (e.g., fix critical bug)
• Feature already validated
• Regulatory requirement
• Confidence > 80% with solid data

---

The Experiment Design Workflow

Step 1: Capture the Hypothesis

Ask: "What do you believe to be true that you want to test?"

Help them form a clear hypothesis in IF-THEN format:

Template:

IF [we do X action],
THEN [Y measurable result will happen]
BECAUSE [assumption about users]

Examples:

✅ Good hypothesis: "IF we show dashboard prototype to 10 CFOs, THEN 8/10 will say they'd use it weekly BECAUSE CFOs waste 30min/week on manual reports"

❌ Bad hypothesis (too vague): "IF we build dashboard, THEN users will like it"

Push for specificity:

• What EXACTLY are we testing?
• What EXACT result would confirm it?
• What's the underlying assumption?

Store: hypothesis (IF-THEN-BECAUSE statement)

---

Step 2: Assess Current Confidence

Ask: "How confident are you (0-100%) that this will work?"

This determines which experiment method to use:

| Confidence | What it means | Experiment needed | | ----- | ----- | ----- | | 0-30% | Pure speculation | Heavy validation (interviews, prototype) | | 30-50% | Weak signal | Moderate validation (wizard of oz, concierge) | | 50-70% | Some evidence | Light validation (feature flags, beta) | | 70-100% | Strong evidence | Minimal/no experiment, just build |

Store: confidence_level (percentage)

If confidence > 70%: Suggest: "Confidence is already high. You might not need an experiment. What makes you hesitant?"

---

Step 3: Determine Constraints

Ask: "What are your constraints?"

Key questions:

1. Time: "How quickly do you need an answer?"

• Urgent (3x/week for 2 weeks, build real API"

Output: Usage frequency + completion rate

---

Method 3: Concierge

Best for: Testing willingness to pay Confidence target: 50% → 80% Time: 2-4 weeks Cost: $5K-$15K (time doing manual service)

When to use:

• Want to validate willingness to pay
• Testing pricing
• B2B features with few users

How it works:

1. Offer manual service for price
2. User KNOWS it's manual
3. Track: Do they pay? For how long?

Example:

Feature: Automated reporting system
Concierge: "We'll generate your reports manually for $200/month"
Track: How many sign up? How many renew month 2?

Success criteria example: "If 8/10 users pay for 2+ months, build automated version"

Output: Conversion rate + retention rate + pricing validation

---

Method 4: Landing Page + Ads

Best for: Demand validation at scale Confidence target: 20% → 60% Time: 1 week Cost: $1K-$3K (ads + page)

When to use:

• Testing if problem exists
• Validating market size
• No direct user access

How it works:

1. Create landing page describing solution
2. Run ads to target audience
3. Track: Email signups, "early access" clicks

Example:

Feature: AI-powered analytics
Landing page: "Get AI insights for your data. Join waitlist."
Track: Click-through rate, email signups

Success criteria example: "If >5% of traffic signs up for early access, proceed"

Output: Demand signal (click rate, signups)

---

Method 5: Feature Flag / Beta

Best for: Testing real usage with minimal risk Confidence target: 60% → 85% Time: 2-4 weeks (after build) Cost: Full build cost

When to use:

• Already decided to build
• Want to validate with small % of users first
• Reduce risk of bad rollout

How it works:

1. Build feature
2. Roll out to 10% users
3. Track: Adoption, retention, NPS

Example:

Feature: New dashboard
Beta: Launch to 10% of users
Track: % who use >1x/week, NPS impact

Success criteria example: "If adoption >50% and NPS doesn't drop, roll to 100%"

Output: Real usage data + retention

---

Step 5: Define Success Criteria

Critical: Define BEFORE running experiment

Ask: "What result would make you build this?"

Good success criteria:

Quantitative + Threshold:

✓ "If 7/10 users say 'I'd use this weekly'"
✓ "If 5/10 users use it >3x/week for 2 weeks"
✓ "If >5% landing page visitors sign up"
✓ "If 8/10 pay for 2+ months"

Bad success criteria:

✗ "If users like it"  (not measurable)
✗ "If it seems promising"  (no threshold)
✗ "We'll see how it goes"  (no criteria)

The Two-Threshold System

Define TWO thresholds:

1. Green light (build): Clear success
2. Red light (stop): Clear failure

Example:

Green: ≥7/10 users say "yes, I'd use weekly" → Build
Yellow: 4-6/10 → Iterate design, test again
Red: ≤3/10 → Abandon or pivot

Store: success_threshold (green), failure_threshold (red)

---

Step 6: Design the Experiment Plan

Output a complete 1-page experiment plan:

═══════════════════════════════════════════
EXPERIMENT PLAN
═══════════════════════════════════════════

HYPOTHESIS:
IF [action]
THEN [result]
BECAUSE [assumption]

CURRENT CONFIDENCE: [%]%
TARGET CONFIDENCE: [%]%

METHOD: [Prototype / Wizard of Oz / Concierge / etc]

TIMELINE:
• Week 1: [Tasks]
• Week 2: [Tasks]
• Week 3: [Results + decision]

WHAT WE'LL DO:
1. [Step 1]
2. [Step 2]
3. [Step 3]

WHAT WE'LL MEASURE:
• Metric 1: [description]
• Metric 2: [description]

SUCCESS CRITERIA:
✅ Green (build): [threshold]
⚠️ Yellow (iterate): [threshold]
❌ Red (stop): [threshold]

COST:
• Time: [X days/weeks]
• Money: $[amount]
• People: [who's involved]

NEXT STEPS:
□ [Action 1]
□ [Action 2]
□ Decision by: [date]
═══════════════════════════════════════════

---

Real Examples

Example 1: Dashboard Feature

Input from user: "CEO wants dashboard of metrics. Not sure if users will use it."

Hypothesis built:

IF we show Figma prototype to 10 active users,
THEN 8/10 will say "I'd check this daily/weekly"
BECAUSE users currently waste time searching for data

Confidence: 40%

Constraints:

• Time: Need answer in 2 weeks
• Budget: $1K (designer)
• Access: Have user contact list

Recommended method: Prototype + Interviews

Experiment plan:

WEEK 1:
• Day 1-3: Create Figma prototype (3 variants)
• Day 4-5: Schedule 10 user interviews

WEEK 2:
• Day 1-3: Run interviews
• Day 4: Analyze results
• Day 5: Decision

MEASURE:
• % who say "I'd use daily/weekly"
• Which variant resonates most
• What data they want to see

SUCCESS CRITERIA:
✅ ≥7/10 say "yes" + agreement on which data → Build
⚠️ 4-6/10 → Iterate design
❌ ≤3/10 → Dashboard not the solution

COST: $1K (designer) + 15 hours PM time

---

Example 2: API Integration

Input from user: "3 enterprise clients asked for ERP integration. Should we build API?"

Hypothesis:

IF we offer manual CSV integration to 3 clients,
THEN all 3 will use it >5x/week
BECAUSE they claim it's blocking their workflow

Confidence: 60%

Recommended method: Wizard of Oz

Experiment plan:

WEEK 1-2:
• Set up CSV email integration
• Tell clients: "Send CSV, we'll process in 5x/week → Build API (real need)
⚠️ 1-2 use regularly → Partial need, validate more
❌ 3x/week

Template: Concierge

MONTH 1:
□ Offer service at $[price]
□ 20 users invited
□ Manual fulfillment

TRACK:
• Conversion rate
• Month 2 retention
• Feature requests

SUCCESS: ≥60% convert + ≥80% retain month 2

---

Final Checklist

Before running ANY experiment, verify:

| ✓ | Item | | ----- | ----- | | ☐ Hypothesis is clear (IF-THEN-BECAUSE) | | ☐ Success criteria defined (green/yellow/red) | | ☐ Timeline < 25% of build time | | ☐ Costs < 10% of build cost | | ☐ You have user access (or plan to get it) | | ☐ Results will influence decision (not just validating pre-made choice) |

---

Remember: The goal of experiments is to LEARN FAST and CHEAPLY, not to prove you're right. Be willing to kill ideas that fail validation.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Grafuja
• Source: Grafuja/Product-Manager-Skills
• License: MIT

Install and usage instructions live in the source repository linked above.