# Experiment Designer

> Design quick validation experiments for product features. Use this skill when the user needs to validate a hypothesis, test a feature idea before building, has low confidence in an initiative, or asks "should I build this" or "how do I validate this". Also trigger when user mentions prototypes, MVPs, testing, validation, or wants to reduce risk before investing in development. Guide them through…

- **Type:** Skill
- **Install:** `agentstack add skill-grafuja-product-manager-skills-experiment-designer`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [Grafuja](https://agentstack.voostack.com/s/grafuja)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [Grafuja](https://github.com/Grafuja)
- **Source:** https://github.com/Grafuja/Product-Manager-Skills/tree/main/skills/experiment-designer

## Install

```sh
agentstack add skill-grafuja-product-manager-skills-experiment-designer
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Experiment Designer

Help product managers design lightweight validation experiments to test hypotheses before committing to full development.

## Core Philosophy

**Don't build to learn. Experiment to learn, then build.**

Experiments answer: "Should we build this?" before investing weeks/months.

---

## When to Use This Skill

Trigger experiment design when:
- Confidence 4 weeks effort)
- Stakeholder request without user validation
- New product/market with uncertainty

**Don't experiment when:**
- Obvious must-have (e.g., fix critical bug)
- Feature already validated
- Regulatory requirement
- Confidence > 80% with solid data

---

## The Experiment Design Workflow

### Step 1: Capture the Hypothesis

Ask: **"What do you believe to be true that you want to test?"**

Help them form a clear hypothesis in IF-THEN format:

**Template:**
```
IF [we do X action],
THEN [Y measurable result will happen]
BECAUSE [assumption about users]
```

**Examples:**

✅ **Good hypothesis:**
"IF we show dashboard prototype to 10 CFOs,
THEN 8/10 will say they'd use it weekly
BECAUSE CFOs waste 30min/week on manual reports"

❌ **Bad hypothesis (too vague):**
"IF we build dashboard,
THEN users will like it"

**Push for specificity:**
- What EXACTLY are we testing?
- What EXACT result would confirm it?
- What's the underlying assumption?

**Store:** `hypothesis` (IF-THEN-BECAUSE statement)

---

### Step 2: Assess Current Confidence

Ask: **"How confident are you (0-100%) that this will work?"**

This determines which experiment method to use:

| Confidence | What it means | Experiment needed |
| ----- | ----- | ----- |
| **0-30%** | Pure speculation | Heavy validation (interviews, prototype) |
| **30-50%** | Weak signal | Moderate validation (wizard of oz, concierge) |
| **50-70%** | Some evidence | Light validation (feature flags, beta) |
| **70-100%** | Strong evidence | Minimal/no experiment, just build |

**Store:** `confidence_level` (percentage)

**If confidence > 70%:**
Suggest: "Confidence is already high. You might not need an experiment. What makes you hesitant?"

---

### Step 3: Determine Constraints

Ask: **"What are your constraints?"**

**Key questions:**
1. **Time:** "How quickly do you need an answer?"
   - Urgent (3x/week for 2 weeks, build real API"

**Output:** Usage frequency + completion rate

---

### Method 3: Concierge
**Best for:** Testing willingness to pay  
**Confidence target:** 50% → 80%  
**Time:** 2-4 weeks  
**Cost:** $5K-$15K (time doing manual service)

**When to use:**
- Want to validate willingness to pay
- Testing pricing
- B2B features with few users

**How it works:**
1. Offer manual service for price
2. User KNOWS it's manual
3. Track: Do they pay? For how long?

**Example:**
```
Feature: Automated reporting system
Concierge: "We'll generate your reports manually for $200/month"
Track: How many sign up? How many renew month 2?
```

**Success criteria example:**
"If 8/10 users pay for 2+ months, build automated version"

**Output:** Conversion rate + retention rate + pricing validation

---

### Method 4: Landing Page + Ads
**Best for:** Demand validation at scale  
**Confidence target:** 20% → 60%  
**Time:** 1 week  
**Cost:** $1K-$3K (ads + page)

**When to use:**
- Testing if problem exists
- Validating market size
- No direct user access

**How it works:**
1. Create landing page describing solution
2. Run ads to target audience
3. Track: Email signups, "early access" clicks

**Example:**
```
Feature: AI-powered analytics
Landing page: "Get AI insights for your data. Join waitlist."
Track: Click-through rate, email signups
```

**Success criteria example:**
"If >5% of traffic signs up for early access, proceed"

**Output:** Demand signal (click rate, signups)

---

### Method 5: Feature Flag / Beta
**Best for:** Testing real usage with minimal risk  
**Confidence target:** 60% → 85%  
**Time:** 2-4 weeks (after build)  
**Cost:** Full build cost

**When to use:**
- Already decided to build
- Want to validate with small % of users first
- Reduce risk of bad rollout

**How it works:**
1. Build feature
2. Roll out to 10% users
3. Track: Adoption, retention, NPS

**Example:**
```
Feature: New dashboard
Beta: Launch to 10% of users
Track: % who use >1x/week, NPS impact
```

**Success criteria example:**
"If adoption >50% and NPS doesn't drop, roll to 100%"

**Output:** Real usage data + retention

---

## Step 5: Define Success Criteria

**Critical:** Define BEFORE running experiment

Ask: **"What result would make you build this?"**

### Good success criteria:

**Quantitative + Threshold:**
```
✓ "If 7/10 users say 'I'd use this weekly'"
✓ "If 5/10 users use it >3x/week for 2 weeks"
✓ "If >5% landing page visitors sign up"
✓ "If 8/10 pay for 2+ months"
```

**Bad success criteria:**
```
✗ "If users like it"  (not measurable)
✗ "If it seems promising"  (no threshold)
✗ "We'll see how it goes"  (no criteria)
```

### The Two-Threshold System

Define TWO thresholds:

1. **Green light (build):** Clear success
2. **Red light (stop):** Clear failure

**Example:**
```
Green: ≥7/10 users say "yes, I'd use weekly" → Build
Yellow: 4-6/10 → Iterate design, test again
Red: ≤3/10 → Abandon or pivot
```

**Store:** `success_threshold` (green), `failure_threshold` (red)

---

## Step 6: Design the Experiment Plan

Output a complete 1-page experiment plan:

```
═══════════════════════════════════════════
EXPERIMENT PLAN
═══════════════════════════════════════════

HYPOTHESIS:
IF [action]
THEN [result]
BECAUSE [assumption]

CURRENT CONFIDENCE: [%]%
TARGET CONFIDENCE: [%]%

METHOD: [Prototype / Wizard of Oz / Concierge / etc]

TIMELINE:
• Week 1: [Tasks]
• Week 2: [Tasks]
• Week 3: [Results + decision]

WHAT WE'LL DO:
1. [Step 1]
2. [Step 2]
3. [Step 3]

WHAT WE'LL MEASURE:
• Metric 1: [description]
• Metric 2: [description]

SUCCESS CRITERIA:
✅ Green (build): [threshold]
⚠️ Yellow (iterate): [threshold]
❌ Red (stop): [threshold]

COST:
• Time: [X days/weeks]
• Money: $[amount]
• People: [who's involved]

NEXT STEPS:
□ [Action 1]
□ [Action 2]
□ Decision by: [date]
═══════════════════════════════════════════
```

---

## Real Examples

### Example 1: Dashboard Feature

**Input from user:**
"CEO wants dashboard of metrics. Not sure if users will use it."

**Hypothesis built:**
```
IF we show Figma prototype to 10 active users,
THEN 8/10 will say "I'd check this daily/weekly"
BECAUSE users currently waste time searching for data
```

**Confidence:** 40%

**Constraints:**
- Time: Need answer in 2 weeks
- Budget: $1K (designer)
- Access: Have user contact list

**Recommended method:** Prototype + Interviews

**Experiment plan:**
```
WEEK 1:
• Day 1-3: Create Figma prototype (3 variants)
• Day 4-5: Schedule 10 user interviews

WEEK 2:
• Day 1-3: Run interviews
• Day 4: Analyze results
• Day 5: Decision

MEASURE:
• % who say "I'd use daily/weekly"
• Which variant resonates most
• What data they want to see

SUCCESS CRITERIA:
✅ ≥7/10 say "yes" + agreement on which data → Build
⚠️ 4-6/10 → Iterate design
❌ ≤3/10 → Dashboard not the solution

COST: $1K (designer) + 15 hours PM time
```

---

### Example 2: API Integration

**Input from user:**
"3 enterprise clients asked for ERP integration. Should we build API?"

**Hypothesis:**
```
IF we offer manual CSV integration to 3 clients,
THEN all 3 will use it >5x/week
BECAUSE they claim it's blocking their workflow
```

**Confidence:** 60%

**Recommended method:** Wizard of Oz

**Experiment plan:**
```
WEEK 1-2:
• Set up CSV email integration
• Tell clients: "Send CSV, we'll process in 5x/week → Build API (real need)
⚠️ 1-2 use regularly → Partial need, validate more
❌ 3x/week
```

### Template: Concierge
```
MONTH 1:
□ Offer service at $[price]
□ 20 users invited
□ Manual fulfillment

TRACK:
• Conversion rate
• Month 2 retention
• Feature requests

SUCCESS: ≥60% convert + ≥80% retain month 2
```

---

## Final Checklist

Before running ANY experiment, verify:

| ✓ | Item |
| ----- | ----- |
| ☐ Hypothesis is clear (IF-THEN-BECAUSE) |
| ☐ Success criteria defined (green/yellow/red) |
| ☐ Timeline < 25% of build time |
| ☐ Costs < 10% of build cost |
| ☐ You have user access (or plan to get it) |
| ☐ Results will influence decision (not just validating pre-made choice) |

---

**Remember:** The goal of experiments is to LEARN FAST and CHEAPLY, not to prove you're right. Be willing to kill ideas that fail validation.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [Grafuja](https://github.com/Grafuja)
- **Source:** [Grafuja/Product-Manager-Skills](https://github.com/Grafuja/Product-Manager-Skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-grafuja-product-manager-skills-experiment-designer
- Seller: https://agentstack.voostack.com/s/grafuja
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.