# Experiment Designer

> Help PMs design rigorous experiments (A/B tests, feature rollouts). Use before any A/B test, feature experiment, pricing test, or phased rollout.

- **Type:** Skill
- **Install:** `agentstack add skill-hardiktiwari-pm-operating-os-experiment-designer`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [hardiktiwari](https://agentstack.voostack.com/s/hardiktiwari)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [hardiktiwari](https://github.com/hardiktiwari)
- **Source:** https://github.com/hardiktiwari/PM-operating-OS/tree/main/skills/experiment-designer

## Install

```sh
agentstack add skill-hardiktiwari-pm-operating-os-experiment-designer
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Experiment Designer

Help PMs design rigorous experiments with clear hypotheses, variants, metrics, and decision criteria. Ensures experiments are statistically sound and actionable.

## When to Use

- Before any A/B test
- Feature experiment design
- Pricing or packaging test
- Phased rollout planning
- When asked "how do we test this?"
- Before setting up an experiment in an experimentation platform

## Process / Template

### 1. Capture the Hypothesis

Use format: **"We believe [action] will [outcome] for [segment] because [rationale]"**

- **Action** — what you're changing (feature, copy, flow)
- **Outcome** — expected metric movement
- **Segment** — who you're testing with
- **Rationale** — why you believe this (research, prior data)

### 2. Define Control vs. Treatment

- **Control** — baseline experience (no change)
- **Treatment** — the change being tested
- Be explicit about what differs between variants

### 3. Define Success Metrics

- **Primary metric** — main success signal (one metric)
- **Secondary metrics** — supporting signals
- **Guardrail metrics** — must not regress (e.g., revenue, retention)

### 4. Determine Experiment Type

- **A/B test** — two variants, single change
- **Multivariate** — multiple factors
- **Phased rollout** — gradual % rollout
- **Holdout** — long-term impact test

### 5. Calculate Sample Size and Duration

- **Power** — typically 80%
- **MDE (Minimum Detectable Effect)** — smallest effect you care about
- **Traffic** — daily eligible users
- **Duration** — run until sufficient sample size

### 6. Define Decision Criteria

| Result | Action |
|--------|--------|
| Primary metric improves, guardrails hold | Ship |
| Primary metric flat, secondary improves | Iterate (refine and re-test) |
| Primary metric declines | Kill |
| Inconclusive (low power) | Extend or increase traffic |

### 7. Document Risks

- What could invalidate results? (seasonality, external events)
- Confounding factors?
- Technical implementation risks?

## Output

An **Experiment Design Doc** with:

| Section | Content |
|---------|---------|
| Hypothesis | Full hypothesis statement |
| Variants | Control and treatment definitions |
| Primary Metric | Main success metric |
| Secondary Metrics | Supporting metrics |
| Guardrail Metrics | Metrics that must not regress |
| Sample Size | Required sample per variant |
| Duration | Estimated run length |
| Decision Criteria | Ship / iterate / kill thresholds |
| Risks | What could invalidate results |

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [hardiktiwari](https://github.com/hardiktiwari)
- **Source:** [hardiktiwari/PM-operating-OS](https://github.com/hardiktiwari/PM-operating-OS)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-hardiktiwari-pm-operating-os-experiment-designer
- Seller: https://agentstack.voostack.com/s/hardiktiwari
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.