# Experiment Design > Help the user design a well-formed GrowthBook experiment before it's launched. Use when the user asks to "design an A/B test", "set up an experiment", "test X vs Y", "configure an experiment", or "what should we measure". Produces a complete spec — hypothesis, variations, primary metric, guardrails, sample size — ready to hand off. Does not create the experiment in GrowthBook. For launching, use… - **Type:** Skill - **Install:** `agentstack add skill-growthbook-skills-experiment-design` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [growthbook](https://agentstack.voostack.com/s/growthbook) - **Installs:** 0 - **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [growthbook](https://github.com/growthbook) - **Source:** https://github.com/growthbook/skills/tree/main/skills/experiment-design ## Install ```sh agentstack add skill-growthbook-skills-experiment-design ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # experiment-design Help the user produce an experiment spec that's actually launchable. Walk them through hypothesis, variations, metrics, and sample-size sanity. This skill does **not** write to GrowthBook — it ends with a ready-to-launch spec that `experiment-launch` consumes. All API calls go through the bundled helper: `${CLAUDE_PLUGIN_ROOT}/scripts/gb-call`. It needs `GB_API_KEY` — set in your shell, or written to `~/.config/growthbook/.env` by `/growthbook:setup`. If unset or invalid, gb-call's error message points back at `/growthbook:setup`. ## Workflow 1. **Frame the hypothesis.** Falsifiable, if/then/because format: > If we change X, then Y will improve, because Z. Push for specificity if the hypothesis is vague. "We think users will like it" doesn't say which metric — engagement could mean five different things. "If we move the CTA above the fold, then click-through will increase, because users decide whether to engage before they scroll" gives the prediction something concrete to land against. 2. **Define variations.** Default to two: control (current state) and treatment (the change). Three or more variations are valid but cost statistical power; ask the user whether they really need a third. Number variations from 0 (control) to N. 3. **Pick goal metrics (ideally one, two max).** List available templates and available metrics: ```bash gb-call GET /api/v1/api/v1/experiment-templates gb-call GET /api/v1/metrics gb-call GET /api/v1/fact-metrics ``` Help the user choose a template if one exists and applies, otherwise help the user choose metrics based on what the hypothesis predicts will move. Note the metric type (proportion, mean, ratio, quantile) — affects sample-size math. Push back at three or more goal metrics and demote the rest to secondary or guardrail: the GrowthBook decision framework treats goal metrics as plural by design, but each additional goal dilutes power and complicates the ship/kill decision. 4. **Pick guardrails (1–3).** Metrics that _shouldn't_ regress. Common examples: signup rate, page error rate, latency. Push back if the user skips guardrails; often it's good for experiments to have at least 1 or 2 guardrails. Guardrails are excluded from multiple-comparison correction by design, so don't over-stack them. 5. **Estimate sample size.** Need three inputs from the user: - Baseline rate (or mean) of the primary metric - Minimum detectable effect (MDE) the user cares about, in relative terms ("a 2% lift in conversion"). - Daily traffic on the affected surface. Use a back-of-envelope estimate to gut-check, then point the user at GrowthBook's in-app Power Calculator for the real number. A common rule-of-thumb GrowthBook documents is **≥ 200 conversions per variation** for proportion metrics; the formula `n ≈ 16 × p × (1 - p) / (p × MDE)^2` per variation lands in roughly the same place for 80% power. Don't quote three significant figures from either — they're estimates. Round up and surface the inputs. Compute the expected experiment duration: `2 × n / daily_traffic`. Flag the duration on both ends: - **> 4 weeks** — likely underpowered for practical use; consider a larger MDE, a more sensitive metric, or higher-traffic surface. - ** **Hypothesis:** If , then , because . **Variations:** - 0: Control — - 1: Treatment — **Primary metric:** () — baseline **Guardrails:** , **MDE:** **Estimated sample size:** per variation **Estimated duration:** days at visitors/day on the affected surface **Project:** **Tracking key suggestion:** ``` Ask the user to confirm before handing off to `experiment-launch`. ## Guardrails - **Ideally one goal metric, two max.** GrowthBook's decision framework treats goal metrics as plural by design and the power calculator supports up to five, but each additional goal dilutes power and complicates the ship/kill decision. Push back at three or more; demote the rest to secondary. - **At least one guardrail.** Push back if the user skips guardrails. Multiple-comparison correction does **not** apply to guardrails (intentionally — a guardrail signal is meant to block shipping), so don't over-stack them either; 1–3 is the sweet spot. - **Falsifiable hypothesis, if/then/because format.** Push the user to make the prediction concrete enough to interpret results against. "Users will engage more" doesn't say which metric. - **Sample-size math is approximate.** Use it as a gut check; route the user to the in-app Power Calculator for the real number. Round up and surface the inputs you used so the user can check. - **Watch out for activation-metric bias.** Activation metrics downstream of variation differences silently bias results without tripping SRM. If the user picks one, name the risk explicitly. - **Suggest an A/A test for first-time experimenters.** If the org has no stopped experiments (check via `flag-discovery` or `experiment-brainstorm`), GrowthBook recommends an A/A test first to validate the implementation before running a real one. - **Day-of-week effects matter.** Push back on experiment durations under one full week. Weekend traffic and behavior differ from weekday traffic. - **Don't launch from this skill.** Final spec → user confirms → hand off to `experiment-launch`. Resist scope creep. - **Tracking-key naming is permanent.** Suggest kebab-case derived from the experiment name. The launch step will use this as `trackingKey`; it lands in event data and can't be cleanly changed later. ## Endpoints used - `GET /api/v1/metrics` and `/api/v1/fact-metrics` — list candidate primary + guardrail metrics - `GET /api/v1/metrics/` — fetch baseline value for sample-size estimation - `GET /api/v1/projects` — resolve project name to ID - `GET /api/v1/data-sources` — list available datasources (used by launch) ## Handoffs - `experiment-launch` — consumes the spec and creates the draft experiment in GrowthBook. - Manual metric creation — if the primary metric doesn't exist yet, the user needs to create it in the GrowthBook UI at `/metrics` (or `/fact-tables` for fact metrics) before launching. No skill for that yet; derive `` from `GB_API_URL` by swapping `api.` → `app.`. - `experiment-brainstorm` — if the user came in without a specific hypothesis, route back here to ground a new idea in past results. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [growthbook](https://github.com/growthbook) - **Source:** [growthbook/skills](https://github.com/growthbook/skills) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** yes - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-growthbook-skills-experiment-design - Seller: https://agentstack.voostack.com/s/growthbook - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.