# Experiment Framework

> Designs statistically rigorous A/B experiments with sample size calculation, guardrail metrics, and results analysis. Use when planning experiments, validating feature impact, or analyzing test results.

- **Type:** Skill
- **Install:** `agentstack add skill-vibbs-company-os-experiment-framework`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [vibbs](https://agentstack.voostack.com/s/vibbs)
- **Installs:** 0
- **Category:** [Data & Analytics](https://agentstack.voostack.com/c/data-and-analytics)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [vibbs](https://github.com/vibbs)
- **Source:** https://github.com/vibbs/company-os/tree/main/.claude/skills/experiment-framework

## Install

```sh
agentstack add skill-vibbs-company-os-experiment-framework
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Experiment Framework

## Reference
- **ID**: S-QA-08
- **Category**: QA / Experimentation
- **Inputs**: PRD (success metrics), feature flag spec, analytics config, baseline conversion rates
- **Outputs**: experiment spec → artifacts/experiments/, experiment results → artifacts/experiments/
- **Used by**: QA & Release Agent
- **Tool scripts**: ./tools/artifact/validate.sh, ./tools/qa/experiment-report.sh

## Purpose

Bring statistical rigor to product experiments. Most teams "ship and see" -- they launch a feature, watch a dashboard, and declare victory if a metric goes up. This leads to false conclusions, wasted engineering time, and product decisions driven by noise rather than signal. The experiment framework ensures every test has a clear hypothesis, proper sample size, controlled execution, and structured analysis.

Move beyond intuition-driven development to hypothesis-driven development. Every experiment answers a specific question with a quantified answer. When a feature "wins," you know the effect size, the confidence level, and whether it came at the cost of degrading something else. When a feature "loses," you learn something concrete rather than wondering if you just didn't run the test long enough.

## When to Use

- Planning an A/B test for a new feature
- Validating whether a change actually improves metrics
- Feature flag strategy is `full` or feature uses experiment-type flags
- Product Agent defines success metrics that need causal validation
- Activation experiments from activation-onboarding skill need statistical design

## Experiment Framework Procedure

### Step 0.5: Traffic Feasibility Check

Before designing any experiment:
1. If `analytics.provider` is configured, estimate current DAU from available data
2. Calculate minimum feasible MDE (Minimum Detectable Effect) given traffic and 90-day max duration
3. If DAU 2 variants)

**Sample Size Lookup Table** (two-sided test, alpha=0.05, power=0.80):

| Baseline Rate | MDE (relative) | Sample Size per Variant |
|--------------|-----------------|------------------------|
| 1% | 20% | ~130,000 |
| 5% | 10% | ~30,000 |
| 10% | 10% | ~14,000 |
| 20% | 10% | ~6,400 |
| 50% | 5% | ~6,400 |

For custom calculations, use the full formula or reference an online sample size calculator. Always round UP to the nearest hundred.

### Step 4: Estimate Duration

1. Calculate daily eligible traffic (users who enter the experiment)
2. Duration = (sample_size_per_variant x num_variants) / daily_traffic
3. Add buffer for weekday/weekend variation (multiply by 1.2)
4. Minimum duration: 7 days (to capture weekly cycles)
5. Maximum recommended: 90 days (if longer, the MDE is too small or traffic too low -- reconsider)
6. If duration exceeds `experiments.max_concurrent` capacity, flag scheduling conflict

### Step 5: Design Experiment Spec

Produce an experiment specification artifact:

```yaml
---
id: EXP-XXX
type: experiment
status: draft
parent: PRD-XXX
depends_on: [RFC-XXX]
---
```

Include:
- Hypothesis (from Step 2)
- Primary metric + baseline + MDE
- Secondary metrics
- Guardrail metrics with thresholds
- Sample size per variant
- Estimated duration
- Variant descriptions (control = current, treatment = change)
- Feature flag name (links to feature-flags skill output)
- Assignment method (random, hash-based, segment-based)
- Exclusion criteria (new users only, specific segments, etc.)

### Step 6: Define Experiment State Machine

Every experiment follows this lifecycle:

```
draft -> approved -> running -> analyzing -> concluded -> archived
```

| State | Entry Criteria | Exit Criteria | Who Owns |
|-------|---------------|---------------|----------|
| draft | Hypothesis + metrics defined | Spec reviewed by Product + Engineering | QA |
| approved | Spec reviewed, sample size validated | Feature flag configured, tracking verified | QA |
| running | Flag activated, users being assigned | Duration elapsed OR early termination triggered | QA + Engineering |
| analyzing | Experiment stopped, data collected | Analysis complete, recommendation made | QA |
| concluded | Decision made (ship/no-ship/iterate) | Flag cleaned up, code path removed or made permanent | Engineering |
| archived | Learnings documented | -- | Product |

### Step 7: Define Early Termination Rules

- **Harm detected**: If guardrail metric degrades >20% relative to control, stop immediately
- **Clear winner**: If primary metric shows >99% probability of being better (sequential testing), can stop early
- **No-effect**: If 80% of planned duration has passed and effect size is 5%, tracking failures >10%, stop and investigate

### Step 8: Define Analysis Plan

1. When to analyze: only after planned duration OR early termination trigger
2. NEVER peek at results before planned duration (peeking inflates false positive rate)
3. Statistical test:
   - For proportions: two-proportion z-test or chi-squared test
   - For continuous metrics: Welch's t-test or Mann-Whitney U
   - For duration/time metrics: log-transform then t-test
4. Report: effect size, confidence interval, p-value, practical significance
5. Segmentation analysis: check if effect varies by user segment (new vs returning, plan tier, geography)
6. Guardrail check: verify no guardrail metric degraded significantly

### Step 9: Anti-Patterns

Document these to prevent common mistakes:

- **Peeking**: Checking results daily and stopping when "significant" -- inflates false positives to >20%
- **Multiple testing**: Testing 5 metrics without correction -- one will be "significant" by chance
- **Survivorship bias**: Only analyzing users who completed the flow, ignoring drop-offs
- **Underpowered**: Running experiments too short because "we need to move fast"
- **History effects**: External events (holidays, marketing campaigns) contaminating results
- **Novelty effects**: Short-term engagement bump from any change, not real improvement

### Step 10: Validate and Save

1. Save experiment spec to `artifacts/experiments/`
2. Run `./tools/artifact/validate.sh` to verify frontmatter
3. Link to parent PRD and RFC via `./tools/artifact/link.sh`

## Quality Checklist

- [ ] Hypothesis is clear and falsifiable
- [ ] Primary metric is defined with baseline rate
- [ ] MDE is specified and justified
- [ ] Sample size is calculated (not guessed)
- [ ] Duration is estimated with weekly cycle buffer
- [ ] Guardrail metrics are defined with degradation thresholds
- [ ] Early termination rules are documented
- [ ] Analysis plan specifies exact statistical test
- [ ] Anti-patterns section is included
- [ ] Experiment spec has valid artifact frontmatter

## Cross-References

- **feature-flags** skill: Experiment-type flags from the feature-flags skill should have a corresponding experiment spec from this skill. The flag controls assignment; this skill designs the experiment.
- **activation-onboarding** skill: Activation experiments designed in the activation skill should use this framework for statistical rigor.
- **instrumentation** skill: Event tracking must be in place before an experiment can run. Verify tracking with the instrumentation skill's event taxonomy.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [vibbs](https://github.com/vibbs)
- **Source:** [vibbs/company-os](https://github.com/vibbs/company-os)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-vibbs-company-os-experiment-framework
- Seller: https://agentstack.voostack.com/s/vibbs
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.