# Ab Test Analysis

> Analyze A/B test results with statistical significance, sample size validation, confidence intervals, and ship/extend/stop recommendations. Use when evaluating experiment results or deciding whether to ship a variant.

- **Type:** Skill
- **Install:** `agentstack add skill-ericwang915-data-scientist-skills-ab-test-analysis`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [ericwang915](https://agentstack.voostack.com/s/ericwang915)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [ericwang915](https://github.com/ericwang915)
- **Source:** https://github.com/ericwang915/data-scientist-skills/tree/main/ds-business-analytics/skills/ab-test-analysis

## Install

```sh
agentstack add skill-ericwang915-data-scientist-skills-ab-test-analysis
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# A/B Test Analysis

## Purpose
Evaluate A/B test results with statistical rigor and translate findings into clear product decisions.

## How It Works

### Step 1: Validate Test Setup
- Sample size adequacy (power analysis)
- Duration (≥1-2 full business cycles)
- Randomization check (sample ratio mismatch)
- Novelty/primacy effect assessment

### Step 2: Calculate Results
- Conversion rates for control and variant
- Relative lift: (variant - control) / control × 100
- p-value (two-tailed z-test or chi-squared)
- 95% confidence interval for the difference
- Statistical and practical significance

### Step 3: Interpret

| Outcome | Recommendation |
|---------|---------------|
| Significant positive lift, no guardrail issues | **Ship it** |
| Significant positive, guardrail concerns | **Investigate** |
| Not significant, positive trend | **Extend the test** |
| Not significant, flat | **Stop** — no effect |
| Significant negative | **Don't ship** — revert |

### Step 4: Report
```
## A/B Test: [Name]
**Hypothesis**: [Expected outcome]
**Duration**: [X days] | **Sample**: [N control / M variant]

| Metric | Control | Variant | Lift | p-value | Significant? |
|--------|---------|---------|------|---------|-------------|
| [Primary] | X% | Y% | +Z% | 0.0X | Yes/No |
| [Guardrail] | ... | ... | ... | ... | ... |

**Recommendation**: [Ship / Extend / Stop]
```

## Usage Examples

```
"Analyze our checkout A/B test: control 3.2% conversion (n=5000), variant 3.8% (n=5000)"
```

## Output Format

- **Results Table**: Metrics with significance indicators
- **Recommendation**: Ship/extend/stop with reasoning
- **Python Code**: Statistical calculations

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [ericwang915](https://github.com/ericwang915)
- **Source:** [ericwang915/data-scientist-skills](https://github.com/ericwang915/data-scientist-skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-ericwang915-data-scientist-skills-ab-test-analysis
- Seller: https://agentstack.voostack.com/s/ericwang915
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.