# Experiment Analysis

> Analyze experiment data and draw conclusions. Use when an A/B test or product experiment has concluded and you need to interpret the results.

- **Type:** Skill
- **Install:** `agentstack add skill-uthumany-uthy-legacy-os-experiment-analysis`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [uthumany](https://agentstack.voostack.com/s/uthumany)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [uthumany](https://github.com/uthumany)
- **Source:** https://github.com/uthumany/uthy-legacy-os/tree/main/skills/measuring/experiment-analysis
- **Website:** https://uthumany.github.io/uthy-legacy-os/

## Install

```sh
agentstack add skill-uthumany-uthy-legacy-os-experiment-analysis
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# Experiment Analysis

## Overview

Running the experiment is only half the work — analyzing the results correctly is where the value lives. This skill helps you interpret experiment data, avoid common statistical pitfalls, and make confident go/kill decisions.

## When to Use

- An A/B test or experiment has concluded
- You need to decide whether to ship, iterate, or kill
- You want to understand not just the outcome but WHY it happened
- Don't use for: pre-experiment planning (use experiment-design skill), exploratory data analysis (different approach)

## Instructions

### 1. Check Experiment Validity

Before analyzing results, verify:
- **Sample size**: Did you reach the required sample size?
- **Duration**: Did the experiment run long enough? (minimum 1 full business cycle, usually 1-2 weeks)
- **Novelty effect**: Did the effect change over time? (compare first few days vs. last few days)
- **Sample ratio mismatch**: Are the control/treatment group sizes as expected?
- **Data quality**: Any tracking issues, bugs, or outliers?

### 2. Read the Primary Metric

- **Statistical significance**: p-value < 0.05 (or your pre-defined threshold)
- **Effect size**: How big is the impact? (absolute and relative)
- **Confidence interval**: What's the range of plausible effects?
- **Practical significance**: Even if statistically significant, is this effect worth building?

### 3. Analyze Guardrail Metrics

What metrics should NOT have degraded?
- Check ALL guardrail metrics for statistically significant negative changes
- A negative guardrail doesn't automatically kill the experiment, but it requires discussion
- Trade-off decision: Is the primary lift worth the guardrail degradation?

### 4. Segment Analysis

- Break down results by key segments (new vs. existing users, platform, plan type)
- A negative overall result might hide a strong positive in a specific segment
- Be cautious of over-segmenting (too many slices = false positives)

### 5. Qualitative Signals

- What did user feedback say during the experiment?
- Any support tickets related to the change?
- Any unexpected user behavior observed?

### 6. Make the Decision

| Signal | Decision |
|--------|----------|
| ✅ Significant positive, no guardrail damage | Ship it |
| ✅ Significant positive, minor guardrail impact | Ship with monitoring |
| 🔄 Inconclusive (not significant) | Need more data or iterate |
| ❌ Significant negative | Kill or iterate |
| ❌ No effect, large sample | Don't ship |

## Sample Output

**Experiment**: New onboarding flow vs. current flow
**Primary metric**: 7-day activation rate
**Result**: +12% activation (p=0.003, CI: 5%-19%)
**Guardrails**: Support tickets +3% (not significant, p=0.21)
**Segments**: Stronger effect on new users (+18%) than existing feature explorers (+4%)
**Decision**: ✅ SHIP. Roll out to 100% of new users. Monitor support volume for 2 weeks.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [uthumany](https://github.com/uthumany)
- **Source:** [uthumany/uthy-legacy-os](https://github.com/uthumany/uthy-legacy-os)
- **License:** MIT
- **Homepage:** https://uthumany.github.io/uthy-legacy-os/

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-uthumany-uthy-legacy-os-experiment-analysis
- Seller: https://agentstack.voostack.com/s/uthumany
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.