# Experiment Analyze

> Fetch results for a GrowthBook experiment, refresh the snapshot only when the cached data is over 24 hours old, then interpret. Use when the user asks "what are the results of X", "analyze this experiment", "is X winning", "did the test work", "show me the results", or "dig into the dimensions". Reads only — does not stop or modify the experiment. For stopping after you've seen results, use exper…

- **Type:** Skill
- **Install:** `agentstack add skill-growthbook-skills-experiment-analyze`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [growthbook](https://agentstack.voostack.com/s/growthbook)
- **Installs:** 0
- **Category:** [Agent Skills](https://agentstack.voostack.com/c/agent-skills)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [growthbook](https://github.com/growthbook)
- **Source:** https://github.com/growthbook/skills/tree/main/skills/experiment-analyze

## Install

```sh
agentstack add skill-growthbook-skills-experiment-analyze
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# experiment-analyze

Fetch results, refresh the snapshot only when the cached data is over 24 hours old or the user wants a different phase/dimension cut, then interpret. This skill is the heaviest in the catalog because of the conditional polling loop and the statistical interpretation — slow down and do each step deliberately.

All API calls go through the bundled helper: `${CLAUDE_PLUGIN_ROOT}/scripts/gb-call`. It needs `GB_API_KEY` — set in your shell, or written to `~/.config/growthbook/.env` by `/growthbook:setup`. If unset or invalid, gb-call's error message points back at `/growthbook:setup`. The skill also uses `sleep` between poll calls.

## Workflow

1. **Fetch results + experiment metadata in one call.** `/results` returns `{ experiment, result }` — the same payload that powers the GrowthBook UI's results view, so there's no need for a separate metadata call.

   ```bash
   gb-call GET /api/v1/experiments//results
   # or, when the user asks for a specific phase / dimension cut:
   gb-call GET '/api/v1/experiments//results?phase=1&dimension=exp:country'
   ```

   The `phase` and `dimension` query params filter to the latest snapshot taken with those settings; omit both for the default "how is it doing" view.

   From `experiment`, capture:

   - `status` — `running` or `stopped` both make sense for analysis. If `draft`, there are no results to interpret; tell the user.
   - `type` — if `"multi-armed-bandit"`, halt and tell the user this skill targets standard A/B tests. Bandits report differently (per-arm probabilities, dynamic traffic allocation) and shouldn't be read with the standard winner/loser framing.
   - `settings.statsEngine` — `"bayesian"` (default) or `"frequentist"`. Drives the metric-interpretation step below.
   - `regressionAdjustmentEnabled` (CUPED) and `sequentialTestingEnabled` — affect what to report.

   From `result`, capture:

   - `id` — the snapshot ID; used in step 3 if a refresh is needed.
   - `dateUpdated` — ISO timestamp of when this snapshot was created. Drives the staleness check in step 2.
   - The per-variation metric data itself (lift estimates, intervals, sample sizes) — what step 4 interprets.

   If the response errors with `"No results found for that experiment"`, the experiment has been started but no snapshot exists yet (the auto-refresh hasn't run, or you've filtered to a phase/dimension that's never been snapshotted). Skip step 2 and jump straight to step 3.

2. **Decide whether to refresh.** Branch on `result.dateUpdated`:

   - **Under 24 hours old, and the existing snapshot matches the user's requested phase/dimension** → skip step 3, jump to step 4.
   - **Over 24 hours old, or the user explicitly asked for a fresh snapshot** → step 3.

   The server auto-refreshes snapshots every 6 hours by default (`EXPERIMENT_REFRESH_FREQUENCY`), so anything under 24 hours has typically been refreshed at least once recently. The 24-hour bar is deliberately conservative — don't burn snapshot-compute budget on data that hasn't moved.

3. **Trigger a fresh snapshot, poll, then re-fetch results.**

   **3a. POST a snapshot.** Optional body fields shape what gets computed; pass the same `phase` / `dimension` the user asked for in step 1:

   - `phase` (integer, 0-indexed): pick a specific phase if the experiment has multiple — e.g., re-analyze the pre-ramp phase after the experiment has ramped up. Defaults to the latest phase.
   - `dimension` (string): break the results down. Built-in: `"pre:date"`, `"pre:activation"`. For a configured Unit Dimension, use its ID (e.g. `"dim_abc123"`). For an Experiment Dimension, prefix with `"exp:"` (e.g. `"exp:country"`).

   ```bash
   echo '{}' | gb-call POST /api/v1/experiments//snapshot -
   # or with phase + dimension:
   echo '{"phase": 1, "dimension": "exp:country"}' \
     | gb-call POST /api/v1/experiments//snapshot -
   ```

   The response is `{ snapshot: { id, experiment, status } }`. Capture `snapshot.id`.

   **3b. Poll for completion.** Snapshot creation is asynchronous:

   ```bash
   for i in $(seq 1 60); do
     gb-call GET /api/v1/snapshots/
     # Parse snapshot.status. Stop when it is "success" or "error".
     sleep 5
   done
   ```

   Cap the loop at 60 iterations (5 minutes). If it hasn't finished by then, stop polling and tell the user the snapshot is still in progress — they can retry the skill in a few minutes. Don't loop forever.

   **3c. Re-fetch results** with the same phase/dimension args used in 3a, then re-capture the fields listed in step 1:

   ```bash
   gb-call GET '/api/v1/experiments//results'
   # or, if 3a passed phase/dimension:
   gb-call GET '/api/v1/experiments//results?phase=1&dimension=exp:country'
   ```

4. **Run the data-quality checks first, then interpret.** GrowthBook surfaces six health checks; any failing one changes how the result should be read. Surface failures prominently — don't bury them.

   **Data-quality checks (in order):**

   - **SRM (Sample Ratio Mismatch).** Observed traffic split vs. configured split. Failure invalidates downstream results — say so prominently and stop interpreting until the user understands the bias risk. Common causes: bot traffic, ad-blockers blocking client-side tracking, mid-experiment targeting changes, activation-metric bias.
   - **Multiple Exposures.** Users assigned to more than one variation. A small rate (95% Chance to Win as a strong positive signal;  ()
   - Status: 
   - Type: 
   - Stats engine: 
   - Adjustments: 
   - Phase: 
   - Dimension: 
   - Sample size:  across N variations
   - Snapshot timestamp: 

   ### Data-quality checks
   - SRM: 
   - Multiple Exposures: 
   - Minimum Data Thresholds: 
   - Variation ID Mismatch: 
   - Suspicious Uplift: 

   ### Primary metric: 
   - Variation 0 (Control): 
   - Variation 1 (Treatment): 
   - **Bayesian:** Chance to Win , relative lift , 95% CrI [a, b]
   - **Frequentist:** lift , 95% CI [a, b]
   - Verdict: 

   ### Guardrails
   - : 

   ### Secondary metrics
   - :  

   ### Recommendation
   
   ```

6. **Link to the experiment, then suggest the next step.** Always surface the direct UI link so the user can review the live results, dimensional cuts, and historical snapshots beyond what `/results` returns:

   ```
   View in GrowthBook: /experiment/
   ```

   Derive `` from `GB_API_URL` by swapping `api.` → `app.` (matches `experiment-launch`'s convention; on the default cloud host this produces `https://app.growthbook.io`).

   Then suggest a next action based on `experiment.status` from step 1:

   - `running` and conclusive → suggest `experiment-stop` with the chosen variation.
   - `running` and inconclusive → suggest waiting or extending.
   - `stopped` → point at flag cleanup via `flag-targeting` to update or remove the `experiment-ref` rule on the linked flag.

## Guardrails

- **All six data-quality checks come before interpretation.** SRM, Multiple Exposures, Minimum Data Thresholds, Variation ID Mismatch, Suspicious Uplift, and Guardrails. A failure in any of them changes how (or whether) to interpret the result. Don't bury them under the primary-metric heading.
- **Branch interpretation on `settings.statsEngine`.** Bayesian (default) reports Chance to Win + Credible Intervals; frequentist reports lift + Confidence Intervals. Don't manufacture a p-value the API didn't return, and don't claim "95% CI" on a Bayesian result (it's a Credible Interval, not a Confidence Interval).
- **Multiple-comparison correction is frequentist-only and excludes guardrails.** When reporting secondaries, note the correction status. On Bayesian, the prior provides implicit shrinkage; no correction is applied.
- **CUPED and sequential testing change how to read CIs.** If `regressionAdjustmentEnabled` is true, point estimates may differ from raw means and CIs are typically narrower. If `sequentialTestingEnabled` is true, CIs are intentionally wider — say so, so the user doesn't compare them apples-to-oranges with non-sequential results.
- **Don't peek-and-decide.** Under-powered experiments mean interim numbers are noisy. Frequentist peeking inflates false-positive rates; Bayesian is more robust but still benefits from hitting the planned sample size.
- **Bandits are out of scope.** `type === "multi-armed-bandit"` reports per-arm probabilities and dynamically reallocates traffic. Halt and tell the user to read bandit results in the UI.
- **Activation-metric bias hides as "passing SRM."** If the experiment uses an activation metric that is downstream of variation differences (e.g., "completed signup" when variations affect signup completion), the overall split can look fine while the activated cohort is biased. The dashboard surfaces this; flag it when you spot the pattern in metadata.
- **Don't promote a secondary to a primary.** If the primary didn't move, the experiment didn't move — secondaries are exploratory.
- **Polling has a ceiling.** 60 iterations × 5s = 5 minutes. If the snapshot isn't done by then, stop and report. Don't run for hours.
- **Snapshot timestamp matters.** Always surface `result.dateUpdated` when reporting results — it's both how step 2 decides whether to refresh and how the user judges whether a slow-traffic experiment has moved. Stale snapshots are common; don't hide them.
- **24h is a deliberate ceiling, not the auto-refresh cadence.** The server auto-refreshes every 6h by default, so a snapshot under 24h has almost always been refreshed at least once. The skill's 24h bar gives the user a useful conservative window without paying snapshot compute on data that hasn't moved. Don't drop it to "any data older than a minute" — that's how you pin a busy org against the 60 rpm limit.
- **Rate limit awareness.** Happy path is a single `/results` call. Worst case (no snapshot or stale + 60-iteration poll + re-fetch) is ~63 calls spread over 5 minutes (~13/min), well under 60 rpm. If multiple users invoke this concurrently in the same org the limit can still bite — surface clearly if `gb-call` returns a 429.
- **Read-only.** This skill never stops or modifies the experiment. Hand off to `experiment-stop` when the user wants to act.

## Endpoints used

- `GET /api/v1/experiments//results` — primary entry point; returns `{ experiment, result }` so step 1 grabs metadata, status, and the snapshot timestamp (`result.dateUpdated`) in a single call. Accepts `phase` / `dimension` query params.
- `POST /api/v1/experiments//snapshot` — trigger a fresh snapshot when results are over 24h old or the user wants a phase/dimension cut the cached snapshot doesn't cover. Body accepts `phase` (integer) and `dimension` (string).
- `GET /api/v1/snapshots/` — poll for snapshot completion (5s interval, 60 iteration cap). Returns `{ snapshot: { id, experiment, status } }`.

## Handoffs

- `experiment-stop` — when the user is ready to act on a conclusive result.
- `flag-targeting` — after stopping with a winner, the linked flag (if any) needs its `experiment-ref` rule updated or removed.
- `experiment-brainstorm` — to ground ideas for the next test in results from past experiments.
- `experiment-statistics` (when shipped) — for deeper questions about CUPED, sequential testing, multiple comparisons, dimensional analysis.

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [growthbook](https://github.com/growthbook)
- **Source:** [growthbook/skills](https://github.com/growthbook/skills)
- **License:** MIT

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** yes
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-growthbook-skills-experiment-analyze
- Seller: https://agentstack.voostack.com/s/growthbook
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.