Experiment Analyze

experiment-analyze

Fetch results, refresh the snapshot only when the cached data is over 24 hours old or the user wants a different phase/dimension cut, then interpret. This skill is the heaviest in the catalog because of the conditional polling loop and the statistical interpretation — slow down and do each step deliberately.

All API calls go through the bundled helper: ${CLAUDE_PLUGIN_ROOT}/scripts/gb-call. It needs GB_API_KEY — set in your shell, or written to ~/.config/growthbook/.env by /growthbook:setup. If unset or invalid, gb-call's error message points back at /growthbook:setup. The skill also uses sleep between poll calls.

Workflow

1. Fetch results + experiment metadata in one call. /results returns { experiment, result } — the same payload that powers the GrowthBook UI's results view, so there's no need for a separate metadata call.

``bash gb-call GET /api/v1/experiments//results # or, when the user asks for a specific phase / dimension cut: gb-call GET '/api/v1/experiments//results?phase=1&dimension=exp:country' ``

The phase and dimension query params filter to the latest snapshot taken with those settings; omit both for the default "how is it doing" view.

From experiment, capture:

• status — running or stopped both make sense for analysis. If draft, there are no results to interpret; tell the user.
• type — if "multi-armed-bandit", halt and tell the user this skill targets standard A/B tests. Bandits report differently (per-arm probabilities, dynamic traffic allocation) and shouldn't be read with the standard winner/loser framing.
• settings.statsEngine — "bayesian" (default) or "frequentist". Drives the metric-interpretation step below.
• regressionAdjustmentEnabled (CUPED) and sequentialTestingEnabled — affect what to report.

From result, capture:

• id — the snapshot ID; used in step 3 if a refresh is needed.
• dateUpdated — ISO timestamp of when this snapshot was created. Drives the staleness check in step 2.
• The per-variation metric data itself (lift estimates, intervals, sample sizes) — what step 4 interprets.

If the response errors with "No results found for that experiment", the experiment has been started but no snapshot exists yet (the auto-refresh hasn't run, or you've filtered to a phase/dimension that's never been snapshotted). Skip step 2 and jump straight to step 3.

1. Decide whether to refresh. Branch on result.dateUpdated:

• Under 24 hours old, and the existing snapshot matches the user's requested phase/dimension → skip step 3, jump to step 4.
• Over 24 hours old, or the user explicitly asked for a fresh snapshot → step 3.

The server auto-refreshes snapshots every 6 hours by default (EXPERIMENT_REFRESH_FREQUENCY), so anything under 24 hours has typically been refreshed at least once recently. The 24-hour bar is deliberately conservative — don't burn snapshot-compute budget on data that hasn't moved.

1. Trigger a fresh snapshot, poll, then re-fetch results.

3a. POST a snapshot. Optional body fields shape what gets computed; pass the same phase / dimension the user asked for in step 1:

• phase (integer, 0-indexed): pick a specific phase if the experiment has multiple — e.g., re-analyze the pre-ramp phase after the experiment has ramped up. Defaults to the latest phase.
• dimension (string): break the results down. Built-in: "pre:date", "pre:activation". For a configured Unit Dimension, use its ID (e.g. "dim_abc123"). For an Experiment Dimension, prefix with "exp:" (e.g. "exp:country").

``bash echo '{}' | gb-call POST /api/v1/experiments//snapshot - # or with phase + dimension: echo '{"phase": 1, "dimension": "exp:country"}' \ | gb-call POST /api/v1/experiments//snapshot - ``

The response is { snapshot: { id, experiment, status } }. Capture snapshot.id.

3b. Poll for completion. Snapshot creation is asynchronous:

``bash for i in $(seq 1 60); do gb-call GET /api/v1/snapshots/ # Parse snapshot.status. Stop when it is "success" or "error". sleep 5 done ``

Cap the loop at 60 iterations (5 minutes). If it hasn't finished by then, stop polling and tell the user the snapshot is still in progress — they can retry the skill in a few minutes. Don't loop forever.

3c. Re-fetch results with the same phase/dimension args used in 3a, then re-capture the fields listed in step 1:

``bash gb-call GET '/api/v1/experiments//results' # or, if 3a passed phase/dimension: gb-call GET '/api/v1/experiments//results?phase=1&dimension=exp:country' ``

1. Run the data-quality checks first, then interpret. GrowthBook surfaces six health checks; any failing one changes how the result should be read. Surface failures prominently — don't bury them.

Data-quality checks (in order):

• SRM (Sample Ratio Mismatch). Observed traffic split vs. configured split. Failure invalidates downstream results — say so prominently and stop interpreting until the user understands the bias risk. Common causes: bot traffic, ad-blockers blocking client-side tracking, mid-experiment targeting changes, activation-metric bias.
• Multiple Exposures. Users assigned to more than one variation. A small rate (95% Chance to Win as a strong positive signal; ()
• Status:
• Type:
• Stats engine:
• Adjustments:
• Phase:
• Dimension:
• Sample size: across N variations
• Snapshot timestamp:

### Data-quality checks

• SRM:
• Multiple Exposures:
• Minimum Data Thresholds:
• Variation ID Mismatch:
• Suspicious Uplift:

### Primary metric:

• Variation 0 (Control):
• Variation 1 (Treatment):
• Bayesian: Chance to Win , relative lift , 95% CrI [a, b]
• Frequentist: lift , 95% CI [a, b]
• Verdict:

### Guardrails

• :

### Secondary metrics

• :

### Recommendation

```

1. Link to the experiment, then suggest the next step. Always surface the direct UI link so the user can review the live results, dimensional cuts, and historical snapshots beyond what /results returns:

`` View in GrowthBook: /experiment/ ``

Derive ` from GBAPIURL by swapping api. → app. (matches experiment-launch's convention; on the default cloud host this produces https://app.growthbook.io`).

Then suggest a next action based on experiment.status from step 1:

• running and conclusive → suggest experiment-stop with the chosen variation.
• running and inconclusive → suggest waiting or extending.
• stopped → point at flag cleanup via flag-targeting to update or remove the experiment-ref rule on the linked flag.

Guardrails

• All six data-quality checks come before interpretation. SRM, Multiple Exposures, Minimum Data Thresholds, Variation ID Mismatch, Suspicious Uplift, and Guardrails. A failure in any of them changes how (or whether) to interpret the result. Don't bury them under the primary-metric heading.
• Branch interpretation on settings.statsEngine. Bayesian (default) reports Chance to Win + Credible Intervals; frequentist reports lift + Confidence Intervals. Don't manufacture a p-value the API didn't return, and don't claim "95% CI" on a Bayesian result (it's a Credible Interval, not a Confidence Interval).
• Multiple-comparison correction is frequentist-only and excludes guardrails. When reporting secondaries, note the correction status. On Bayesian, the prior provides implicit shrinkage; no correction is applied.
• CUPED and sequential testing change how to read CIs. If regressionAdjustmentEnabled is true, point estimates may differ from raw means and CIs are typically narrower. If sequentialTestingEnabled is true, CIs are intentionally wider — say so, so the user doesn't compare them apples-to-oranges with non-sequential results.
• Don't peek-and-decide. Under-powered experiments mean interim numbers are noisy. Frequentist peeking inflates false-positive rates; Bayesian is more robust but still benefits from hitting the planned sample size.
• Bandits are out of scope. type === "multi-armed-bandit" reports per-arm probabilities and dynamically reallocates traffic. Halt and tell the user to read bandit results in the UI.
• Activation-metric bias hides as "passing SRM." If the experiment uses an activation metric that is downstream of variation differences (e.g., "completed signup" when variations affect signup completion), the overall split can look fine while the activated cohort is biased. The dashboard surfaces this; flag it when you spot the pattern in metadata.
• Don't promote a secondary to a primary. If the primary didn't move, the experiment didn't move — secondaries are exploratory.
• Polling has a ceiling. 60 iterations × 5s = 5 minutes. If the snapshot isn't done by then, stop and report. Don't run for hours.
• Snapshot timestamp matters. Always surface result.dateUpdated when reporting results — it's both how step 2 decides whether to refresh and how the user judges whether a slow-traffic experiment has moved. Stale snapshots are common; don't hide them.
• 24h is a deliberate ceiling, not the auto-refresh cadence. The server auto-refreshes every 6h by default, so a snapshot under 24h has almost always been refreshed at least once. The skill's 24h bar gives the user a useful conservative window without paying snapshot compute on data that hasn't moved. Don't drop it to "any data older than a minute" — that's how you pin a busy org against the 60 rpm limit.
• Rate limit awareness. Happy path is a single /results call. Worst case (no snapshot or stale + 60-iteration poll + re-fetch) is ~63 calls spread over 5 minutes (~13/min), well under 60 rpm. If multiple users invoke this concurrently in the same org the limit can still bite — surface clearly if gb-call returns a 429.
• Read-only. This skill never stops or modifies the experiment. Hand off to experiment-stop when the user wants to act.

Endpoints used

• GET /api/v1/experiments//results — primary entry point; returns { experiment, result } so step 1 grabs metadata, status, and the snapshot timestamp (result.dateUpdated) in a single call. Accepts phase / dimension query params.
• POST /api/v1/experiments//snapshot — trigger a fresh snapshot when results are over 24h old or the user wants a phase/dimension cut the cached snapshot doesn't cover. Body accepts phase (integer) and dimension (string).
• GET /api/v1/snapshots/ — poll for snapshot completion (5s interval, 60 iteration cap). Returns { snapshot: { id, experiment, status } }.

Handoffs

• experiment-stop — when the user is ready to act on a conclusive result.
• flag-targeting — after stopping with a winner, the linked flag (if any) needs its experiment-ref rule updated or removed.
• experiment-brainstorm — to ground ideas for the next test in results from past experiments.
• experiment-statistics (when shipped) — for deeper questions about CUPED, sequential testing, multiple comparisons, dimensional analysis.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: growthbook
• Source: growthbook/skills
• License: MIT

Install and usage instructions live in the source repository linked above.