# 19 Ab Test Setup Global

> Design valid A/B tests for global marketing — hypothesis formulation, sample size calculation, statistical significance, multi-arm testing, primary vs secondary metrics. Tools: Optimizely, VWO, Google Optimize (sunset 2023, alternatives), built-in platform tests (Meta, Google). Trigger: 'A/B test', 'split test', 'multivariate test', 'experiment design', 'statistical significance', 'sample size ca…

- **Type:** Skill
- **Install:** `agentstack add skill-minhnv0807-ai-business-skills-19-ab-test-setup-global`
- **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior
- **Seller:** [minhnv0807](https://agentstack.voostack.com/s/minhnv0807)
- **Installs:** 0
- **Category:** [AI & ML](https://agentstack.voostack.com/c/ai-and-ml)
- **Latest version:** 0.1.0
- **License:** MIT
- **Upstream author:** [minhnv0807](https://github.com/minhnv0807)
- **Source:** https://github.com/minhnv0807/ai-business-skills/tree/master/skills/en/19-ab-test-setup-global
- **Website:** https://opa.business

## Install

```sh
agentstack add skill-minhnv0807-ai-business-skills-19-ab-test-setup-global
```

Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent.

## About

# A/B Test Setup (Global)

> Run experiments that produce decisions, not noise. Most "A/B tests" in marketing are underpowered, peeked-at, and badly hypothesized — meaning the team learns nothing and ships the louder variant.

---

## For Newbies

A valid A/B test answers one question: "Did this change cause a real improvement, or am I seeing noise?"

To answer it credibly you need four things:
1. A **specific hypothesis** with a numeric prediction
2. **One variable changed** (everything else identical)
3. **Enough sample** to detect the effect you care about
4. **Statistical significance** before you call a winner (typically p  0.10 → no evidence of an effect — keep control or test something else

For high-stakes tests (pricing, branding) consider 99% confidence (p  **Note:** Google Optimize was sunset in September 2023. Migration paths: GA4 + a third-party platform (Optimizely, VWO, Convert) or PostHog/GrowthBook for product-led teams.

---

## Setup Without Dedicated Tools

```
1. Build two versions of the page: /landing-a and /landing-b
2. Split traffic 50/50:
   - Meta Ads: 2 ad sets, identical audience, different destination URLs
   - Google Ads: 2 ads in the same ad group, identical targeting, different URLs
   - Email: list-split feature in your ESP
3. Track conversions per variant:
   - Meta Pixel custom event with parameter: page_version = "A" / "B"
   - GA4 event with custom dimension
   - PostHog feature flag exposure event
4. Run for the planned duration. Don't peek mid-test.
5. Export raw counts. Run significance test (calculator below).
```

---

## Result Analysis

### Statistical significance

Use a calculator. Recommended:
- **Evan Miller's calculator** — `evanmiller.org/ab-testing/chi-squared.html`
- **AB Testguide** — `abtestguide.com/calc/`
- **Optimizely's calculator** — built into platform
- **Survey Monkey calculator** — for sample size pre-test

**Inputs:**
- Variant A: visitors + conversions
- Variant B: visitors + conversions

**Outputs:**
- p-value (need  5%       | B wins — implement and document |
|  10%    | Borderline — extend test if feasible |
| > 0.10  | any         | No evidence — keep A or design a stronger test |

### Common Pitfalls

1. **Peeking and stopping early.** Most common cause of false positives. If the platform shows "B is winning" on day 3, the platform is misleading you (unless it's specifically designed for sequential testing).
2. **Uneven splits.** If split is 30/70 instead of intended 50/50, your delivery infrastructure has a bug. Investigate before trusting results.
3. **Seasonality.** Tests run only on weekdays vs weekends produce different results. Always run for whole weeks.
4. **Novelty effect.** New variants attract attention for the first 2–3 days, then performance regresses. Long enough tests absorb this.
5. **Sample ratio mismatch (SRM).** Even with 50/50 intent, if visitor counts diverge significantly (e.g. 8,400 vs 11,600), there's likely a tracking or assignment bug. Tools like PostHog and Optimizely flag this automatically.
6. **Mixing traffic sources mid-test.** Don't add a new ad campaign halfway through — it changes audience composition.
7. **Multiple comparison problem.** Running 20 simultaneous tests means ~1 will look "significant" by chance alone. Adjust thresholds (Bonferroni correction) or pre-register hypotheses.

---

## Output Template

```markdown
# A/B Test: [test name]
Created: [YYYY-MM-DD]
Owner: [name]

## 1. Hypothesis
"If we [change X], [metric Y] will increase by [Z%] because [reason]."

## 2. Variants
- Variant A (Control): [current state description]
- Variant B (Challenger): [changed state description]
- Single change: [the one element that differs]

## 3. Metrics
- Primary: [e.g. conversion rate]
- Secondary (guardrails): [e.g. bounce rate, time on page, AOV]

## 4. Sample size & duration
- Baseline (p): [%]
- Minimum detectable effect (MDE): [%]
- Sample needed per variant: [N]
- Daily traffic to test surface: [N]
- Estimated days to complete: [N]

## 5. Setup
- Tool: [Optimizely / VWO / PostHog / Meta built-in / custom]
- Variant A URL or asset: [...]
- Variant B URL or asset: [...]
- Tracking events: [list]
- Split ratio: 50/50

## 6. Timeline
- Start: [date]
- End (planned): [date]
- Review meeting: [date]

## 7. Results (filled in after test ends)
| Variant | Visitors | Conversions | Rate | Lift vs A |
|---------|----------|-------------|------|-----------|
| A | | | | — |
| B | | | | +X% |

p-value: [x]
95% CI on lift: [lower%, upper%]
Significant (p < 0.05): [Yes / No]

## 8. Decision
[Ship B / Keep A / Inconclusive — extend or redesign]

## 9. Action
[Implement variant B globally / Roll back / Schedule next iteration]

## 10. Lessons
[What this teaches generalizable for future tests]
```

---

## Quality Checklist

- [ ] Exactly one variable changed
- [ ] Hypothesis is specific and includes a numeric prediction
- [ ] Sample size calculated up front (minimum 100 conv/variant)
- [ ] Duration is at least 1 full week, ideally 2 weeks
- [ ] No peeking — end date defined and honored
- [ ] p-value calculated before declaring a winner (target p < 0.05)
- [ ] Result documented (winner or not — both are learning)
- [ ] Secondary metrics checked (no guardrail violations)
- [ ] Sample ratio verified (no SRM red flags)
- [ ] Next test identified based on what this one taught

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [minhnv0807](https://github.com/minhnv0807)
- **Source:** [minhnv0807/ai-business-skills](https://github.com/minhnv0807/ai-business-skills)
- **License:** MIT
- **Homepage:** https://opa.business

Install and usage instructions live in the source repository linked above.

## Pricing

- **Free** — Free

## Security capabilities

Automated source analysis of v0.1.0 — what this tool can access:

- **Network access:** no
- **Filesystem access:** no
- **Shell / process execution:** no
- **Environment & secrets:** no
- **Dynamic code execution:** no

*"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.*


## Versions

- **0.1.0** — security scan: passed — Imported from the upstream source.

## Links

- Listing page: https://agentstack.voostack.com/l/skill-minhnv0807-ai-business-skills-19-ab-test-setup-global
- Seller: https://agentstack.voostack.com/s/minhnv0807
- Browse the marketplace: https://agentstack.voostack.com/browse

---
Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.