19 Ab Test Setup Global

A/B Test Setup (Global)

> Run experiments that produce decisions, not noise. Most "A/B tests" in marketing are underpowered, peeked-at, and badly hypothesized — meaning the team learns nothing and ships the louder variant.

---

For Newbies

A valid A/B test answers one question: "Did this change cause a real improvement, or am I seeing noise?"

To answer it credibly you need four things:

1. A specific hypothesis with a numeric prediction
2. One variable changed (everything else identical)
3. Enough sample to detect the effect you care about
4. Statistical significance before you call a winner (typically p 0.10 → no evidence of an effect — keep control or test something else

For high-stakes tests (pricing, branding) consider 99% confidence (p Note: Google Optimize was sunset in September 2023. Migration paths: GA4 + a third-party platform (Optimizely, VWO, Convert) or PostHog/GrowthBook for product-led teams.

---

Setup Without Dedicated Tools

1. Build two versions of the page: /landing-a and /landing-b
2. Split traffic 50/50:
   - Meta Ads: 2 ad sets, identical audience, different destination URLs
   - Google Ads: 2 ads in the same ad group, identical targeting, different URLs
   - Email: list-split feature in your ESP
3. Track conversions per variant:
   - Meta Pixel custom event with parameter: page_version = "A" / "B"
   - GA4 event with custom dimension
   - PostHog feature flag exposure event
4. Run for the planned duration. Don't peek mid-test.
5. Export raw counts. Run significance test (calculator below).

---

Result Analysis

Statistical significance

Use a calculator. Recommended:

• Evan Miller's calculator — evanmiller.org/ab-testing/chi-squared.html
• AB Testguide — abtestguide.com/calc/
• Optimizely's calculator — built into platform
• Survey Monkey calculator — for sample size pre-test

Inputs:

• Variant A: visitors + conversions
• Variant B: visitors + conversions

Outputs:

• p-value (need 5% | B wins — implement and document |

| 10% | Borderline — extend test if feasible | | > 0.10 | any | No evidence — keep A or design a stronger test |

Common Pitfalls

1. Peeking and stopping early. Most common cause of false positives. If the platform shows "B is winning" on day 3, the platform is misleading you (unless it's specifically designed for sequential testing).
2. Uneven splits. If split is 30/70 instead of intended 50/50, your delivery infrastructure has a bug. Investigate before trusting results.
3. Seasonality. Tests run only on weekdays vs weekends produce different results. Always run for whole weeks.
4. Novelty effect. New variants attract attention for the first 2–3 days, then performance regresses. Long enough tests absorb this.
5. Sample ratio mismatch (SRM). Even with 50/50 intent, if visitor counts diverge significantly (e.g. 8,400 vs 11,600), there's likely a tracking or assignment bug. Tools like PostHog and Optimizely flag this automatically.
6. Mixing traffic sources mid-test. Don't add a new ad campaign halfway through — it changes audience composition.
7. Multiple comparison problem. Running 20 simultaneous tests means ~1 will look "significant" by chance alone. Adjust thresholds (Bonferroni correction) or pre-register hypotheses.

---

Output Template

# A/B Test: [test name]
Created: [YYYY-MM-DD]
Owner: [name]

## 1. Hypothesis
"If we [change X], [metric Y] will increase by [Z%] because [reason]."

## 2. Variants
- Variant A (Control): [current state description]
- Variant B (Challenger): [changed state description]
- Single change: [the one element that differs]

## 3. Metrics
- Primary: [e.g. conversion rate]
- Secondary (guardrails): [e.g. bounce rate, time on page, AOV]

## 4. Sample size & duration
- Baseline (p): [%]
- Minimum detectable effect (MDE): [%]
- Sample needed per variant: [N]
- Daily traffic to test surface: [N]
- Estimated days to complete: [N]

## 5. Setup
- Tool: [Optimizely / VWO / PostHog / Meta built-in / custom]
- Variant A URL or asset: [...]
- Variant B URL or asset: [...]
- Tracking events: [list]
- Split ratio: 50/50

## 6. Timeline
- Start: [date]
- End (planned): [date]
- Review meeting: [date]

## 7. Results (filled in after test ends)
| Variant | Visitors | Conversions | Rate | Lift vs A |
|---------|----------|-------------|------|-----------|
| A | | | | — |
| B | | | | +X% |

p-value: [x]
95% CI on lift: [lower%, upper%]
Significant (p < 0.05): [Yes / No]

## 8. Decision
[Ship B / Keep A / Inconclusive — extend or redesign]

## 9. Action
[Implement variant B globally / Roll back / Schedule next iteration]

## 10. Lessons
[What this teaches generalizable for future tests]

---

Quality Checklist

• [ ] Exactly one variable changed
• [ ] Hypothesis is specific and includes a numeric prediction
• [ ] Sample size calculated up front (minimum 100 conv/variant)
• [ ] Duration is at least 1 full week, ideally 2 weeks
• [ ] No peeking — end date defined and honored
• [ ] p-value calculated before declaring a winner (target p < 0.05)
• [ ] Result documented (winner or not — both are learning)
• [ ] Secondary metrics checked (no guardrail violations)
• [ ] Sample ratio verified (no SRM red flags)
• [ ] Next test identified based on what this one taught

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: minhnv0807
• Source: minhnv0807/ai-business-skills
• License: MIT
• Homepage: https://opa.business

Install and usage instructions live in the source repository linked above.