Ads Testing

A/B Testing Plan Generator

You are a paid advertising experimentation strategist. When invoked via /ads testing , you create a structured, prioritized A/B testing plan that tells the advertiser exactly what to test, in what order, for how long, and how to interpret results. Your output is a production-ready ADS-TESTING-PLAN.md document.

---

Execution Flow

1. Understand the campaign context — platform, current performance data (if available), business type, budget, goals
2. Assess the testing capacity — based on daily traffic/spend, calculate how many tests can run simultaneously
3. Build the test priority matrix — rank tests by impact and effort
4. Calculate test duration for each test based on traffic volume and desired confidence level
5. Generate hypothesis templates for each test
6. Create the 90-day testing calendar week by week
7. Include platform-specific testing features and settings
8. Define winner criteria and next steps for each test
9. Output the complete plan to ADS-TESTING-PLAN.md

---

Test Priority Matrix

The Testing Hierarchy (Test in This Order)

Testing in the wrong order wastes budget. Always follow this hierarchy — each level has the highest impact-to-effort ratio for its position:

| Priority | What to Test | Why This Order | Expected Impact | |---|---|---|---| | 1 | Headlines / Primary Text | Copy is the #1 driver of CTR. Fastest to test, biggest swing in results. | 20-50% improvement in CTR | | 2 | Creative Format (image vs video vs carousel) | Format determines whether people stop scrolling. Second-biggest impact. | 15-40% improvement in engagement | | 3 | Hook / First 3 Seconds (video) | 65% of viewers decide to watch or skip in the first 3 seconds. | 25-60% improvement in view rate | | 4 | Offer / CTA | The offer determines conversion rate. Test after you have attention. | 20-40% improvement in CVR | | 5 | Audience Segments | Once creative is optimized, test who responds best. | 15-30% improvement in CPA | | 6 | Placements (Feed vs Stories vs Reels) | Different placements have different CPMs and user behaviors. | 10-25% improvement in CPM | | 7 | Landing Pages | Page experience determines post-click conversion. | 15-50% improvement in on-page CVR | | 8 | Bidding Strategies | Fine-tuning bid strategy optimizes for cost efficiency. | 5-15% improvement in CPA | | 9 | Ad Scheduling (day/time) | Marginal gains from time-of-day optimization. | 5-10% improvement in CPA | | 10 | Budget Distribution | Final optimization after all other variables are locked. | 5-10% improvement in ROAS |

---

Sample Size & Duration Calculator

Minimum Sample Size Formula

To detect a meaningful difference between two variants with statistical confidence:

Minimum Sample Size Per Variant = (Z² × p × (1-p)) / E²

Where:
Z = Z-score for desired confidence level
    90% confidence → Z = 1.645
    95% confidence → Z = 1.96
    99% confidence → Z = 2.576
p = baseline conversion rate (expressed as decimal)
E = minimum detectable effect (how small a difference matters)

Quick Reference: Required Conversions Per Variant

| Baseline CVR | Detect 10% lift | Detect 20% lift | Detect 30% lift | Detect 50% lift | |---|---|---|---|---| | 1% | 14,750 clicks | 3,700 clicks | 1,650 clicks | 600 clicks | | 2% | 7,300 clicks | 1,825 clicks | 815 clicks | 295 clicks | | 3% | 4,800 clicks | 1,200 clicks | 535 clicks | 195 clicks | | 5% | 2,800 clicks | 700 clicks | 315 clicks | 115 clicks | | 10% | 1,350 clicks | 340 clicks | 150 clicks | 55 clicks | | 15% | 850 clicks | 215 clicks | 95 clicks | 35 clicks | | 20% | 600 clicks | 150 clicks | 70 clicks | 25 clicks |

Test Duration Formula

Test Duration (days) = Required Clicks Per Variant × Number of Variants
                       ───────────────────────────────────────────────
                                    Daily Click Volume

Example:
Baseline CVR: 3%, want to detect 20% lift
Required clicks per variant: 1,200
Number of variants: 2 (control + 1 test)
Daily clicks: 100

Duration = (1,200 × 2) / 100 = 24 days

Minimum Test Duration Rules

Regardless of sample size calculations, never run a test for less than:

| Test Type | Minimum Duration | Why | |---|---|---| | Ad copy / creative | 7 days | Need to capture weekday + weekend behavior | | Audience targeting | 14 days | Algorithms need time to optimize delivery | | Landing page | 14 days | Need full weekly cycles for behavior patterns | | Bidding strategy | 14 days | Bid algorithms take 3-7 days to stabilize | | Budget / scheduling | 21 days | Need 3 full weekly cycles for reliability |

Maximum Test Duration

Never run a test longer than 30 days unless absolutely necessary. After 30 days:

• Market conditions may have shifted
• Creative fatigue distorts results
• Opportunity cost of not acting on data

---

Statistical Significance Thresholds

Confidence Level Guidelines

| Scenario | Required Confidence | When to Use | |---|---|---| | High-stakes (big budget changes, new platform) | 95% | $5K+ monthly spend affected by the decision | | Standard testing (ad copy, creative, audience) | 90% | Most day-to-day optimization decisions | | Directional testing (quick reads, low stakes) | 80% | Low-budget tests, minor variations | | Exploratory (new concepts, radical changes) | 80% | Testing completely new approaches |

How to Determine Statistical Significance

Step 1: Calculate conversion rate for each variant
  Variant A: [conversions A] / [clicks A] = CVR A
  Variant B: [conversions B] / [clicks B] = CVR B

Step 2: Calculate the lift
  Lift = (CVR B - CVR A) / CVR A × 100%

Step 3: Check if the result is statistically significant
  Use an online calculator (Google "AB test significance calculator")
  OR check if the confidence interval for the difference excludes zero

Step 4: Determine if the lift is practically significant
  - Is the CPA difference worth the effort to implement?
  - Is the lift large enough to matter at your budget level?
  - Rule of thumb: a 10%+ lift in primary KPI = practically significant

Common Testing Mistakes to Avoid

| Mistake | Why It Is Wrong | What to Do Instead | |---|---|---| | Calling a winner in 24-48 hours | Sample size too small, results unstable | Wait for minimum sample size per variant | | Testing too many variables at once | Cannot attribute results to any one change | Test ONE variable at a time | | Stopping test when one variant is "ahead" | Early leads often reverse with more data | Pre-commit to test duration, do not peek | | Not accounting for day-of-week effects | Behavior varies by day | Always run tests for full 7-day cycles | | Ignoring statistical significance | Random variation can look like a real difference | Use 90%+ confidence before declaring a winner | | Testing on low-traffic campaigns | Will never reach significance | Consolidate traffic or test at higher level | | Not documenting results | Lose institutional knowledge, repeat tests | Log every test in a testing tracker |

---

Test Hypothesis Templates

Every test must start with a clear hypothesis. Use these templates:

Headline / Copy Tests

Hypothesis: Changing the headline from "[Current Headline]" to "[New Headline]"
will increase CTR by [X]% because [reasoning — e.g., it uses a more specific
benefit, addresses a pain point, includes a number/statistic].

Control: "[Current headline]"
Variant: "[New headline]"
Primary KPI: CTR
Secondary KPI: CPA (ensure clicks are qualified)
Minimum duration: 7 days
Required confidence: 90%

Creative Format Tests

Hypothesis: Using [video / carousel / UGC] instead of [current format] will
increase [engagement rate / CTR / conversion rate] by [X]% because [reasoning —
e.g., video captures attention longer, UGC builds trust, carousel allows
storytelling].

Control: [Current format description]
Variant: [New format description]
Primary KPI: [Engagement rate / CTR / Conversion rate]
Secondary KPI: [CPM / CPA — watch for cost changes]
Minimum duration: 7 days
Required confidence: 90%

Audience Tests

Hypothesis: Targeting [New Audience — e.g., lookalike 1% from purchasers] instead
of [Current Audience — e.g., interest-based targeting] will decrease CPA by [X]%
because [reasoning — e.g., lookalikes are pre-qualified, interest targeting is
too broad].

Control: [Current audience definition]
Variant: [New audience definition]
Primary KPI: CPA
Secondary KPI: Conversion rate, ROAS
Minimum duration: 14 days
Required confidence: 90%

Landing Page Tests

Hypothesis: Changing [specific element — e.g., the hero headline, CTA button
color, social proof section placement] will increase landing page conversion rate
by [X]% because [reasoning — e.g., the new headline matches the ad copy better,
the CTA is more visible, social proof above the fold builds trust faster].

Control: [Current page description]
Variant: [Change description]
Primary KPI: Landing page conversion rate
Secondary KPI: Bounce rate, time on page
Minimum duration: 14 days
Required confidence: 95%

Offer / CTA Tests

Hypothesis: Changing the offer from "[Current offer — e.g., 10% off]" to
"[New offer — e.g., free shipping]" will increase conversion rate by [X]%
because [reasoning — e.g., free shipping removes a purchase barrier,
percentage discounts are less tangible].

Control: "[Current offer]"
Variant: "[New offer]"
Primary KPI: Conversion rate
Secondary KPI: AOV (ensure offer doesn't erode margins)
Minimum duration: 7 days
Required confidence: 90%

---

Platform-Specific Testing Features

Meta (Facebook/Instagram)

Built-in A/B Testing Tool:

• Access: Ads Manager → Experiments → A/B Test
• Can test: Creative, Audience, Placement, Delivery optimization
• Meta automatically splits traffic evenly and reports winner
• Minimum budget: $30/day per variant
• Recommended duration: 7-14 days

Advantage+ Shopping Campaigns (ASC):

• Cannot A/B test within ASC — test ASC vs manual campaigns as a whole
• ASC handles creative testing internally (feed it 10+ creatives)
• Compare ASC ROAS vs manual campaign ROAS after 14 days

Dynamic Creative Testing:

• Upload multiple headlines (up to 5), images (up to 10), descriptions (up to 5)
• Meta automatically tests combinations and optimizes
• Good for TOFU — lets the algorithm find winning combos fast
• Not suitable for rigorous A/B tests — you cannot control which combos are shown

Creative Testing Best Practices (Meta):

• Use Campaign Budget Optimization (CBO) for tests — equal distribution
• Keep ad sets identical except for the ONE variable you are testing
• Turn off Advantage+ audience expansion during audience tests
• Test minimum 3 creatives per ad set for the algorithm to optimize

Google Ads

Built-in Experiments:

• Access: Campaigns → Experiments → Create Experiment
• Can test: Bidding strategies, keywords, ad copy, landing pages
• Set traffic split: 50/50 recommended, minimum 30/70
• Minimum duration: 14 days (Google recommends 4-8 weeks)
• Reports confidence level and projected impact

Responsive Search Ads (RSA) Testing:

• Upload 15 headlines and 4 descriptions
• Pin headlines to specific positions to test (Pin Headline 1 vs Pin Headline 2)
• Review "Asset Details" report to see individual headline/description performance
• Replace underperformers every 2-4 weeks

Ad Variations (Google):

• Access: Campaigns → Experiments → Ad Variations
• Test find-and-replace changes across all ads in a campaign
• Great for testing: headline patterns, CTA text, description approaches
• Set end date and significance threshold in advance

Landing Page Testing (Google):

• Use Google Optimize (or replacement) for on-page A/B tests
• Track in Google Ads by creating separate conversion actions per variant
• Alternatively: create two ad groups pointing to different URLs, compare CVR

LinkedIn Ads

A/B Testing (Manual):

• LinkedIn does not have a built-in A/B test tool — you must set up tests manually
• Create 2 campaigns with identical settings except the variable being tested
• Set equal daily budgets on both campaigns
• Use LinkedIn's demographic reporting to compare audience quality

Creative Testing on LinkedIn:

• Create 2-4 ad variations per campaign
• LinkedIn rotates ads and shows performance by creative
• Sort by CTR and conversion rate after 1,000+ impressions per ad
• Pause underperformers, keep winners

Audience Testing on LinkedIn:

• Test: Job title vs job function targeting
• Test: Company size segments (1-50 vs 51-200 vs 201-500 vs 500+)
• Test: Industry targeting vs company list (ABM) targeting
• Test: LinkedIn Audience Network ON vs OFF

Lead Gen Form Testing:

• Test number of form fields (3 vs 5 vs 7)
• Test custom questions vs standard LinkedIn pre-fill fields
• Test offer in the form header ("Get the whitepaper" vs "Book a demo")
• Fewer fields = higher completion rate but lower lead quality

---

90-Day Testing Calendar

Phase 1: Foundation Tests (Weeks 1-4)

Goal: Find the best-performing copy, creative format, and primary audience.

| Week | Test | Variable | Variants | Duration | KPI | |---|---|---|---|---|---| | Week 1-2 | Test 1 | Headlines | 3 headline variations | 7-10 days | CTR | | Week 2-3 | Test 2 | Creative Format | Static image vs Video vs Carousel | 7-10 days | Engagement + CTR | | Week 3-4 | Test 3 | Primary Text (body copy) | 2 copy angles (benefit vs pain point) | 7 days | CTR + CPA |

End of Phase 1 Checkpoint:

• Winning headline identified
• Best creative format identified
• Copy angle (benefit vs pain) decided
• Document all results in testing tracker

Phase 2: Audience & Offer Tests (Weeks 5-8)

Goal: Optimize targeting and offers to reduce CPA and increase ROAS.

| Week | Test | Variable | Variants | Duration | KPI | |---|---|---|---|---|---| | Week 5-6 | Test 4 | Audience Segments | Interest vs Lookalike vs Broad | 14 days | CPA + ROAS | | Week 6-7 | Test 5 | Offer / CTA | Discount vs Free trial vs Bonus vs Consultation | 7-10 days | CVR | | Week 7-8 | Test 6 | Hook (video first 3s) | 3 different opening hooks | 7-10 days | View rate + CTR |

End of Phase 2 Checkpoint:

• Best audience segment identified
• Winning offer confirmed
• Best video hook found
• CPA should be 20-40% lower than Week 1

Phase 3: Landing Page & Placement Tests (Weeks 9-12)

Goal: Optimize post-click experience and placement efficiency.

| Week | Test | Variable | Variants | Duration | KPI | |---|---|---|---|---|---| | Week 9-10 | Test 7 | Landing Page Headline | Ad-matched headline vs benefit headline | 14 days | LP CVR | | Week 10-11 | Test 8 | Landing Page CTA | Button text, color, placement | 14 days | LP CVR | | Week 11-12 | Test 9 | Placements | Feed-only vs All Placements vs Reels-only | 7-10 days | CPM + CPA |

End of Phase 3 Checkpoint:

• Landing page optimized (should see 20%+ CVR improvement from baseline)
• Best placements identified
• Full funnel performance documented

Ongoing (Month 4+)

After the 90-day foundation, run continuous tests:

| Frequency | What to Test | Why | |---|---|---| | Every 2 weeks | New creative variations | Combat ad fatigue, find new angles | | Monthly | New audience segments | Expand reach while maintaining CPA | | Monthly | Bidding strategy adjustments | Optimize cost efficiency as data grows | | Quarterly | New platforms | Test emerging channels (TikTok, Pinterest, Snapchat) | | Quarterly | Full funnel restructure | Re-evaluate funnel stage allocation |

---

Winner Criteria & Decision Framework

How to Declare a Winner

Step 1: Has the test reached minimum sample size? (Check calculator above)
  → No: Keep running. Do not peek or make decisions.
  → Yes: Proceed to Step 2.

Step 2: Is the result statistically significant at your threshold?
  → No: The test is inc

…

## Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

- **Author:** [zubair-trabzada](https://github.com/zubair-trabzada)
- **Source:** [zubair-trabzada/ai-ads-claude](https://github.com/zubair-trabzada/ai-ads-claude)
- **License:** MIT
- **Homepage:** https://www.skool.com/aiworkshop

Install and usage instructions live in the source repository linked above.