Ab Test Analyze

A/B Test Analyze

Structure, run, and analyze A/B tests on social content to determine what actually drives better performance. Social media optimization without experimentation is superstition. This skill turns "I think this works" into "this works, and here's the data."

Purpose

Every content decision is a guess until you test it. Hook variation A vs B, thread format vs short post, morning posting vs afternoon — this skill takes the guesswork out by running controlled experiments and declaring clear winners.

When to Use

• Testing hook variations for the same content
• Testing posting times
• Testing formats (thread vs short post vs carousel)
• Testing CTAs
• After 2+ weeks with controlled variations

Inputs Required

• Test hypothesis: "We believe [change] will improve [metric] by [X]%"
• Control condition description
• Variant condition description
• Result data (from metrics-track)

Quick Reference

| Test Type | Control | Variant | Min Sample | Duration | |-----------|---------|---------|-----------|---------| | Hook A/B | Current hook style | New hook, same body | 2 per condition, same day of week, different weeks | 2 weeks | | Time A/B | Current best time | Alternative time | 2 per condition | 2 weeks | | Format A/B | Current format | Different format, same message | 2 per condition | 2 weeks | | CTA A/B | Current CTA | Different CTA, same post | 2 per condition | 2 weeks |

Procedure

1. Define hypothesis clearly:

• "We believe [change] will improve [specific metric] by [X]%"
• Example: "We believe leading with a statistic instead of a question will improve ER by 15%"
• Bad hypothesis: "See what works better" (too vague)

1. Design the test:

• Independent variable: what you're changing
• Dependent variable: what you're measuring
• Control: current version
• Variant: new version
• Controlled factors: same platform, day of week, topic weight

1. Run minimum duration:

• Single-week tests unreliable (algorithm luck)
• Min: 2 variations per condition over 2 weeks (4 data points per condition)
• Never run two A/B tests simultaneously on the same platform (variable contamination)

1. Analyze results:

• Average performance for control vs variant
• Calculate % difference
• Assess significance: >20% difference = significant; 10-20% = directional; 20%
• Directional: 10-20% — worth re-testing
• Tie/conclusive: <10% — keep control
• Loser: revert, document why

1. Document learning: What was tested, result, what to change, what to test next.

Output Format

# A/B Test: [Test Name]

## Design
**Hypothesis:** [Specific prediction]
**Variable:** [What changed]
**Metric:** [What was measured]
**Control:** [Condition]
**Variant:** [Condition]
**Platform:** [X / LinkedIn / IG / TikTok]
**Duration:** [Start to end]
**Sample:** [N posts per condition]

## Results
| Condition | Avg [Metric] | vs Control |
|-----------|-------------|------------|
| Control | [N] | — |
| Variant | [N] | [+/-N]% |

**Winner:** [Control / Variant / Inconclusive]

## Analysis
**What happened:** [Plain-English summary]
**Why:** [Theory]
**Confidence:** [High / Medium / Low]

## Decision & Next Steps
**Action:** [Adopt / Keep / Re-test]
**Implementation:** [What changes to make]
**Next test:** [What to test next]

Done Criteria

The skill is complete when:

1. Hypothesis is specific and measurable
2. Test was designed with proper controls
3. Minimum sample size was met (4+ data points per condition)
4. Winner is declared with confidence assessment
5. Learning is documented for the always-on learnings doc

Pitfalls

• Declaring winner from 1 data point per condition (anecdote, not test)
• Changing multiple variables at once (can't attribute result)
• Testing during abnormal weeks (holiday, launch, crisis)
• Cherry-picking data (confirmation bias)

Verification

Would you bet $1,000 of the founder's money on this result being reproducible? If not, don't change strategy based on it yet.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: gokulb20
• Source: gokulb20/Crewm8-Social-Media-Manager-Skill-Graph
• License: MIT
• Homepage: https://crewm8.ai

Install and usage instructions live in the source repository linked above.