Experimentation
Designs, runs, and reads A/B tests and growth experiments — hypothesis, sample size, duration, and honest interpretation. Use this to plan a test, judge whether a result is real, build an experimentation program, decide what to test next, or diagnose why tests keep producing inconclusive or non-replicating results.
Install
$ agentstack add skill-cbrock84-headcount-experimentation ✓ scanned · ✓ verified, works with Claude Code, Cursor, and more.
Security review
✓ PassedNo issues found. Passed automated security review. · v0.1.0 How review works →
- ✓ Prompt-injection patterns
- ✓ Secret / credential exfiltration
- ✓ Dangerous shell & filesystem operations
- ✓ Untrusted network calls
- ✓ Known-malicious package signatures
What it can access
- ✓ Network access No
- ✓ Filesystem access No
- ✓ Shell / process execution No
- ✓ Environment & secrets No
- ✓ Dynamic code execution No
From automated source analysis of v0.1.0. “Used” means the capability is present in the source — more access means more to trust, not that it’s unsafe.
Verified badge
Passed review? Show it. Paste this badge into your README, it links to the public security report.
Links to your public security report.[](https://agentstack.voostack.com/security/report/skill-cbrock84-headcount-experimentation) Reliability & compatibility
Declared compatibility
Compatibility is declared by the source manifest. End-to-end runtime verification is coming, see below.
We're building live execution health for every listing: tool-call success rate, median latency, uptime, and last-checked timestamps, measured, not self-reported. It isn't live yet, so we don't show numbers we can't stand behind.
How agent discovery & health will work →About
Experimentation
Most A/B testing programs produce confident conclusions from insufficient data. The discipline is almost entirely in what you do before launch.
Before running
- Hypothesis with a mechanism. "Moving the pricing table above the fold will raise trial starts,
because visitors currently leave before seeing pricing." Not "let's try a green button."
- One primary metric, chosen in advance. Secondary metrics are context, never the verdict.
- Sample size calculated in advance, from your baseline rate and the smallest lift that would
change a decision. If the required sample is unreachable, do not run the test — decide by judgment and say so.
- Duration set in advance, covering at least one full weekly cycle, and two if the buying cycle
is long.
- Guardrail metrics that would make you reject a win: refunds, support volume, downstream
retention.
While running
Do not look at results and act on them mid-flight. Peeking and stopping at significance is the single most common way to generate false positives, and it is very effective at it.
Check only that the test is running correctly — even split, no broken variant, tracking firing.
Reading
- At the pre-set duration, not before, and not extended because it is nearly significant.
Extending until significance manufactures it.
- Significance is not size. A statistically significant 0.3% lift may not be worth shipping.
- Inconclusive is a real result and the most common one. It means the change did not matter
enough to detect, which is useful.
- Check the guardrails before declaring a win.
- Segment afterward for hypotheses only, never for verdicts. Slice enough ways and something is
always significant.
Program level
Test where the traffic and the leverage are. Most sites can only run a handful of adequately powered tests a year — spend them on structural questions, not button colors.
Keep a log of every test: hypothesis, result, decision. Without it, teams re-run the same tests every eighteen months and re-learn the same things.
Source & license
This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.
- Author: cbrock84
- Source: cbrock84/headcount
- License: MIT
- Homepage: https://cbrock84.github.io/headcount/
Install and usage instructions live in the source repository linked above.
Reviews
No reviews yet, be the first.
Write a review
Versions
- v0.1.0 Imported from the upstream source.