Statistics

Core Principles

1. Question first, method second — Define what you're trying to learn before selecting a test. The question determines the method, not the other way around.
2. Check assumptions before computing — Every test has assumptions (normality, independence, equal variance). Violating them silently produces misleading results.
3. Effect size matters more than p-values — A statistically significant result can be practically meaningless. Always report the magnitude of the effect alongside significance.
4. Confidence intervals over point estimates — A mean of 42 is less useful than "42 ± 5 (95% CI)". Always quantify uncertainty.
5. Don't test what you can describe — If the question is "what does the data look like?", use descriptive statistics. Hypothesis tests answer "is this difference real?", not "what is the data?"
6. One question, one test — Running multiple tests on the same data inflates false positives. If you must run multiple comparisons, correct for it (Bonferroni, Holm, or FDR).
7. Honest reporting — Report all results, not just significant ones. Pre-register your hypothesis when possible. State limitations clearly.

---

Method Selection Guide

Step 1: Classify Your Question

| Question Type | What You're Asking | Go To | |--------------|-------------------|-------| | Describe | What does the data look like? | Descriptive Statistics | | Compare | Are these groups different? | Comparison Tests | | Relate | Do these variables move together? | Correlation & Regression | | Predict | What will happen next? | Regression & Modeling (beyond this skill — use ML) | | Classify | Does this differ from expected? | Goodness-of-Fit Tests |

Step 2: Identify Your Data Structure

| Factor | Options | Why It Matters | |--------|---------|---------------| | Number of groups | 1, 2, or 3+ | Determines which test family to use | | Paired or independent? | Same subjects measured twice vs. different subjects | Paired tests are more powerful but require matched data | | Variable type | Continuous (interval/ratio) vs. categorical (nominal/ordinal) | Determines parametric vs. non-parametric | | Sample size | Small ( 1 = substantially skewed | | Kurtosis | scipy.stats.kurtosis | > 0 = heavy tails, Desired power: 0.80 (standard) or 0.90 (high confidence) Significance level (α): 0.05 Minimum detectable effect (MDE): Required sample size per group: Expected duration to collect:

**Rules of thumb**:
- Power 
**Duration**: 
**Sample size**: Control: , Variant: 

**Primary metric**: 
| Group | Value | 95% CI |
|-------|-------|--------|
| Control |  | [, ] |
| Variant |  | [, ] |

**Relative lift**:  [, ]
**p-value**: 
**Effect size**: 

**Guardrail metrics**: 

**Decision**: 
**Reasoning**: 

---

Assumption Validation Checklist

Before applying any parametric test, verify:

| Assumption | How to Check | If Violated | |-----------|-------------|-------------| | Normality | Shapiro-Wilk test, Q-Q plot, histogram | Use non-parametric alternative | | Equal variance | Levene's test, F-test | Use Welch's t-test or rank-based test | | Independence | Study design (not a statistical test) | Use paired/repeated measures test | | Linearity (regression) | Scatter plot of residuals vs. fitted | Transform variables or use non-linear model | | No multicollinearity (regression) | VIF Population: Hypothesis (if testing):

• H₀:
• H₁:

Practical significance threshold:

### Step 2: Profile the Data

```markdown
## Data Profile

**Source**: 
**Sample size**: 
**Variables**: 
  - :  — 
  - :  — 
**Missing data**: 
**Distribution**: 
**Assumption checks**: 

Step 3: Select the Method

Use the selection guide above. Document:

## Method Selection

**Test**: 
**Why**: 
**Alternative considered**: 
**Correction applied**: 

Step 4: Compute and Interpret

## Results

**Test statistic**:  = 
**p-value**: 
**Effect size**:  =  ()
**Confidence interval**: [, ] at 

**Interpretation**: 

**Limitations**: 

---

Output Format

Every statistical analysis should produce:

1. Question — What you're trying to learn, with hypothesis if applicable
2. Data Profile — Summary of the data, assumption checks
3. Method Selection — Which test, why, and what alternatives were considered
4. Results — Test statistic, p-value, effect size, confidence interval
5. Interpretation — Plain-language meaning and practical significance
6. Limitations — What the result doesn't tell you

---

Regression Basics

Regression is the most common applied statistics method. Use this decision guide:

| Outcome Variable | Predictor(s) | Method | Use When | |-----------------|-------------|--------|----------| | Continuous | 1 continuous | Simple linear regression | Predicting one variable from another (e.g., revenue from ad spend) | | Continuous | Multiple | Multiple linear regression | Predicting with several factors; controlling for confounders | | Binary (0/1) | Any | Logistic regression | Predicting yes/no outcomes (e.g., churn, conversion) | | Count (0, 1, 2...) | Any | Poisson regression | Predicting event counts (e.g., support tickets per day) |

Before running regression, check:

• Linearity (scatter plot of residuals vs. fitted)
• Independence of residuals
• Homoscedasticity (constant variance of residuals)
• No multicollinearity (VIF X?" directly |

| Sequential testing / continuous monitoring | Bayesian A/B tests allow peeking without inflating error rates | | Stakeholders struggle with p-values | "95% probability the effect is between 2% and 8%" is more intuitive |

Practical tools: pymc, arviz for general Bayesian analysis; Bayesian A/B testing via bayesian-testing or built-in platform features (Optimizely, LaunchDarkly).

Default stance: Use frequentist methods unless one of the above signals is present. Don't switch to Bayesian for complexity's sake.

---

Tooling Reference

| Library | Language | Best For | |---------|----------|----------| | scipy.stats | Python | Hypothesis tests, distributions, descriptive stats | | statsmodels | Python | Regression, ANOVA, time series, assumption diagnostics | | pingouin | Python | Clean API for t-tests, ANOVA, correlation, effect sizes | | scikit-learn | Python | Train/test splits, cross-validation, preprocessing | | pymc | Python | Bayesian modeling and inference | | power_analysis / statsmodels.stats.power | Python | Sample size and power calculations |

---

Common Pitfalls

| Pitfall | Why It Fails | |---------|-------------| | p-hacking | Running multiple tests and reporting only significant results inflates false positives. Pre-register your hypothesis. | | Ignoring effect size | p = 0.001 with Cohen's d = 0.05 means "we're very sure about a trivially small difference." Not actionable. | | Small sample, big claims | A study with n = 12 that finds p = 0.04 is fragile. One outlier changes the conclusion. | | Violating independence | Using the same users in both groups, or multiple measurements without paired tests. Results are invalid. | | Peeking at A/B tests | Checking daily and stopping when p < 0.05 dramatically inflates false positive rate. Use sequential testing if you must peek. | | Treating non-significant as "no effect" | Absence of evidence ≠ evidence of absence. You may be underpowered. Report power. | | Applying parametric tests to ordinal data | A Likert scale (1–5) is not continuous. Use non-parametric methods. | | Confusing correlation with causation | Pearson r = 0.8 does not mean X causes Y. It means they move together. | | Cherry-picking subgroups | "It wasn't significant overall, but it was significant for users aged 25–30 on Tuesdays." This is noise. | | Reporting without uncertainty | "Conversion rate is 4.2%" is less useful than "4.2% ± 0.8% (95% CI)." Always show the interval. |

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: michaelsvanbeek
• Source: michaelsvanbeek/personal-agent-skills
• License: MIT

Install and usage instructions live in the source repository linked above.