Statistical Analysis

Statistical Analysis

Overview

Statistical analysis is a systematic process for testing hypotheses and quantifying relationships. Conduct hypothesis tests (t-test, ANOVA, chi-square), regression, correlation, and Bayesian analyses with assumption checks and APA reporting. Apply this skill for academic research.

When to Use This Skill

This skill should be used when:

• Conducting statistical hypothesis tests (t-tests, ANOVA, chi-square)
• Performing regression or correlation analyses
• Running Bayesian statistical analyses
• Checking statistical assumptions and diagnostics
• Calculating effect sizes and conducting power analyses
• Reporting statistical results in APA format
• Analyzing experimental or observational data for research

---

Core Capabilities

1. Test Selection and Planning

• Choose appropriate statistical tests based on research questions and data characteristics
• Conduct a priori power analyses to determine required sample sizes
• Plan analysis strategies including multiple comparison corrections

2. Assumption Checking

• Automatically verify all relevant assumptions before running tests
• Provide diagnostic visualizations (Q-Q plots, residual plots, box plots)
• Recommend remedial actions when assumptions are violated

3. Statistical Testing

• Hypothesis testing: t-tests, ANOVA, chi-square, non-parametric alternatives
• Regression: linear, multiple, logistic, with diagnostics
• Correlations: Pearson, Spearman, with confidence intervals
• Bayesian alternatives: Bayesian t-tests, ANOVA, regression with Bayes Factors

4. Effect Sizes and Interpretation

• Calculate and interpret appropriate effect sizes for all analyses
• Provide confidence intervals for effect estimates
• Distinguish statistical from practical significance

5. Professional Reporting

• Generate APA-style statistical reports
• Create publication-ready figures and tables
• Provide complete interpretation with all required statistics

---

Workflow Decision Tree

Use this decision tree to determine your analysis path:

START
│
├─ Need to SELECT a statistical test?
│  └─ YES → See "Test Selection Guide"
│  └─ NO → Continue
│
├─ Ready to check ASSUMPTIONS?
│  └─ YES → See "Assumption Checking"
│  └─ NO → Continue
│
├─ Ready to run ANALYSIS?
│  └─ YES → See "Running Statistical Tests"
│  └─ NO → Continue
│
└─ Need to REPORT results?
   └─ YES → See "Reporting Results"

---

Test Selection Guide

Quick Reference: Choosing the Right Test

Use references/test_selection_guide.md for comprehensive guidance. Quick reference:

Comparing Two Groups:

• Independent, continuous, normal → Independent t-test
• Independent, continuous, non-normal → Mann-Whitney U test
• Paired, continuous, normal → Paired t-test
• Paired, continuous, non-normal → Wilcoxon signed-rank test
• Binary outcome → Chi-square or Fisher's exact test

Comparing 3+ Groups:

• Independent, continuous, normal → One-way ANOVA
• Independent, continuous, non-normal → Kruskal-Wallis test
• Paired, continuous, normal → Repeated measures ANOVA
• Paired, continuous, non-normal → Friedman test

Relationships:

• Two continuous variables → Pearson (normal) or Spearman correlation (non-normal)
• Continuous outcome with predictor(s) → Linear regression
• Binary outcome with predictor(s) → Logistic regression

Bayesian Alternatives: All tests have Bayesian versions that provide:

• Direct probability statements about hypotheses
• Bayes Factors quantifying evidence
• Ability to support null hypothesis
• See references/bayesian_statistics.md

---

Assumption Checking

Systematic Assumption Verification

ALWAYS check assumptions before interpreting test results.

Use the provided scripts/assumption_checks.py module for automated checking:

from scripts.assumption_checks import comprehensive_assumption_check

# Comprehensive check with visualizations
results = comprehensive_assumption_check(
    data=df,
    value_col='score',
    group_col='group',  # Optional: for group comparisons
    alpha=0.05
)

This performs:

1. Outlier detection (IQR and z-score methods)
2. Normality testing (Shapiro-Wilk test + Q-Q plots)
3. Homogeneity of variance (Levene's test + box plots)
4. Interpretation and recommendations

Individual Assumption Checks

For targeted checks, use individual functions:

from scripts.assumption_checks import (
    check_normality,
    check_normality_per_group,
    check_homogeneity_of_variance,
    check_linearity,
    detect_outliers
)

# Example: Check normality with visualization
result = check_normality(
    data=df['score'],
    name='Test Score',
    alpha=0.05,
    plot=True
)
print(result['interpretation'])
print(result['recommendation'])

What to Do When Assumptions Are Violated

Normality violated:

• Mild violation + n > 30 per group → Proceed with parametric test (robust)
• Moderate violation → Use non-parametric alternative
• Severe violation → Transform data or use non-parametric test

Homogeneity of variance violated:

• For t-test → Use Welch's t-test
• For ANOVA → Use Welch's ANOVA or Brown-Forsythe ANOVA
• For regression → Use robust standard errors or weighted least squares

Linearity violated (regression):

• Add polynomial terms
• Transform variables
• Use non-linear models or GAM

See references/assumptions_and_diagnostics.md for comprehensive guidance.

---

Running Statistical Tests

Python Libraries

Primary libraries for statistical analysis:

• scipy.stats: Core statistical tests
• statsmodels: Advanced regression and diagnostics
• pingouin: User-friendly statistical testing with effect sizes
• pymc: Bayesian statistical modeling
• arviz: Bayesian visualization and diagnostics

Example Analyses

T-Test with Complete Reporting

import pingouin as pg
import numpy as np

# Run independent t-test
result = pg.ttest(group_a, group_b, correction='auto')

# Extract results
t_stat = result['T'].values[0]
df = result['dof'].values[0]
p_value = result['p-val'].values[0]
cohens_d = result['cohen-d'].values[0]
ci_lower = result['CI95%'].values[0][0]
ci_upper = result['CI95%'].values[0][1]

# Report
print(f"t({df:.0f}) = {t_stat:.2f}, p = {p_value:.3f}")
print(f"Cohen's d = {cohens_d:.2f}, 95% CI [{ci_lower:.2f}, {ci_upper:.2f}]")

ANOVA with Post-Hoc Tests

import pingouin as pg

# One-way ANOVA
aov = pg.anova(dv='score', between='group', data=df, detailed=True)
print(aov)

# If significant, conduct post-hoc tests
if aov['p-unc'].values[0]  group2
prob_greater = np.mean(trace.posterior['difference'].values > 0)
print(f"P(μ₁ > μ₂ | data) = {prob_greater:.3f}")

# Plot posterior
az.plot_posterior(trace, var_names=['difference'], ref_val=0)

---

Effect Sizes

Always Calculate Effect Sizes

Effect sizes quantify magnitude, while p-values only indicate existence of an effect.

See references/effect_sizes_and_power.md for comprehensive guidance.

Quick Reference: Common Effect Sizes

| Test | Effect Size | Small | Medium | Large | |------|-------------|-------|--------|-------| | T-test | Cohen's d | 0.20 | 0.50 | 0.80 | | ANOVA | η²_p | 0.01 | 0.06 | 0.14 | | Correlation | r | 0.10 | 0.30 | 0.50 | | Regression | R² | 0.02 | 0.13 | 0.26 | | Chi-square | Cramér's V | 0.07 | 0.21 | 0.35 |

Important: Benchmarks are guidelines. Context matters!

Calculating Effect Sizes

Most effect sizes are automatically calculated by pingouin:

# T-test returns Cohen's d
result = pg.ttest(x, y)
d = result['cohen-d'].values[0]

# ANOVA returns partial eta-squared
aov = pg.anova(dv='score', between='group', data=df)
eta_p2 = aov['np2'].values[0]

# Correlation: r is already an effect size
corr = pg.corr(x, y)
r = corr['r'].values[0]

Confidence Intervals for Effect Sizes

Always report CIs to show precision:

from pingouin import compute_effsize_from_t

# For t-test
d, ci = compute_effsize_from_t(
    t_statistic,
    nx=len(group1),
    ny=len(group2),
    eftype='cohen'
)
print(f"d = {d:.2f}, 95% CI [{ci[0]:.2f}, {ci[1]:.2f}]")

---

Power Analysis

A Priori Power Analysis (Study Planning)

Determine required sample size before data collection:

from statsmodels.stats.power import (
    tt_ind_solve_power,
    FTestAnovaPower
)

# T-test: What n is needed to detect d = 0.5?
n_required = tt_ind_solve_power(
    effect_size=0.5,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative='two-sided'
)
print(f"Required n per group: {n_required:.0f}")

# ANOVA: What n is needed to detect f = 0.25?
anova_power = FTestAnovaPower()
n_per_group = anova_power.solve_power(
    effect_size=0.25,
    ngroups=3,
    alpha=0.05,
    power=0.80
)
print(f"Required n per group: {n_per_group:.0f}")

Sensitivity Analysis (Post-Study)

Determine what effect size you could detect:

# With n=50 per group, what effect could we detect?
detectable_d = tt_ind_solve_power(
    effect_size=None,  # Solve for this
    nobs1=50,
    alpha=0.05,
    power=0.80,
    ratio=1.0,
    alternative='two-sided'
)
print(f"Study could detect d ≥ {detectable_d:.2f}")

Note: Post-hoc power analysis (calculating power after study) is generally not recommended. Use sensitivity analysis instead.

See references/effect_sizes_and_power.md for detailed guidance.

---

Reporting Results

APA Style Statistical Reporting

Follow guidelines in references/reporting_standards.md.

Essential Reporting Elements

1. Descriptive statistics: M, SD, n for all groups/variables
2. Test statistics: Test name, statistic, df, exact p-value
3. Effect sizes: With confidence intervals
4. Assumption checks: Which tests were done, results, actions taken
5. All planned analyses: Including non-significant findings

Example Report Templates

Independent T-Test

Group A (n = 48, M = 75.2, SD = 8.5) scored significantly higher than
Group B (n = 52, M = 68.3, SD = 9.2), t(98) = 3.82, p  1000).

---

Bayesian Statistics

When to Use Bayesian Methods

Consider Bayesian approaches when:

• You have prior information to incorporate
• You want direct probability statements about hypotheses
• Sample size is small or planning sequential data collection
• You need to quantify evidence for the null hypothesis
• The model is complex (hierarchical, missing data)

See references/bayesian_statistics.md for comprehensive guidance on:

• Bayes' theorem and interpretation
• Prior specification (informative, weakly informative, non-informative)
• Bayesian hypothesis testing with Bayes Factors
• Credible intervals vs. confidence intervals
• Bayesian t-tests, ANOVA, regression, and hierarchical models
• Model convergence checking and posterior predictive checks

Key Advantages

1. Intuitive interpretation: "Given the data, there is a 95% probability the parameter is in this interval"
2. Evidence for null: Can quantify support for no effect
3. Flexible: No p-hacking concerns; can analyze data as it arrives
4. Uncertainty quantification: Full posterior distribution

---

Resources

This skill includes comprehensive reference materials:

References Directory

• testselectionguide.md: Decision tree for choosing appropriate statistical tests
• assumptionsanddiagnostics.md: Detailed guidance on checking and handling assumption violations
• effectsizesand_power.md: Calculating, interpreting, and reporting effect sizes; conducting power analyses
• bayesian_statistics.md: Complete guide to Bayesian analysis methods
• reporting_standards.md: APA-style reporting guidelines with examples

Scripts Directory

• assumption_checks.py: Automated assumption checking with visualizations
• comprehensive_assumption_check(): Complete workflow
• check_normality(): Normality testing with Q-Q plots
• check_homogeneity_of_variance(): Levene's test with box plots
• check_linearity(): Regression linearity checks
• detect_outliers(): IQR and z-score outlier detection

---

Best Practices

1. Pre-register analyses when possible to distinguish confirmatory from exploratory
2. Always check assumptions before interpreting results
3. Report effect sizes with confidence intervals
4. Report all planned analyses including non-significant results
5. Distinguish statistical from practical significance
6. Visualize data before and after analysis
7. Check diagnostics for regression/ANOVA (residual plots, VIF, etc.)
8. Conduct sensitivity analyses to assess robustness
9. Share data and code for reproducibility
10. Be transparent about violations, transformations, and decisions

---

Common Pitfalls to Avoid

1. P-hacking: Don't test multiple ways until something is significant
2. HARKing: Don't present exploratory findings as confirmatory
3. Ignoring assumptions: Check them and report violations
4. Confusing significance with importance: p < .05 ≠ meaningful effect
5. Not reporting effect sizes: Essential for interpretation
6. Cherry-picking results: Report all planned analyses
7. Misinterpreting p-values: They're NOT probability that hypothesis is true
8. Multiple comparisons: Correct for family-wise error when appropriate
9. Ignoring missing data: Understand mechanism (MCAR, MAR, MNAR)
10. Overinterpreting non-significant results: Absence of evidence ≠ evidence of absence

---

Getting Started Checklist

When beginning a statistical analysis:

• [ ] Define research question and hypotheses
• [ ] Determine appropriate statistical test (use testselectionguide.md)
• [ ] Conduct power analysis to determine sample size
• [ ] Load and inspect data
• [ ] Check for missing data and outliers
• [ ] Verify assumptions using assumption_checks.py
• [ ] Run primary analysis
• [ ] Calculate effect sizes with confidence intervals
• [ ] Conduct post-hoc tests if needed (with corrections)
• [ ] Create visualizations
• [ ] Write results following reporting_standards.md
• [ ] Conduct sensitivity analyses
• [ ] Share data and code

---

Support and Further Reading

For questions about:

• Test selection: See references/testselectionguide.md
• Assumptions: See references/assumptionsanddiagnostics.md
• Effect sizes: See references/effectsizesand_power.md
• Bayesian methods: See references/bayesian_statistics.md
• Reporting: See references/reporting_standards.md

Key textbooks:

• Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences
• Field, A. (2013). Discovering Statistics Using IBM SPSS Statistics
• Gelman, A., & Hill, J. (2006). Data Analysis Using Regression and Multilevel/Hierarchical Models
• Kruschke, J. K. (2014). Doing Bayesian Data Analysis

Online resources:

• APA Style Guide: https://apastyle.apa.org/
• Statistical Consulting: Cross Validated (stats.stackexchange.com)

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: zLanqing
• Source: zLanqing/codex-claude-academic-skills
• License: MIT

Install and usage instructions live in the source repository linked above.