Statistics Fundamentals

Statistics Fundamentals

Conventions and Decision Rules

Sample variance: use n-1

When estimating variance or standard deviation from a sample of returns, divide by n - 1 (Bessel's correction), not n. Dividing by n systematically underestimates dispersion. Standard deviation of returns is "volatility"; annualize with sigma_annual = sigma_period * sqrt(periods_per_year) (e.g., * sqrt(12) for monthly, * sqrt(252) for daily).

Normality testing: Jarque-Bera and its limits

JB = (n/6) * (skew^2 + excess_kurtosis^2 / 4), distributed chi-squared with 2 df under the null of normality (5% critical value: 5.99).

Low-power caveat: with small samples (n below roughly 50), JB rarely rejects even for clearly non-normal data — failing to reject is weak evidence of normality, not confirmation. With large samples, financial return series almost always reject due to fat tails and (for equities) negative skewness. Treat the test as a screen, and pair it with a look at the actual skew/kurtosis magnitudes and extreme observations.

Covariance estimation and Ledoit-Wolf shrinkage

The sample covariance matrix Sigma_hat = (1/(n-1)) (X - X_bar)^T (X - X_bar) becomes poorly conditioned or singular when the number of assets p approaches the number of observations n. Plugging it into a mean-variance optimizer then produces extreme, unstable weights that flip with small data changes.

Shrinkage blends the sample matrix toward a structured target:

$$\hat{\Sigma}_{shrunk} = \delta \cdot F + (1 - \delta) \cdot \hat{\Sigma}$$

where F is the target (e.g., scaled identity) and delta is the shrinkage intensity. Ledoit-Wolf (2004) derives the delta that minimizes expected squared Frobenius distance to the true covariance matrix, trading a little bias for a large variance reduction — yielding better-conditioned, invertible matrices and stable portfolio weights.

Note: the bundled script's shrunk_covariance implements a simplified shrinkage-intensity estimate, not the full Ledoit-Wolf estimator. For production work use sklearn.covariance.LedoitWolf.

Regression diagnostics (CAPM and factor models)

For the single-factor CAPM regression R_i - R_f = alpha + beta * (R_m - R_f) + epsilon:

• beta = rho * sigma_i / sigma_m (market sensitivity); alpha is the risk-adjusted excess return.
• In a single-factor regression, R^2 = rho^2.
• Judge coefficients by t-statistics (t = coefficient / SE); with n - 2 df, |t| above roughly 2 indicates 5% significance. A positive alpha point estimate with |t| s = 2.195% per month

Ann. vol = 2.195% * sqrt(12) = 7.60% Skewness = -0.045 (bias-corrected; near symmetric) Ex. kurt = -0.42 (bias-corrected; lighter tails than normal)

JB = (12/6) * ((-0.045)^2 + (-0.42)^2 / 4) = 0.09

JB = 0.09   t(alpha) = 0.236 / 0.452 = 0.52
SE(beta)  = 2.213% / (3.8% * sqrt(23)) = 0.121  ->  t(beta) = 0.939 / 0.121 = 7.74

With 22 df, the 5% two-tailed critical t is 2.074. Beta is highly significant (7.74 >> 2.074); alpha is not significant (0.52 < 2.074) — despite the positive point estimate, the sample cannot distinguish it from zero.

Common Pitfalls

• Using population variance instead of sample variance: always use n - 1 (Bessel's correction) when estimating from a sample.
• Assuming normality when financial returns have fat tails: equity returns typically show negative skewness and positive excess kurtosis; normal-based models (standard VaR) underestimate tail risk. Use Student-t or non-parametric methods.
• Ignoring non-stationarity: return distributions shift over time (regime changes, volatility clustering). Rolling-window estimation or GARCH may be more appropriate than full-sample statistics.
• Overfitting with too many regressors: R-squared always rises with added factors; use adjusted R-squared, information criteria, or cross-validation.
• Unstable covariance matrices with small samples: when p approaches or exceeds n, apply Ledoit-Wolf shrinkage or factor-based covariance models before optimizing.

Running the Script

scripts/statistics_fundamentals.py provides descriptive_stats, covariance_matrix, correlation_matrix, shrunk_covariance (simplified Ledoit-Wolf — see note above), ols_regression, rolling_regression, bootstrap_mean, and jarque_bera_test.

• Run: uv run scripts/statistics_fundamentals.py (PEP 723 inline metadata resolves numpy and scipy), or python3 scripts/statistics_fundamentals.py with numpy/scipy installed.
• Bare invocation (or --verify) prints a demo on synthetic data and asserts the Example 1 worked-example values above (mean 0.8167, std 2.195, JB 0.09 on the 12-month series), exiting nonzero on any mismatch.
• --help lists the available functions and import usage.
• For programmatic use, import rather than run: from statistics_fundamentals import descriptive_stats, ols_regression.

Cross-References

• return-calculations (core plugin, Layer 0): Arithmetic and geometric mean returns, log returns for statistical modeling
• time-value-of-money (core plugin, Layer 0): Discount rate estimation via CAPM regression; NPV and IRR calculations use statistical inputs

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: JoelLewis
• Source: JoelLewis/finance_skills
• License: MIT

Install and usage instructions live in the source repository linked above.