# Overfitting Prevention > A Claude skill from brainbytes-dev/everything-claude-trading. - **Type:** Skill - **Install:** `agentstack add skill-brainbytes-dev-everything-claude-trading-overfitting-prevention` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [brainbytes-dev](https://agentstack.voostack.com/s/brainbytes-dev) - **Installs:** 0 - **Category:** [Finance & Payments](https://agentstack.voostack.com/c/finance-and-payments) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [brainbytes-dev](https://github.com/brainbytes-dev) - **Source:** https://github.com/brainbytes-dev/everything-claude-trading/tree/main/skills/backtesting/overfitting-prevention ## Install ```sh agentstack add skill-brainbytes-dev-everything-claude-trading-overfitting-prevention ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Overfitting Prevention ## When to Activate - Evaluating whether a backtest result reflects genuine alpha or data mining artifacts - Applying statistical tests to adjust for multiple hypothesis testing - Implementing cross-validation methods designed for financial time series - Assessing the minimum backtest length and statistical significance of strategy performance - Distinguishing between robust strategies and curve-fitted illusions ## Core Concepts ### What Is Overfitting in Trading? **Definition:** A strategy is overfit when it captures noise (random patterns) in historical data rather than signal (persistent patterns), resulting in poor live performance despite excellent backtest results. **Symptoms of Overfitting:** ``` 1. Exceptional backtest performance (Sharpe > 2.5 for daily strategies is suspicious) 2. Many tuned parameters relative to data points 3. Performance degrades sharply out-of-sample 4. Strategy only works on one specific asset or time period 5. Small parameter changes cause large performance changes 6. Strategy requires frequent re-optimization to maintain performance 7. Complex, unintuitive logic with no economic rationale 8. Backtest includes survivorship bias, lookahead bias, or unrealistic fills ``` **Overfitting Probability:** ``` With N independent strategy configurations tested: P(at least one appears significant at p=0.05) = 1 - (0.95)^N N=1: 5% false positive rate N=10: 40% false positive rate N=20: 64% false positive rate N=50: 92% false positive rate N=100: 99.4% false positive rate Implication: If you test 100 parameter combinations, you are almost guaranteed to find one that "works" even on random data. This is why multiple testing adjustment is essential. ``` ### Deflated Sharpe Ratio (Bailey & Lopez de Prado) **Problem:** Standard Sharpe ratio does not account for the number of strategies tested (selection bias) or the statistical properties of returns (skewness, kurtosis). **Deflated Sharpe Ratio (DSR):** ``` DSR adjusts the Sharpe ratio for: 1. Number of trials (strategies tested) 2. Data length (more data = more reliable) 3. Skewness (negative skew inflates Sharpe) 4. Kurtosis (fat tails inflate Sharpe) DSR = Prob(SR > SR_benchmark | trials, skew, kurtosis, T) Using the haircut formula: SR_required = SR_observed * sqrt(1 - gamma * skewness / (6 * T) + (kurtosis - 3) / (24 * T)) Where gamma depends on the number of trials and significance level Practical interpretation: - If you tested 50 strategies and the best has Sharpe = 1.5: DSR might show this is only significant at p=0.15 (not significant) - If you tested 3 strategies and the best has Sharpe = 1.2: DSR might show this is significant at p=0.02 (significant) - Rule of thumb: record ALL strategies tested, not just the ones that worked ``` **Minimum Required Sharpe:** ``` Given N trials and desired significance p=0.05: SR_min ≈ sqrt(2 * ln(N)) * (1/sqrt(T)) + E[max(SR)] under null Approximate minimum Sharpe for significance (T=252 daily obs per year, 5 years): N=1: SR_min ≈ 0.40 N=10: SR_min ≈ 0.70 N=50: SR_min ≈ 0.95 N=100: SR_min ≈ 1.10 N=1000: SR_min ≈ 1.40 This means: if you tested 100 strategies, only those with Sharpe > 1.10 should be considered potentially significant. ``` ### Combinatorial Purged Cross-Validation (CPCV) **Problem with Standard k-Fold CV:** - Financial returns are autocorrelated — standard CV leaks information between folds - Training on data adjacent to test data creates lookahead bias - Standard CV overestimates performance **CPCV (Lopez de Prado):** ``` Key innovations: 1. Purging: remove observations from training set that are near the test set boundary - Purge window = max holding period of the strategy - Prevents lookahead: if strategy holds for 5 days, purge 5 days around test boundaries 2. Embargo: additional buffer after purging - Prevents serial correlation from leaking information - Embargo period = 1-2x the autocorrelation decay period 3. Combinatorial: instead of sequential folds, use all possible combinations - N groups, select k for testing, use remaining for training - C(N,k) combinations provides many more test paths - Each path is a valid walk-forward simulation Implementation: - Split data into N groups (e.g., N=10) - For each combination of k test groups (e.g., k=2): - Remove test groups from training data - Purge observations near test group boundaries - Apply embargo period - Train on remaining data - Evaluate on test groups - Average performance across all C(N,k) paths ``` **CPCV vs Standard Walk-Forward:** ``` Walk-forward: produces 1 OOS path (single sequence of OOS results) CPCV: produces C(N,k) OOS paths (many sequences) Advantages of CPCV: - More statistically robust (larger sample of test paths) - Can estimate distribution of performance, not just mean - Probability of loss (% of paths with negative return) is directly observable - Less dependent on specific IS/OOS split choice Example: N=10, k=2 -> C(10,2) = 45 test paths If 40/45 paths are profitable: strong evidence of genuine edge If 25/45 paths are profitable: strategy likely fragile or overfit ``` ### White's Reality Check and Hansen's SPA Test **White's Reality Check (2000):** ``` Purpose: Test if the BEST strategy among many is significantly better than a benchmark, accounting for data snooping. Method: 1. Define null hypothesis: best strategy is no better than benchmark 2. Bootstrap the data (resample returns with replacement) 3. For each bootstrap sample, find the best strategy's performance 4. Build distribution of "best strategy performance under null" 5. Compare actual best strategy to this distribution 6. p-value = fraction of bootstrap samples where best >= actual If p 1.5. For daily strategies: multiply by 252 to get minimum observations For monthly strategies: multiply by 12 ``` ### Degrees of Freedom Penalty **Concept:** Each additional parameter or rule in a strategy consumes a degree of freedom, reducing the reliability of the backtest. ``` Adjusted Sharpe ≈ SR * sqrt(1 - k/N) Where k = number of free parameters, N = number of independent observations Example: - Strategy with 5 parameters, 1000 daily observations - SR = 1.5 - Adjusted SR = 1.5 * sqrt(1 - 5/1000) = 1.5 * 0.9975 = 1.496 (minimal penalty — sufficient data) - Strategy with 20 parameters, 200 daily observations - SR = 2.0 - Adjusted SR = 2.0 * sqrt(1 - 20/200) = 2.0 * 0.949 = 1.90 (noticeable penalty — not enough data for this many parameters) Rule of thumb: observations per parameter should be > 50 (ideally >100) for reliable estimation ``` ## Methodology ### Overfitting Detection Checklist 1. **Count all trials** — every parameter combination, strategy variant, and data exploration step 2. **Calculate DSR** — adjust the best Sharpe ratio for number of trials and return distribution 3. **Run CPCV** — evaluate the distribution of OOS paths, not just the best path 4. **Check parameter stability** — do parameters change drastically in walk-forward windows? 5. **Assess economic rationale** — does the strategy exploit a known, persistent inefficiency? 6. **Test on alternative data** — different time periods, assets, or frequencies 7. **Simplify** — can you reduce parameters and maintain most of the performance? ### Strategy Acceptance Criteria ``` Tier 1 (Deploy with full allocation): - DSR significant at p 80% of paths profitable - Walk-forward WFE > 0.6 - Economic rationale is clear and documented - Works on at least 2 independent datasets/assets Tier 2 (Deploy with reduced allocation, monitor closely): - DSR significant at p 65% of paths profitable - Walk-forward WFE > 0.4 - Economic rationale exists but is less clear Tier 3 (Paper trade only, continue research): - DSR significant at p 50% of paths profitable - Walk-forward WFE SR_min (1.25): passes DSR threshold - But: with negative skew (-0.8) and excess kurtosis (4.2): DSR adjusts down further to SR_required ≈ 1.45 - 1.85 > 1.45: still passes (strategy is likely genuine) If best SR were 1.35 instead of 1.85: - 1.35 > 1.25 but 1.35 C(8,2) = 28 paths Purge window: 5 days (max holding period) Embargo: 2 days Results across 28 paths: - Paths with positive return: 24/28 (86%) - Paths with Sharpe > 0.5: 20/28 (71%) - Paths with Sharpe > 1.0: 12/28 (43%) - Average Sharpe across paths: 0.82 - Std of Sharpe across paths: 0.55 - Worst path Sharpe: -0.35 - Best path Sharpe: 1.95 Assessment: 86% profitable paths with average Sharpe 0.82 is strong evidence. The worst path (-0.35) shows the strategy can underperform for extended periods but losses are limited. Strategy meets Tier 1 acceptance criteria. ``` ### Example 3: Simplification Test ``` Original strategy: 8 parameters, Sharpe = 2.1 Simplified versions: - Remove 2 parameters (fix at defaults): Sharpe = 1.85 (12% drop) - Remove 4 parameters: Sharpe = 1.55 (26% drop) - Remove 6 parameters (2 remaining): Sharpe = 1.20 (43% drop) Analysis: - Core edge is captured by 4 parameters (Sharpe 1.55) - Additional 4 parameters add 0.55 Sharpe — much of this is likely overfitting - With 4 parameters, DSR is more lenient (fewer trials per parameter) - 4-parameter version has better parameter stability in walk-forward Decision: deploy the 4-parameter version (Sharpe 1.55) instead of the 8-parameter version (Sharpe 2.1). Expected live Sharpe: 1.0-1.2 (after realistic degradation) vs 8-parameter expected live Sharpe: 0.8-1.0 (more degradation from overfitting) ``` ## Quality Gate Before accepting a strategy as not overfit, verify: - [ ] Total number of configurations tested is documented honestly (including those tried and discarded) - [ ] Deflated Sharpe Ratio is significant at p 80% of paths profitable (with proper purging and embargo) - [ ] Walk-forward efficiency ratio > 0.6 across multiple windows - [ ] Minimum backtest length is satisfied for the observed Sharpe ratio - [ ] Parameter count is justified by data quantity (observations/parameters > 50) - [ ] Strategy has a plausible economic rationale — not just a statistical artifact - [ ] Simplification tested — performance survives removal of non-essential parameters - [ ] Results validated on at least one independent dataset or time period - [ ] Return distribution is examined — negative skew and fat tails are accounted for in significance testing - [ ] No obvious biases: survivorship bias, lookahead bias, unrealistic execution assumptions ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [brainbytes-dev](https://github.com/brainbytes-dev) - **Source:** [brainbytes-dev/everything-claude-trading](https://github.com/brainbytes-dev/everything-claude-trading) - **License:** MIT Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-brainbytes-dev-everything-claude-trading-overfitting-prevention - Seller: https://agentstack.voostack.com/s/brainbytes-dev - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.