Trading Strategy Review

Trading Strategy & Backtest Review

A methodology for reviewing a trading strategy, a backtest, or strategy code (Python, Pine, or any language) for the failure modes that make a backtest look great in a chart and lose money in production. The output is a severity-rated findings report with a concrete fix for each problem.

This skill is grounded in the established backtest-overfitting literature, primarily the work of David H. Bailey and Marcos López de Prado — "The Probability of Backtest Overfitting" (PBO), "The Deflated Sharpe Ratio", "Pseudo-Mathematics and Financial Charlatanism", and "The Seven Sins of Quantitative Investing." The central insight of that literature: if you try enough strategy configurations, a spectacular in-sample backtest is the expected outcome even when the strategy has zero real edge. Most of the checklist below exists to detect exactly that.

> Not financial advice. This skill performs an educational, methodological review of > a strategy or backtest. It is not investment advice, not a recommendation to trade, and > not a guarantee of any outcome. Past backtest performance — even after every fix here — > does not guarantee live results. The user is solely responsible for their own > trading and risk decisions.

---

When to Activate

Activate this skill when the user:

• Shares a trading strategy idea, ruleset, or signal and asks whether it is sound.
• Shares backtest code (Python/pandas/backtrader/vectorbt/zipline, Pine Script,

or any language) and asks for a review.

• Shares backtest results — an equity curve, a metrics table (Sharpe, CAGR, max

drawdown, win rate), or a screenshot from TradingView / a backtesting platform.

• Asks for a "pre-deployment check," "is this overfit?", "why might this fail live?", or a

risk review before putting real capital behind a strategy.

If the user only shares results with no code, you can still run Steps 1, 2, 4 and 5 and the metrics sanity check; flag that code-level confirmation is needed for several findings.

---

Step 1: Scope & Context

Before reviewing, establish context. Ask for anything missing — these answers change the severity of nearly every finding. If the user cannot answer, note the gap as an Info finding and assume the worst reasonable case.

| Question | Why it matters | |---|---| | Asset class & instrument (equities, futures, FX, crypto, options) | Drives cost model, liquidity, borrow/funding, session boundaries. | | Timeframe / bar size (tick, 1m, 1h, daily) | Sets the realistic Sharpe ceiling and how dangerous look-ahead is. | | Backtest window & in-sample period | Short windows + one regime = fragile. | | Number of parameters in the strategy | More knobs → more overfitting capacity. | | Number of strategies / configs tried (the "trials") | The single biggest driver of false discovery; needed for Deflated Sharpe / PBO. | | Out-of-sample / walk-forward done? | If everything is in-sample, results are unverified. | | Live or paper track record? | Live > paper > backtest in evidentiary weight. | | Capital, position size, leverage | Determines risk-of-ruin and market-impact realism. | | Costs modeled? (commission, slippage, spread, borrow, funding) | Frictionless backtests are the most common silent killer. |

---

Step 2: Severity Model

Rate every finding with this scale. The driving question is always: "Does this change the decision to trade, or just the precision of the estimate?"

| Severity | Definition | Examples | |---|---|---| | Critical | Invalidates the results entirely, or exposes the account to ruin. The backtest cannot be trusted and/or the strategy can blow up. | Look-ahead bias in the signal; zero transaction costs on a high-turnover strategy; no risk limit / unbounded position sizing; survivorship-biased universe; results are 100% in-sample with hundreds of trials. | | High | Materially inflates performance or materially understates risk; likely flips the strategy from "edge" to "no edge" once corrected. | Optimistic fills (mid/close, no slippage); over-parameterization without OOS; ignoring borrow/funding; regime dependence on a single bull market. | | Medium | Biases results in a meaningful but recoverable way; needs correction before sizing real money. | Curve-fit stop/target levels; survivorship in a secondary filter; small trade count weakening significance; timezone/session edge cases. | | Low | Minor realism or robustness gap; unlikely to flip the conclusion but should be fixed. | Rounding/contract-size handling; minor parameter sensitivity; cost estimate slightly optimistic. | | Info | Context, missing information, or good practice to confirm — not a defect per se. | "Number of trials not disclosed"; "confirm data is point-in-time"; "consider regime-tagging the equity curve." |

A single Critical finding means do not deploy until fixed, regardless of how good the metrics look.

---

Step 3: Failure-Mode Checklist

Walk every item. For each, you have: what it is → how to detect it → why it inflates performance → the fix. Cover at minimum the 16 modes below.

3.1 Look-ahead bias (usually Critical)

• What: The strategy uses information at bar t that was not actually available until

t+1 or later — future prices, end-of-bar values used intrabar, same-bar close used to decide a same-bar entry, full-series statistics (mean, z-score, scaler) computed over data that includes the future.

• Detect: Signals that index the future (close[t+1], df['close'].shift(-1),

.shift(-n) feeding a feature); indicators computed on the entire series then used at earlier bars; fillna(method='bfill'); a StandardScaler/MinMaxScaler fit on the whole dataset before the train/test split; using a bar's high/low to decide entries that also assume the favorable extreme was filled; resampling that leaks the closing bar.

• Why it inflates: Trading with tomorrow's information is the most powerful "edge" there

is — and it is entirely fake. It typically produces unrealistically smooth equity curves and Sharpe ratios that are physically impossible.

• Fix: Use point-in-time data and a frozen, as-of view of every input. Lag all

signals by one bar relative to execution. Fit any normalizer/scaler inside the walk-forward fold on training data only. Decide on bar t's closed values, execute at bar t+1's open (or next available price).

```python # BAD — decides and fills on the same bar's close (look-ahead) signal = (df['close'] > df['sma']).astype(int) df['ret'] = signal * df['close'].pct_change() # uses same-bar close

# GOOD — decide on closed bar t, execute next bar signal = (df['close'] > df['sma']).astype(int) df['position'] = signal.shift(1) # act on prior bar's signal df['ret'] = df['position'] * df['close'].pct_change() ```

3.2 Survivorship bias (High–Critical)

• What: The universe only contains instruments that survived to today — delisted,

bankrupt, merged, or relegated names are missing.

• Detect: Universe pulled from a current index membership / current ticker list;

equities backtest with no delisting returns; crypto backtest excluding dead coins; using "today's S&P 500" over a 20-year history.

• Why it inflates: You systematically exclude the losers. Buy-and-hold and especially

mean-reversion strategies look far better than reality because the names that went to zero were never in the test.

• Fix: Use a point-in-time, survivorship-free universe with historical constituents

and delisting/bankruptcy returns. Reconstruct index membership as of each date.

3.3 Data-snooping / overfitting (too many params, in-sample tuning) (Critical)

• What: Parameters were chosen by maximizing performance on the same data used to report

it; or the strategy has so many free parameters it can fit noise.

• Detect: Grid/optimizer over many parameters reporting only the best result; no

held-out data; parameter count high relative to trade count; "we tried a few and kept the best."

• Why it inflates: With enough knobs and enough trials, you can fit historical noise

perfectly. Per Bailey–López de Prado, the expected maximum in-sample Sharpe across N independent trials grows roughly like √(2·ln N) even when true Sharpe is zero — so a great backtest is the default outcome of a search, not evidence of edge.

• Fix: Out-of-sample holdout + walk-forward analysis. Minimize parameters

(parsimony). Report the Deflated Sharpe Ratio and PBO (below). Pre-register the rule before testing where possible.

3.4 Multiple testing / p-hacking (Critical)

• What: Many configurations, indicators, or universes were tried; only the winner is

reported. The reported Sharpe is a maximum over trials, not a single draw.

• Detect: "Selected the best of K"; many indicators/lookbacks scanned; the count of

trials is undisclosed or large.

• Why it inflates: The more you try, the higher the best in-sample result purely by

chance. A nominal Sharpe of 2.0 over 1,000 trials may be statistically indistinguishable from luck.

• Fix: Deflated Sharpe Ratio (DSR) — adjust the observed Sharpe for the number of

trials, the variance of trial Sharpes, and the non-normality (skew/kurtosis) of returns; DSR is the probability the true Sharpe > 0 after that haircut. Probability of Backtest Overfitting (PBO) via Combinatorially Symmetric Cross-Validation (CSCV): the fraction of splits where the in-sample-best config underperforms the median out-of-sample. PBO > 0.5 means the selection process is overfit. Always report how many trials were run.

3.5 Unrealistic fills (High–Critical)

• What: Orders assumed to fill at prices you could not actually get — mid, close,

best-case high/low of the bar, or instantly with no queue.

• Detect: Backtest fills at the signal bar's close; limit orders assumed filled

whenever price merely touched the level; market-on-open with no gap handling; no spread.

• Why it inflates: Best-case fills quietly add returns every single trade; the effect is

largest for high-frequency / high-turnover strategies.

• Fix: Model the bid/ask spread; fill market orders at the next bar's open (or

worse); for limit orders, require the price to trade through the level, not just touch it, and account for queue position. Add conservative slippage (below).

3.6 Slippage & commission omitted/under-modeled (High–Critical)

• What: No (or token) per-trade costs.
• Detect: Cost = 0; commission set but slippage = 0; costs not scaled by turnover.
• Why it inflates: A strategy with a 5 bps gross edge and 200% monthly turnover is

guaranteed to lose after realistic costs, yet looks profitable frictionless.

• Fix: Apply realistic commission + spread + slippage per fill, scaled by turnover.

A useful sanity check: compute the break-even cost (the per-trade cost at which the strategy's edge disappears) and ask whether real-world costs exceed it.

``python COST_BPS = 5 # round-trip commission + half-spread + slippage, in basis points df['turnover'] = df['position'].diff().abs() df['net_ret'] = df['gross_ret'] - df['turnover'] * (COST_BPS / 1e4) ``

3.7 Market impact & liquidity ignored (High)

• What: Assumes you can trade any size at the quoted price with no impact.
• Detect: Position sizes large vs. average daily volume (ADV); thin instruments; no cap

on participation rate.

• Why it inflates: Real orders move the price against you; the backtest captures alpha

that evaporates at size.

• Fix: Cap order size to a fraction of ADV (e.g. ≤1–5% participation), add a

size-dependent impact cost model (e.g. square-root impact), and stress-test at the capital you actually intend to deploy.

3.8 Regime dependence (High)

• What: The strategy only works in one market regime (one long bull market, one

low-vol period, one rate environment) present in the backtest.

• Detect: Backtest spans a single regime; equity curve is one smooth uptrend; no

drawdowns through 2008/2020/2022-type stress.

• Why it inflates: It is a bet on the regime continuing, not a durable edge.
• Fix: Test across multiple regimes (bull/bear/high-vol/low-vol/rate cycles), tag

performance by regime, and run out-of-sample on a different regime than the one tuned on.

3.9 No out-of-sample / walk-forward validation (Critical)

• What: All reported performance is in-sample.
• Detect: No train/test split; no holdout; no walk-forward.
• Why it inflates: In-sample performance is an upper bound, not an estimate of the

future.

• Fix: Walk-forward analysis — repeatedly tune on a rolling/anchored window and

evaluate on the immediately following untouched window; concatenate the out-of-sample segments to get an honest equity curve. Keep a final lockbox holdout never used for tuning.

3.10 Position sizing & risk-of-ruin (Critical)

• What: Sizing is ad hoc, fixed-fractional too large, or unbounded; no account-level

risk limit.

• Detect: "All-in" sizing; no max position; bet size that risks a large fraction of

capital per trade; ignoring the probability of a terminal drawdown.

• Why it inflates / endangers: Even a positive-edge strategy bankrupts with too-large

bets due to the variance drag and path dependence; risk-of-ruin can be high even when expectancy is positive.

• Fix: Volatility targeting (size to a target portfolio vol) and/or a **fractional

Kelly stake (e.g. ¼–½ Kelly, never full Kelly), with hard per-trade and per-account risk caps. Estimate risk-of-ruin** explicitly given edge, variance, and bet size.

3.11 Leverage & path dependence (High–Critical)

• What: Leverage applied to the return series without modeling margin, intraday path,

or forced liquidation.

• Detect: Returns simply multiplied by a leverage factor; no margin call / liquidation

logic; ignoring volatility drag of leveraged compounding.

• Why it inflates: A 3× return series ignores that an intraday spike can liquidate you

before the favorable close; leveraged compounding suffers volatility decay.

• Fix: Model margin requirements, intraday path, and liquidation; account for

volatility drag; cap leverage; backtest on the actual path, not just close-to-close.

3.12 Funding & borrow costs ignored (High for shorts/crypto/leverage)

• What: Short rebate/borrow fees, hard-to-borrow constraints, perpetual-swap **funding

rates**, and carry are omitted.

• Detect: Short strategy with no borrow cost; crypto perp strategy with no funding;

long-short with no financing.

• Why it inflates: Borrow/funding can exceed the gross edge, especially on shorts and

crypto perps where funding flips sign and is paid continuously.

• Fix: Apply borrow fees (and hard-to-borrow availability) on shorts and **funding

rates** on perps; include financing on leveraged/long-short books.

3.13 Timezone / session-boundary bugs (Medium–High)

• What: Bars aligned to the wrong timezone, daylight-saving shifts, or session

open/close handled incorrectly; mixing exchange time and UTC.

• Detect: Naive timestamps; DST not handled; daily bars rolled at the wrong hour;

overnight gaps treated as intraday moves; 24/7 crypto vs. session-based equities mismatch.

• Why it inflates: Misaligned bars can accidentally introduce look-ahead or attribute

the overnight gap to an intraday signal.

• Fix: Use timezone-aware timestamps, define sessions explicitly per instrument,

handle DST, and verify bar boundaries against the exchange calendar.

3.14 Repainting indicators (Pine / charting)

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Viprasol-Tech
• Source: Viprasol-Tech/trading-strategy-review
• License: MIT

Install and usage instructions live in the source repository linked above.