Backtesting

Backtesting Best Practices

The mechanics of submitting a backtest are in the main SKILL.md under "Backtest Workflow". This page is about the judgment calls — picking a window that means something, telling signal from noise in the result, and knowing when to give up vs. iterate.

The trade-count bar (sample size first)

Trade count is the single most important number on a result page. Look at it before PnL, before Sharpe, before win rate.

| Trade count | Verdict | | ----------- | ---------------------------------------------------------------------------------------- | | A great backtest over the wrong window is a great fiction.

The window should answer: "if I had deployed this strategy on day one of this window, what would have happened?" — not "what's the prettiest curve I can fit?"

Cover at least one regime change

Pure bull, pure bear, sideways chop — your window should include at least two of the three. A 90-day backtest in a one-direction market is a 90-day cherry-pick. A momentum strategy that prints +50% over a +60% trending window has told you nothing about itself; it's just measured beta.

Useful default windows

• 6 months of 1h data, or
• 18 months of 1d data

Less than that and you're really looking at noise. More than that and Hyperliquid's history may not cover the pair (HL was launched in 2023; many alts have 500K | Warn user "could take 10+ minutes", longer poll intervals. |

Reference points:

• 1 pair × 15m × 90 days = 8.6K candles (fast)
• 3 pairs × 5m × 90 days = 78K candles (fast)
• 12 pairs × 1m × 90 days = 1.5M candles (slow — warn upfront)

Always allow the user to proceed. Just set expectations. Don't block on size.

Exponential polling cadence

When polling backtest_status on long runs:

• Polls 1-3: every 10s
• Polls 4-6: every 20s
• Polls 7-9: every 30s
• Polls 10+: every 60s

This prevents hitting the 20-step tool limit on million-candle backtests.

What headline numbers to trust (and not)

| Metric | Trust at | Notes | | ------------------ | --------------------------------------------------------- | --------------------------------------------------------------------------------- | | Total trades | Always look first. | Below 30 = ignore everything else. | | Total profit % | Useful for ranking, weak for forecasting. | Large windows + small per-trade edge can produce big PnL from luck. | | Win rate | OK above 50 trades. | A 60% WR on 8 trades is one good week, not an edge. | | Sharpe ratio | Above 1.0 = good, above 2.0 = excellent. | But fragile under 50 trades; don't quote "Sharpe 2.0" off a 12-trade run. | | Profit factor | Gross gains / gross losses. > 1.3 is the practical floor. | Penalizes hidden tail losses better than Sharpe. | | Max drawdown | > 20% is risky for retail-sized accounts. | A 5% Sharpe-1 strategy with 30% DD is unrunnable for most users. | | Avg holding | Sanity check — does it match the strategy's intent? | A "scalp" with 12h avg holding is misnamed; a "swing" with 5min holding likewise. |

Walk-forward (the honest validation)

Before recommending live deployment, run the strategy on out-of-sample data at least once:

1. Out-of-sample period: take the most recent 30 days that weren't in your tuning window. Re-run unchanged. Numbers should be in the same ballpark — not better, not dramatically worse.
2. Out-of-sample pair: run on a sibling pair (ETH if you tuned on BTC). Allow ±30% degradation; anything beyond that means the parameters were pair-specific not regime-specific.

If both walk-forwards survive, you have a defensible recommendation. If either falls apart, you have a backtest, not a strategy.

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: Superior-Trade
• Source: Superior-Trade/superior-skills
• License: MIT

Install and usage instructions live in the source repository linked above.