Lesson T10.3 · 3 of 6 · Strategy, testing and the trading journal
Backtesting a trading strategy: the traps and how to avoid them
Backtesting applies a strategy’s rules to past prices to see how it would have done. The result only means something if the rules were fixed beforehand, checked on data not used to choose them, run on every company that existed at the time, and charged realistic costs.
Terms in this lesson
- Sharpe ratio
- How much extra return you got for each unit of risk.
- Slippage
- The gap between the price you expected and the price your order filled at.
- Trading commission
- What it costs you to buy, sell and hold shares at a broker.
- Maximum drawdown
- The largest fall in price from a peak to the lowest point after it.
What backtesting is, and in-sample versus out-of-sample data
Definition
What are in-sample and out-of-sample data?
The data you choose your rules on, and the data you check them on.
You split the history in two. On the first part (in sample) you try out rules and pick one. The second part (out of sample) you set aside and use once, at the end, like an exam. If you look at it, tweak the rules and test again, it has become part of the sample too.
Entry, exit, stop, position size and costs, before the first test.
Pick the settings on the first part of the history and record how many versions you tried.
Run the unchanged rules on the held-back data and accept the result, whatever it is.
The rest of this lesson is in the subscription
The subscription unlocks every chapter of Intermediate, Trading school, Investor psychology and Advanced, with their exercises and unlimited reviews.
Create a free accountBacktest overfitting: what Bailey, Borwein, López de Prado and Zhu showed
In 2014, David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu published “Pseudo-Mathematics and Financial Charlatanism” in the Notices of the American Mathematical Society. Their point: try enough versions of a strategy on the same history and the best one will look excellent by pure chance. The yardstick they use is the annualised Sharpe ratio: the return above the risk-free rate, divided by volatility.
- 45 versionsWith only 5 years of data, the authors show that no more than 45 independent configurations should be tried: after that many, the best one has an average Sharpe ratio of 1 in the test, even if it has no real edge and its expected out-of-sample Sharpe is 0.
- 7 versionsOn a backtest of just 2 years, trying 7 configurations is enough for the best one to show an average Sharpe ratio of 1, by chance.
- 8,800 combinationsOn 1,000 days of randomly generated prices with no real effect in them, they searched for the best “seasonal” rule (entry day, holding period, stop, direction). They found one with an annualised Sharpe ratio of 1.27.
Their conclusion: a backtest that doesn’t say how many versions were tried can’t be judged, and the more versions were tried, the higher the result you should demand. The authors suspect that overfitting is a large part of the reason so many systematic hedge funds fail to live up to expectations.
Survivorship bias, look-ahead bias and forgotten costs
| Trap | What it means | How to avoid it |
|---|---|---|
| Survivorship bias | you only test on companies that exist today, leaving out those that went bust or were delisted | use the list of companies as it stood on each past date, delisted ones included |
| Look-ahead bias | the rule uses information that wasn’t available when the decision was made | check when each piece of data you use was actually published |
| Forgotten costs | the test doesn’t deduct the spread, commission and slippage | deduct realistic costs, preferably a little higher than today’s |
Tyler Shumway (Journal of Finance, 1997) showed that the CRSP database, used in a great deal of research on US shares, was missing delisting returns for most firms removed for negative reasons such as bankruptcy.
Here is a backtest report of the kind you see on forums. Every sentence has a problem:
Walk-forward testing: checking a strategy step by step
Definition
What is walk-forward testing?
You choose the settings on one window of data, test them on the period that follows, then move the window on.
The method is closely linked with Robert Pardo, who devotes a full chapter to it in “The Evaluation and Optimization of Trading Strategies” (Wiley, 2008), the second edition of his 1992 book “Design, Testing and Optimization of Trading Systems”. An example with data from 2014-2025: choose the settings on 2014-2016 and test them on 2017; then choose on 2015-2017 and test on 2018; and so on up to 2025. Put together, the nine test periods show how the strategy would have done had you redone the choice every year, without seeing the future. CFA Institute’s materials describe rolling-window backtesting as a proxy for actual investing.
- 2014-2016 → 2017choose the settings, then test the following year
- 2015-2017 → 2018move the window on a year
- …repeat
- 2022-2024 → 2025the last window
Lessons are educational. They are not a recommendation to buy or sell. Investing involves risk and you can lose money. Sources: D. H. Bailey, J. M. Borwein, M. López de Prado, Q. J. Zhu, “Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance”, Notices of the American Mathematical Society, May 2014 (and the AMS press release of 10 April 2014); T. Shumway, “The Delisting Bias in CRSP Data”, Journal of Finance 52(1), 1997; R. Pardo, “The Evaluation and Optimization of Trading Strategies”, Wiley, second edition, 2008 (chapter “Walk-Forward Analysis”; first edition: “Design, Testing and Optimization of Trading Systems”, Wiley, New York, 1992); CFA Institute, refresher reading “Backtesting and Simulation”. The backtest report and the cost calculations are illustrative examples.