Hi, i've written an ea and if i backtest 2010 to today it has a profit factor of 3.1. But if i expand the test to 2008 it is pretty much flat for 2008 - 2010 and the factor drops to 2.4.
How far back do you backtest? Do you worry about results from years ago and just run with something that works know with the understanding that one day it will stop working, or should a good ea always be profitable over a 12 month window?
How Far Back to Backtest?
-
garyfritz
Re: How Far Back to Backtest?
Welcome monkeyzu,
You'll find several camps here with regard to backtesting.
* Some think it is utterly useless because the future won't be like the past. (But many/most of those think it's useful to live-test a strategy, assuming that if it live-tests OK that it'll keep trading OK after the live test. Which of course is just "backtesting in realtime.")
* Some think backtesting has value, but don't trust the Empty4 backtester. (I fit in this group.)
* Some think Empty4 backtests are useful and valuable.
How reliable your Empty4 backtest is, depends on the strategy and the data you backtest with. If your strategy only acts on the close of a bar, Empty4 does pretty well. If it pays attention to each tick, there's a very good chance it won't be accurate. Empty4 tends to assume the most optimistic possible result, which means your live results will look a lot worse than your backtests.
Assuming you have a reliable backtest, from whatever source...
Most people will say you need a minimum of 30 or so trades to have any statistical validity. I think that's WAY too few, since every trade will hit different conditions. I'm not comfortable with a backtest until it has at least 100 trades.
You also should test your system across a variety of market conditions. Some systems look great as long as the market goes up, but crash in sideways or down markets. Some do well in high volatility but fall apart when the markets quiet down. Often the only way to find out what kind of markets your system likes, is to test it. And once you find where the system fails, you might be able to find out WHY it fails, and improve its performance. But you have to do that carefully so you don't introduce over-fitting.
Many EAs will show long periods of losses or sideways equity. Many people run multiple EAs, with the hope that EA 2 will make profits when EA 1 stumbles. The EAs have to be different and uncorrelated for that to work.
You'll find several camps here with regard to backtesting.
* Some think it is utterly useless because the future won't be like the past. (But many/most of those think it's useful to live-test a strategy, assuming that if it live-tests OK that it'll keep trading OK after the live test. Which of course is just "backtesting in realtime.")
* Some think backtesting has value, but don't trust the Empty4 backtester. (I fit in this group.)
* Some think Empty4 backtests are useful and valuable.
How reliable your Empty4 backtest is, depends on the strategy and the data you backtest with. If your strategy only acts on the close of a bar, Empty4 does pretty well. If it pays attention to each tick, there's a very good chance it won't be accurate. Empty4 tends to assume the most optimistic possible result, which means your live results will look a lot worse than your backtests.
Assuming you have a reliable backtest, from whatever source...
Most people will say you need a minimum of 30 or so trades to have any statistical validity. I think that's WAY too few, since every trade will hit different conditions. I'm not comfortable with a backtest until it has at least 100 trades.
You also should test your system across a variety of market conditions. Some systems look great as long as the market goes up, but crash in sideways or down markets. Some do well in high volatility but fall apart when the markets quiet down. Often the only way to find out what kind of markets your system likes, is to test it. And once you find where the system fails, you might be able to find out WHY it fails, and improve its performance. But you have to do that carefully so you don't introduce over-fitting.
Many EAs will show long periods of losses or sideways equity. Many people run multiple EAs, with the hope that EA 2 will make profits when EA 1 stumbles. The EAs have to be different and uncorrelated for that to work.
-
monkeyzu
- Posts: 3
- Joined: Sat Sep 01, 2012 8:01 am
Re: How Far Back to Backtest?
Hi gary,
thanks for the response, very helpful.
MZ
thanks for the response, very helpful.
MZ