The key to developing successful EAs/robots/systems is to find some behavior that works reasonably well most of the time, for a long time. I'll take that over one that does spectacularly well for a little while and then fails spectacularly -- since without exception, it always fails spectacularly RIGHT after I start trading it.
If you find something that has a genuine edge, you hope that edge will continue to work. Things can shift and change that -- all my very successful systems quit working after the 9/11 attacks in New York, for example -- but in general you hope to find something that works in most market conditions.
It seems to me that this is not much use unless we know what x market conditions were that led to the EA being profitable between A-date and B-date, but unprofitable between C-date and D-date.
Yes, it would be very valuable to know what those conditions were. And how would you be able to figure that out if you're just forward testing? You might not hit A-B *or* C-D conditions during that time.
If someone brings you a system that has been successful for 6 months, how can you be certain that A-B period is typical? How can you be sure you've accurately implemented it (and that they've accurately described it, i.e. they weren't "cheating" and breaking their rules all the time) without seeing how your EA performed in those last 6 months? You can watch it for a month and see if it does what it's supposed to do, but what if that month never presents the conditions you need to test it? And wouldn't you rather be able to verify it NOW, instead of waiting a month or two? (Obviously Tradestation backtesting doesn't help you there unless you shifted over to using TS. But you get the idea. And a good TS backtest would tell you if it's worth writing/testing at all.)
I much prefer building a system and seeing if it works
right away, so I can improve it or discard it and move on, rather than "implement 47 EAs in a row, debug them all in parallel, wait for months to see any of them worked," etc. (Not naming any names of course.
)
So, is not a month of forward testing that says that the EA has worked recently more useful than a backtest that says that it worked sometimes, but not others, but did not say why it worked/did not work?
The month says "it worked for a month." You don't know if that's the only month it
ever worked, or if it's been working like that consistently for years. Which would you rather trade? Wouldn't you like to KNOW which one you're trading? I wouldn't touch a system that only worked for the last month or two and lost before that.
True, the backtest doesn't tell
why it worked or didn't work. That's your job as system developer, to try to see what was different in those times, try different changes to your system to help it work better all the time or step aside when it doesn't work, etc. Which of course you can only develop if you can backtest your changes. (Of course you do have to be careful not to overfit your system to the historical data.)
Backtest Scoobs' FR for example, and you'll quickly see it blows up when the market trends strongly against it. Sure we know that now, but say you just got the idea for FR and implemented it. Your backtests would tell you it works fabulously, except in trending markets. So you can try adding trend filters,
then see if they worked. If you're forward testing, you'd have to wait for a strong trending market. You might hit a few small trends and conclude your changes were successful. Then Spain implodes, Europe slips into a morass like the US did in 2008, and your "fixed" robot blows up in your face. Maybe you could have avoided that -- or at least known what to expect and adjusted your risk accordingly -- if you'd backtested back through 2008. That's what I do, because I want to know how my system is likely to behave if (when) the world goes into another financial meltdown.
I think understanding the risk is one of the most important values of backtesting,
especially with all the recovery systems around here. People run them for a few weeks and start getting dollar/pound/euro signs in their eyes, so they think "This thing hardly has any drawdowns at all! I'll size it up and really start making some money!" But they have
no idea what risk they're taking on because they have no idea how the EA performs in the long run. Even though Empty4 backtests are crap, they can (if properly done) at least give you an idea of the risks when your recovery system hits a panic-exit trade -- and how often that happens. That's pretty valuable in itself.
)
