Realistic Back Testing and Goals for an EA or manual strat

[Jonathan2FI]

I would like to hear from the forum both discretionary and algorithmic trading on realistic targets for an EA and or manual trading systems with backtesting.
But first a little background (comments on this again would be appreciated:

1. Clearly there are many pitfalls to backtesting :
1.1 Poor historical data !! plus time frame inconsistencies
1.2 Inconsistencies on Empty4
1.3 Spreads - over optimistic settings for the test and specifically for time dependent EAs for
example using an EA that takes advantage of the volatile period between 2100 and 2300 on
say EURGBP but setting the spread at 1.6 avg for the day...but for 21-23 the spread
is 1.8 to 3.5pips.
1.4 Too many parameters . and then setting these to get best results over a limited period see 1.5
1.5 Testing over too short a period with too few trades ... need 5 years and >100 trades -
exceptions to this?
1.6 Using Net Profit as the only selection criteria ... would be a big mistake use MAR returns
adjusted for risk used by hedge funds basically divide the annual returns by the max drawdown
and look for strategy with highest ratio
1.7 Check Equity curve .... smooth or does it do nothing for 4 years then go exponential?
1.8 and if the above is performed manually with eye paper and pencil then that brings a whole
lot of new biases form the brain.
(so you may conclude that backtesting is misleading at the best.....but how to get long term perf. stats without going grey in the process e.g. 5 years live or fwd testing.. so we need to use back testing and stats). I know Steve is anti backtesting on this forum and I can understand why he would not want 1000s of ill conceived backtested results with no common data or methodology covering the site.

OK so now for the debate Barclays hedge fund perf. analysis last 10 -20 years indicates that even with the best neural networks and wide awake sober traders MAR is limited to 3. So lets look at a situation where a trader says his system gives him 12% pm every month = 390% / 3 = max potential drawdown = 130% .....but margin call at 100% and he leaves with tail between his get rich quick legs!
My openers
A So what is realistic from a strategy with a Long Term Edge - no not fire and forget but nevertheless a long term edge?
B How do prof. traders distinguish between a system in drawdown and one that is DEAD?
C How does a portfolio of strategies change any of the above and are there tools to assess this?

Jonathan2FI

[garyfritz]

You've raised some very good points, Jonathan.

1. Clearly there are many pitfalls to backtesting :
1.1 Poor historical data !! plus time frame inconsistencies
1.2 Inconsistencies on Empty4
1.3 Spreads -

All true. Data issues can be addressed. Spreads can be dealt with -- add an additional cost per trade to your data and see if you still like it. (Though I don't think there's a way to do that with the Empty4 backtester -- you'd have to do it by hand, after the fact.)

Empty4 backtesting issues is a big one. Many people in Empty4-world think backtesting is useless, primarily I think because the Empty4 backtester is near useless. In fact, according to the experts in this field, proper backtesting is the ONLY way to verify a strategy. Walk-forward testing is a very weak alternative.

I believe Empty4 backtesting can be done safely and accurately, but it's not easy. It's too easy to get totally false and inflated results. This is one big reason Steve banned Empty4 backtests on the forum.

1.4 Too many parameters . and then setting these to get best results over a limited period see 1.5
1.5 Testing over too short a period with too few trades ... need 5 years and >100 trades - exceptions to this?

These are BIG. The more parameters you tune on your strategy, the less likely it will continue to work in the future. With many finely-tuned parameters, it becomes more likely that you've curve-fit to the specific conditions during your test period, and those conditions aren't likely to repeat.

You can reduce this effect by testing over a longer period to get lots of trades. I think 100 trades is a good lower end for what you want in your backtest, and more is better. If your strategy works over a 100-trade test, it's less likely that you curve-fit to a few specific trades.

1.6 Using Net Profit as the only selection criteria ... would be a big mistake use MAR returns
adjusted for risk used by hedge funds basically divide the annual returns by the max drawdown
and look for strategy with highest ratio
1.7 Check Equity curve .... smooth or does it do nothing for 4 years then go exponential?

Exactly right. Ideally you want something that cranks out consistent profits through the entire history, but of course that's mighty hard to do. You DON'T want a strategy that made all its profits in a short period. You probably wouldn't have stuck with the strategy during the years of flat or losing results, and even if you did, will the conditions that produced your burst of profit ever repeat? Better to have something that makes money more consistently, in all types of market conditions. Sharpe ratio is a good measure for that, but you want to combine it with returns -- a bank account that makes one penny every month will have a huge (actually infinite) Sharpe, but not much profit.

1.8 and if the above is performed manually with eye paper and pencil then that brings a whole lot of new biases form the brain.

Yup. The eye/brain are famous for seeing what they want to see, and not seeing the cases where the strategy didn't work. I don't trust hand backtests unless they're driven entirely by computer-generated signals.

A So what is realistic from a strategy with a Long Term Edge - no not fire and forget but nevertheless a long term edge?

Hard to answer in general. Depends on many things: the strategy, the trading instrument, the timeframe, the trader's risk tolerance, ...

B How do prof. traders distinguish between a system in drawdown and one that is DEAD?

That's really tough. You can track performance statistics so you understand how the system behaved in the past, and say "if it gets outside those parameters by X% I'll pull the plug." So maybe if the DD exceeds the historical max by X%, or if the win rate drops below previous lows by X%, or similar. There are statistical measures (Z-score if I remember right? Or t-test?) that are supposed to indicate if a system's behavior has changed, but I don't have any experience using those. I don't know if they really tell you more than you can see by looking at your equity curve.

C How does a portfolio of strategies change any of the above and are there tools to assess this?

Trading a portfolio of strategies can hugely improve the stability of your results, **IF** they're uncorrelated. When one is in drawdown, the others can fill in the holes. I forget the exact equation but if I remember right, if you trade N totally-uncorrelated strategies that all have Sharpe of S, the resulting Sharpe of the portfolio is sqrt(N)*S. That can be a huge improvement. Finding those N totally-uncorrelated profitable strategies is the hard part. Partially-correlated strategies will help, but they won't help as much -- they'll tend to go into drawdowns at the same time.

[Jonathan2FI]

Thanks Gary,
Very useful ....I feel like I am on right track stretching testing out to 5 years plus applying the rigor identified above and of course with a healthy skepticism for any EA/Strategy - I ll back off the multi x multi parameters look for smooth equity curves and low drawdown (these are often but not always correlated together from my experience) and of course a useful profit. I will look at the work that has been done on currency correlation as well as looking at strategy correlation e.g. envelope range trading m15 compared to trend retracements on H1 H4 D1... I understand Sharpes ratio and bringing things back to the risk free level for judging a % return - e.g. 20 years ago we may have turned our nose up at a strategy that gave 10% pa when you could get returns from banks of 5-8% ...but now when they give you 1-3% pa suddenly 10% does not look too bad! I think I will use MAR ratio (but look at CAGR with a min. hurdle of say 10%)
I would still like to push a little for an opinion on the realistic target or range at least with respect to a ratio between net profit and max drawdown given the current economic times for say a 5 year test - I take your comment on risk appetite so maybe I can fill in some of the unknowns 5x$3k accounts, trades<2% risk and avoid >20% RDD
Jonathan

Ps For anyone reading this thinking of buying a commercial EA with too good to be true results aaaaaaah think how easy it is to show miracle EAs from backtesting data ..

[forextrader-radioman]

[...]
1. Clearly there are many pitfalls to backtesting :
1.1 Poor historical data !! plus time frame inconsistencies
[...]

Hy Jonathan2FI,

interesting post! ... I make my M T 4 backtesting with this software, it converts Tick-data from Dukascopy and so I can have 99,90% results ...

http://tickstory.com/

It´s easy to use, read the online help and FAQ ...

all the best to you and yours,
Dietmar