Why Empty4 backtesting does not work

Why Empty4 backtesting does not work

Postby SteveHopwood » Mon Dec 29, 2014 6:17 pm

Note. In this document:
  • M T4 = Empty4, because it is rubbish.
  • Crapergy Tester = Stratergy Tester, because it is utter bollocks arising out of rubbish.
  • Craptesting = backtesting, because of all the above.

The purpose of this document is to explain why craptesting in crapergy tester does not help us.

One of the hardest things that new traders have to grasp about the charts they are viewing is this: the only information each candle holds in its database is:
  • the candle Open price.
  • the candle High price.
  • the candle Low price.
  • the candle Close price.
  • the candle open time and date.

That's it folks. Now think of the things that are vital to a viewing the likely outcome of trading that is not held in each candle's database:
  • the spread. Think about this for a few seconds, folks. By default, Empty4's Crapergy Tester uses the current spread. Ouch. If you are 'paper testing' by manually going back over the charts, you probably won't even bother to include any kind of spread calculation. Backtesting in crapergy tester ignores the spread at the time the trade would have been taken because it has no record of it. Consider this:
    • if the trade was taken at the time a major news event was released and the spread jumped by a factor of 10, this will not show up in your backtested trade result.
    • spreads widen at the weekend, sometimes by a factor of 20 or more. Imagine what that does to EA craptesting results.
    • EA's craptesting their results use the current spread. This changes all the time even without what I previously described. A craptesting run at different times will give different results.
  • swap costs. Crapergy Tester does not factor this in at all when craptesting an EA. For sure, it would take a really dedicated paper trader to factor them in, even if he/she knows what they were. Just imagine the difference that this makes to the headline profit/loss.
  • commission charged by 'real' as opposed to market-makers and bucket-shop brokers. This is ignored by Crapergy Tester. Imagine the difference that makes to profitability.

Have a quick think about the implications of a candle only holding the OHLC in its database, on looking back to see what indicators are showing you. They are working with each candle's close price. Ok, so they might be working with their open price, or some combination of HL, but the big thing to remember is this; what an indicator shows you on a chart is a fixed point in time, which is the candle close time.

Take an indi that we are all familiar with - Stochastic. The damn thing might have zoomed up and down like an elevator on drugs during the course of a candle and played merry Hell with your trading. This will not show up on your chart because Stoch can only work the the candle close price. Imagine what this does to an EA that depends on Stoch to trigger trade opening and closing. :arrrg:

So, if you run your Crapergy Tester in visual every tick mode, where do all those ticks come from? Suppose you are craptesting an EA on the H1. You will see the "Using M1", "Using M5" etc messages. Crapergy Tester is taking the OHLC of each candle it has in its database and constructing a grid of ticks based on the direction these candles took. It is guesswork.

Fine/ish if you are testing a long term strategy that only makes trading decisions at the close of each candle. Bit of a sod if you are craptesting an M5 scalping system. Absolute sod if your long term trading includes a tick-by-tick trailing stop.

And then we get to the issue of missing candles. Empty4 ships with incomplete candle OHLC data, let alone missing tick-by-tick data. Noobs do not tend to know this.

Added to all the above is part of a post by Renexxxx:
"If you are referring to back testing using Empty4's Strategy Tester, then this is not going to give you much information about the profitability of your EA and strategy. Empty4's strategy tester is notoriously inaccurate, even if supplied with accurate tick data. Particularly, what Empty4's strategy tester can't do is:

1. Refer to price information on other timeframes eg. iClose(_Symbol, PERIOD_D1, shift) if the timeframe of the backtest is other than the daily timeframe.
2. Similarly, refer to indicator information on other timeframes than the backtest timeframe.
3. Refer to price information of other symbols (for a multi-symbol EA)."

So guys, this is why back testing is informally banned at SHF and why trying to discuss it will get you into trouble an any of my threads, and most that I merely contribute to.

I use it to test that EA code is working. I would not even bother to have nightmares about using it to test an EA's potential profitability - that is a waste of time.

:xm:

Subsequent note: this thread is about why those of us who have been around this stuff for years know that craptesting does not work. Posts by dimwits trying to insist it does will be deleted - guess what prompted this edit. Posts adding to the knowledge offered up here will be added to this post.
Read the effing manual, ok?

Global Prime is the official SHF broker. Click here to sign up for a live account with Global Prime and join the 600+ Steve Hopwood members who choose GP as their broker of choice.

I still suffer from OCCD. Good thing, really.

Anyone here feeling generous? My paypal account is always in the market for a tiny donation. [email protected] is the account.

To see The Weekly Roundup of stuff you guys might have missed Click here

My special thanks to Tommaso (milanese) for all the incredible work he does here.
User avatar
SteveHopwood
 
Posts: 9060
Joined: Tue Nov 15, 2011 8:43 am
Location: Misterton - an insignificant village in England. Very pleasant to live in.

Why Empty4 backtesting does not work

Postby milanese » Mon Dec 29, 2014 6:22 pm

Thanks Steve :clap: :clap: a MUST READ for all :xm: :xm: :xm:
Global Prime is the official SHF broker :yahoo:
Searching for Servers and Workstations with individual configuration?
Just PM
:smile: Click here to go to the BoardKnowledgeBase
NOTE: Cookies and JavaScript are required for the using the board, with full functionality
User avatar
milanese
 
Posts: 3369
Joined: Wed Jan 09, 2013 9:02 am
Location: btr rdx, r8 +

Why Empty4 backtesting does not work

Postby bazze » Mon Dec 29, 2014 6:35 pm

Great information,Steve...Thanks :good:
bazze
 

Why Empty4 backtesting does not work

Postby bEtTy » Tue Dec 30, 2014 12:50 pm

Thank you, Steve......
I really do not want to regret it later with a waste of time.
bEtTy
 
Posts: 7
Joined: Sat May 11, 2013 5:47 am

Why Empty4 backtesting does not work

Postby nc! » Tue Dec 30, 2014 1:29 pm

I run an ea called RobinVol which passed a 30 years backtest, wcs and all the stats jargon you can think of. Well all lost money including me. I do not even look at them past test results. Same ea on different accounts will have different results. So backtesting which feed? Dukas Copy (DC) backtest data is a scum since DC feed has nothing to do with their backtest data they supply the market. Had FIX API sperads at news 50 pips wide, that was not in the backtest data. Backtest is only good to test if the ea works, open closes trades that's all.

Happy New Year Steve and forum users. :yahoo: :clap:
nc!
 
Posts: 7
Joined: Fri Mar 09, 2012 5:59 pm

Why Empty4 backtesting does not work

Postby nika » Wed Feb 18, 2015 10:41 pm

elevator on drugs :smile:

seems true
nika
 
Posts: 8
Joined: Wed Feb 18, 2015 6:02 am

Why Empty4 backtesting does not work

Postby Chorlton » Sat Jul 25, 2015 4:51 am

Hi Steve,

What method would you recommend to test a strategy? Is there any form of backtesting (ignoring strategy tester) that would be worthwhile?

I've asked this question on many forums and normally receive the reply "run it on a demo account". Although I can understand why this would be preferred to relying on the Empty4 Strat Tester, the issue for me is that any results are provided in real-time so if I want to get a good sample size I may need to run the EA for many months.

Just interested in how you guys validate any new strategy.

Cheers,

Chorlton

SteveHopwood » Tue Dec 30, 2014 4:17 am wrote:Note. In this document:
  • Empty4 = Empty4, because it is rubbish.
  • Crapergy Tester = Stratergy Tester, because it is utter bollocks arising out of rubbish.
  • Craptesting = backtesting, because of all the above.

The purpose of this document is to explain why craptesting in crapergy tester does not help us.

One of the hardest things that new traders have to grasp about the charts they are viewing is this: the only information each candle holds in its database is:
  • the candle Open price.
  • the candle High price.
  • the candle Low price.
  • the candle Close price.
  • the candle open time and date.

That's it folks. Now think of the things that are vital to a viewing the likely outcome of trading that is not held in each candle's database:
  • the spread. Think about this for a few seconds, folks. By default, Empty4's Crapergy Tester uses the current spread. Ouch. If you are 'paper testing' by manually going back over the charts, you probably won't even bother to include any kind of spread calculation. Backtesting in crapergy tester ignores the spread at the time the trade would have been taken because it has no record of it. Consider this:
    • if the trade was taken at the time a major news event was released and the spread jumped by a factor of 10, this will not show up in your backtested trade result.
    • spreads widen at the weekend, sometimes by a factor of 20 or more. Imagine what that does to EA craptesting results.
    • EA's craptesting their results use the current spread. This changes all the time even without what I previously described. A craptesting run at different times will give different results.
  • swap costs. I am not sure that Crapergy Tester factors this in at all when craptesting an EA. For sure, it would take a really dedicated paper trader to factor them in, even if he/she knows what they were. Just imagine the difference that this makes to the headline profit/loss.

Have a quick think about the implications of a candle only holding the OHLC in its database, on looking back to see what indicators are showing you. They are working with each candle's close price. Ok, so they might be working with their open price, or some combination of HL, but the big thing to remember is this; what an indicator shows you on a chart is a fixed point in time, which is the candle close time.

Take an indi that we are all familiar with - Stochastic. The damn thing might have zoomed up and down like an elevator on drugs during the course of a candle and played merry Hell with your trading. This will not show up on your chart because Stoch can only work the the candle close price. Imagine what this does to an EA that depends on Stoch to trigger trade opening and closing. :arrrg:

So, if you run your Crapergy Tester in visual every tick mode, where do all those ticks come from? Suppose you are craptesting an EA on the H1. You will see the "Using M1", "Using M5" etc messages. Crapergy Tester is taking the OHLC of each candle it has in its database and constructing a grid of ticks based on the direction these candles took. It is guesswork.

Fine/ish if you are testing a long term strategy that only makes trading decisions at the close of each candle. Bit of a sod if you are craptesting an M5 scalping system. Absolute sod if your long term trading includes a tick-by-tick trailing stop.

And then we get to the issue of missing candles. Empty4 ships with incomplete candle OHLC data, let alone missing tick-by-tick data. Noobs do not tend to know this.

So guys, this is why back testing is informally banned at SHF and why trying to discuss it will get you into trouble an any of my threads, and most that I merely contribute to.

I use it to test that EA code is working. I would not even bother to have nightmares about using it to test an EA's potential profitability - that is a waste of time.

:xm:

Subsequent note: this thread is about why those of us who have been around this stuff for years know that craptesting does not work. Posts by dimwits trying to insist it does will be deleted - guess what prompted this edit. Posts adding to the knowledge offered up here will be added to this post.
Chorlton
 
Posts: 5
Joined: Sun Apr 27, 2014 2:54 am

Why Empty4 backtesting does not work

Postby SteveHopwood » Sat Jul 25, 2015 8:57 am

Chorlton » Sat Jul 25, 2015 4:51 am wrote:Hi Steve,

What method would you recommend to test a strategy? Is there any form of backtesting (ignoring strategy tester) that would be worthwhile?

I've asked this question on many forums and normally receive the reply "run it on a demo account". Although I can understand why this would be preferred to relying on the Empty4 Strat Tester, the issue for me is that any results are provided in real-time so if I want to get a good sample size I may need to run the EA for many months.

Just interested in how you guys validate any new strategy.

Cheers,

Chorlton


There is no quick and easy way that I know, to evaluate a strategy using Empty4. The only way to properly evaluate a strategy is to trade it on a live, real money account.

Even demo trading has its limitations. There is no slippage. Order sending/closing usually works perfectly even in the sort of storm that freezes live cash accounts with the less honest 'brokers'. The one thing that demo trading proves is that a strategy does not work. If it is not profitable on demo, it will get murdered live. If your 'broker' is in fact a criminal, then you strategy could well run perfectly on demo and then get murdered when you go live.

:xm:
Read the effing manual, ok?

Global Prime is the official SHF broker. Click here to sign up for a live account with Global Prime and join the 600+ Steve Hopwood members who choose GP as their broker of choice.

I still suffer from OCCD. Good thing, really.

Anyone here feeling generous? My paypal account is always in the market for a tiny donation. [email protected] is the account.

To see The Weekly Roundup of stuff you guys might have missed Click here

My special thanks to Tommaso (milanese) for all the incredible work he does here.
User avatar
SteveHopwood
 
Posts: 9060
Joined: Tue Nov 15, 2011 8:43 am
Location: Misterton - an insignificant village in England. Very pleasant to live in.

Why Empty4 backtesting does not work

Postby Chorlton » Sun Jul 26, 2015 4:05 am

There is no quick and easy way that I know, to evaluate a strategy using Empty4. The only way to properly evaluate a strategy is to trade it on a live, real money account.

Even demo trading has its limitations. There is no slippage. Order sending/closing usually works perfectly even in the sort of storm that freezes live cash accounts with the less honest 'brokers'. The one thing that demo trading proves is that a strategy does not work. If it is not profitable on demo, it will get murdered live. If your 'broker' is in fact a criminal, then you strategy could well run perfectly on demo and then get murdered when you go live.

:xm:



Thanks Steve,

Appreciate the reply.

Given your opinion of Empty4, which platform do you use?
Chorlton
 
Posts: 5
Joined: Sun Apr 27, 2014 2:54 am

Why Empty4 backtesting does not work

Postby SteveHopwood » Sun Jul 26, 2015 9:06 am

Chorlton » Sun Jul 26, 2015 4:05 am wrote:
Thanks Steve,

Appreciate the reply.

Given your opinion of Empty4, which platform do you use?

Empty4. It is the only one offered by Global Prime. Being able to trade through an honest broker does a lot to alleviate the Empty4 lunacy.

:xm:
Read the effing manual, ok?

Global Prime is the official SHF broker. Click here to sign up for a live account with Global Prime and join the 600+ Steve Hopwood members who choose GP as their broker of choice.

I still suffer from OCCD. Good thing, really.

Anyone here feeling generous? My paypal account is always in the market for a tiny donation. [email protected] is the account.

To see The Weekly Roundup of stuff you guys might have missed Click here

My special thanks to Tommaso (milanese) for all the incredible work he does here.
User avatar
SteveHopwood
 
Posts: 9060
Joined: Tue Nov 15, 2011 8:43 am
Location: Misterton - an insignificant village in England. Very pleasant to live in.

Next

Return to Know your MT4 platform

Who is online

Users browsing this forum: No registered users and 1 guest

cron