Simple Shelley Backtester

[NeoTrader]

Yes, that's the kind of thing I'd like to be able to backtest. Unfortunately my situation right now means I can barely keep up with the forum posts, let alone do any coding or design. I don't know when I'm going to be able to get around to creating this indicator-backtester framework.

hehe... I feel you... it's the same with me...so many ideas so little time...

But nevertheless if i can help you in any way feel free to contact me... i think this backtester would be a great asset in our toolkit.

happy trading,

NeoTrader

[Thunder123]

Being able to backtest EA that operate on multiple pairs would be invaluable. With good quality historical data (for example dukascopy), it might even become an excellent proxy for live test results, which would significantly speed up the development time for EAs. This is an excellent idea and I hope you guys can pull it off.

[NeoTrader]

Hi gary,

i came across this article Visual Testing of the Profitability of Indicators and Alerts and thought it might be of use... it is not a library but maybe you can fish a few ideas out of the concept...

happy trading,

NeoTrader

-

[NeoTrader]

hi Gary,

just wanted to ask if you would share your preliminary Backtester as I am really interested to see how your concept is.

The link I posted in the last message shows also an interesting concept with source code included that is easy adaptable to test any indicator visually or with little work as a exported table.

Since I am desperately looking for a reliable testing solution I would be very grateful if you could comment on what you think about the concept of the previously posted solution.

Thanks in advance,

NeoTrader

-

[garyfritz]

I guess I never posted it, did I? Probably just as well, because it was an ad-hoc hack (an ad-hack?) as I figured out what I was doing. I'm not sure it's going to help you any, in fact I suspect it might confuse the issue more than it helps, but here 'tis.

I'm not sure how applicable that ZigZag example is. It's a specialized tester written ground-up for ZZ. Some of the concepts will be similar but the general solution I have in mind (assuming we can actually make it work!!!) will be much more difficult.

Here's my idea: we want this to work for any EA. The first thing I would do is to write a "wrapper" library that people would call instead of calling the MQL code directly. So instead of calling OrderSend() they would call BTOrderSend() or SHFOrderSend() or whatever. (Unfortunately I don't think MQL supports replacing the original OrderSend() with our own OrderSend() and still allowing us to access the original OrderSend().) So e.g. Steve would write his EAs as before, but calling functions with the BT (backtester) prefix.

In the "wrapper" library, each BTxxx() function would just call the xxx() function. You would load in the "wrapper" library when you just wanted to run the EA normally.

Then the BackTester library would provide a different set of BTxxx() functions, but instead of calling the xxx() functions directly, it would perform the backtesting function.

But actually....... now that I think through this a bit.....

The "wrapper" library and BTxxx() functions may not be necessary! Let's say you write your EA using the normal OrderSend() function. Everything works normally, and for the most part you don't have to do anything differently.

But the Backtester will run as an **indicator***, NOT an EA! You will copy your EA code into an indicator to test it. So it doesn't NEED to call EA functions like OrderSend(), and in fact it would probably be an error if it did. So the Backtester library will provide functions that replace the order functions used in an EA. Instead of re-calling the "real" OrderSend(), the backtester OrderSend() will record the details of the order. Then as the EA-turned-indicator executes, the Backtester will keep a list of all open orders and open positions, track any positions that open and close, and "do the right thing" as order prices are hit, etc. It will track the open and closed equity and probably chart it and/or dump it to a file for analysis.

Make sense so far? This approach has the huge advantage that it can be applied to ANY existing EA, with very little change to the EA source code. You'll probably have to make a few changes as you convert your EA into a backtester-indicator -- copy the EA source into an indicator, add some #include lines, add a BT_init() call in init(), add a BT_start() call in start(), things like that -- but that's a lot more straightforward than the "replace all xxx() calls by BTxxx()" approach that I envisioned originally.

So ... if you want to start making progress on this, the first thing we need to do is to draw up a list of all the EA functions (and anything else) we'll need to replace. Anything that is EA-specific, anything that we'll need to re-implement to track orders & positions, etc. That will include all OrderXXX() and AccountXXX() functions... anything else?

Then we'll need to start designing the logic behind them. That will be the tricky bit.

[garyfritz]

So, we figure out all the functions we need to replace. Then we start thinking about the logic.

Indicators execute very differently to EAs. EA start() gets called on every tick in realtime, and that's what the user's EA start() function will assume.

Indicator start() functions get called once per bar for history, and then once per tick in realtime, and also several recent bars may get re-called whenever it feels like it. That's messy and we don't want the backtester to work that way. We don't care about realtime updates of the chart, we're just interested in history.

I think the way to work this is that the backtester provides its own start() function, and then the user (when he converts his EA to a backtester-indicator) renames his start() to EA_start(). Then the backtester start() "simulates" the once-per-tick start() call of an EA by calling EA_start() for each bar.

I don't know if there is any way to step through tick-by-tick history on, say, an H4 chart. Maybe we could do M1-level backtesting -- step through the M1 data in the chart and call EA_start() once for each minute, instead of once for each tick the way an actual realtime EA would execute.

Hmm, problem. The EA is likely to refer to High, Low, Close, etc. In realtime EA execution, those refer to the current tick. How do we get them to refer to the H/L/C of the backtested M1 bar that we're calling EA_start() on?

Bigger problem: the EA will probably use iClose(), iHigh(), etc. The EA will expect those to be relative to realtime. I.e. shift=0 will refer to the currently-building bar. But when we're calling EA_start() on a historic M1 bar, shift=0 needs to refer to the current M1 bar. So I suspect the backtester will have to re-implement iClose() &etc. That's a problem because, unlike OrderSend() &etc, we still want to call the original iClose(). So we might have to implement wrapper functions for iClose() & similar, and EA writers will have to use (or convert when they use the backtester) BT_iClose(), BT_iOpen(), etc.

Ewww. And we'll have to do similar things for Time[], TimeCurrent(), and probably others. Ick. This is going to be messier than I hoped.

[NeoTrader]

Hi Gary,

I finally found time to get back here...

first thanks for your source and your comments...gave me lots of thoughts...and brainwrenching..

Now I will give it back to you...

I guess I never posted it, did I? Probably just as well, because it was an ad-hoc hack (an ad-hack?) as I figured out what I was doing. I'm not sure it's going to help you any, in fact I suspect it might confuse the issue more than it helps, but here 'tis.

No confusing at all.. at least you documented it a little...

I'm not sure how applicable that ZigZag example is. It's a specialized tester written ground-up for ZZ. Some of the concepts will be similar but the general solution I have in mind (assuming we can actually make it work!!!) will be much more difficult.

The ZigZag is only a prove of concept...in the source there is also a MACD example and a Fantasies example commented out. I also like the visualization part of this concept.
But it is only a really simple version of that what we want to accomplish here.

Here's my idea: we want this to work for any EA. The first thing I would do is to write a "wrapper" library that people would call instead of calling the MQL code directly. So instead of calling OrderSend() they would call BTOrderSend() or SHFOrderSend() or whatever. (Unfortunately I don't think MQL supports replacing the original OrderSend() with our own OrderSend() and still allowing us to access the original OrderSend().) So e.g. Steve would write his EAs as before, but calling functions with the BT (backtester) prefix.

I also think that this would be the way to go (Wrapper Library for Functions with equal Naming + prefix).
I did some little tests with creating some Functions with the same Naming as OrderSend, OrdersTotal... this seems not to work as the compiler and our beloved Empty4 Platform does not support any kind of overriding/overloading MQL functions.

In the "wrapper" library, each BTxxx() function would just call the xxx() function. You would load in the "wrapper" library when you just wanted to run the EA normally.
Then the BackTester library would provide a different set of BTxxx() functions, but instead of calling the xxx() functions directly, it would perform the backtesting function.

Yep this kind of logic could work...
another approach/idea could be that we only have one library and we could use a external Parameter "backtesting=true/false" to switch between the modes backtesting/real trading.

But actually....... now that I think through this a bit.....
The "wrapper" library and BTxxx() functions may not be necessary! Let's say you write your EA using the normal OrderSend() function. Everything works normally, and for the most part you don't have to do anything differently.
But the Backtester will run as an **indicator***, NOT an EA! You will copy your EA code into an indicator to test it. So it doesn't NEED to call EA functions like OrderSend(), and in fact it would probably be an error if it did. So the Backtester library will provide functions that replace the order functions used in an EA. Instead of re-calling the "real" OrderSend(), the backtester OrderSend() will record the details of the order. Then as the EA-turned-indicator executes, the Backtester will keep a list of all open orders and open positions, track any positions that open and close, and "do the right thing" as order prices are hit, etc. It will track the open and closed equity and probably chart it and/or dump it to a file for analysis.
Make sense so far? This approach has the huge advantage that it can be applied to ANY existing EA, with very little change to the EA source code. You'll probably have to make a few changes as you convert your EA into a backtester-indicator -- copy the EA source into an indicator, add some #include lines, add a BT_init() call in init(), add a BT_start() call in start(), things like that -- but that's a lot more straightforward than the "replace all xxx() calls by BTxxx()" approach that I envisioned originally.

It's a nice Idea, but as I said above no function overloading of any kind in Empty4. This is also not possible in Indicators and in scripts...it just won't let you compile...
To run the EA as Indicator could still be possible but we would still need the library.

So ... if you want to start making progress on this, the first thing we need to do is to draw up a list of all the EA functions (and anything else) we'll need to replace. Anything that is EA-specific, anything that we'll need to re-implement to track orders & positions, etc. That will include all OrderXXX() and AccountXXX() functions... anything else?

I think that the OrderXXX() and AccountXXX() functions should be enough...no other Function as of now comes to my mind.

Then we'll need to start designing the logic behind them. That will be the tricky bit.

In the example of the article I posted he used an array just to get hold of the signals to buy, sell and close of the orders.
I also think that we should have a table (array) where all open/pending orders are stored that we can access through the library.
The Problem I see here is that within an indicator...also with a library that we call inside an indicator we have no access at all to the normal AccountXXX() functions and therefore need also to code the logic of this. This also means that we need Parameters to collect (accountsize, leverage,...)

Indicators execute very differently to EAs. EA start() gets called on every tick in realtime, and that's what the user's EA start() function will assume.
Indicator start() functions get called once per bar for history, and then once per tick in realtime, and also several recent bars may get re-called whenever it feels like it. That's messy and we don't want the backtester to work that way. We don't care about realtime updates of the chart, we're just interested in history.
I think the way to work this is that the backtester provides its own start() function, and then the user (when he converts his EA to a backtester-indicator) renames his start() to EA_start(). Then the backtester start() "simulates" the once-per-tick start() call of an EA by calling EA_start() for each bar.

This could be a reasonable solution...just need to check it though.

I don't know if there is any way to step through tick-by-tick history on, say, an H4 chart. Maybe we could do M1-level backtesting -- step through the M1 data in the chart and call EA_start() once for each minute, instead of once for each tick the way an actual realtime EA would execute.

I don't think tick-by-tick is available in indicators...rather bar-by-bar, but I am not sure myself. If we have the logic in the indicator this should not be a great problem since the indicator also inits with each change of the period...so we can see what timeframe would be appropriate for our testing purposes.

Hmm, problem. The EA is likely to refer to High, Low, Close, etc. In realtime EA execution, those refer to the current tick. How do we get them to refer to the H/L/C of the backtested M1 bar that we're calling EA_start() on?
Bigger problem: the EA will probably use iClose(), iHigh(), etc. The EA will expect those to be relative to realtime. I.e. shift=0 will refer to the currently-building bar. But when we're calling EA_start() on a historic M1 bar, shift=0 needs to refer to the current M1 bar. So I suspect the backtester will have to re-implement iClose() &etc. That's a problem because, unlike OrderSend() &etc, we still want to call the original iClose(). So we might have to implement wrapper functions for iClose() & similar, and EA writers will have to use (or convert when they use the backtester) BT_iClose(), BT_iOpen(), etc.
Ewww. And we'll have to do similar things for Time[], TimeCurrent(), and probably others. Ick. This is going to be messier than I hoped.

That is really a Problem and would complicate things a lot.

The easy way of copying and pasting an EA into an Indicator and editing a few lines is not going to happen as I see now. The limitation and problems we face are not easy to overcome and would take a lot of work.

I think we should reassess what we want to accomplish.

Our goal as I understand it is to be able to test multi-currency and multi-timeframe strategies and to have a result if this strategy could/would be successful or not.

What about another way of doing this...
The rule sets of a strategy are most times not that complex...and could be transfered to an indicator with much less effort. Why not split the strategy in two pieces...
- 1st part is a Custom Indicator with the rule sets that rely on indicator- or chart-action (like in the article I posted before)... this could work like a signaling service (buy/sell/closeBuy/closeSell).
- 2nd Part would be an EA that has the rule sets for Order/Trade/Money Management .
Both could communicate with each other via Global Variables or via File read/write and can also log their results in one log/csv that can later be analyzed with Excel.
We still could have a library for the EA Part where we can switch the Order Part from backtesting to real trading.
The Rule set and indicator calls from the Indicator should be easy to implement into the final EA if needed...or you just release both Files.

I know that we maybe loose Steve with this concept. But eventually we could find a solution that his EAShell can still be used or we setup a new Base for the EA Shell. This Part depends upon Steve if he wants to jump on the wagon...

What do you think about this? Or is this too far away from what you want to achieve?

happy trading,

NeoTrader

-

[garyfritz]

It's a nice Idea, but as I said above no function overloading of any kind in Empty4. This is also not possible in Indicators and in scripts...it just won't let you compile...

Oh. Crud. You're right. I hadn't tested it, just assumed it would be possible to overload a function.

So I guess we always need to use wrappers, which means the EA-writer will need to use BT_xxxx() functions for ALL EA functions. That's unfortunate, but not a fatal problem.

It would probably not be too hard to write a "backtester preprocessor" that accepts EA code and does the OrderXXX() -> BT_OrderXXX() &etc changes. That would make it easy for someone to use the backtester without any code changes for their EA. Steve wouldn't have to do anything different.

I also think that we should have a table (array) where all open/pending orders are stored that we can access through the library.
The Problem I see here is that within an indicator...also with a library that we call inside an indicator we have no access at all to the normal AccountXXX() functions and therefore need also to code the logic of this. This also means that we need Parameters to collect (accountsize, leverage,...)

Yes, we'd need tables to store orders, positions, etc. And yes, we'd have to simulate the AccountXXX() functions. We'd have to track the open & closed equity so the various AccountXXX() functions worked right.

Hmm, problem. The EA is likely to refer to High, Low, Close, etc. In realtime EA execution, those refer to the current tick. How do we get them to refer to the H/L/C of the backtested M1 bar that we're calling EA_start() on?
Ewww. And we'll have to do similar things for Time[], TimeCurrent(), and probably others. Ick. This is going to be messier than I hoped.

That is really a Problem and would complicate things a lot.

Well, it's still workable. If we require them to use BT_High, BT_Low, BT_iClose(), etc, it can still work. We just need to keep track of the current bar we're backtesting on, and have the iClose() "shift" variable based off that bar instead of bar 0. BT_High / BT_Time / etc would be relative to the current backtest bar instead of bar 0.

Our goal as I understand it is to be able to test multi-currency and multi-timeframe strategies and to have a result if this strategy could/would be successful or not.

Yes.

What about another way of doing this...
The rule sets of a strategy are most times not that complex...and could be transfered to an indicator with much less effort. Why not split the strategy in two pieces...

That sounds like a very complex approach, and you're right we'd lose Steve. Plus I'm not convinced it would work for most EAs. The rules/logic in most EAs are spread throughout the EA code. I'm not confident the resulting split backtest EA would implement the same logic as the actual EA you want to trade.

I don't think it would be a simple matter to extract the EA logic. I think the entire EA has to be pasted into an indicator, with the BT_ library providing the appropriate behavior in the indicator environment.

[NeoTrader]

Hi Gary,

So I guess we always need to use wrappers, which means the EA-writer will need to use BT_xxxx() functions for ALL EA functions. That's unfortunate, but not a fatal problem.

It would probably not be too hard to write a "backtester preprocessor" that accepts EA code and does the OrderXXX() -> BT_OrderXXX() &etc changes. That would make it easy for someone to use the backtester without any code changes for their EA. Steve wouldn't have to do anything different.

No this is not a fatal Problem and also the "Preprocessor" should be not a great problem. How do you envision this? An external script (windows or MQL) that should be run on the EA to make a simple find and replace?

Yes, we'd need tables to store orders, positions, etc. And yes, we'd have to simulate the AccountXXX() functions. We'd have to track the open & closed equity so the various AccountXXX() functions worked right.

puuhhh...but if you say so...

Well, it's still workable. If we require them to use BT_High, BT_Low, BT_iClose(), etc, it can still work. We just need to keep track of the current bar we're backtesting on, and have the iClose() "shift" variable based off that bar instead of bar 0. BT_High / BT_Time / etc would be relative to the current backtest bar instead of bar 0.

It is workable...thats true... we just could use a candle array where all the values of the current candle of the backtest loop are copied to and the BT_functions work only with this array...

Yes.

At least we are on the same Page..

That sounds like a very complex approach, and you're right we'd lose Steve. Plus I'm not convinced it would work for most EAs. The rules/logic in most EAs are spread throughout the EA code. I'm not confident the resulting split backtest EA would implement the same logic as the actual EA you want to trade.
I don't think it would be a simple matter to extract the EA logic. I think the entire EA has to be pasted into an indicator, with the BT_ library providing the appropriate behavior in the indicator environment.

If I think again about it I must admit that you are right... it would be much easier if we just can copy and paste the whole thing into an indicator and edit a few parameters and we can let the baby run...


Ok...then lets do it...where do we start...


Do you use any kind of Source control Tools? like Git or Subversion?
w633 told me about the site bitbucket.org that we could use as repository for our code....It is a private Git Repository that up to 5 People can use for free...what do you think?

I don't know about your time but I would like to commit some time in this project because I really want/need some reliable testing and results that I can work with...And this is the one topic that bothers me most with Empty4...

If you want you can also send me a PM on this topic...


happy trading,

NeoTrader

-

[garyfritz]

No this is not a fatal Problem and also the "Preprocessor" should be not a great problem. How do you envision this? An external script (windows or MQL) that should be run on the EA to make a simple find and replace?

I'd say an external script. You could do a simple text find&replace, or it would be very simple to do it in Perl or something like that. Ideally it would be best to use something that actually parsed the code so you only changed the things you were really supposed to change -- e.g. if the EA had a string that said "The High value is:", you wouldn't want to change it to "The BT_High value is:" -- but I think a Perl parser would be good enough.

Well, it's still workable. If we require them to use BT_High, BT_Low, BT_iClose(), etc, it can still work.

It is workable...thats true... we just could use a candle array where all the values of the current candle of the backtest loop are copied to and the BT_functions work only with this array...

I don't think it's that hard. e.g. iClose() has a "shift" parameter. So BT_iClose() will also take a "shift" parameter, and we'll just add the current Backtest_Bar to Shift before passing it to iClose(). So if the EA wants the Close of 1 bar ago, it would pass a value of 1 to iClose(). If we're currently backtesting on bar 100, we'll just pass a value of 100+1 to iClose(). High and Time &etc would be similar, but BT_High() and BT_Time() &etc would just pass Backtest_Bar to the corresponding iHigh() or whatever function.

Ok...then lets do it...where do we start...

Well, we're starting already, figuring out some of these issues... and I'm sure we'll run into more!

Do you use any kind of Source control Tools? like Git or Subversion?

No, and I really should. I've got Tortoise SVN on my system but I never started using it. A shared repository like that bitbucket.org is probably a good idea for a shared project.

I don't know about your time but I would like to commit some time in this project because I really want/need some reliable testing and results that I can work with...And this is the one topic that bothers me most with Empty4...

Unfortunately my time is extremely limited -- I'm already spending too much time here and falling far behind on my paying job -- and it looks like it's about to get much worse. I might have to let you run with this, and I just act as a sounding board for design ideas.