First all all, I am fully aware of the impact of optimisation and over-optimisation on a system. I'm also aware of the pitfalls of the back tester in Empty4. Having said that, I'm curious if anyone has ever tried this approach to an EA, automated or manual. A manual approach would be optimising every day or week and adjusting the EA inputs. This link below discusses how that can be done automatically.
Again, just looking to chat about this this, nothing serious here folks.
https://www.mql5.com/en/articles/1467
Automated Optimizer
-
nonlinear
- Trader
- Posts: 293
- Joined: Fri Jun 01, 2012 2:56 pm
- Location: Washington DC
Automated Optimizer
I have gone down this road sort of, as have many others in much greater depth than me. It has some merit, for some systems, in my opinion. But the chosen windows have to be carefully selected, again based on the specific trading system, I think. I have had so many tell me that I am wasting my time on this, but I am still trying. Maybe I am stubborn and have to learn a hard lesson, I figure another year and I will know lol
-
AnotherBrian
Automated Optimizer
Well we all know the market can change behavior and no matter what system you're running, it gets hit.
So same old story, if you protect yourself with stops you should be able to survive the turn. I guess you could argue that a trader could do this optimization manually every so many periods, if you can read charts, and tune the next week based on recent weeks.
You know when you backtest a system over years and you see one year it makes a killing, the next it fails, and you could optimize for each year based on the past weeks... and this leads back to your comment about how far back do we automate the optimization.
A moving average cross over would be an interesting experiment just to see the result compared to a one time optimized. Someones probably already done that, would be a good read. Ever heard of such a test?
So same old story, if you protect yourself with stops you should be able to survive the turn. I guess you could argue that a trader could do this optimization manually every so many periods, if you can read charts, and tune the next week based on recent weeks.
You know when you backtest a system over years and you see one year it makes a killing, the next it fails, and you could optimize for each year based on the past weeks... and this leads back to your comment about how far back do we automate the optimization.
A moving average cross over would be an interesting experiment just to see the result compared to a one time optimized. Someones probably already done that, would be a good read. Ever heard of such a test?
-
nonlinear
- Trader
- Posts: 293
- Joined: Fri Jun 01, 2012 2:56 pm
- Location: Washington DC
Automated Optimizer
Yes, Pardo did it, but I heard it failed (unconfirmed). Regardless, his book is a good read, although somewhat dated now. http://www.amazon.com/The-Evaluation-Op ... 0470128011
-
nonlinear
- Trader
- Posts: 293
- Joined: Fri Jun 01, 2012 2:56 pm
- Location: Washington DC
Automated Optimizer
I should also add that I don't rely on or trust Empty4's optimizer (it's horrible as many here have noted). I run my test windows outside of Empty4's optimizer based on OHLC data only, so I understand exactly what my results mean (hopefully). Even then it takes me forever to run studies and I'm not nearly as advanced as some running tests with GPU's. My personal view is that I want as short a test window as possible so long as it is statistically significant (sometimes that is many years, sometimes a quarter year - another topic for debate). Many many algo traders disagree with me on this point and strive to find a system that works over much larger Windows (e.g., 10 years), but I am not trying to fit a square peg in a round hole. I want to recognize that I have a round hole right now and use a round peg for it until it changs to square, if you follow. Anyway, I have yet to find anyone prove me wrong with real data, primarily I think because the primary way to prove this wrong is to trade it live as it cannot be accurately modeled easily....
-
RedLineFred
- Trader
- Posts: 52
- Joined: Wed May 01, 2013 6:15 am
- Location: Brisbane, Australia
Automated Optimizer
I agree Empty4 sucks at most things and especially back-testing and optimizing, so what do you use to test?
I am having some good success using simple old Excel (admittedly with VBA code to assist), and it also allows me to do portfolio testing.
Using templates to manage the trades allows testing to be run fairly quickly. Once the strategy has been coded, a change in parameters can be run over a portfolio of 40 instruments, looking at ten years of daily bars, and produce an output result/graph in less than 15 minutes
With regard to the look back time frame (round hole square peg) i think it comes down to sample size. If you have a significant number of trades, you should be able to gauge the strategy's effectiveness. How many samples are needed? not sure, but i look for at least 100.
Also by testing across a portfolio, you effectively increase sample size without needing to drill back too far in time.
I am having some good success using simple old Excel (admittedly with VBA code to assist), and it also allows me to do portfolio testing.
Using templates to manage the trades allows testing to be run fairly quickly. Once the strategy has been coded, a change in parameters can be run over a portfolio of 40 instruments, looking at ten years of daily bars, and produce an output result/graph in less than 15 minutes
With regard to the look back time frame (round hole square peg) i think it comes down to sample size. If you have a significant number of trades, you should be able to gauge the strategy's effectiveness. How many samples are needed? not sure, but i look for at least 100.
Also by testing across a portfolio, you effectively increase sample size without needing to drill back too far in time.
-
nonlinear
- Trader
- Posts: 293
- Joined: Fri Jun 01, 2012 2:56 pm
- Location: Washington DC
Automated Optimizer
We are essentially doing the same thing, sort of. This is going to sound odd, but I take Empty4 offline (logout of account, disable internet access, and delete all bars). Then I load my HST files into Empty4 and confirm they are accurate. Then I have an ea work off of the HST files to generate thousands of arrays of statistical data (and load them in the file folder of Empty4). I then take the array results and load them to another online version of Empty4 for use in trading. So I am using Empty4, but not as intended, and I am not restricted to crap Empty4's methods, which is the primary reason I took this path many years ago (yes, it may have been a complete waste of effort and time). One issue I have is that I occasionally run into Empty4's memory limit, which really sucks - should have used a dll. As for sample size, I am still testing, but I think 30-40 really might be enough, depending on the system (again, many disagree with me here). All that matters is that it walk forward somewhat well, in my view anyway. Here is a test I am running using a trade sample size of 30 on a 15M system. Still too early to gauge anything. https://www.myfxbook.com/members/nonlin ... 15/1405353 If I get this to work, I may try to get Empty4 to use the recently received live bars and update the arrays every weekend, but I am not there yet.
-
Timid
- Posts: 6
- Joined: Mon Dec 07, 2015 5:32 pm
Automated Optimizer
Interesting topic!
Maybe this is not relevant and then I am glad to be informed (always happy to learn :hi: ), but the people at StrategyQuant http://www.strategyquant.com/ have as one of their methods the suggestion to re-optimise a trading method every so often. Their system will suggest how often to re-optimise and over which period. The example they have in their documentation suggests to re-optimise that particular example every 6 months with a period of 12 months history.
Timid
Maybe this is not relevant and then I am glad to be informed (always happy to learn :hi: ), but the people at StrategyQuant http://www.strategyquant.com/ have as one of their methods the suggestion to re-optimise a trading method every so often. Their system will suggest how often to re-optimise and over which period. The example they have in their documentation suggests to re-optimise that particular example every 6 months with a period of 12 months history.
Timid
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
Automated Optimizer
I'm still working towards this, although I have probably wasted more time than most, but I'm going slowly forward on the following basis:
Currency liquidity and price swings have changed year by year, such that some 'great' EAs of mine on previous years data are useless today.
The best non-curve fitted EA that I have so far works on some pairs/indexes/stocks based on continuous long term direction producing either HH, or LL. How do I know that I am not curve fitting: Use exactly the same EA on different assets without any tweaking and the results are similar. It would be great if the same method also works on shorter time scales - but it'll be some time before I can test that.
A difficulty with optimization that I have done in the past is that in the beginning I have a simple entry method which generates a lot of trades, so we can say the first version produces statistically sufficient data, but optimising produces two few trades to be statistically reliable (the curve fitting conundrum).
Perhaps, keeping the number of entries high but focusing on smaller losses and bigger gains could be more reliable than a high win to loss ratio (which is paraphrasing what we read in most books on trading: keep your winners running and cut your losses early - so perhaps I am restating the obvious?). I have found it hard to take the recommended approach on shorter time scales without excessive stop-outs. The longer the time-frame the less influence price manipulation has on the method, so perhaps this is why the stacked entries based on daily HH/LL is the the most promising candidate for optimization.
Nevertheless, your thoughts on this have prompted me to think about revisiting the DFEM (Daily Fibonacci Entry Method) which produces a lot of trades (statistically significant).
So the issue of automated optimisation could mean different things on different time scales. Are we discussing the same kind of optimization?
It looks like NL has done a lot of hard work trying to work with different sources of data. I also had the issue of totally different results on different brokers with the faster trades but less impact with the slower daily approach.
So are we talking about making a particular entry/exit method better or about avoiding some pairs and selecting others?
I am happy with my trading algorithm but optimization means choosing the best pairs to trade.
So far I am part way in to a C++ optimization framework, there is an (open-source) SQL library which might be good for querying tables of results, there is a (open-source) 2-D & 3-D graphics library to help produce visually interesting plots of multiple pairs. It is going to be a very long time before any of this actually does anything useful. I am still just reading dirty data and trying workout what to do about it...
Currency liquidity and price swings have changed year by year, such that some 'great' EAs of mine on previous years data are useless today.
The best non-curve fitted EA that I have so far works on some pairs/indexes/stocks based on continuous long term direction producing either HH, or LL. How do I know that I am not curve fitting: Use exactly the same EA on different assets without any tweaking and the results are similar. It would be great if the same method also works on shorter time scales - but it'll be some time before I can test that.
A difficulty with optimization that I have done in the past is that in the beginning I have a simple entry method which generates a lot of trades, so we can say the first version produces statistically sufficient data, but optimising produces two few trades to be statistically reliable (the curve fitting conundrum).
Perhaps, keeping the number of entries high but focusing on smaller losses and bigger gains could be more reliable than a high win to loss ratio (which is paraphrasing what we read in most books on trading: keep your winners running and cut your losses early - so perhaps I am restating the obvious?). I have found it hard to take the recommended approach on shorter time scales without excessive stop-outs. The longer the time-frame the less influence price manipulation has on the method, so perhaps this is why the stacked entries based on daily HH/LL is the the most promising candidate for optimization.
Nevertheless, your thoughts on this have prompted me to think about revisiting the DFEM (Daily Fibonacci Entry Method) which produces a lot of trades (statistically significant).
So the issue of automated optimisation could mean different things on different time scales. Are we discussing the same kind of optimization?
It looks like NL has done a lot of hard work trying to work with different sources of data. I also had the issue of totally different results on different brokers with the faster trades but less impact with the slower daily approach.
So are we talking about making a particular entry/exit method better or about avoiding some pairs and selecting others?
I am happy with my trading algorithm but optimization means choosing the best pairs to trade.
So far I am part way in to a C++ optimization framework, there is an (open-source) SQL library which might be good for querying tables of results, there is a (open-source) 2-D & 3-D graphics library to help produce visually interesting plots of multiple pairs. It is going to be a very long time before any of this actually does anything useful. I am still just reading dirty data and trying workout what to do about it...
-
nonlinear
- Trader
- Posts: 293
- Joined: Fri Jun 01, 2012 2:56 pm
- Location: Washington DC
Automated Optimizer
One can optimize in different ways for different pairs, but I try to keep it as simple as possible, basically focusing on win rate, reward to risk, and the number of trades (honestly not much different than PF in Empty4). I also embrace that I am curve fitting in many respects (I am not trying to avoid it), but my systems have only a few variables. Among the difficult parts are (i) having faith to test live and (ii) setting the test and forward windows. I reoptimize weekly at the moment (for live testing) and try to keep my test windows as short as possible while also statistically significant (a delicate balance). I am not recommending anyone take my approach and most knowledgeable traders will tell one not to do what I am doing (particularly some very experienced quants strongly disagree with my approach). I just grew tired of seeing solid systems fail simply because the markets changed. You can read various thoughts on how many trades are statistically significant and even some heavy math on this, but I don't know if it really is of much use (although I am very curious how some professionals approach this).