Daily Fibonacci Signals

[tomele]

Hi.

He explained to me how to do it yourself. Load tick data from Dukascopy with "StrategyQuant Tick Data Downloader". Then convert CSV to FXT format with a script. You find everything you need if you google for "eareview tick data".

But even with correct tickdata, Craptester is still a mess and backtesting will lead to curvefitting if you dont do out-of-sample verification. You surely know. Others reading this post should continue here: http://www.stevehopwoodforex.com/phpBB3 ... ng#p108983

Cheers

[snailbeard]

Actually, this is not a fair comparison, because we need to take into account commissions. For a more useful comparison I should include the commission and convert it and add it to the spread, then it would be a more realistic comparison.

However, before we compare several brokers, we need to understand what the data and colours represent.

First lets look at 24 hours for 28 pairs. The code calculates a rolling 60 sample average for each spread, then half way through each hour I save the average to a CSV file. After 24 hours I then have an average for each hour of the day for 28 pairs. I also calculate the (D1 ATR / spread) and save that periodically as well.

Note that server time is GMT+2, so we should avoid trading after 10pm London time and next month that will change to 9pm. However, more countries and states seem to be proposing an end to DST, so next year might be different. Many of the successful EA trades are entered before I wake up so it would be difficult for me to enter the same trades manually.

To visually understand a large amount of data quickly, I used the conditional formatting to colour-code the numbers from deep blue (expensive spread) to deep red (thin spreads). Day traders need red spreads, whereas position traders holding trades for weeks can also trade blue spreads.

The following numbers are not spreads, they are the (D1 ATR / spread) - so the more price swings in relation to the spread the better (provided that we capture a significant part of that swing):

fxAtrToSpread-for-15Feb2017.png

So it is helpful to know the worst hours to enter and leave trades, but perhaps the data is incomplete because we really need to relate the spreads for each hour to the change in price for each hour as well, not just the daily range.

However, if you just want to know which pairs to avoid for day trading then just avoid anything bluish.

There are a number of assumptions buried in the calculations which are undermined by super price spikes which mess up the calculations, I will probably have to go back and scrap ATR and use something based on average price change which is less damaged by super-price-spikes than ATR, or I'll have to create a variant of ATR which isn't distorted by spikes.

In the next post I show a snapshot comparison between three popular brokers...

[snailbeard]

Hi.

He explained to me how to do it yourself. Load tick data from Dukascopy with "StrategyQuant Tick Data Downloader". Then convert CSV to FXT format with a script. You find everything you need if you google for "eareview tick data".

But even with correct tickdata, Craptester is still a mess and backtesting will lead to curvefitting if you dont do out-of-sample verification. You surely know. Others reading this post should continue here: http://www.stevehopwoodforex.com/phpBB3 ... ng#p108983

Cheers

Yes, back testing can lead to curve fitting. It has taken me a long time to discover most of the idiosyncrasies of CrappyTester. When people say back testing does not work, they mean they have discovered a number of issues with it and given up. However, we need to understand what the tool can and can't do and how to break up testing and data into parcels, so that code changes are generic and not curve fitted.

Optimisation is even more dangerous if one doesn't understand when and when not to use it.

Since pairs go through various mood swings over time, there cannot be a a single trading method or single SLTP choice. I use optimisation only to find out if there is an SLTP which works at the moment for a particular pair. It is curve fitting to the current mood and if the pair becomes more spiky or smooth then we need to re-optimise SLTP or stop trading that pair.

So there are a lot of hiccups and pitfalls to avoid, but it is the best way to find bugs and check the validity of an idea for a wide range of data. There are lots of optional filters in the EA, some of them fall into the curve fitted category and some are generally useful.

There are a lot of comments in the code along the lines of 'this filter kills too many good trades'.
So I would recommend persevering and learning how to avoid curve fitting.

Having said that it is quite possible that my EA is simply a collection of curve fitting algorithms, hopefully not but time will tell...

[snailbeard]

I have been intrigued by the progress of Greedy Wotsit and wondered if I am missing out on some good profits.

However, I didn't want to jump in the deep end without understanding how it would perform over several years and on a number of pairs. In particular, what happens if the EA is running across a major event like the EURCHF event of 15Jan2015?

It is difficult to test this without the exact data stream from the live feed since pending orders and stop losses can fail to trigger in these kind of events. There are limits to what we can know, but it is no good making profits for months and months only to have the whole account wiped out in one super event. So although we can't be sure what will happen to our orders, we might be able to find out what orders and stop losses are in the system when a major event happens.

I added some code to my EA which records unusually large bars from the D1 history and produces a summary at the end of a test. I make sure I run the EA over these spikes to see if trades are active during these extra large price swings.

However, I have been diverted from my gold, oops, I meant goal of automatically collecting and organising M1 data for a dozen pairs and gone off at a tangent developing a Greedy Wotsit clone, because I wasn't able to test the official version with Crappytester. Having thought about a grid trader several times myself and doing my own mental simulation I had previously concluded that poor trading conditions would eventually end up triggering both buys and sells resulting in lots of trades for little gain or worse. Unfortunately, I am not able simulate a basket grid trader in my head or with CrappyTester, so I am stuck looking at the behaviour of a single pair over a longer period of time of about two years (provided that I cut corners and use M15 instead of M1). These results will help me find issues but not give an accurate overall result.

My initial grid trader was pretty awful at first and given my initial prejudice about the long term out come of grid
trading I almost stopped working on it, but being a bit like one of those dogs that gets it's teeth into something and won't let go, I kept banging away at the keys until something interesting happened. If your hoping to read that I have discovered the secret of unlimited profits, well, no that isn't what I meant by interesting.

For quick tests I chose March 2015 because it contains a good mix of trending and ranging:

USDCHF-D1-Bars-PGX-trend-vs-ranging-Mar2015.png

I focused my initial development on this month and I needed some way to see if pending orders, buys, sells, stop losses and profit taking were actually doing what was expected. Some of the bugs in this code did not show up until I ran on over a years worth of data.

One would ideally like to see all these operations take place in simulated real time like the visual testing mode but it is a headache to keep track of multiple overlapping trades so I introduced an end of day running snapshot in a spreadsheet format:

USDCHF-running-PGT-nett-PL-per-day-March2015.png

where:
cDoY - current day of year (DOY) at about 23:45 server time
sDoY - the DOY on which the trade opened (assume same day as pending order)
Days - count of days running
Buys - count of running buy orders from a particular grid
Sells - count of running sell orders
B&S - total running grid trades from
Profit - running profit trades from a particular grid
Loss - running loss trades from a particular grid
Tickets - the tickets still open from a particular grid

So we can have trades open from several days starting from different days, but just consider a reverse bar against a long running trend.
We have only one simple method of opening trades but I have introduced a number of ways of cutting trades or trailing profit, but I have also added the option of only trading trends, and another option for half grid trading, which does not send the pending orders which are opposite to the dominant trend.

Now we start to get into the problem having too many choices and whether or not these are short term or long term options.

However, what is evident is that a day trading grid does not seem to be profitable for a single pair in the long term although baskets might produce different results. However, long running trades over days or weeks of trend accumulate pips and make up for the bad days, this is particularly good when each new day adds a new set of orders.

USDCHF doesn't have good clean trends very often and spends most of its time hitting stop losses due to the size of reversals, so over two years it turned into a leaky toilet.

I managed to turn off trading USDCHF for most poor trading cycles by checking for low volume and was able to end up in profit after running with two years of data:

USDCHF-2Y-sucess-hockey-stick-run16Mar17.png

Well this doesn't really look very exciting for two years of trading, however, most EAs left to run for two years unattended turn into leaky toilets so from that perspective it is a significant success.

It is important to note that I did have double the nett profit in just a few weeks initially, which was due to curve fitting for the initial test month.

USDCHF price movements over the last few years have not been good for trend traders due to the large reverse swings. The ideal trending pair has bars with short wicks, so I was optimistic that running this on a better pair like XAUUSD would produce better results.

However, with trading there is always a sting in the tail somewhere.
After some further bug fixing, long term grid trading XAUUSD looked like this:

XAUUSD-2years-small-profit-for-D1AtrSL-run19Mar17.png

This is surprising, I still need to see if it is due to a flaw or some aspect of curve fitting to USDCHF.

However, I also wanted to know how the code changes impacted on the previous trend following method.
Keep in mind that Grid trading is promiscuous whereas the previous method is extra picky, too picky and misses a lot of good entries, but never the less has more and stronger profitable runs, avoids deep draw down and finally makes strong gains towards the end which might be because previous test data was M1 , not M15 and only covered relatively recent history:

XAUUSD-bktst-M15-Jun15toMar17-run21Mar17.png

So what can we conclude from all these experiments if anything, was it a big waste of time?

It was definitely not a waste of time, as firstly the work uncovered some deficiencies in existing code, which should improve the breakout-momentum trend trading results.

However, the straight lines up and then down tell us that we have a great trading method if only we could turn it off in lousy periods.

So we might never make consistent profits unless we can calculate when to turn off trading completely in lousy trading conditions without turning off trading during good quality trends.

Does the Grid trading still hold any interest or is it just a good way of generating commission for brokers?
Having trades open most of the time exposes us to more major events which could wipe out our accounts altogether, whereas as selective entries reduce that risk, but fewer entries mean the results are statistically not significant.

Finally, the old method does not have the ability to manage overlapping trades from different days like the Grid trader and the one thing which made the Grid trader interesting was its ability to keep winning trades running. That is the one gem which could be fed back into the breakout-momentum trader to enhance it's profitability!

Both methods have the same problem in that we need to determine when to turn off trading altogether.

The question here for experienced traders is can you manually stop trading and do this better by looking at charts than it can be calculated automatically? And if you can do it manually why can it not be expressed mathematically?

[TraderJoeForex]

Hello,

I read this post with interest as working out a mechanical way to turn on and off grid trading is something we have been testing with the Flying Buddha indicator (which is based on price action around 5ema) and/or SuperSlope to highlight setups, followed by a 5ema cross, or HGI to trigger and only holding the grid for a maximum of 2ATR clear profit, then waiting out for the next setup and trigger. It would be very interesting to rerun your analysis with these filters. I focus on the higher volatility pairs rather than CHF crosses too.

Let me know if you want to do any more with this. Would be happy to help.

[snailbeard]

Hello,

I read this post with interest as working out a mechanical way to turn on and off grid trading is something we have been testing with the Flying Buddha indicator (which is based on price action around 5ema) and/or SuperSlope to highlight setups, followed by a 5ema cross, or HGI to trigger and only holding the grid for a maximum of 2ATR clear profit, then waiting out for the next setup and trigger. It would be very interesting to rerun your analysis with these filters. I focus on the higher volatility pairs rather than CHF crosses too.

Let me know if you want to do any more with this. Would be happy to help.

I am definitely interested to explore this further but a limit of spare time means being disciplined about focusing on a limited set of goals.

My major concern is what will happen to live accounts of grid traders at the next Swiss Central bank type price jump, so I prefer to wait until we know the answer to that and focus on something which is only trading occasionally, in the hope that it won't be trading the wrong way when one of these account busting events happen.

[snailbeard]

In a departure from the normal trend trading, I noticed some set-ups which provide us with an opportunity to enter some short term trades based on breakouts. The trend trader has been focused on a two-to-one profit ratio, which out of two years of data had some short periods of excellent returns. However, my risk aversion has found it's way into the code to such an extent that too many winning trades are filtered out and I now have to spend time finding an acceptable balance between winning and losing periods.

In the meantime, how about some shorter trades with a higher profit to loss ratio and a smaller ratio of wins to losses with the purpose of actually having some regular gain?

I recently added an H1 based OBV divergence filter which filters out entries when price is likely to start moving in the opposite direction, but I also thought this might be an opportunity to trade short range reverse breakouts.
In the past I have never been able to make OBV/price divergence reliable by itself, but perhaps if it is combined with other patterns some interesting things might happen?

The key limitations of this approach are a combination of spread and the size of back moves. Only highly liquid assets can be considered viable and only if it has small price reverses after entry.

The classic example is a price wedge followed by a breakout, but as experienced traders know, a lot of these patterns precede a false breakout just before the main price move.

Below is an example of a conforming breakout after a wedge:

XAUUSD-M15-3H1-breakout-setup-11Jan2017.png

This is an interesting day in the sense that it is the worst kind of day for a trend trader due to a almost equal measures of a swing down followed by a swing up.

Clearly, we will get burned if we are late entering or fail to take profit early, but we might get a 3 or 4 to 1 profit ratio which means we can stay ahead with a higher failure rate.

It is easy to produce a curve-fitted entry method to this particular pattern but how often will it win/fail over a long period of testing?

The coloured rectangles help to put the key features into perspective.
Indigo: The first sign of a serious breakout.
Brown: The spread is narrow enough
Teal: A suitable stop distance
Blue: An optimistic expectation of usable range

Rules:
1. the stop level must be several times the spread
2. the stop needs to be close but (say) above the recent M15 high
3. the usable range needs to be 3 or 4 time the stop distance
4. there needs to be an exit method or two
a) could use a multiple of the stop distance
b) could look for signs of exhaustion such as the doji bar at the bottom

We might experience a lot of failures when the D1 is showing a regular trend and the entry is in the opposite direction.

There are often examples of price swinging in the wrong direction after a breakout so we can expect a high proportion of failures.

In addition, if we over estimate the usable range this will also reduce profits or increase losses.
In this particular example our range appears to be about
a) 3 x the breakout bar
b) 4 x ATR(3)
c) 6 x ATR(100)
but our mileage might vary

[snailbeard]

This tongue in cheek implementation of short range range trading was run over about two years of M15 bars and as expected when left to run by itself with no intervention or calibration of SL/TP managed to produce a majority of losing trades, but still with a significant proportion of successful trades.

Can the calculation of stop range and target range be calculated in way that reverses the proportion of wins to losses, so that there is a cumulative gain rather than a cumulative loss?

In the following example from a different year than the first example we can see that a small adjustment is enough to move a loss from a loss to a win:

XAUUSD-M15-example-to-convert-loss-to-win-3Mar2016.png

So the trigger point is fixed but, there is not enough range for the stop loss and even the usable range is pessimistic.

There is a need to increase the stop range. The corresponding TP adjustment can be either
a) to accept a lower profit to loss ratio
b) increase the TP range

Since we have no idea what this means in terms of results over two years, we have a number of options
a) set these once based on two years of history
b) change them periodically based on a more recent history
c) try to turn off trading in the worst periods
d) some combination of adjustment and turning off trading

Note that the more tweaking we do the more we are moving into the realm of curve fitting and away from statistical or probabilistic out comes. So the preferred approach is shift parameters once for a long period of time.

[snailbeard]

Following on from the previous post, where a short range trade failed because the range was too short and needed a slightly longer range, we can contrast this to the main trend trading method which has an entry for the 4th March 2016, which allows more than necessary range for the stop loss and fails to reach the target price.

xauusd-M15-failing-trend-trade-4Mar2016.png

So here we see a stop loss distance which is twice the requirement and take profit price which would normally be well within range according to previous swings but it turns out to be a ranging day.

There are clues in the candle pattern after the peak and a confirmation by a second lower peak followed by a doji. This forms a falling M. Not visible on this chart is the volume which has dropped down as well.

A smart exit pattern detector could take profit here.

So we have two trading methods on subsequent days with two narrowly failing trades and perhaps some possibility of finding a better out comes for both methods?

[snailbeard]

Following on from the previous post, I have added a 10% increase and the improvement for the first 3 months of 2016 is dramatic:

XAUUSD-M15SRT-3months-successes-tilmidMarch2016.png

So clearly there are times when this adjustment is perfect.

Lets image that I don't trust back testing (which is very reasonable) but I do trust forward testing and lets say I started this on a demo in January and then after 3 months decided to start trading live. Unfortunately I would have missed the best period for these settings and I would be entering a new market phase for the next three months and by the time I reached June I would be in a very depressed state.

What is the way forward if we cannot trust back-testing or forward-testing?

How could we know that a winning method will suddenly become a losing method?
Are there are a new set of adjustments?
How would I know what they are in advance of a change in the market?

Although, the longer term results seem discouraging, we can take some comfort from an overall improvement to a final balance of 4845.47 after a previous final balance of 3976.84 from an initial 5000. So in the second attempt the EA practically came through over 12 months without blowing the account from a very crude implementation of a simple idea.