Daily Fibonacci Signals

[snailbeard]

In an earlier post there is a chart showing Fibonacci relationships between MN1, W1 and D1 and I proposed a method of detecting poor trading conditions when price movements contract for several weeks. This method has been successful for a small range of dates and so far just EURUSD, I have yet to find out if it interferes with normal trading conditions.

The following table shows a much reduced number of trades between late August and mid October:

eurusd-trades-after-fibonacci-filtering-SeptOct2016.png

So above are the results using the EA's default stop-loss and take profit, but using the optimiser I find that due to large price swings we need to use much higher values of SL and TP. Using a significantly higher value we end up with even fewer trades:

Code: Select all

1	2016.10.11	09:51:00	sell	1	0.09	1.11227	1.13205	1.08647	0	5000
2	2016.10.13	10:12:00	close	1	0.09	1.10012	1.13205	1.08647	99.31	5099.31
3	2016.10.14	07:00:00	sell	2	0.08	1.1031	1.1247	1.07418	0	5099.31
4	2016.10.14	18:25:00	close	2	0.08	1.09951	1.1247	1.07418	25.73	5125.04
5	2016.10.18	17:57:00	sell	3	0.08	1.09744	1.11842	1.06984	0	5125.04
6	2016.10.21	18:25:00	close	3	0.08	1.08692	1.11842	1.06984	77.43	5202.47

[/size]

So we have succeeded in turning off wasteful trading under certain conditions, these changes are making the EA very picky, perhaps too picky and so the next step is to check the impact of these changes on Gold and other pairs...

[snailbeard]

In the following chart we can see strong downward momentum, the large olive rectangle shows the previous weeks range and the small green rectangle shows the previous days range, the new Fibonacci filters can block range trades, but there is significant momentum going up, would you trade the breakout?

gbpchf-would-u-trade-the-breakout-0400-12Jul2016.png

Perhaps there is no correct answer to this question, sometimes the breakout will have enough momentum, but sometimes it won't.

The following chart could help make a decision, it shows Fibonacci support and resistance levels and also the first part of the chart helps us gauge momentum. I have only read briefly the description of CommanderComfy's method but perhaps you can read this chart in a similar way? That is: price should have momentum and price should not be approaching a significant Fibonacci level:

GBPCHF-H1-momentum-in-a Fib-context-12Jul2016.png

I don't won't to stretch the analogy too far because the duration of trades might be different, so I have ignored the faster time-frames.

Once again it is important to reflect on stop-loss and take profit:
If we allow range trades and the pair we are trading needs a large SL/TP then our range trades might never reach their target.

[snailbeard]

Every new piece of code and analysis can have unexpected consequences. The recent work to detect dead weeks or months by Fibonacci relationships has had some interesting consequences.

Whenever I come up with a new kind of filter I know that it is likely to block some valuable trades, so each filter has to have a yang to it's ying, the EA's internal universe must remain balanced. So as soon as I was happy with results on EURUSD, I went to GBPCHF to find out what damage would result from this new filter. As expected some of the good trades had disappeared.

So how do you compensate for a dead zone filter?
The answer must be that a dead zone implies a lack of sustained momentum so we need the corresponding sustained momentum analysis and detection code. The entry signal detection is looking for shorter momentum, but longer term momentum can be determined over several time-frames.

In this case D1 might be ranging but H4 and faster should be racing along. Most of the code to do this analysis already exits, it just requires some boolean logic to produce either: yes or no.

The following extract is still in the testing phase:

gbpchf-strong-reversal-detection-by-fibs-May-Jul2016.png

The first column is the day-of-year, then the date, followed by flags for strong down or strong up.
The rows correspond to the first time during a day that the state becomes true, on a good day it will remain true for most of the day.

The first surprise was that these flags were true for so many days (possibly a bug) because it takes a lot of other conditions to be true for either of these flags to be true. It is not surprising that down is more prevalent than up, since we know that the long term trend for GBP against CHF is down.

However, we should only have a flag set if price is moving in the opposite direction to the previous month and week. You might have noticed that most values are clustered, which is to be expected when the current week is running strongly opposite to the previous week and month.

What is not expected is to suddenly go from up to down and vice a versa (possibly due to bugs).

In the table above the first entry is for the first of June, it is quite common for a new month to kick off a new trend, but the analysis was originally expected to find strong counter moves in a longer term trend, however, according to the following chart it appears to have detected a full reversal:

GBPCHF-H4-strong-reversal-detection-may-jun2016.png

On the left side, in purple is May, clearly finishing higher, B is the last week of May also finishing up.
The thin green rectangle is part of the current week of interest, and it is also the previous day to the current day of interest. It is clear from the fast moving averages that H4 has considerable downward momentum.
However, since we have to rely on H4 we don't get a flag until 12:00 although with some extra work we might be able to get it at 08:00.

The previous runs for GBPCHF could never get an entry through the filters on this day because of the upward momentum but an unexpected consequence is that this new bit of code might just change that...

[snailbeard]

Last week I started making some changes which went deeper than expected, but in the process I uncovered some subtle problems in estimating the daily range. Sharing code between the FBATT script and the EA has also resulted in brain strain, but with the help of excessive amounts of coffee, FBATT is hopefully a little more accurate than before and I had some dramatically better results for GBPCHF than expected. GBPCHF has been a pair that has rarely been good for the account balance in the past, so it was very rewarding to retest the EA from July to October.

This was the resulting balance/profit:

Code: Select all

32	2016.10.27 23:59	close at stop	16	0.09	1.20808	1.25167	1.16607	30.20	5713.26

[/size]
This is higher than my target but at the same time you might have noticed that there were only 16 trades over a period of 5 months. Does that seems a little on the low side?

The experienced EA developer never wholly trusts the results of back testing especially when they are better than expected. Any deep code changes are going to introduce unexpected consequences which could remain hidden for sometime. Certain run time parameters can mask a particular kind of bug. Do you terminate you open trades before the weekend? Sometimes it makes sense to avoid the cost and risk of leaving trades open all weekend. However, if you are developing an EA setting this parameter to false might uncover some bugs.

I was so pleased with the balance I almost didn't look closely at the monthly break down. In addition the recent work to skip over poor trading periods could convince me that this was not unreasonable.:

gbpchf-super-long-trade-29Jul-20Sep16.png

Does 7 weeks of poor trading conditions sound unlikely?
Another difficulty here is the large difference between the shown stop-loss, take profit prices and the real stealth prices which can been extracted from the log file.

If we study the following chart it might be possible to come up with some theories about the nature of this bug:

gbpchf-D1-chart-shows-missed-stp-ssl-29Jul-Sep16.png

The period before A is one of those bad trading periods that we are trying to skip over. At A the EA determines it is a good time to allow trading again. The entry looks reasonable on the D1 chart.
Would you consider the stealth-take-profit (STP) price to be too far away from the entry?
So is this the bug or is it a secondary issue?
That depends whether or not the stealth-stop-loss (SSL) is also larger than necessary.
According to the D1 bars at B we should have had an exit by SSL so that is a confirmed bug.
A few days later price travels past the STP level and again nothing happens, so I suspect B and C are the result of a single bug.
We can observe that the D1 bars are small in comparison to the size of take profit, but that might be the value I put in after running the optimization for these values, so not necessarily a bug but something to reflect on.
There appear to be some good trading periods between B and D so it looks like we are missing out on hoovering up some easy pips.
Eventually, price reaches E and the STP code is suddenly working again after a long summer holiday.

Do you have any theories about the nature of the bug?
Where would be a good place to start?

I suspect this is a recent bug and therefore a consequence of new trading hibernation code,
in one of two ways
a) STP, STP checks never get executed
b) They are executed but are dependent on data generated for trading signal generation
It might be enough just to review the source code to pin point the problem.

[snailbeard]

So the above problem of a super long lasting trade, was because checkForExits() was not called in that period, but it wasn't so much a bug in the code as a bug in the data. There is already code to detect inconsistent bar data, and large gaps can occur around Easter holidays and Christmas holidays, so we don't won't to abort testing just because of these gaps, but now I have added a check for a gap of 5 or more trading days and a pop up message (forcing me to notice that something unusual happened).

In the log file there is now a detailed record of the hole:

Code: Select all

2016.09.20 19:22  FxPro GBPCHF,M1: Chart symbol: GBPCHF
2016.09.20 19:22  FxPro GBPCHF,M1: using Empty4_WMCMD_REMOVE_EXPERT
2016.09.20 19:22  FxPro GBPCHF,M1: About to drop expert from chart
2016.09.20 19:22  FxPro GBPCHF,M1: New-Day-Bar: Bar count on symbol: GBPCHF with PERIOD_D1 is 1085
2016.09.20 19:22  FxPro GBPCHF,M1: ExpertRemove function called
2016.09.20 19:22  FxPro GBPCHF,M1: forceAppExit(TRACEPOSITION)
Hole between minutes is >= 5 days 
2016.09.20 19:22  FxPro GBPCHF,M1: Previous time: 2016.07.29 14:39, New time: 2016.09.20 19:22
2016.09.20 19:22  FxPro GBPCHF,M1: Aborting testing due large hole in M1 data!
2016.07.29 12:00  FxPro GBPCHF,M1: 

[/size]

So this back test runs using period M1. To properly test your EA you will have to run it for several brokers and different pairs, but keeping track of all these combinations is hard work.

There is plenty of data available for free from Dukascopy/Tickstory but it has Sunday bars and a different server time offset to other brokers, so I need a simple program to shift the bars as required.
Should I start with Minutes and regenerate the other time frames or apply a-read-shift-write independently to each time-frame?

Those of you that follow my earlier posts will know that I already did some work on reading data, analysing it and filling some issues with missing minutes and extra minutes, but it does not have a time shift operation as yet.

Before assuming that this will be easy lets look at some data from different sources, and lets say that at any one time we are interested in one weeks worth of bars, because a broker can change their server time offset several times a year but this is likely to be at the weekend.

The following samples show the kind of data that we might have to work with:

data-variation-in-bar-start-of-day-2016.png

So from this we can see that forex pairs, including gold, are much more straight forward than indexes and shares, but gold needs a tweak because D1 should always start at 00:00 even though the other time frames start at 01:00, in most cases H1 and H4 start on the same offset except for UK100.

FxPro never provides the first minute of the day, we could always set it to a copy of 00:01. So for forex pair a pipeline read-shift-write should be fairly straight forward, but we need to have a look at the Dukascopy data before we make too many assumptions.

[snailbeard]

So I have finally taken the time to take advantage of the charting features of a spread sheet
In the following chart I have overlaid the results (balance/net-profit) of three separate runs, with the only change being the ATR based stop-loss. The results are slightly skewed by a favourable price surge due to weakness in GBP. The important thing here is that the EA doesn't try to capture any short-range counter trades which could be caught out by the price spike in the general trend:

GBPCHF-variation-in-profits-by-SL-July-Oct2016.png

So in addition to generally positive results, we shouldn't be complacent about some of the losses which are occurring in all runs. We should try to understand the factors that lead to a sequence of losing trades. In particular there is a period in July which is not at all good for trading and being able to classify the generic aspects of this period should help to turn off trading for a week or two.

[snailbeard]

So there could be several ways to classify bars as stalled, but in essence we are looking for a price confined within a range. In the following sheet we can see the bad zones coloured salmon pink, the improvements are green and the unwanted filtering of winning trades are marked orange:

GBPCHF-trades-before-and-after-kill-stalled-bars-4Hov2016.png

The loss of some winning trades might be a necessary compromise unless I can determine a consistent difference between stalled and not stalled.

[snailbeard]

Most traders here will be familiar with the triangle (or pennant) breakout pattern. The price range of each successive bar size reduce to a minimum, followed eventually by a breakout, when the market has determined the dominant price direction.

In the following chart we see a type of triangle which might be tagged as angelfish because the directions alternate giving the appearance of stripes:

gbpchf-D1-angelfish-pattern-July2016.png

If I was trading manually I would easily identify the ranging behaviour and avoid trading until the market determines the preferred price direction. Initially the high speed red and blue moving averages indicates price has been moving upwards, however, price is approaching the slower green falling moving average.
The EA (not having logic to spot every warning pattern) initially finds upward price momentum for the bar labelled B, in the context of recent daily upward momentum, but observe that the preceding bars have a middle price which is approximately the same. We can use simple arithmetic to indicate that the bars have stalled, but will such a simple test kill off some winning breakout trades?

[snailbeard]

It is has been a while since I talked about the work done on riding out the EURUSD, which continues to jump around in very unhelpful ways. I have only just completed the work on GOLD and GBPCHF, which required some deep analysis of ATR. There is enough work on this to fill a small book and if anyone is interested I'll post about some of the insights I have gained from it. These insights have helped me achieve better filtering and additional winning trades, ultimately we are interested in net profit over a given period, so how does ATR figure in the mix?

Anyway, back to EURUSD, we are now interested to know whether or not I was just curve-fitting to GOLD and GBPCHF or do the changes affect the results of EURUSD? We are trying to find that fine balance between ranging and genuine breakouts.

In the following chart we can see the EURUSD has a long ranging period, followed by some strong movements, but also some killer reversals:

eurusd-D1-mostly-hopeless-trading-conditions-Sep-Nov-Oct-2016.png

Firstly, the ranging filter is doing a great job of protecting our capital against trading in a poor trading period, but then we see price developing a downward bias and building up some momentum, however, the EA holds off until 'A' where on the 11th October the range filters are cleared and we have a satisfying entry, but we get one good day before again the price is ranging and jumping around, B is potentially a good trading day after a pull back, but we run into several days of whip-lashing prices at C, although according to momentum we should be getting some sell opportunities, the EA struggles to find good entries and gets caught by some of the whip-lashing, fortunately, it was not caught by the giant reverse spike on the 9th of November, as that could have really exceeded the stop-loss. I hope none of the other EAs from this forum got caught be this super spike!

So the question now, is it a matter of manual judgement to avoid trading some pairs for the time being, or are there technical aspects to these reversals which can warn the EA of impending doom?

[snailbeard]

In good times when there is a steady trend ATR provides a good way to make the take-profit and stop-loss dynamic, but how do we survive these roller coaster rides and are our days numbered by these sometimes unexplained and sudden surges in price?

The normal or popular way to determine the daily range is to use D1 ATR14 but unfortunately this gets skewed by long periods of low volatility or sudden giant swings this can confuse many EAs. One way round this is to calculate variations on ATR on several time-frames, often there is a relationship between the size of bars on a particular time frame and the daily range. To begin with I started comparing D1 and H4 (gold):

Predicting_the_daily_range_by_ATR_Nov2016.png

As we would expect H4 moves around a lot more than D1 but this chart does not tell us whether or not H4 is more accurate or more inaccurate.

Would it be useful to know if your daily range forecasting method is accurate or not?
This should be straight forward with quality software and data but it turns out not to be straight forward with MetaCrap and CrappyTester...

So the theory is that I add a function to write all this at the end of the current day, however, when I tried to do this I was reminded of the movie: "The edge of tomorrow". It is easy to detect the start of a new day and write into the start of the new log file but writing at the edge of tomorrow was tricky because sometimes several minutes or even several hours of minutes are missing at the end of the day. In retrospect, I would have coded this differently, but here is the first attempt at comparing range forecasts to reality:

gold-compare-predictions-to-actual-Nov2016.png

Unfortunately there is rather too much information to process to get a quick understanding of the chart. However, the first observation is that actual range (yellow) jumps around a lot more than than D1 forecasts, perhaps this in itself tells something about the erratic nature of the market. We can also see that sometimes the forecasts are close to reality. If we look at successful months and unsuccessful months and different pairs will we notice any correlations? We are not even interested in all the forecasts - if we have already decided not trade for other reasons then the forecast is irrelevant so perhaps we are only interested in forecast for the days we actual chose to trade?

From studies of ATR and range forecasting I know how the daily forecast can have more impact on profit and losses than the actual method of trading. Adapting the SL/TP values has shifted loss making periods into profit making periods. There are periods when ATR is reducing day by day which can still cause some potentially profitable trades to over shoot so perhaps there is more to learn...