stevehopwoodforex.com
https://www.stevehopwoodforex.com/phpBB3/
Print view

Daily Fibonacci Signals
https://www.stevehopwoodforex.com/phpBB3/viewtopic.php?t=4100		 Page 6 of 21
Author:  snailbeard [ Sun Jan 10, 2016 8:17 am ]
Post subject:  Currency Preference Indicator
Its a new year and time to think about making steady profits. It is time to stop beating myself up for programming in MQL instead of C++. Why would anyone want to struggle with C++ when they could use Java or Python. Now there are new languages like Scala and Haskell. Well if I need to work with either Empty4 or Zorro then I'll probably have to make DLLs and if I want to test ideas out on lots of data and several pairs then speed might be necessary for testing and if I need to jump into a trade partly based on second by second price movements then perhaps speed is essential.

There are also cases where speed is not required, for example, I have some files and I just want to extract some information, do something with it and save the result in another file. Python is a great language for quickly converting an idea into a working program with a lot less typing.

I'm not suggesting that anyone start learning C++, rather I would say don't learn C++ or Haskell unless you understand the commitment of marriage, because a lot of C++ programmers never discover the real power of it, by learning how to design templates - but that is a topic for a completely different forum, but if you are reading this you are probably interested in doing some programming, perhaps just at the edge knowing just enough to tweak MQL or more seriously. For those with the time and interest knowing two serious and radically different languages can be very useful.

So my New Year resolution is to brush up C++ and learn Haskell. Why Haskell? Haskell is a pure functional programming language and also like Python you can do more with less effort (once you have learnt it). Meaning that solving a problem in Haskell requires a different way of thinking about the problem and being forced to do this might hurt in the beginning but adds value to your mental tool-kit.

Anyway the first thing I need to code in C++ is currency preference indicator. It sounds like the currency strength indicator, but the CSI looks at all currencies, whereas the CPI is intended to show which currency is in the driving seat and which other currency is the better choice for our next trade. It is a pre-selector and an anti-correlator (only trade the best pair for a given set of possible pairs).

At the moment it only exists in my head and I imagine that it would look something like this:
LiveCurrencySurgeIndicatorProposal.png
In this example, USD is the driver and either EUR or GBP are more promising than the alternatives.
At the moment, this idea is based on real-time price changes looking back in seconds, minutes or hours. We know that the big moves of the day can sometimes be over in minutes. So as well as preference indicator it might be used to exit some bad trades.

For long running trades (weeks), a daily surge might not be relevant, or we might just have hit one of those hidden super SR reversal prices...
Author:  snailbeard [ Sun Jan 24, 2016 10:26 am ]
Post subject:  All About Time
I am a small step closer to the CPI (Currency Preference Indicator). Several traders in here have experimented with Zorro, so I took a look at the file format for GBPUSD_2014.bar, each bar consists of 4 x floats and 1 double, but no volume data.

Code: Select all

typedef struct ZTICK
{
	float fOpen, fClose; // 2 x 4 bytes
	float fHigh, fLow;    // 2 x 4 bytes
	DATE	time;	// time stamp, GMT, resolves to 8bytes = 64 bits
} ZTICK1; // .bar file content
For some perverse reason DATE requires 8 bytes of data to represent bars which resolve to minutes.
In C++ cross-platform time is available in two main flavours
a) clock time - which resolves to seconds since 01/01/1970 upto 2038
b) hi-res-clock - which resolves to nano-seconds since 01/01/1970 which is useful to those folks trying to create black-holes in CERN (perhaps they would say nano-seconds is too slow for their purposes).

clock-time only requires 4 bytes, and nano time requires 8 bytes.

When MetaQuotes updated between versions a couple of years ago they broke my DLL because I didn't realise they had changed their clock-time from 4 bytes to 8 bytes.

Zorro time is a hangover from MSW OLE which is more weird than quantum mechanics. Somebody who must have been high at the time decided they would represent time as a combination of whole days and fractions of days, and they thought that the 01/01/1970 was just arbitrary, so where do you suppose MSW time starts?
0000 AD??? or 0000 BB (big-bang)???
Wrong, it starts 1899, 30 December - wait, I must have ms-remembered that, let me go and double check,
maybe it was 1900, 31 December, or was it 1900, 01 January.
Good job I wrote it done in my notes - its the first one 1899, 30 December

So to recap
a) 8 bytes instead of 4 bytes
b) nice easy to remember start date
So if we look inside the binary file and display the data in various formats...
hex: 40E4829949F49F4A
double: 42004.7902777777781
long: 4676005907999792970

This is not a very convenient format when converting to M1, M5, M15,...

Nor did I find some ready made code for translating perverse time to integral seconds.
You might find it helpful to know that there are 257.63125 trading days in a Zorro year.

Anyway, if any one here needs to load a Zorro file into memory in a more concise format, there is some code to load it as array of 20-byte bars.

Well at least we now have some high quality data to work with... or do we?
Some years ago I said it would be useful to have a program that goes through history data and cleans it up, but I thought it would be a lot of work. However, the CPI won't be very good unless the data from several currency pairs line up.

So how bad is the Zorro data? - probably no worse than most other broker data.
After a first go at scanning GBPUSD 2014 it has the following quirks:

Frequent duplicate minutes
Missing minutes
Extra minutes - hours with more than 60 minutes

So what is the best way to trundle the the stream of minutes - should we stop at errors and clean up before carrying on, or should we restructure the data into a tree first, then go back and repair bottom up from minutes to hours?

Something I haven't mentioned is checking the quality of the price data, that requires some extra code to check the correctness of OHLC and log suspicious price spikes which might not be real values.
Author:  snailbeard [ Thu Feb 04, 2016 6:32 pm ]
Post subject:  Tick Tick Boom
Are you struggling with bad data from your broker?
Do you pay for good quality data?
Could we gauge how dishonest our broker is by looking at the errors in the price information?

From extensive use of CrappyTester I have had to work around small gaps, large gaps and extreme values.

Are we being paranoid when we believe the data providers are deliberately messing with our trading platform?

Perhaps on windy days the bits in their mainframes get blown around and end up in different bytes or perhaps on rainy days some decimal points float off by an order of magnitude.

Perhaps there should be an award for the most dishonest and corrupt price data?

I have recently being experimenting with loading data from Zorro Trader and some of the data is bad in new ways that I had not imagined. So now I am having doubts about their claims to be helping the retail trader. Another thing to consider is the default broker which is FXCM, quite a few of their customers have complained about dodgy practices.

Can TickRadar help cheated customers?
I came across this tool by accident from an article in financemagnates.com, I have registered for the free version of the tool, but haven't tried it.

Back to dodgy Zorro data... in addition to the usual issue of missing bars, there are also duplicate bars and you can end up with more than 60 M1 bars in an hour if you have a tree like data structure which adds M1 bars to the current hour (of the current day of the current week) until it finds a minute with a time-stamp from the next hour.

What would be your approach to loading a years worth of M1 bars?
Would you load it into a single linear array, ignoring gaps and duplicates?
Would you try to modify and rectify the data on the fly?
Perhaps load it first then start fixing it?
Move on to a different source like Dukascopy?

For speed of access data is normally held in a single dimension array.
So why have I gone to the trouble of creating a tree like structure?
I thought it would be useful for locating issues and massaging the data into the correct number of bars, before converting back into a simple array.

Until we measure the quality of any particular source of data then we don't know to what extent we can trust it.

For me, the easiest way to produce an Empty4 compatible DLL is to use an C/C++ IDE like CodeBlocks. Although it is quite sophisticated it is not as 'bloated' as Eclipse. Don't get me wrong I use Eclipse as well, but there is a bit too much of it.

Dukascopy has JForex for Java programmers, plus apparently you can also make DLLs from Java according to this post:
http://findnerd.com/list/view/How-to-cr ... java/1934/

C-Sharp is a better starting point for occasional programmers and MS-VS easily produces DLLs, this would be my first choice if I didn't already know C++.

Now what can a great IDE do for us?
I use a few defensive techniques which help catch bugs such as using 'assert( some-expression )' which halts the program if the expression is false.
Lots of trace logging in debug-mode so we know what happened leading up to a problem.
Finally judicious use of break-points when debugging.
In the following example assert was about to halt the program:
HourNode-AddM1-sanity-checks.png
This 'sanity' cross-checks the number of minutes counted in the current hour and program previously halted because assert check that the maximum minutes should be 60 and it was 61.
Now we want to know how we arrived at 61 bars in one hour.
That only makes sense if the same minute occurs several times or the code missed the start of a new hour.
We can also examine the call history:
HourNode-call-stack.png
Clicking on the previous function 'unwinds' the program and we can inspect variables from the previous scope.
When I was writing the code I thought it would be useful to have a view-point from the current minute backward and another view-point forwards to the next M1 bar, as its a convenient way of determining 'edges' between gaps, weekends, holidays, hours, days and weeks.
A good debugger lets you examine nested data structures in a watch window:
HourNode-AddM1-debug-watch-window.png
Note that the time in human readable form also helps us understand that we have already added 61 minutes and next minute corresponds to 06:54. However, the current minute is 53 and previous was 52, so perhaps the problem duplicate bars occurred earlier.

Now we know what to look for, we can look in the log file to confirm the theory that there are duplicate bars:

Code: Select all

2016/02/04 06:21:08 44704937922	INFO [datehelper.cpp L: 826]	"2014-01-02 06:00:00
"
2016/02/04 06:37:42 45698045987	DEBUG [MinuteNode.cpp L: 30]	"MinuteNode::updateM1()... "
2016/02/04 06:37:42 45698046047	INFO [datehelper.cpp L: 826]	"2014-01-02 06:00:00
"
2016/02/04 06:37:53 45709077341	DEBUG [MinuteNode.cpp L: 30]	"MinuteNode::updateM1()... "
2016/02/04 06:37:53 45709077392	INFO [datehelper.cpp L: 826]	"2014-01-02 06:00:00
"
2016/02/04 06:37:53 45709077407	DEBUG [MinuteNode.cpp L: 30]	"MinuteNode::updateM1()... "
2016/02/04 06:37:53 45709077417	INFO [datehelper.cpp L: 826]	"2014-01-02 06:00:00
"
2016/02/04 06:37:53 45709077430	DEBUG [MinuteNode.cpp L: 30]	"MinuteNode::updateM1()... "
2016/02/04 06:37:53 45709077440	INFO [datehelper.cpp L: 826]	"2014-01-02 06:00:00
"
2016/02/04 06:37:53 45709077450	DEBUG [MinuteNode.cpp L: 30]	"MinuteNode::updateM1()... "
2016/02/04 06:37:53 45709077477	INFO [datehelper.cpp L: 826]	"2014-01-02 06:00:00
"
2016/02/04 06:37:53 45709077487	DEBUG [MinuteNode.cpp L: 30]	"MinuteNode::updateM1()... "
2016/02/04 06:37:53 45709077510	INFO [datehelper.cpp L: 826]	"2014-01-02 06:01:00
"
2016/02/04 06:37:53 45709077520	DEBUG [MinuteNode.cpp L: 30]	"MinuteNode::updateM1()... "
2016/02/04 06:37:53 45709077537	INFO [datehelper.cpp L: 826]	"2014-01-02 06:01:00
[/size][/size]

Having got this far should we abandon working with Zorro history or spend time working out methods of fixing dodgy data. Imagine another possible scenario:
There are the correct number of bars but half-a-dozen duplicates.
So we can't just blindly drop duplicate bars or we would end with a short hour.
And another possible scenario:
We have some duplicate bars but also missing some minutes.
Such possibilities, if not handled carefully, would lead to even more problems and we might think about a multi-pass approach with a prioritized set of rules for tackling one issue at a time:

Do we have enough bars?
If not fill in the gaps with similar values
Do we have too many bars?
AND do we have some duplicates?
THEN drop duplicates while we have too many bars
Do we have the right number of bars
AND some duplicates
THEN change the minutes to conform to 0...59

Note that we haven't mentioned the case of extreme (invalid) prices:
I leave it as an exercise for the reader to adjust the rules for this possibility...
Author:  nonlinear [ Sat Feb 06, 2016 4:03 pm ]
Post subject:  Daily Fibonacci Signals
After trying different data sources (like FXCM), I use Tickstory, which pulls tick data from Dukascopy. The data seems decent. Have you tried it yet?
Author:  snailbeard [ Thu Feb 11, 2016 7:53 am ]
Post subject:  Daily Fibonacci Signals
nonlinear » Sat Feb 06, 2016 4:03 pm wrote:After trying different data sources (like FXCM), I use Tickstory, which pulls tick data from Dukascopy. The data seems decent. Have you tried it yet?
Yes I have used Tickstory, and it is probably the best data for our purposes but has anyone crossed checked it? Since the aim is to monitor multiple pairs, synchronize them, detecting and allowing for gaps and duplicate bars so that the comparisons are valid, I would still need to do the same kind of scrutiny of the data.
Author:  snailbeard [ Thu Feb 11, 2016 12:12 pm ]
Post subject:  Zorro - unmasked
Now that my Zorro data analyser is mostly working well, its time to unmask the mess inside a Zorro history file. One surprise is the absence of ridiculous price glitches. Many of the brokers history files I have used in the past have these. So now I'm thinking about a more generic approach with the ability to load data from different assets or different sources.

Imagine if you wanted to know if two or more assets were correlated or anti-correlated, how would you check?

What about "DNA finger-printing" each file?

There are several issues to work around, firstly the time of each bar can be shifted by several hours and prices will not be identical, and then there is the spurious data and gaps.

Here is an extract from the Zorro report

" sumMinutes : 370988"
" sumDuplicateMins : 36154"
" sumMissingMins : 1941"
" sumSpuriousMins : 0"
" sumExtraMins : 2487"

I had some doubts about the size of sumDuplicateMins - I'm going to cross-check that just to be sure.
sumExtraMins is when more than 60 minutes are found for the same hour.

The answer to finger-printing might be to locate several common 'genes' - I might be stretching the analogy a bit far. The interesting parts of each history file are when the major price changes occur. The absolute changes might be different but they should stand out as key moments, which should line up when data from two sources is overlaid.

Here is an example: as I'm examining the bars I track the biggest price jumps as part of looking for spurious prices, what if I keep the 10 biggest moves - should they have corresponding moves in other history files?

Again from the report:

Significant Biggest M1 price jump : 0.00726, priceAtAnomaly: 1.6351,
M1Change: 0.00726, percentChange: 0.49628, TS: 2014-01-17 09:30:00"

The percentage change is just a rough estimate - it demands on how you calculate it.

The following chart demonstrates how the raw Zorro spike needs shifting to line-up with GlobalPrime history prices:
Zorro-unmasked-17Jan2014.png
If only it was that simple, if anyone wants to have a more detailed look at the Zorro report then I can post a copy it.
Author:  nonlinear [ Thu Feb 11, 2016 1:31 pm ]
Post subject:  Daily Fibonacci Signals
One could write an EA to collect live tick data, let it run a while, and then test against that data. I have heard of some doing this, but is sounds like a serious pain and memory drain, depending on how much one needs for trading purposes. Even then, sometimes the live feed has errors, not sure how one corrects that. For me, I gave up on using tick data, and try to get my systems to work off of M1 bars, which are much easier to confirm against my live broker feed.
Author:  snailbeard [ Tue Feb 16, 2016 2:22 pm ]
Post subject:  Daily Fibonacci Signals
Thanks to NL's suggestion, I restarted Strategy Quant's (SQ) tick down loader on another PC, so I have a large amount of M1 data from them. I also looked at FxOpen and GlobalPrime data, but so far I have only finger-printed the Zorro data as I need to add functions to read Empty4 and CSV bars.

Now I have added another feature to save the most significant M1 Bar moves, which is an essential part of monitoring changes in prices of a basket of currencies simultaneously, or rather finding interesting areas.

It is important to remember there are distinct problems
a) trying out an idea in real time
b) developing and testing an idea with historical data
but when (a) is over and we want to know what happened we might have to rerun with data from (a) but now (a) has become part of the historical data.

So the finger-printing also finds the interesting parts of (a) once it moves into history,
whereas the real time code only uses recent history bars, but if we get duplicate bars, missing bars and repeat bars in real time, we need to know about it or comparing different currencies in real-time will produce spurious results.

So the data that came from Zorro is so bad I doubt that it was ever real time data.
I have since change the analysis to distinguish between repeat bars (bars with the same time-stamp) and duplicate bars which are completely identical, but using bad data helps make code more robust.

So this is kind of data we get from finger-printing:

Code: Select all

Major M1 Bar moves:
Time: 2014-09-07 22:02:00 Abs%:  0.97 Price: 1.61727 Change: 0.01595 dir: -1, Prev Time: 2014-09-05 22:00:00 Prev Price: 1.63301
Time: 2014-01-17 09:30:00 Abs%:  0.46 Price: 1.64243 Change: 0.00726 dir: +1, Prev Time: 2014-01-17 09:29:00 Prev Price: 1.63508
Time: 2014-01-10 13:30:00 Abs%:  0.37 Price: 1.64599 Change: 0.00488 dir: +1, Prev Time: 2014-01-10 13:29:00 Prev Price: 1.63914
Time: 2014-01-10 09:30:00 Abs%:  0.21 Price: 1.64168 Change: 0.00326 dir: -1, Prev Time: 2014-01-10 09:29:00 Prev Price: 1.64527
Time: 2014-01-05 22:00:00 Abs%:  0.13 Price: 1.63996 Change: 0.00214 dir: -1, Prev Time: 2014-01-03 22:00:00 Prev Price: 1.64210
Time: 2014-01-02 15:00:00 Abs%:  0.13 Price: 1.64245 Change: 0.00185 dir: -1, Prev Time: 2014-01-02 15:00:00 Prev Price: 1.64437
Time: 2014-01-02 09:07:00 Abs%:  0.12 Price: 1.65876 Change: 0.00174 dir: +1, Prev Time: 2014-01-02 09:06:00 Prev Price: 1.65700
[/size]

The rules for collecting major events are currently simple and we need several variations on this, so that we could analyse different pairs/brokers in useful ways.i.e. find out which brokers we should really avoid,
and which history files to delete.

If you take one event such as 2014-09-07 22:02:00 we can easily relate this to say FxOpen, GlobalPrime and SQ

Strategy Quant:

Code: Select all

Strategy Quant
2014.09.05	20:59	1.63236	1.63257	1.63225	1.63241	62
2014.09.07	21:00	1.61687	1.61696	1.61615	1.61682	128
1.61687-1.63257 = −0.01570, (0.01570)*100/1.63257= 0.96%
[/size]

The following screen-shot is from FxOpen which started a new week at 00:00 without any missing bars,
whereas GlobalPrime had some difficulties and didn't get going until 01:05.
GP_Zr_events_and_missing_bars_20140908_0000.png
So in summary SQ and FxOpen cleanly handled this price gap.

If you rely on indicators or EAs for signals have you checked the impact of gaps and dodgy real-time data on those signals?
Author:  nonlinear [ Tue Feb 16, 2016 3:43 pm ]
Post subject:  Daily Fibonacci Signals
My personal view is that one can rely on M1 historical bars from some providers, at least to some degree, but only if the system (1) targets hourly or daily moves that typically last a half day or more and (2) the system has many trades per test window. So even if the test M1 bars have a few errors, gaps, or whatever here and there, its somewhat okay because due to typical targets most anomalies get corrected after a few M1 bars (so long as the time stamps are correct). And even if there is an occasional issue with some historical data that actually affects the entry and exit criteria to a significant degree, that should just be a few among a much larger and accurate sample (hopefully) and the system should still work despite these anomalies if it really is targeting moves on a higher time frame. I would thus not hold this view if a system sought tighter stops and targets -- then you have to rely on tick data and it gets much more complicated to model, highly broker-dependent, etc.
Author:  snailbeard [ Thu Feb 18, 2016 1:09 pm ]
Post subject:  Memories are made of this
I agree with NL about the trading periods, for robust indicators and EAs, these errors might be smoothed out, however, if you are brave enough to scalp though, you might be entering into a quantum mechanical world of strangeness. I doubt there are any retail traders like us who can survive long term doing this, unless you have a really good liquidity provider and really thin spreads.

The practical issues of running several copies of Empty4 from several brokers do cause problems. I could probably run ten copies of Empty4 if none of those copies had any charts, indicators or EAs running.

So where does all the memory and CPU resource go?
Why does Empty4 store prices as doubles and would floats suffer from rounding errors?

mqlRates requires 60 bytes of memory per bar

a cvs file requires less than this: ~ 50 bytes of data per bar:
"2014.12.31,23:59,1.55756,1.55762,1.55734,1.5574,71"

A Zorro bar has ***no volume*** information and uses floats.

I'm currently using floats, because that was what came out of a Zorro file

Code: Select all

typedef struct SSETICK
{
	sse32time_t	ssetime;	// 4 bytes, time stamp, GMT(+/-)h
	float fOpen;			// 4 bytes
	float fHigh;			// 4 bytes
	float fLow;			// 4 bytes
	float fClose;			// 4 bytes
	int32_t volume; 		// 4 bytes
} SSETICK1; // .bar file content reduced to 24 bytes
Do we really need to be concerned about this?

"Starting value: 1.55756"
"Sample: double: 1.5575600000, float: 1.5575599670"
"sizeof(double): 8, sizeof(float): 4"
"set precision to 5 places: double: 1.55756, float: 1.55756"

I found that simple division then multiplication by a range of values does not affect the result.
So why use doubles, it consumes more memory, CPU and disk-space?
Has anyone experienced a rounding issue with floats?
Under what circumstances are floats not accurate enough (keeping in mind: spread, slippage and dirty data)?

A don't believe that Empty4's bar size is by itself a cause of memory creep, even if your charts use M1 bars,
you could limit the chart to say '1000' bars, but we all know the huge burden that indicators add to the CPU and memory, which is why some of you limit back bars to 100 on some indicators.

In theory a years worth of M1 Bars does not consume that much memory:

Initial free memory say 4GB
60 bytes x 371000 = ~ 22,260,000 Bytes
remaining memory after storing 1Y of M1 bars: (4,000 MB - 22 MB)

Clearly this is not where Empty4 eats up memory, and for many people who run one copy of Empty4 and only use a few indicators there is no issue.

I ran into difficulties as the my indicators and EAs performed more and more calculations every tick. CPU issues were cured by saving the results of calculations that only needed updating according to a particular time frame but then this gradually impacted on memory.

It is good to start from scratch with no baggage occasionally, keeping in mind that we might run low on CPU and memory eventually, but without putting too much effort into premature optimization.
All times are UTC Page 6 of 21

Powered by phpBB® Forum Software © phpBB Limited