Applying what we have learnt about predicting the daily range and analysing ATR for EURUSD between October and November 2016, where price falls for a couple of weeks and then rises for couple of weeks but ranging quite a bit in between, the EA struggles to distinguish between trends and weak trends, so I have applied additional analysis to the ATR data. The easy part is achieving a blanket switch off of trading because conditions are weak, but there are some good growth spurts in the midst of the weak trending.
The following results of the current technique need to be tested on a lot more data but the general principle should be generic without being too curve fitted to the current data:
So the first attempt produced the best net profit but when I looked at the chart I suspected that some of the wins were not entirely safe and allowing for slippage/spread widening I deemed some of the trades to be at the edge.
In the second table I introduced a deeper analysis of falling price momentum but on checking out the missing winning trades it was clear that some of the filtering was due to the lagging nature of ATR.
Finally, the third table is the result of being quite picky, but also compensating for the limitations of a blunt instrument, it results in a drop in the net profit. However, since we have insufficient data and trades to draw conclusions, more testing is required.
The 1st Nov 2016 was a perfect day for entering a trade but without a different kind of analysis there is no way to let this through the range filters. The one that got away:
So there was a strong reverse break out at the end of October, we see what appears to be a confident trend going up, but we can also see the longer term falling moving averages. Not apparent in the chart is that the daily moves are weaker than the long term average, price action is weaker than it has been, so price is weakly moving up in the shadow of recent past down and in the context of a longer term down - it would be simpler not to trade this counter move at all, but some markets rarely get better than this. Perhaps, trend hunting is the wrong approach for EURUSD between October and November. Perhaps traders using range trading or grid trading had better success, if you did then please share.
Finally, we can compare the actual day's range against the EA's forecasts:
EoD Range: 0.01087 (Actual End-of-Day value)
D1AtrEst: 0.00675,
H4AtrEst: 0.00580,
H1AtrEst: 0.00718
Clearly D1 and H4 are very pessimistic and H1 is closer to the action but still predicting less than the long term average: D1-Atr100: 0.00814
The EA becomes more interested in trading when the forecast range is more than the long term average.
Daily Fibonacci Signals
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
Adaptive range filter
You do not have the required permissions to view the files attached to this post.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
USDJPY doesn't disappoint
It is time to see how the latest EA performs on a new range of pairs in order to make sure that I haven't just been curve fitting to test data. This is the first time I have tried USDJPY for a long time.
What are we looking for?
a) The ability to not trade during weak price action
b) The detection of good entry points during better trading conditions
So how did it perform?
The EA has become quite picky and you could argue that it should spot more entries during stronger price action. I would have to agree that there is potential to change the trading method, perhaps entering more, smaller and shorter trades spaced over a trading day, but for the time being I'll have to work with what I have.
In the following chart we see about 3 months of history: August and September might be trade-able by other methods but not by trend. This EA has correctly prevented trading in the weak period. In early November we start to see a genuine trend emerge. I would like to see trading start a day earlier, but the EA has it's reasons for not entering, so we only get 4 entries in November.
Now I am wondering if I spliced the inhibition filters onto one of my old EAs, like a version of NB, whether or not I would get the best of both? That NB EA produced some fantastic results during proper trends but didn't know when to turn off, so it might produce more trades and higher net profits? Unfortunately I don't have enough time to work on two separate trading methods at the moment.
So November produced a net profit of just under 5% over a period of 3 months. For this EA to be useful there needs to be some other asset trending in September, and October, then it can produce a nice steady profit each month.
The draw-down is very small for the this selective sparse trading method which means one account might be able to monitor more pairs without getting into danger. However, I need to find out if there are any issues with correlated trades.
What are we looking for?
a) The ability to not trade during weak price action
b) The detection of good entry points during better trading conditions
So how did it perform?
The EA has become quite picky and you could argue that it should spot more entries during stronger price action. I would have to agree that there is potential to change the trading method, perhaps entering more, smaller and shorter trades spaced over a trading day, but for the time being I'll have to work with what I have.
In the following chart we see about 3 months of history: August and September might be trade-able by other methods but not by trend. This EA has correctly prevented trading in the weak period. In early November we start to see a genuine trend emerge. I would like to see trading start a day earlier, but the EA has it's reasons for not entering, so we only get 4 entries in November.
Now I am wondering if I spliced the inhibition filters onto one of my old EAs, like a version of NB, whether or not I would get the best of both? That NB EA produced some fantastic results during proper trends but didn't know when to turn off, so it might produce more trades and higher net profits? Unfortunately I don't have enough time to work on two separate trading methods at the moment.
So November produced a net profit of just under 5% over a period of 3 months. For this EA to be useful there needs to be some other asset trending in September, and October, then it can produce a nice steady profit each month.
The draw-down is very small for the this selective sparse trading method which means one account might be able to monitor more pairs without getting into danger. However, I need to find out if there are any issues with correlated trades.
You do not have the required permissions to view the files attached to this post.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
Better range prediction
I am feeling more confident that we are heading towards a safe EA, but at the same time the size and speed of a sudden price spike gives me cause for concern. These price spikes might be artificially induced or a consequence of competing HFT robots, but for us small fish the end is the same, losses which have nothing to do with normal trading. Many people see HFT as insider trading by the back door, but since corruption and criminal activity are prevalent at the highest levels we might have to wait a long time before HFT becomes a criminal offence. So my optimism is in equilibrium with an equal measure of pessimism.
So what does that have to do with range predictions? Obviously, a price spike does several bad things, in addition to mystery stop-outs our historical data is corrupted by price swings which are included in indicators and calculations but are not relevant to a normal days trading. If we include anomalies our predictions will be skewed.
Does your EA adjust for anomalous price spikes?
To understand the following image it will help if you have read the earlier posts about predicting the daily range: I am starting to wonder if these pretty patterns are only useful for turning into to Christmas decorations?
However, the numbers on the outside are days since the start of sampling and I have put an A against 32 because it was a valid prediction (yellow:actual range; orange:D1 based prediction; blue:H4 based prediction).
You can also see that the blue square (H4) was hopelessly wrong. Those who are interested and paying attention will be able to work out from the chart why the H4 prediction was so wrong. I invite readers to contribute the reason or request me to give a better explanation of the chart.
Is this work important?
I believe it is an important area of focus and shifts the net return more positive.
The EA now makes profits on different pairs but only by paying close attention to the daily range.
So what does that have to do with range predictions? Obviously, a price spike does several bad things, in addition to mystery stop-outs our historical data is corrupted by price swings which are included in indicators and calculations but are not relevant to a normal days trading. If we include anomalies our predictions will be skewed.
Does your EA adjust for anomalous price spikes?
To understand the following image it will help if you have read the earlier posts about predicting the daily range: I am starting to wonder if these pretty patterns are only useful for turning into to Christmas decorations?
However, the numbers on the outside are days since the start of sampling and I have put an A against 32 because it was a valid prediction (yellow:actual range; orange:D1 based prediction; blue:H4 based prediction).
You can also see that the blue square (H4) was hopelessly wrong. Those who are interested and paying attention will be able to work out from the chart why the H4 prediction was so wrong. I invite readers to contribute the reason or request me to give a better explanation of the chart.
Is this work important?
I believe it is an important area of focus and shifts the net return more positive.
The EA now makes profits on different pairs but only by paying close attention to the daily range.
You do not have the required permissions to view the files attached to this post.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
USDCHF A rift in time
I have posted regularly about the difficulty of having a ready, accurate and up-to-date extensive collection of price data. The amount of manual effort needed eats into time that could be used productively elsewhere.
This post is only going to be of interest to those determined to do a lot of testing with different pairs and I only use C++ because of familiarity, Dukascopy like Java and others like C#.
Now that extensive testing on 4 pairs seems to be complete for the recent past, what about the other twenty-something pairs of interest, and what about the data coming in a weeks time, when I need to rerun the tests with the latest data?
There is no way I can rely on brokers to keep providing data, even Global Prime's data is full of problem gaps, the only way forward is to constantly pull data from Dukascopy and chain a series of separate steps into series of automated operations.
So far I have a C++ program from last year which can read Zoro data (very poor quality), Tickstory CSV and StrategyQuant CSV. At the end of 2016 I modified Birt's PHP download script and now I can see a way to pull data without StrategyQuant (MS Windows only).
The master plan is to have a little “Raspberry Pie” sitting quietly in corner somewhere constantly pulling data, processing it and having it in a ready format for Empty4.
That assumes I have solved some subtle problems with shifting of time. It is very dangerous to ask questions about clocks and time-zones in the SH forum, but I have some sympathies if you find yourself getting a bit lost.
In this example we have a little programming headache which relates to C++ and might not be a problem (or the same problem) in other languages.
D1: Open hour???: 2016.10.30 01:00:00 tm_hour: 0"
D1: Open hour???: 2016.10.31 00:00:00 tm_hour: 0"
So where am I ?
I am both in the past and in the present, I am in the UK and also in Eastern Europe. Confused?
That is the problem when you want to take some time dependent data from one source and convert it to another format. The final data should be UTC+2 and what about day-light-saving offsets?
C++ has some basic functions for converting time from one representation to another but the only representation which we can be sure about is Epoch or UTC, once we convert it into a string or other format we are open to misinterpretations.
The C++ functions were implemented for expediency and at that time the concept of storing time and date as a string would be insanely expensive, since it was necessary to save every byte of data and cram a date into 32 bits.
When it comes to reading data from CSV, converting it to epoch time and converting back to a string, by default these functions don't know anything about the original time zone and the target time-zone, but they do know about the current real-time time-zone and might make some un-useful assumptions about what we are trying to do with our time values.
The latest C++ standard (C++17) has a new feature which (when fully implemented on all platforms) will make our lives at little easier:
std::timespec_get(&ts, TIME_UTC);
which you might have guessed explicitly uses UTC instead of assuming locale.
However, until then we might need to thoroughly test data which crosses DST dates.
The following debugging shows the subtle nature of the problem:
"Part I(a) "
"convertStrDateToEpoch DST is not in effect"
" stdTm.tm_isdst: 0"
" stdTm.tm_hour 0"
"Part I(b) "
"Was stdTm changed by std::mktime() ??? : "
" stdTm.tm_isdst: 0"
" stdTm.tm_hour 0"
" -----------------------------------"
"Part II "
"convertStrDateToEpoch DST is not in effect"
" Tm.tm_isdst: 0"
"tEpochDate: 1458864000 Tm.tm_hour 0"
"convert stdTm To : convertTmToString(): 2016.03.25 00:00"
"convert Tm To : convertTmToString(): 2016.03.25 00:00"
"lEpochTime: 1458864000 Tm.tm_hour 0"
"UTC: Fri Mar 25 00:00:00 2016 GMT"
"local: Fri Mar 25 00:00:00 2016 GMT"
" -----------------------------------------------------------------"
" -----------------------------------------------------------------"
"Part I(a) "
"convertStrDateToEpoch DST is not in effect"
" stdTm.tm_isdst: 0"
" stdTm.tm_hour 0"
"Part I(b) "
"Was stdTm changed by std::mktime() ??? : "
" stdTm.tm_isdst: 1"
" stdTm.tm_hour 1"
" -----------------------------------"
"Part II "
"convertStrDateToEpoch DST is not in effect"
" Tm.tm_isdst: 0"
"tEpochDate: 1459123200 Tm.tm_hour 0"
"convert stdTm To : convertTmToString(): 2016.03.28 01:00"
"convert Tm To : convertTmToString(): 2016.03.28 00:00"
"lEpochTime: 1459123200 Tm.tm_hour 0"
"UTC: Mon Mar 28 00:00:00 2016 GMT"
"local: Mon Mar 28 01:00:00 2016 BST"
" -----------------------------------------------------------------"
I am still using C++11 and I don't know when C++17 will be generally available and in common use, so for the time being I have added my own work around, which looks like this:
The result of testing confirms that the we get our original historical date back without any unwanted time zone adjustment:
2016.03.25 00:00 : sseTime1 1458864000 2016.03.25 00:00
2016.03.28 00:00 : sseTime2 1459123200 2016.03.28 00:00
In conclusion, extensive testing could be undermined by subtle errors in bar prices and volumes, which might be confusing when we have to look at charts in Empty4 and find the bars, prices and volumes don't quite match.
We could end up with bars from one time frame not quite lined up with another time-frame and the entries or exits not occurring quite where we thought they would.
This post is only going to be of interest to those determined to do a lot of testing with different pairs and I only use C++ because of familiarity, Dukascopy like Java and others like C#.
Now that extensive testing on 4 pairs seems to be complete for the recent past, what about the other twenty-something pairs of interest, and what about the data coming in a weeks time, when I need to rerun the tests with the latest data?
There is no way I can rely on brokers to keep providing data, even Global Prime's data is full of problem gaps, the only way forward is to constantly pull data from Dukascopy and chain a series of separate steps into series of automated operations.
So far I have a C++ program from last year which can read Zoro data (very poor quality), Tickstory CSV and StrategyQuant CSV. At the end of 2016 I modified Birt's PHP download script and now I can see a way to pull data without StrategyQuant (MS Windows only).
The master plan is to have a little “Raspberry Pie” sitting quietly in corner somewhere constantly pulling data, processing it and having it in a ready format for Empty4.
That assumes I have solved some subtle problems with shifting of time. It is very dangerous to ask questions about clocks and time-zones in the SH forum, but I have some sympathies if you find yourself getting a bit lost.
In this example we have a little programming headache which relates to C++ and might not be a problem (or the same problem) in other languages.
D1: Open hour???: 2016.10.30 01:00:00 tm_hour: 0"
D1: Open hour???: 2016.10.31 00:00:00 tm_hour: 0"
So where am I ?
I am both in the past and in the present, I am in the UK and also in Eastern Europe. Confused?
That is the problem when you want to take some time dependent data from one source and convert it to another format. The final data should be UTC+2 and what about day-light-saving offsets?
C++ has some basic functions for converting time from one representation to another but the only representation which we can be sure about is Epoch or UTC, once we convert it into a string or other format we are open to misinterpretations.
The C++ functions were implemented for expediency and at that time the concept of storing time and date as a string would be insanely expensive, since it was necessary to save every byte of data and cram a date into 32 bits.
When it comes to reading data from CSV, converting it to epoch time and converting back to a string, by default these functions don't know anything about the original time zone and the target time-zone, but they do know about the current real-time time-zone and might make some un-useful assumptions about what we are trying to do with our time values.
The latest C++ standard (C++17) has a new feature which (when fully implemented on all platforms) will make our lives at little easier:
std::timespec_get(&ts, TIME_UTC);
which you might have guessed explicitly uses UTC instead of assuming locale.
However, until then we might need to thoroughly test data which crosses DST dates.
The following debugging shows the subtle nature of the problem:
"Part I(a) "
"convertStrDateToEpoch DST is not in effect"
" stdTm.tm_isdst: 0"
" stdTm.tm_hour 0"
"Part I(b) "
"Was stdTm changed by std::mktime() ??? : "
" stdTm.tm_isdst: 0"
" stdTm.tm_hour 0"
" -----------------------------------"
"Part II "
"convertStrDateToEpoch DST is not in effect"
" Tm.tm_isdst: 0"
"tEpochDate: 1458864000 Tm.tm_hour 0"
"convert stdTm To : convertTmToString(): 2016.03.25 00:00"
"convert Tm To : convertTmToString(): 2016.03.25 00:00"
"lEpochTime: 1458864000 Tm.tm_hour 0"
"UTC: Fri Mar 25 00:00:00 2016 GMT"
"local: Fri Mar 25 00:00:00 2016 GMT"
" -----------------------------------------------------------------"
" -----------------------------------------------------------------"
"Part I(a) "
"convertStrDateToEpoch DST is not in effect"
" stdTm.tm_isdst: 0"
" stdTm.tm_hour 0"
"Part I(b) "
"Was stdTm changed by std::mktime() ??? : "
" stdTm.tm_isdst: 1"
" stdTm.tm_hour 1"
" -----------------------------------"
"Part II "
"convertStrDateToEpoch DST is not in effect"
" Tm.tm_isdst: 0"
"tEpochDate: 1459123200 Tm.tm_hour 0"
"convert stdTm To : convertTmToString(): 2016.03.28 01:00"
"convert Tm To : convertTmToString(): 2016.03.28 00:00"
"lEpochTime: 1459123200 Tm.tm_hour 0"
"UTC: Mon Mar 28 00:00:00 2016 GMT"
"local: Mon Mar 28 01:00:00 2016 BST"
" -----------------------------------------------------------------"
I am still using C++11 and I don't know when C++17 will be generally available and in common use, so for the time being I have added my own work around, which looks like this:
Code: Select all
//---------------------------------------------------
sse32time_t mktimeUTC(std::tm* stdTmPtr, bool bDebug)
{
std::tm* tmp_tm_ptr;
std::tm tmp_tm;
sse32time_t tEpochDate = 0;
tmp_tm_ptr = (std::tm*) &tmp_tm;
// make a copy to avoid timezone adjustment in the original data
memcpy( reinterpret_cast<void*>(tmp_tm_ptr), reinterpret_cast<void*>(stdTmPtr), sizeof(std::tm));
tEpochDate = mktime( tmp_tm_ptr ); // watch out for local BST/DST
if( bDebug )
{
std::string tmpCStr = calendar_ptr_to_str( (const std::tm*) stdTmPtr);
}
return tEpochDate;
}
//---------------------------------------------------
void TestmktimeUTC()
{
bool bDebug = true;
std::tm stdTm1;
std::tm stdTm2;
std::string strDate1 = "2016.03.25 00:00";
std::string strDate2 = "2016.03.28 00:00";
std::string strDate1b;
std::string strDate2b;
convertStrDateToTm( strDate1, stdTm1);
sse32time_t sseTime1 = mktimeUTC( & stdTm1, bDebug);
convertTmToString( stdTm1, strDate1b);
convertStrDateToTm( strDate2, stdTm2);
sse32time_t sseTime2 = mktimeUTC( & stdTm2, bDebug);
convertTmToString( stdTm2, strDate2b);
LOG(DEBUG) << strDate1 << " : sseTime1 "<< sseTime1 << " " << strDate1b << std::endl;
LOG(DEBUG) << strDate2 << " : sseTime2 "<< sseTime2 << " " << strDate2b << std::endl;
}
2016.03.25 00:00 : sseTime1 1458864000 2016.03.25 00:00
2016.03.28 00:00 : sseTime2 1459123200 2016.03.28 00:00
In conclusion, extensive testing could be undermined by subtle errors in bar prices and volumes, which might be confusing when we have to look at charts in Empty4 and find the bars, prices and volumes don't quite match.
We could end up with bars from one time frame not quite lined up with another time-frame and the entries or exits not occurring quite where we thought they would.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
data and quality control
All this effort going into collecting price data to automatically set up Empty4's price history would be a waste of time if we don't thoroughly check each step of the process, starting with our original data via Strategy-Quant, note that they do not provide W1 and MN1 data:
So we need to be sure that we can extract a window of data from CSV file by using various run time parameters and create an audit trail for the load and conversion process:
In the first case I set the extraction dates before the start of the price data and then I set dates for the full CSV file so that I can cross check the first and last bars loaded perfectly match the original data.
Everything appears to be consistent with no issues on crossing the end of BST. So we can move on to the next step of being able to generate the missing W1 and MN1 files.
Code: Select all
brian@brian-q4 /srv/ftp/dukas_price_data/tickdata/ASUS/USDCHF
$ ls
USDCHF_D1_UTC+2_00.csv USDCHF_H4_UTC+2_00.csv
USDCHF_M1.dat USDCHF_M30_UTC+2_00.csv
USDCHF_H1_UTC+2_00.csv USDCHF_M15_UTC+2_00.csv
USDCHF_M1_UTC+2_00.csv USDCHF_M5_UTC+2_00.csv
brian@brian-q4 /srv/ftp/dukas_price_data/tickdata/ASUS/USDCHF
$ head USDCHF_H1_UTC+2_00.csv
2016.09.01,03:00,0.98369,0.98382,0.98324,0.98356,10076
2016.09.01,04:00,0.98357,0.98391,0.98301,0.98332,11518
2016.09.01,05:00,0.98332,0.9836,0.98281,0.98329,10534
2016.09.01,06:00,0.9833,0.98348,0.98284,0.98337,8459
2016.09.01,07:00,0.98337,0.98372,0.98333,0.9836,8826
2016.09.01,08:00,0.9836,0.98383,0.98348,0.98354,10462
2016.09.01,09:00,0.98354,0.98419,0.98331,0.98338,18213
2016.09.01,10:00,0.98337,0.98845,0.98331,0.98836,43506
2016.09.01,11:00,0.98836,0.98843,0.98535,0.98606,46655
2016.09.01,12:00,0.98606,0.98648,0.98436,0.98438,34585
brian@brian-q4 /srv/ftp/dukas_price_data/tickdata/ASUS/USDCHF
$ tail USDCHF_H1_UTC+2_00.csv
2016.12.14,16:00,1.01048,1.01086,1.00973,1.01005,11403
2016.12.14,17:00,1.01003,1.01021,1.00832,1.01017,13624
2016.12.14,18:00,1.01016,1.01021,1.00932,1.00998,9602
2016.12.14,19:00,1.00998,1.01052,1.00952,1.01037,7886
2016.12.14,20:00,1.01038,1.01301,1.01028,1.01092,10919
2016.12.14,21:00,1.01203,1.01903,1.01203,1.01875,28048
2016.12.14,22:00,1.01874,1.02228,1.01828,1.02004,21008
2016.12.14,23:00,1.02003,1.02099,1.01964,1.02011,7819
2016.12.15,00:00,1.02001,1.02087,1.01956,1.02054,7899
2016.12.15,01:00,1.02051,1.02218,1.02012,1.02215,12387
Everything appears to be consistent with no issues on crossing the end of BST. So we can move on to the next step of being able to generate the missing W1 and MN1 files.
You do not have the required permissions to view the files attached to this post.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
Gold shines brightly
Although I am too cautious to trade for real in the first week of 2017, the EA took a slightly risky buy (XAUUSD) on GlobalPrime:
Was it high risk or medium risk?
Price is rising strongly but under the shadow of fairly strong and continuous downward momentum, definitely one for the brave-hearts!
Also consider the four variants of ATR which were all falling but with a turning point in late December.
Does this reversal have further to go?
Price is rising strongly but under the shadow of fairly strong and continuous downward momentum, definitely one for the brave-hearts!
Also consider the four variants of ATR which were all falling but with a turning point in late December.
Does this reversal have further to go?
You do not have the required permissions to view the files attached to this post.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
Sunday or Monday?
Gremlins are still at work in the data conversion, or are they in the source D1 bars?
A minor tweak is required to clear any non-zero minutes or hours to zero,
but did you notice the Halloween bar is missing?
It seems we are still having problems at the end of DST with 2016.10.30 being Sunday when we actually expected 2016.10.31.
Those of you who trade manually using weekly bars will have noticed that all weekly bars always start on Sunday and this has nothing to do with DST.
So it is actually my W1 bars that start on Monday that need to follow the convention of starting on Sunday at 00:00.
Now a closer look at the original D1 bar output from Strategy-Quant: So we have ended up with 6 D1 bars at the boundary instead of 5!
All brokers I am using provide 5 D1 bars starting on Monday, so why is StrategyQuant different, are they following the original Dukascopy data?
What is the point of exporting StrategyQuant data using UTC+2 if Sunday bars gets left behind?
Would be better to generate higher time frames myself?
It would be useful to have a peek at the data on these Sunday/Monday boundaries. I have added a new loader for the M1 format but even just days of M1 bars is a lot of data to sift through. The log file currently gives me a rough idea:
so I am able to load a sub sample from a long file but it would be useful to pick a day and just get the boundary information. For binary M1 files we can find the start of a day then move the file pointer back by the size of the bar structure.
Code: Select all
2016.09.05 00:00:00,0.9794,0.98136,0.96491,0.97518,2538310
2016.09.12 00:00:00,0.97464,0.98178,0.96899,0.97943,2802757
2016.09.19 00:00:00,0.97935,0.98185,0.96595,0.96987,2526897
2016.09.26 00:01:00,0.96914,0.9755,0.96397,0.97125,2457050
2016.10.03 00:00:00,0.97115,0.9839,0.9697,0.97702,2495563
2016.10.10 00:00:00,0.97696,0.99093,0.97628,0.99011,2210930
2016.10.17 00:00:00,0.99044,0.99616,0.98426,0.99304,1952076
2016.10.24 00:00:00,0.99384,0.99985,0.98576,0.98731,740856
2016.10.30 00:00:00,0.9867,0.99073,0.96745,0.96808,800721
2016.11.07 00:00:00,0.97764,0.98963,0.95493,0.98745,1140780
2016.11.14 00:00:00,0.98889,1.01228,0.98838,1.00993,1176043
2016.11.21 00:02:00,1.01009,1.01914,1.00673,1.01261,1036884
2016.11.28 00:00:00,1.01139,1.02046,1.00729,1.00998,1155250
2016.12.05 00:00:00,1.01193,1.02133,1.002,1.01656,953866
but did you notice the Halloween bar is missing?
It seems we are still having problems at the end of DST with 2016.10.30 being Sunday when we actually expected 2016.10.31.
Those of you who trade manually using weekly bars will have noticed that all weekly bars always start on Sunday and this has nothing to do with DST.
So it is actually my W1 bars that start on Monday that need to follow the convention of starting on Sunday at 00:00.
Now a closer look at the original D1 bar output from Strategy-Quant: So we have ended up with 6 D1 bars at the boundary instead of 5!
All brokers I am using provide 5 D1 bars starting on Monday, so why is StrategyQuant different, are they following the original Dukascopy data?
What is the point of exporting StrategyQuant data using UTC+2 if Sunday bars gets left behind?
Would be better to generate higher time frames myself?
It would be useful to have a peek at the data on these Sunday/Monday boundaries. I have added a new loader for the M1 format but even just days of M1 bars is a lot of data to sift through. The log file currently gives me a rough idea:
Code: Select all
"history file: /srv/ftp/dukas_price_data/tickdata/ASUS/USDCHF/USDCHF_M1.dat"
"first bar read: 2016.09.01 03:00:00,0.98369,0.98374,0.98365,0.98365,462"
"first bar stored: 2016.10.28 00:00:00,0.99332,0.99336,0.99324,0.99336,14"
"last bar stored: 2016.10.31 00:00:00,0.98743,0.98766,0.98722,0.98722,104"
"last bar read: 2016.10.31 00:01:00,0.98724,0.98736,0.9872,0.98721,62"
You do not have the required permissions to view the files attached to this post.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
Pushing the boundaries
The following C++ code scans the bar-data loaded into memory, pauses when it reaches the nearest bar on or after the date given and writes the details to the log file:
There is some 'fluff' code which is unnecessary, but since I run it via the debugger, having a few extra variables pulled out can be helpful when stepping through the code.
The code should work for any time frame, if the corresponding bars have been loaded.
There is no user interface, so at the moment the functions are called using hard coded parameters:
From the log file we can now see what is in buried in the original M1 data:
So we now know that there is one hour of data which we could shift to Monday if we want to do our own data conversion.
Code: Select all
void BarData::FindBoundary(int iTimeframe, sse32time_t sseTimeOfNewGroup)
{
sse32time_t tStamp;
int iDayOfWeek = Sunday;
int iTotalBarsCounted = 0;
int iDayOfMonth = 0;
int iFirstOfNewGroup = 0;
int iLastOfOldGroup = 0;
std::tm TimeInfo = {0};
std::vector<SSETICK>& rToBarData = getBarsByTimeframe( iTimeframe );
for( auto & currentBar : rToBarData )
{
tStamp = currentBar.ssetime;
sse32time_to_tm( tStamp, &TimeInfo, __LINE__);
//---------------------------------------
iDayOfWeek = TimeInfo.tm_wday;
iDayOfMonth = TimeInfo.tm_mday;
bool bDebug = false;
if( (tStamp >= sseTimeOfNewGroup) )
{
iFirstOfNewGroup = iTotalBarsCounted;
iLastOfOldGroup = iFirstOfNewGroup - 1;
break;
}
iTotalBarsCounted++;
}
LOG(INFO) << "Find Boundary: time frame:" << iTimeframe;
LOG(INFO) << " total bars counted: " << iTotalBarsCounted;
if( iFirstOfNewGroup > iLastOfOldGroup )
{
LogSseBar( rToBarData.at(iLastOfOldGroup));
LogSseBar( rToBarData.at(iFirstOfNewGroup));
} else {
LOG(INFO) << " unable to find boundary. ";
}
}
The code should work for any time frame, if the corresponding bars have been loaded.
There is no user interface, so at the moment the functions are called using hard coded parameters:
Code: Select all
convertStrDateToEpoch( lastWeekDayBeforeGMT, tStartDate, bDebug);
convertStrDateToEpoch( firstWeekDayAfterGMT, tEndDate, bDebug);
tBoundaryDate31 = tEndDate;
myBarDataManager.Asset().SetUpSseTickLoadParameters(
path2AsusCsvHistory,
SseM1Filename,
"M1",
tStartDate,
tEndDate,
iMaxBarsToLoad
);
myBarDataManager.Asset().LoadDukasData();
convertCharDateToEpoch( "2016.10.30 00:00", tBoundaryDate30, bDebug);
myBarDataManager.Asset().FindBoundary( PERIOD_M1, tBoundaryDate30);
myBarDataManager.Asset().FindBoundary( PERIOD_M1, tBoundaryDate31);
You do not have the required permissions to view the files attached to this post.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
tick, tick, tick
How many traders are interested in tick data?
Could my EA be more profitable if it used tick data?
There are too many things to think about already without adding more work.
However, it is useful to know the fundamentals of price data and how tick data gets turned into bar data.
How much storage space will be required if I collect ticks for say 30 pairs for 12 months?
The direct downloaded data from Dukascopy for EURUSD for 1 year requires about 200 MBytes, so for 30 pairs less than 6 GBytes, but what about temporary and working space?
Lets say you use Birt's download and processing scripts. One output tick file in CSV format is a serpiginous 2GB so 30 pairs would require 60GB plus work space.
Two things to bare in mind
a) My Raspberry Pi idea involves a 64 GB card
b) This CVS tick file is just unused fluff, totally pointless for my purposes
However, it is useful for peeking at the raw ticks and we can do this kind of thing:
[/size]
and more useful things like:
[/size]
but why would I need to do that?
It is important that any historical tests can be related to live trades for various brokers.
Here is a challenge or two challenges we need look at some M1 bars
a) one from inside BST/DST
b) one from outside BDT/DST
Sounds easy but there are unhelpful gaps in different places for each broker, so first we must find a specific M1 bar that exists in all histories.
After a few attempts I managed to find these for all brokers:
[/size]
Clearly, the price data is close enough, but what are your thoughts on the volumes?
A factor 4 variation in volumes?
Why the huge difference?
Are these brokers from parallel universes?
Is somebody generating fake ticks or someone else hiding ticks?
Or do liquidity providers use HHGG restaurant mathematics?
So the only thing we can be sure about is that criminals are manipulating everything that we see by the time it gets to our platform.
What are your predictions for the corresponding Dukascopy data?
Now I have to invent my own restaurant mathematics to convert a stream of ticks into an M1 bar.
You might be wondering if there was anything special about 2016.11.09 12:05 and the answer is yes, I made a conscious decision to be slap bang in the middle of a working week, to avoid any boundary issues with Mondays, Fridays and any 24H boundary issues, so you could say I chose the epicentre of a working week. This should mean prices and volumes are closer to each other than off peak times.
Going back to the raw Dukascopy tick data, if we take it literally we end up with:
[/size]
which doesn't match the broker data,
however, some sources say that the data is GMT, whereas and my brokers are UTC+2,
so searching for 10:05 might be better:
[/size]
Referring back to the broker data we have confirmed that we would need to shift the non-DST time stamp by +2 hours.
The Dukascopy volume is closer to the middle range so perhaps FxPro has the least manipulated volume figure?
So the 2GB CSV tick data is useful for manually exploring the raw data, but not something we need to hang onto.
How much space will we need for 12 months of M1 bars?
a) in CSV about 6MB
b) in binary about 3MB
c) in Empty4 .hst, not sure yet but probably less than 10MB
So there is still a good chance that the Raspberry Pi can store what is needed on it's 64GB card without having to slow it down with an external drive.
Could my EA be more profitable if it used tick data?
There are too many things to think about already without adding more work.
However, it is useful to know the fundamentals of price data and how tick data gets turned into bar data.
How much storage space will be required if I collect ticks for say 30 pairs for 12 months?
The direct downloaded data from Dukascopy for EURUSD for 1 year requires about 200 MBytes, so for 30 pairs less than 6 GBytes, but what about temporary and working space?
Lets say you use Birt's download and processing scripts. One output tick file in CSV format is a serpiginous 2GB so 30 pairs would require 60GB plus work space.
Two things to bare in mind
a) My Raspberry Pi idea involves a 64 GB card
b) This CVS tick file is just unused fluff, totally pointless for my purposes
However, it is useful for peeking at the raw ticks and we can do this kind of thing:
Code: Select all
brian@brian-q4 /srv/ftp/birt/test_scripts $ head ./EURUSD_DKC_2016.csv
2016.01.03 22:00:01.446,1.0873,1.0875,0.75,0.75
2016.01.03 22:00:01.799,1.08714,1.08748,0.75,0.75
2016.01.03 22:00:01.981,1.08702,1.08748,0.75,0.75
2016.01.03 22:00:04.548,1.08706,1.0875,0.75,0.75
2016.01.03 22:00:07.478,1.08705,1.08749,0.75,0.75
2016.01.03 22:00:30.293,1.08704,1.08748,0.75,0.75
2016.01.03 22:00:34.876,1.08704,1.08747,0.75,0.75
2016.01.03 22:00:41.479,1.08714,1.08747,0.75,0.75
2016.01.03 22:00:44.739,1.08714,1.08746,0.75,0.75
2016.01.03 22:00:44.789,1.08704,1.08746,0.75,0.75
brian@brian-q4 /srv/ftp/birt/test_scripts $ tail ./EURUSD_DKC_2016.csv
2016.11.30 23:59:45.298,1.05904,1.05907,1.12,1.50
2016.11.30 23:59:50.453,1.05904,1.05906,1.00,1.25
2016.11.30 23:59:50.908,1.05904,1.05908,1.00,2.44
2016.11.30 23:59:51.442,1.05905,1.05908,1.00,1.69
2016.11.30 23:59:53.857,1.05905,1.05907,1.00,1.00
2016.11.30 23:59:54.099,1.05903,1.05907,2.25,1.50
2016.11.30 23:59:54.607,1.05903,1.05906,1.12,2.44
2016.11.30 23:59:57.304,1.05902,1.05906,1.12,3.19
2016.11.30 23:59:57.810,1.05903,1.05906,1.00,2.44
2016.11.30 23:59:58.644,1.05903,1.05907,1.12,3.19
and more useful things like:
Code: Select all
brian@brian-q4 /srv/ftp/birt/test_scripts $ grep "2016.11.09 10:05" ./EURUSD_DKC_2016.csv
2016.11.09 10:05:00.010,1.10598,1.10602,3.75,1.00
2016.11.09 10:05:00.264,1.10598,1.10604,4.87,1.50
2016.11.09 10:05:00.417,1.10599,1.10601,2.32,2.70
2016.11.09 10:05:00.594,1.10598,1.106,1.00,2.70
2016.11.09 10:05:00.949,1.10597,1.10598,1.12,1.35
2016.11.09 10:05:01.303,1.10598,1.10602,1.50,1.00
but why would I need to do that?
It is important that any historical tests can be related to live trades for various brokers.
Here is a challenge or two challenges we need look at some M1 bars
a) one from inside BST/DST
b) one from outside BDT/DST
Sounds easy but there are unhelpful gaps in different places for each broker, so first we must find a specific M1 bar that exists in all histories.
After a few attempts I managed to find these for all brokers:
Code: Select all
FxOpen 2016.11.09 12:05 1.10587 1.10635 1.10576 1.10631 113
FxPro 2016.11.09 12:05 1.10591 1.10646 1.10576 1.10643 261
GPrime 2016.11.09 12:05 1.10603 1.10647 1.10584 1.10645 471
Clearly, the price data is close enough, but what are your thoughts on the volumes?
A factor 4 variation in volumes?
Why the huge difference?
Are these brokers from parallel universes?
Is somebody generating fake ticks or someone else hiding ticks?
Or do liquidity providers use HHGG restaurant mathematics?
So the only thing we can be sure about is that criminals are manipulating everything that we see by the time it gets to our platform.
What are your predictions for the corresponding Dukascopy data?
Now I have to invent my own restaurant mathematics to convert a stream of ticks into an M1 bar.
You might be wondering if there was anything special about 2016.11.09 12:05 and the answer is yes, I made a conscious decision to be slap bang in the middle of a working week, to avoid any boundary issues with Mondays, Fridays and any 24H boundary issues, so you could say I chose the epicentre of a working week. This should mean prices and volumes are closer to each other than off peak times.
Going back to the raw Dukascopy tick data, if we take it literally we end up with:
Code: Select all
Dukas 2016.11.09 12:05:00 1.10877 1.10889 1.10855 1.10857 129
which doesn't match the broker data,
however, some sources say that the data is GMT, whereas and my brokers are UTC+2,
so searching for 10:05 might be better:
Code: Select all
Dukas 2016.11.09 10:05:00 1.10601 1.10671 1.10585 1.10642 237
Referring back to the broker data we have confirmed that we would need to shift the non-DST time stamp by +2 hours.
The Dukascopy volume is closer to the middle range so perhaps FxPro has the least manipulated volume figure?
So the 2GB CSV tick data is useful for manually exploring the raw data, but not something we need to hang onto.
How much space will we need for 12 months of M1 bars?
a) in CSV about 6MB
b) in binary about 3MB
c) in Empty4 .hst, not sure yet but probably less than 10MB
So there is still a good chance that the Raspberry Pi can store what is needed on it's 64GB card without having to slow it down with an external drive.
-
snailbeard
- Trader
- Posts: 615
- Joined: Mon Dec 24, 2012 10:54 am
- Location: Just above water somewhere between Oxford & Cambridge
FindBestAssetsToTrade
Those of you not interested in testing EAs must be quite bored by my recent posts, so how about some real trading. Those of you using the currency slope strength indicator have a method of cherry picking what to trade and this post is just a variation on the cherry picking theme, I gave up using CSS because it didn't work for me, but that might be that I didn't approach it in the right way and I hope at some point I can correlate it with my current method of selecting the best pair to trade.
The past posts on this functionality show a somewhat long winded spread-sheet which takes some effort to understand and wouldn't it be good to have something much simpler, or at least a short form of the spread-sheet?
This has now been completed and the two sets of results are similar but not identical because I use a slightly different approach for the quick overview, which ignores other factors such as spread and swap. So now we can scan down the list of pairs for 'true' and ignore anything which is false. However, you might be wondering why EURUSD is high-lighted mustard, as both values are false. This because the long analysis produces a different result EURUSD and tells us that we should be interested in it.
So it would be a good exercise to look at EURUSD and make up your own mind which analysis best fits with your approach to trading: So there are a number of slower moving averages suggesting falling prices, whereas the high speed moving averages are rising and the current 'trend' is definitely up, but it is a somewhat weak trend against a longer term fall so there might be better pickings else where.
The GBPCHF result might be a bug because when I cross checked the chart the previous days are down, but yesterday's (17Jan17) reverse up should have been taken into account and it invalidates the trend, price is clearly ranging.
Apart from that hiccup we are left with 4 USD based pairs, so trading more than one increase the risk of multiple losses. Which of the four pairs offers the best chance of success?
Now would be a good time to look at the detailed 'FindBestAssetsToTrade' output: So firstly we high-light those pairs which are indicating significant price action in a dominant direction, but then values which are less favourable are marked mustard and those which are harmful are marked red. Orange is used to remind us about swap costs if the trade runs overnight, this cost is mitigated by trading a pair with a 'valid' spread which simply means the price action is expected to be much larger than the spread cost of opening and closing a trade and many times the trade will complete the same day.
There can be many days when checks indicate that nothing should be trend-traded and that could be a big time saver allowing us to spend time more fruitfully else where.
According, to today's analysis (which is now too late to be of interest because it should happen just after notional midnight in time for EA's or us to pull in the analysis and ignore uninteresting pairs and focus on interesting pairs. So had we been awake at say 3am CET what would we have seen?
So it seems to be a toss-up between USDCAD and USDJPY and we need to study the charts to see if we agree with the analysis: Unfortunately, the high speed moving averages intersect a slow moving average which is not a good sign.
So what about USDJPY? As often happens lines of support and resistance are correlated on different pairs, so if you go back another day price had crashed through a slow MA and was a good trading day but that day ended at support and today bounced of that support.
So the daily analysis is only part of the story and we have to look at the price in the context of support and resistance before taking the plunge.
The past posts on this functionality show a somewhat long winded spread-sheet which takes some effort to understand and wouldn't it be good to have something much simpler, or at least a short form of the spread-sheet?
This has now been completed and the two sets of results are similar but not identical because I use a slightly different approach for the quick overview, which ignores other factors such as spread and swap. So now we can scan down the list of pairs for 'true' and ignore anything which is false. However, you might be wondering why EURUSD is high-lighted mustard, as both values are false. This because the long analysis produces a different result EURUSD and tells us that we should be interested in it.
So it would be a good exercise to look at EURUSD and make up your own mind which analysis best fits with your approach to trading: So there are a number of slower moving averages suggesting falling prices, whereas the high speed moving averages are rising and the current 'trend' is definitely up, but it is a somewhat weak trend against a longer term fall so there might be better pickings else where.
The GBPCHF result might be a bug because when I cross checked the chart the previous days are down, but yesterday's (17Jan17) reverse up should have been taken into account and it invalidates the trend, price is clearly ranging.
Apart from that hiccup we are left with 4 USD based pairs, so trading more than one increase the risk of multiple losses. Which of the four pairs offers the best chance of success?
Now would be a good time to look at the detailed 'FindBestAssetsToTrade' output: So firstly we high-light those pairs which are indicating significant price action in a dominant direction, but then values which are less favourable are marked mustard and those which are harmful are marked red. Orange is used to remind us about swap costs if the trade runs overnight, this cost is mitigated by trading a pair with a 'valid' spread which simply means the price action is expected to be much larger than the spread cost of opening and closing a trade and many times the trade will complete the same day.
There can be many days when checks indicate that nothing should be trend-traded and that could be a big time saver allowing us to spend time more fruitfully else where.
According, to today's analysis (which is now too late to be of interest because it should happen just after notional midnight in time for EA's or us to pull in the analysis and ignore uninteresting pairs and focus on interesting pairs. So had we been awake at say 3am CET what would we have seen?
So it seems to be a toss-up between USDCAD and USDJPY and we need to study the charts to see if we agree with the analysis: Unfortunately, the high speed moving averages intersect a slow moving average which is not a good sign.
So what about USDJPY? As often happens lines of support and resistance are correlated on different pairs, so if you go back another day price had crashed through a slow MA and was a good trading day but that day ended at support and today bounced of that support.
So the daily analysis is only part of the story and we have to look at the price in the context of support and resistance before taking the plunge.
You do not have the required permissions to view the files attached to this post.