Tick Capture

[ianj]

Coming soon - a tick capture indicator

I cant show you what it looks like - because it has no presentation element - it merely subscribes to rate updates, opens a network socket and transmits the tick updates. Its part of a Tick Capture mechanism.

A separate process should accept socket updates and, if it chooses, capture tick updates for analysis or external backtesting, or act as a feed for alternate trading capabilities. You deploy 1 indicator per chart you wish to capture. I will be producing a robot version if it offers any advantages over the indicator - once i know the differences :!: for this usage

I have chosen to post to a network socket rather than log to a file to add flexibility -

• it is possible to log to a common capture file using a single capture process, which means you can have a capture file that maintains fairly strict temporal ordering - important if you are looking closely at cross pair temporal ordering of updates
• alternate trading mechanism (trading algo containers, gui trading mechanism) receive the update with very little additional latency (<1ms for a local network call) compared with scanning the tail of a feed file

It is capable of emitting changes to top of book only, or to the whole cTrader aggregated price book. So far, the tick rate of the top of book updates have matched (within 10%) the equivalent Empty4 platform for the broker (IC Markets) but i do receive many more event updates for the book as a whole, which can act to CONFIRM the top of book is still current

I am running against IC Markets and plan to test against Pepper cTrader live when it is available

Here is an example of the messages sent to the network socket:

Code: Select all

2013.06.13 09:26:39.529,cicm,EURUSD,1.33454,1.33456 
2013.06.13 09:26:39.545,cicm,AUDUSD,0.94852,0.94856 
2013.06.13 09:26:39.688,cicm,USDCHF,0.91860,0.91870 
2013.06.13 09:26:39.687,cicm,AUDUSD,0.94852,0.94852 
2013.06.13 09:26:39.801,cicm,USDCHF,0.91861,0.91870 
2013.06.13 09:26:39.796,cicm,AUDUSD,0.94847,0.94847 

I am just going clear some stuff up - reconnection etc before posting the code

TickCaptureIndicator.cs - one pair per chart - deploy indicator on any timeframe but only once per pair - i have 30+ charts, 30+ indicators - It opens a network connection to "Host", "Port" and starts streaming "BookLevels" to that stream - each entry as above where Is defaults to "icmc". If BookLevels=1 then it will only emit the price when the top of book changes, all BookLevels>1 emit the whole book on every tick, whether it changes or not (about 2-300% more ticks than top of book changes)

TickCaptureIndicatorX.cs - in progress - deploy on one chart only - will capture all pairs specified and emit on a single stream

[ianj]

No time to complete yet - but i left it running over weekend - and on monday it just resumed ticking over the socket - excellent. I still have to address reconnections etc but it looks promising

Looks like i will have an Indicator AND a Robot - see http://www.stevehopwoodforex.com/phpBB3 ... =66&t=2523 for the difference

Essentially for tick capture an Indicator is probably fine - its probably lighter weight etc

The eventual aim of the Tick Capture is to support a full trading API on the same network stream (opening, closing positions - accessing history etc) - for that i will need a Robot - so I will extend the Tick Capture Indicator to support trading - I will however have to address a multi pair feed - I have an Empty4 bridge that access all pairs from one EA - i dunno if i can do that in cTrader (access other pairs from a singe Robot) If i cannot i will probably split it into a number of Indicator + a trading Robot - lets hope a single Robot can open multiple currencies or we will end up with 1 Robot per pair

[ianj]

Been away the last couple of months - but am returning to look at cAlgo

I completed the tick capture indicators and have hooked it up alongside my live capture of a handful of Empty4 platforms (all live)

My cAlgo is IC Markets and it sees approx 15-20% more ticks than IC Markets Empty4 - i have yet to analyse relative latency of updates - i will do that in a few days when have some reasonable data

Platform is a bit bloated though - its currently consuming 450MB + ram for 30+ pairs (Empty4 equivalents around nearer to 50MB+) and CPU usage is around 15+% (vs 4-5% for Empty4) - so no lightweight

cAlgo uses slightly more memory than cTrader - but uses less CPU, so i am using cAlgo

Column "icm" is Empty4 IC Markets - "icmc" is cAlgo IC Markets - both live accounts - all other platforms are live Empty4 account - time approc 15:49 GMT (1 minute sample) The grid shows the number of ticks received in the latest M1 interval

PS The cTrader web edition seems to use far less memory (200MB+ on chrome) but it does not seem to allow custom indicators + you have to allow the overhead of the other browser processes - i dont think spotware have any plans to allow custom indicators in the web edition

[ianj]

Some more info if you are interested -

cAlgo supports a DOM - a multi price level order book (prices at levels deeper than top of book) - and it is available to the API

Looking at my LIVE ICM feed the number of relevant tick updates increases greatly if you want to capture a full book - my top of book capture will only emit an update if the top of book changes - and that does not change every time a deeper price changes - in fact i just looked at the last M1 now and over 5 pairs the number of tick updates were as follows (355 updates vs 98)

Code: Select all

	       icmc	icmc1(top only)
AUDUSD	112	37
EURUSD	63	17
GBPUSD	68	9
USDCHF	55	17
USDJPY	57	18
  	   355	98

That leads to 2 conclusions - 350% of the updated frequency (more storage/CPU etc) and also an affirmation that the top of book HAS NOT CHANGED for the intervening tick updates - this might be important - it effectively means its emits 350% more ticks than top of book

If you combine the increased emission of top of book over Empty4 (about 25%) and the full book updates where top of book does not change you get approx 400% more updates

Now that is worth thinking about !

Added a few more samples below to confirm - these were about 18:00 GMT - peak values may be higher (usually about 15:00 GMT)

Code: Select all

	icmc	icmc1
AUDUSD	145	74
EURUSD	87	14
GBPUSD	74	22
USDCHF	62	20
USDJPY	101	21
	469	151

Code: Select all

	icmc	icmc1
AUDUSD	89	30
EURUSD	53	7
GBPUSD	31	9
USDCHF	44	6
USDJPY	121	62
	338	114

[ianj]

I see THINK have cTrader now as well - and i already have a THINK live account - but that is an extra loading on the VPS (400+MB RAM + several GB per day) to store full book data - i have posted a query with spotware to see if the memory footprint can be reduced

I guess that is the cost of being a data hoarder

[ianj]

And more info to keep in mind - my New York VPS is showing 84ms roundtrip - that is because the cTrader (IC Markets) server is based in UK - My local desktop (200mb fibre - am about 40ms from UK) is connected via a proxy in ... SYDNEY !!!

Interesting enough the New York VPS based cTrader is connected to the UK via a proxy in .. wait for it .. Dusseldorf (that is in Germany)

You have to be careful of this stuff - clearly SYDNEY is right next door to Finland (showing 400+ms latency)

[KevinT]

And more info to keep in mind - my New York VPS is showing 84ms roundtrip - that is because the cTrader (IC Markets) server is based in UK - My local desktop (200mb fibre - am about 40ms from UK) is connected via a proxy in ... SYDNEY !!!

Interesting enough the New York VPS based cTrader is connected to the UK via a proxy in .. wait for it .. Dusseldorf (that is in Germany)

You have to be careful of this stuff - clearly SYDNEY is right next door to Finland (showing 400+ms latency)

I´m puzzled You are (DT) 40ms to LDN (HTTP?) via FDDN proxi Down Under (how's that podsiblle!?), but 400ms plus from helsinki (Empty4 Nord server?) please care to explain as I'm really dumb & ignorant

PS: By the way Awesome and stunning fundamental research (building blocks really) also to add to my I never had latency over 140 from low key ADSL half way around the world to Sidney (Go Empty4 server (before the LMax deal!)) :

[ianj]

I´m puzzled You are (DT) 40ms to LDN (HTTP?) via FDDN proxi Down Under (how's that podsiblle!?), but 400ms plus from helsinki (Empty4 Nord server?) please care to explain as I'm really dumb & ignorant

PS: By the way Awesome and stunning fundamental research (building blocks really) also to add to my I never had latency over 140 from low key ADSL half way around the world to Sidney (Go Empty4 server (before the LMax deal!)) :

I live in Finland - i have 200 mb fibre to the home - and my ping from my desktop to UK city (my CNS VPS) is 40ms - I have no idea why cAlgo decides to proxy from my desk to UK (cAlgo servers) via Sydney - that was the point of my statement
The latency was as reported by cAlgo itself (bottom right hand corner) - i just did a manual ping to a couple of sites in Sydney - around 300ms

All these numbers are roundtrip

I have actually been capturing a 5 level live API price book from LMAX since Jan 2013 (from UK VPS) and 10 live Empty4 feeds(top of book) for a couple of months (NY VPS) - i have to make a choice as to where i capture data from - it matters when/if i come to compare the two.

Below are the cAlgo latency reports (desktop is 5 pairs for around 12 hours - the NY VPS is streaming 32 pairs .. via Dusseldorf remember) Because the Sydney link has to come all the way back to the UK the desktop overall latency to server is 763 ms

I will setup cAlgo from CNS UK later - just to compare - but i have not yet decided where to capture from longer term - probably UK but will have to look at VPS spec - as of this moment i am only actually capturing top of book but by the close of today it will be the full book - and thats around 5+ times more data (more updates, more data per update). At the moment i do not encode the data - its just a text dump of each tick update, and its very verbose - i compress/send home daily and have no problem with local storage but VPS intermediate storage and transfer home is an issue

Here is a tick from just a few moments ago - as you can see its pretty verbose

2013.08.13 04:31:28.332,icmc,EURUSD,1.33101/50,1.331/50,1.33098/200,1.33097/600,1.33096/100,1.33095/500,1.33094/200,1.33093/800,1.33092/500,1.33049/8005,;1.33102/50,1.33104/50,1.33105/400,1.33106/300,1.33107/700,1.33108/200,1.33109/300,1.3311/500,1.33111/500,1.3315/8105,

which in "english" is:
2013.08.13 04:31:28.332,icmc,EURUSD,
Bids
1.33101/50, (top of book, 50 contracts - $500k - or 5 "standard lots")
1.331/50,
1.33098/200,
1.33097/600,
1.33096/100,
1.33095/500,
1.33094/200,
1.33093/800,
1.33092/500,
1.33049/8005,;
Offers
1.33102/50, (top of book, 50 contracts)
1.33104/50,
1.33105/400,
1.33106/300,
1.33107/700,
1.33108/200,
1.33109/300,
1.3311/500,
1.33111/500,
1.3315/8105,

* Update - it has now decided to route my desktop (in Finland) thru Dusseldorf as well - and that proxy is a 38ms round trip from me - overall 56ms to the server (9ms each way Dusseldorf to London)
* Update - desktop now routing thru Strasbourg (France, near german border) 42/62ms

[ianj]

OK - a couple of interesting findings - Below you can see a sample of price updates - "icm" is IC Markets Empty4 (US) , "icmc" is IC Markets cTrader (UK)

The logging mechanism is in US - the difference is around 100ms roundtrip, so 50ms one way (via Dusseldorf) so ALL THINGS BEING EQUAL the "icmc" updates should be around 50ms LATER than "icm"(Empty4) updates


So ?

2013.08.12 23:59:58.157,icm,AUDCAD,0.93982,0.93991 Empty4 wins by 50+ms
2013.08.12 23:59:58.213,icmc,AUDCAD,0.93982,0.93991

2013.08.12 23:59:58.590,icmc,AUDCAD,0.93977,0.93983 cTrader wins by 54ms
2013.08.12 23:59:58.644,icm,AUDCAD,0.93977,0.93983

2013.08.12 23:59:58.851,icm,AUDCAD,0.93981,0.93985 Empty4 wins by 18ms
2013.08.12 23:59:58.869,icmc,AUDCAD,0.93981,0.93985

2013.08.12 23:59:58.966,icm,AUDCAD,0.93982,0.93987 Empty4 wins by 90+ms
2013.08.12 23:59:59.067,icmc,AUDCAD,0.93982,0.93987

2013.08.12 23:59:59.289,icmc,AUDCAD,0.93982,0.93989 cTrader wins by 110+ms
2013.08.12 23:59:59.397,icm,AUDCAD,0.93982,0.93989

2013.08.12 23:57:17.440,icmc,EURUSD,1.32923,1.32926 cTrader wins by 68ms
2013.08.12 23:57:17.508,icm,EURUSD,1.32923,1.32926

2013.08.12 23:57:18.647,icm,EURUSD,1.32923,1.32927 Empty4 wins by 13ms
2013.08.12 23:57:18.660,icmc,EURUSD,1.32923,1.32927

2013.08.12 23:57:25.094,icmc,EURUSD,1.32925,1.32927 cTrader extra tick

2013.08.12 23:57:39.646,icmc,EURUSD,1.32922,1.32925 cTrader wins by 92ms
2013.08.12 23:57:39.738,icm,EURUSD,1.32922,1.32925

2013.08.12 23:57:39.886,icmc,EURUSD,1.32923,1.32926 cTrader wins by 70+ms
2013.08.12 23:57:39.957,icm,EURUSD,1.32923,1.32926

2013.08.12 23:57:43.481,icmc,EURUSD,1.32924,1.32927 cTrader wins by 220+ms
2013.08.12 23:57:43.700,icm,EURUSD,1.32924,1.32927

What we are seeing is that, considering cTrader has a 50ms handicap, it manages, in general, to beat Empty4 - sometimes by a good margin - but not always. I would need an automated tool that track sync between the 2 feeds to say anything more conclusive (which is not so easy as they are not identical and will sometimes wander apart for a few updates)

[ianj]

The cAlgo instance in my NY VPS seems to have switched proxy - from Dusseldorf to Strasbourg - a few ms more