Tick Capture

[ianj]

Now this is ANNOYING

CNS UK VPS connection - should be just a few ms - and its routing via .. Dusseldorf !!!

Thats 14ms out to Germany - and then another 20ms roundtrip to the server - and thats just a few miles from the VPS

[ianj]

Quite a few disconnects on the US capture (top of book only) today
At the same time the FULL BOOK streaming from Uk (via Dusseldorf ) seems to be fine

The capture utility outputs stats every minute - here is a side by side view of a recent "minute"

"icm" - Empty4, "icmc" - cAlgo - "icmc" on right hand side is full book capture - notice the increase of ticks over top of book which filters out ticks where the top of book does not change

Ill have to look at capturing the disconnects somehow - to see how reliable the feed is

Oh my poor CPU - a surge of updates earlier on - thats 9128 in 60 seconds across 32 pairs - that is an AVERAGE of 150 ticks per second

[kwanann]

Are u storing all these anywhere?

[ianj]

Are u storing all these anywhere?

Yes, but :

• 1. Some of it i am explicitly NOT allowed to share (LMAX API data) as part of the license - Empty4 feeds/cTrader i am not yet sure of
• 2. Its pretty big stuff and mounts up over time - luckily local storage is not a problem in these days of 12TB raid arrays on the desktop

Size wise :

• The LMAX feed (5 price level DOM, 50+ pairs) is about 60-70 MB per day heavily compressed, about 1GB CSV
• 10 Empty4 live feeds, 32 pairs, (top of book) is about 70 MB per day heavily compressed, about 1 GB CSV
• 1 cTrader full book, 32 pairs ... i haven't completed 1 day yet but i estimate around 100-120MB per day heavily compressed, about 1.5 GB CSV

The files are simple to parse CSV and compressed with Winzip latest version - i could "encode" to something smaller but that requires effort and time - right now capture/storage is good enough

I can upload the cAlgo Indicator (you need to deploy one per pair, so i have deployed 32) that i use when its been in use for a few more days - i need to make sure it is stable. The indicator opens a network connection to a SEPARATE process that stores from multiple feeds (Empty4/cTrader/LMAX mixed) - by storing in 1 place i can retain temporal ordering without worrying overly about syncing clocks and time stamps in files - i get one file that is easier to manage each day- That "store" process is written in Java - i dont mind uploading it here as well - its nothing too complex (The Empty4 captures run 1 EA per chart per account - thats 32*10 = 320 EA's and therefore network streams to the store process)

My eventual aim is to merge it into a single repository, and include Dukascopy & TrueFX - but that is some way off yet. It would be great at that time to offer limit public querying capabilities but it might be a legal minefield so i don't know yet - why do you think there is no similar service out there at the moment ?

[ianj]

1 cTrader full book, 32 pairs ... i haven't completed 1 day yet but i estimate around 100-120MB per day heavily compressed, about 1.5 GB CSV[/list]

Make that 2.8 GB CSV (178 MB compressed) for the first full day (UK) : at a glance not many disconnections - certainly not as many as US

10 Empty4 live feeds, 32 pairs, (top of book) is about 70 MB per day heavily compressed, about 1 GB CSV

Make that 1.4GB CSV including cTrader top of book (US - 91MB compressed) - and plenty of cTrader disconnections, albeit short term

The disconnections in US are disconcerting - The Empty4 captures are not disconnecting - just the cTrader - so its likely not a general VPS stability issue. I will try to monitor bandwidth usage later to see if that might be an issue. My UK VPS has a smaller configuration but is more lightly loaded

The capture does not explicitly log disconnects - they should be apparent from discontinuities in the capture - but i might start storing the disconnects explicitly soon

[ianj]

There have been some proxy changes following a posting i made to IC Markets which they forwarded to spotware - it seemed it did make a difference

My UK VPS connection (CNS) is now connecting directly to the server/proxy (it was routing out from london to dusseldorf and back to london again) It is now routing directly through a london proxy and the roundtrip has dropped to approx 2/5ms - it is still interesting that the proxy latency remains constant at 2ms whilst the server latency oscillates between 5-15ms, but its still a vast improvement on connecting through dusseldorf)

Meanwhile the US connection has also reverted to the London proxy , but it has not improved its disconnection performance. This leads me to suspect that it could be related to the CNS VPS, i will look into the possibility of running it on a separate VPS or trying another provider. The disconnections are still only between the socket from the cAlgo indicator and my capture mechanism - not necessarily from cAlgo to the spotware server - it might be the problem is in the indicator so i cannot lay the blame elsewhere just yet - i will look at the options to distinguish server disconnects from cAlgo indicator disconnections over the coming days. In all this time the connections from Empty4 EA's to my capture have been solid but i cannot asses the quality of the Empty4 EA to the Empty4 server without properly analysing the feeds

[ianj]

On the question of tick handling and comparisons in general:

My aim is to store these ticks in a central repository which i can query for statistical analysis or chart for visual analysis. I have 10 Empty4 top of book streams and two price ladders (IC Markets cAlgo & LMAX) and am planning to add more soon (cAlgo Think and Pepper). I personally have an interest and skills in HFT so i will be looking for alpha generating opportunities in the retail sector

I have development skills and have considered building such a repository from scratch myself, the problem being the time it takes - i just would not get the time to build in easy to access analytics and statistical analysis crossing broker feeds - we are not talking mere charts here. You can get some simple analysis from myfxbook, mt4i, fxintel etc but that analysis is general and canned only - you do not get the chance to perform detailed and/or potential alpha supporting analysis. I have a side project that is building an ultra performant back tester which will tightly encode the data but not necessarily offer it in a form that is ideal for an-hoc analysis - it is intended for out and out speed, but i am considering it as an option - at least access to the data would be very very fast

So i am looking for a tick/market data store that offers itself to supporting comprehensive analysis queries whilst still offering reasonable performance. There are a number of commercial products but only one (i think) that offers a free version to normal mortals who don't wanna spend thousands - KDB by KX (It is used fairly extensively in the industry - they ARE COMMERCIAL but offer a slightly restricted free version - probably enough for my needs)

http://code.kx.com/wiki/Main_Page

I am looking to store the collected data in KDB and give it a try - when i get the time. I will try to let you know how it goes .

Until then, i would welcome any other suggestions for products (free or inexpensive) that will help. I have excluded any general SQL based database, CSV, flat files etc - they just do not offer the specific features required of a temporal data store

[ianj]

Part of the reason for the apparent disconnection on the US VPS was in fact the liquidity dropping out - ie no bid or asks on the book - this lead to a code failure which threw an exception leading to an unnecessary recovery/reconnection to the feed logger. This code path was not taken for a full book capture - hence the full book capture in the UK was fine

I have changed the code to emit "-" instead of the price where there is no bid or ask on the book

I have not questioned WHY there is no bid/ask - it might be an artefact of a real disconnection or simply that the book collapses over news events

I will scan the logs later to see what happens leading up to and away from such an event in the future

[ianj]

Still capturing - the number of "disconnects" on US side has reduced (bug fix above) but they are still there - i will look at the cAlgo API to see if there is anything else to help. The problem is that the cAlgo appears to be restarting so there is nothing i can do to manage the "disconnects" within the indicator - i can log reconnects from the cAlgo indicator to the feed capture but that will only show me so much and probably not help with the underlying problem

I am thinking of producing a cRobot versions - the restart semantics of a robot might be different to an indicator - after all an indicator "state" might not be considered as important as a "robot" state and they might try and preserve the instance for longer

I will try some time soon and report back

Meanwhile i have added a few more pairs - i would like to push into pairs considered a little more exotic - not just illiquid crosses i.e. PLN/TRY/HUF/CZK/HKD/ZGD/MXN/ZAR - the main problem is availability on cAlgo (IC Markets) The pairs have spotty availability of Empty4 as well so i have added support where it is available (about 35-40 extra captures across 10 brokers)

I still seriously thinking about adding cAlgo/cRobot live account with THINK (I have IC Markets right now)

I also have another interest that i might look at introducing here - improved interprocess communication. Currently i use sockets - the advantage being that they can be used to cross process/machine boundaries - anything that supports TCP/IP. They are widely supported and pretty performant (latency WITHIN a machine is 500us to 1ms on the same machine) The trouble is that that consume host machine resources (especially when you have 400+ in use) and could be faster

I am looking at using more efficient IPC (e.g. shared memory) that i can use when the capture and storage process are on the same machine- as well as using less resources it could lead to lower latency, far far lower than 1us, never mind 1ms. It is not really required here but it could be a good place to start testing a mechanism that i would like to use elsewhere (e.g. connecting separate processes in a trading system) If i get anywhere with it i will report here as well

[ianj]

From spotware direct - my gut feeling was correct:
"Indicators are recreated on reconnect, all internal variables are re-initialized. Robots have different life cycle, they are not restarted on disconnects. So in your case most probably it's better to use robots"

Now .. if i can detect the reconnect within a Robot - then i can log the disconnect. If i cannot detect the disconnect in the Robot then i might as be an indicator - at least i am notified if the disconnect - i restart !

As long as it does not lead to long term resource leakage when the socket is pulled - yes i will lose the socket and i must restart but at least it is additional information. I will setup a simple robot+ indicator that runs alongside the capture indicator to see what they can see - and i will look for something in the API that can show me what is happening.