Tick Capture

[ianj]

Are u storing all these anywhere?

Progressing on this slowly - just assembled a simple socket server that will allow a logon and retrieval of tick data for limited periods (including Level 2 where available) - but not bulk download. You can receive the data in clear CSV or more compact compressed forms (i am thinking LZ4 + custom)

The plan is to allow access by hand (using telnet/putty etc) or to a utility i am creating to show a historical DOM (to start with) and tick chart (later) with access to multiple tick sources.

I have considered a HTTP get format retrieval mechanism but would really only want to use it with a presentation capability (say a chart or formatted table) - but not downloading - but am considering it a very low priority - i have looked at google charts with a view on using them to present a tick chart, and its challenging to get something useful

I have serious concerns about offering bulk download because of licensing and bandwidth limitations - the intention will only be to offer limited retrievals

Has anybody got any views on this ?

[ianj]

Updated first post with code: http://www.stevehopwoodforex.com/phpBB3 ... =67&t=2520

[ianj]

Some "interesting" stats for you - i just analysed a "typical" day: 2013-Aug-29

It captured 40 pairs, 6.9M ticks - capture file was 3.3GB - a typical tick with all levels took 400+ bytes

I am analysing it to cut up into chunks - the minimum size i will allow retrieval in my DOM viewer tool - each chunk will essentially be a CSV with the stamp and bid/offers at all levels - compressed

With the compression format i am using right now (chosen for FAST decompression) i get:

ALL pairs: Tick-2013-Aug-28

Code: Select all

TF	Comp%     AvgSz  MaxSz AvgCt
M5	 17%        47k   318k   600
M15   17%       143k   817k  1799

So M5 looks ok - each tick reduced down to <80 bytes, 47k-100k retrievals in most cases - for a 5 minute chunk of data. I can probably squeeze that down another 30% using a better standard compression format (perhaps LZ4) or 50%+ using something custom but that looks fine for now

[ianj]

a*** - I have started to analyse captured files and have noticed "backwardation" in timestamps - a few hundred times a day the local PC timestamp steps back/forward anything from 10ms to 1-2 seconds.

Stepping forward could simply mean no ticks for a few seconds, but stepping back just is not possible (at least right now)

It looks like my clock is moving. I don't rely on standard windows time syncing, i use a dedicated NTP syncing program which, from experience, i know is far more efficient. I just took a look at the console on the time sync and it is showing my time varying by up to 1-2 seconds either way (but usually far less) ...

I think i have some settings i can make to change the time in smaller degrees, more often - but this does not alter the fact that the reported time on a VPS is not so reliable

The tick are recoded in temporal order (that is EXACTLY why i did not rely on time syncing) so they are 99% valid - its just that any processing has to deal with time stepping back, whilst the ticks themselves are correct. I can guesstimate the period it stepped back by taking the short term average tick span leading up to the events but in the end its only guesswork

For now i will mark such "discontinuities" but leave them in place, and look to retrieving my time from an external source - directly from an NTP source, and not from the PC. The stamps should then be far more accurate.

[ianj]

And here we have it - polling an NTP server every minute on the VPS:

05:08:53,086 NTPClient - > 82.113.154.206/82.113.154.206
05:08:53,101 NTPClient - Stratum: 2 (Secondary Reference; e.g. via NTP or SNTP)
05:08:53,101 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-904
05:09:53,195 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1010
05:10:53,286 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1117
05:11:53,391 NTPClient - Roundtrip delay(ms)=15, clock offset(ms)=-1229
05:12:53,478 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1340
05:13:53,562 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1442
05:14:53,644 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1539
05:15:53,724 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1625
05:16:53,818 NTPClient - Roundtrip delay(ms)=16, clock offset(ms)=-1723
05:17:53,894 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1819
05:18:53,967 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1902
05:19:54,039 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-1987
05:20:54,110 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-2066
05:21:52,194 NTPClient - Roundtrip delay(ms)=0, clock offset(ms)=-157
05:22:52,298 NTPClient - Roundtrip delay(ms)=16, clock offset(ms)=-262

Notice the rollover around 2 seconds/05:21:52,194 (05:20:54,110 -> 05:21:52,194) - looks like i have found the culprit

I can apply the NTP offset to the time stamping, but i am not allowed to poll NTP servers too often - perhaps i will maintain a list so i can hit them more often - either way hitting them once per minute at least should reduce backwardation to a hundred ms or so

I am using CNS Windows 2003 in UK, and Windows 2008 in NY, and they both have time backwardation problems - looks like am gonna have to build NTP client into my tick capture and use that instead- I could report the problem to CNS but in all honesty its probably outside of the remit to provide a good Empty4 trading environment - the time is accurate to a couple of seconds

[ianj]

OK - am now testing an NTP compensated capture mechanism in UK - should be accurate down to 1-2ms, i scan the NTP servers once per minute, and maintain time between scans using a performance timer that is not affected by windows time syncing, but needs the NTP time as a base. Its not good enough just to rely on synchronized windows time as it slips 50-100ms per minute, hence am using a combination of NTP and another precision timer.

If this works ok for the next day or two ill deploy in NY and time slip problems should be a thing of the past (and UK/NY timestamps can be compared with some confidence)

[ianj]

Well ... we still have backwardations in the stamps but they are vastly decreased:

Yesterday morning - 1500-2000ms every 10-15 mins

- 1726:2013.09.11 23:28:24.413,icmc,USDSEK,
-> 2013.09.11 23:28:22.687,icmc,EURNOK,
- 1540:2013.09.11 23:48:47.130,icmc,NZDCAD,
-> 2013.09.11 23:48:45.590,icmc,USDCAD,
- 1517:2013.09.12 00:04:41.756,icmc,EURNZD,
-> 2013.09.12 00:04:40.239,icmc,EURNOK,
- 1506:2013.09.12 00:19:33.491,icmc,USDCHF,
-> 2013.09.12 00:19:31.985,icmc,GBPNZD,

To (after NTP+performance timer change) later that day to <20 ms

- 21:13.09.12 16:52:04.143,icmc,GBPJPY,
-> 13.09.12 16:52:04.122,icmc,EURJPY,
- 13:13.09.12 16:53:04.124,icmc,GBPCHF,
-> 13.09.12 16:53:04.111,icmc,EURCHF,
- 10:13.09.12 16:55:06.929,icmc,USDCAD,
-> 13.09.12 16:55:06.919,icmc,EURSEK,
- 19:13.09.12 16:58:06.912,icmc,EURHUF,
-> 13.09.12 16:58:06.893,icmc,USDCHF,
- 6:13.09.12 16:59:06.884,icmc,AUDCHF,
-> 13.09.12 16:59:06.878,icmc,EURJPY,
- 16:13.09.12 17:07:07.260,icmc,USDCHF,
-> 13.09.12 17:07:07.244,icmc,EURJPY,

Not perfect, but a massive improvement

[ianj]

I know this is slightly off subject - but its good to know. My NY VPS is using the built in windows time syncing feature on Windows 2008 (CNS), which checks far less than the NTP service i ran in UK (which was up to 2000ms out), and my NY VPS is currently 6118ms out of sync !!!!

I have deployed the sync correction to NY as well, so UK/NY should be within 10-20 ms of each other from Monday

i can note that the :
* UK VPS (Windows 2003) is slipping 50-100ms per minute before being corrected by my (not builtin Windows) NTP service
* US VPS (Windows 2008) is not really slipping much at all (but is 6 seconds out)

Very odd !

[ianj]

Until now i had been capturing LMAX & ICM cAlgo into separate files - The ICM capture has the time slippage correction but the LMAX does not - so they cannot be reliably compared together. Rather than merely time correct the LMAX capture i have elected to route the LMAX capture to the Java capture process that is capturing ICM output from the indicators as well.

This means it will automatically benefit from time slip correction, but more importantly the temporal ordering between LMAX & ICM cAlgo updates will be established. I now have a single consolidated capture file containing LMAX updates (52 pairs, 5 price levels) and ICM cALgo (40 pairs, 10-15 price levels)

I expect Raw CSV files to be 5+GB per day (LMAX 45%, ICM 55%) - 350-400MB heavily compressed

Not much other progress this week - still capturing - still writing data preparation process that cuts up the CSV above and writes it out into M15 chunks of compressed data, ready to be served to any client that requests them (The multi feed DOM viewer which is part working) By pre-compressing into M15.feed.symbol chunks i can ensure data transfer is minimised which in theory will allow it to support more concurrent users if i decide to open access.

I now automatically package, compress and transfer capture files from the VPS to my server on a daily basis - i hope to process them incrementally to make historical data available within a day. I think its feasible to reduce this interval much further though - to maybe hourly (or more often) - transfers making data available in near time. I could in theory stream from the VPS to the server but it might add unnecessary additional load on the VPS, something i have to be careful of, so i am not planning that right now - batch updating is undoubtedly more efficient

My linux server is in US, my VPS in UK - transfer between the 2 is more costly than local transfer. I have a strong preference to serve data from linux - but i guess that is all detail i don't have to worry about right now

Think its time to look at a live Think cAlgo feed, and Pepper should be available soon - gonna have to try and broker a deal with CNS first - i can't expect to pull down all this data without their support

[ianj]

Not looked at this for quite a while - meanwhile the API has moved on - for the better

I produced TickCaptureIndicatorX.cs this morning - similar to TickCaptureIndicator.cs but it takes a list of symbols, subscribes to L2 for each of them and post updates to a single network stream with all updates interlaced.

This has the advantage of requiring less indicators on a given chart.

TickCaptureIndicatorX.cs can be deployed to any chart - it doesn't have to be one of the symbols requested. It looks like cTrader uses a thread pool so the calling thread for the feed event often changes.

I have performed some tests comparing 25+ TickCaptureIndicator.cs against a single TickCaptureIndicatorX.cs capturing 25+ symbols, and in general TickCaptureIndicator.cs is quicker (less latency) , but i think this is because events are queued for a single threaded invocation of each TickCaptureIndicatorX.cs indicators. In both cases the total number of events captured in a period (say 1 minute) is the same, so there is no LOSS of data

When TickCaptureIndicatorX.cs is used with 1 symbol it is just as fast as TickCaptureIndicator.cs so there is nothing intrisically slower with the X version - its just down to thread pool scheduling in the cTrader/cAlgo engine.

One significant difference is in memory usage. Each TickCaptureIndicator.cs instance takes 5-10Mb memory (so thats 250MB+ for 25+ symbols) whereas a TickCaptureIndicatorX.cs streaming 25+ symbols only uses around 10 MB memory

The upshot of all this is that it looks reasonable to handle 10+ symbols in 1 indicator but it might be more efficient to use a few indicators for 20+ symbols (say 10 symbols per indicator) for a reasonable balance of thread efficiency and memory usage.

To this end i have limited TickCaptureIndicatorX.cs to stream a max number of symbols (currently 30) - the C# code uses some clunky event handling code (my bad) which means copy/paste to increase concurrent support so another reason for limiting the number of symbols supported in each TickCaptureIndicatorX.cs instance

I'll run it for a few days before posting the code to the head of the thread

I am currently running this in cTrader, not in cAlgo - but it should work similarly in either case