How to access Live tick data?

[ericjschroeder]

I see that the Market Watch updates every single Pair individually, the moment a new tick occurs for any Pair. I even see that a Tick Chart is available to view, but how to access that data Live? I also know you can enable DDE to export this to Excel (which is not necessarily what I want to do, unless I can retrieve this data for analysis). I also believe that with an EA or indicator applied to a specific chart, any data it reads for all Pairs will only be updated on each new tick for that specific Pair applied to the chart. What I want to do is to receive and record each tick for every individual Pair (into an(/multiple) array(s)), just like Market watch displays, so I can analyze increases in trading activity, even if just for my own broker. I want this info available in Empty4 so I can display the analysis in my current indicators.

So is it possible to capture and record the incoming tick data accurately for all separate Pairs, can I do this in Empty4, and if not, can I read the exported DDE in Excel back into Empty4 or will I need to learn a new programming language to extract this data as it flows into Excel?

Thanks to anyone who has an answer for this question!

[DigitalCrypto]

So is it possible to capture and record the incoming tick data accurately for all separate Pairs, can I do this in Empty4, and if not, can I read the exported DDE in Excel back into Empty4 or will I need to learn a new programming language to extract this data as it flows into Excel?

Hi Eric,

First let me start by saying that I am responding to this post now to bring closure to this subject. Since no one has answered it they either haven't done it or are already too familiar with the pain that you are about to bring upon yourself and don't want to relive those nightmares.

I will go ahead and save you some headaches from the get go. If you are not sure if you need something like this then you probably don't. If you really think it is something you must absolutely have because it gives you an edge in your trading strategy then continue reading....

First I will start by saying that a lot of Wall St and London firms use kDB for tick data management. It's very expensive and the language was meant to torment you for a couple eternities in hell, revive you, kill you and continue tormenting you more.

I run a tick datastore in house on 2 big 32 core servers with NVMe drives, 128GB of ECC RAM, SSDs and all that jazz connected to 8 data sources (8 separate machines feeding price into it). It works ok. If you want to follow this route you are going to need a server class machine for this project. If you shop around on ebay you can probably find a comparable used out of warranty HP Proliant machine in the 5 to 10k USD range. You will need two of them.

Postgresql is the current tick database that's popular for retailers wanting tick data. It takes about 1GB per pair per day per broker terminal depending more or less on volatility. Higher volatility equals more storage. You will definitely need TimeScaleDB plugin also for this. You should be well versed in SQL, DB management and server management on FreeBSD with ZFS and should have a large UPS (battery backup and generator) to maintain your data integrity. Redundant internet is a must to maintain data integrity. Otherwise what's the point?

If you are thinking something like NoSQL MongoDB, Man Group in the city has successfully created a tick data store across 100s of servers and petabytes of storage and RAM. That's a couple million dollars out of my price range. If you think you can get another column oriented or time series database as a way to get around such large requirements your project will end in tears. Been there. Done that. Got the t-shirt.

As for creating your own tick charts you can write a database/REDIS using something like hiredis client that captures that information and pump that into offline charts or use something like TradingViews API chart library to generate your own charts or you could also use TechanJS. Also keep in mind you are going to need to write your own price feed API or UDF for getting the tick data back out into something else. NodeJS is a popular platform for this. Forget python, it's a joke.

The final way to access live tick data is to pay for it from Quandl or other and have it pumped to you. The last time I checked it was about $7k per month for what I needed and wanted on FX. You will need some kind of message queue like Apache Kafka to handle your routes and storage and you will still need all the heavy industry equipment listed above.

If you are just wanting something simple there are a couple scripts floating around on the net that will capture tick data into a CSV file and build your offline charts for you. But I wouldn't recommend this unless you have a very capable trading station as it will interfere with opening and closing trades on your terminals and you will lose all the data once CrapperQuotes forces an update on you which is usually every week or whenever you close down Empty4 because it's eating up RAM because of the shitty memory leaks.

As for DDE, don't bother. It's a big pain in the ass to get working correctly, doesn't work well on any platform but Windows and is largely undocumented. Instead you should just route ticks from Empty4 to Kafka or write straight to Posgresql which is what I do. You will need another dedicated price feed engine (read other computers) just for this as it consumes a ton of CPU processor and your SSD IOPS will already be pegging from the database I/O. I forgot to mention this is where the second mirror database comes into play to offset the load of the primary write database.

Also not to forget, you are going to need to write all the code for generating time series or tick candles, renkos, range bars etc. That in itself will take you a bit of time to do. Don't foget to automate this task and build tables for all those candles as it will save you CPU load instead of having to pull a query from the DB everytime the chart updates or you switch timeframes.

Now as far as building and trading from the offline charts that you have generated with an EA you would probably be more comfortable having someone pull your lower lip over the back of your head and start punching you relentlessly in your face. Best of luck getting that working. It took me a month.

If all of this sounds a bit daunting to you then it's because it is. If you still feel you need near real time tick data charts, try using another clunky platform like Ninja Trader. From there you could probably write a trading bridge to execute trades across to Empty4. I don't know and will never learn .Net but at least they off a limited tick series chart for whatever you may be curious about.

I was able to do this entire setup on a shoestring budget because this is the type of work that I used to do in FinTech for trading companies so I have a ton of experience that I've gathered since 1997. Most people will not be able to replicate this kind of setup by themselves or with even a handful of people that are not dedicated to the technologies that need to be employed.

However if you have any other questions about creating and storing tick data to run tests against then please first consider that I will in no way provide any further consultation on any project like this unless you've got some really really deep pockets and have paid me well in advance.

I hope this is enough to get you started should you choose to pursue it.

Best of luck to you,
David

[ericjschroeder]

Thanks David. Appreciate the info. I'm not planning to go that complicated, all I really want is tick data from each of the 28 standard pairs, on my single broker is enough, so I can see if the activity I see on my relative strength is matched by order activity, and recent price movement. I found someone else who confirmed the Empty4 documentation, that it will not process additional tick data until it is done with the current tick it's working on, so you can't just get that info from 1 chart. He has 28 individual charts open with an EA on each to collect the tick data from each pair and that would probably work for me, but I'm sure I need a better internet connection and possibly a newer computer.

In the meantime, I think I'll return to studying NB's Holy Grail and the other things Steve mentioned in his last message to members, to see if any of those fit into my current time schedule, which is almost too busy to trade, which is my main problem. Thanks again.

[DigitalCrypto]

No problem Eric. I wasn't trying to be sarcastic or anything like that. It's a LOT of work and takes many hours to get it right. Handling tick data is a big pain in the butt.

Fortunately for you and your schedule Bob has given you a big gift with CSS and HGI.

The more simple you make your trading the easier it will be to succeed at doing this.

-David

[tomele]

Hi.

Part of your problem might be solved at freetickdata.com

Cheers