How to get historical data?

garyfritz · Post by **garyfritz** » Thu Jun 07, 2012 3:04 pm

I want to get some fairly high-resolution historical data. I think M1 is more than good enough -- I don't think I need tick data.

I've read Birt's guide to downloading Dukascopy data, but I have a heck of a time getting the Dukascopy site to cooperate. Today I'm trying it again for the first time in a while and it's working better than it ever has before, but it's been grinding for an hour and it's only loaded about 6 months of one pair's M1 data.

I figured that's crazy. I've got Tradestation data and I can dump out M1 data in a format that Empty4 will accept -- takes a couple of minutes for all 28 pairs. So I've got all the M1 data.

Now how do I get it into Empty4? I deleted all the history files so I'd start fresh, then I imported the data into one pair. Now I have to get it into the other timeframes. I tried using the "All period generator from 1 minute chart" script I found at FF but that seems to be a bit flaky. I loaded M1 back to 6/6/11, and now I have M5 back to 7/26/11, H1 back to 2/9/12, H4 to 2/16/11 and D1 back to 11/9/04 (!? guess it downloaded those?). So it kinda worked and it only took one step for all TFs for that pair, but I ended up with only 4 months of H1 instead of 1 year.

What is the best and simplest way to get and load up data? Is there any simple way that doesn't require doing every pair (and usually every TF) by hand? Has somebody gone through the headache of downloading and converting data who could make it available for download?

Would it work better to dump the data from Tradestation in M1/M5/M15/etc formats and just import each TF by hand??

Thanks!
Gary

Post by **SWG123** » Thu Jun 07, 2012 8:17 pm

Do you use Forex Tester at all? The basic version includes downloadable M1 history from a single source but you can sign up for the premium service which, as I understand it, allows you to choose tick data from a choice of sources.

I am saying this from memory - I tried to use it for a while but its native language is, I think, delphi-based and I'm no coder.

garyfritz · Post by **garyfritz** » Thu Jun 07, 2012 11:07 pm

No I don't. And getting the data isn't the problem -- I've got TS data massaged into a format that the fussy Empty4 importer will accept.

The question is how to reliably turn that M1 CSV data into good solid M1/M5/M15/M30/H1/H4/D1/W1 data in Empty4. I thought the "All period generator" was going to do that but it doesn't seem reliable. Neither did any of the other methods I've looked at before. And they all require an inordinate amount of repetitive by-hand drudgework.

hedgeitall · Post by **hedgeitall** » Fri Jun 08, 2012 11:52 am

Gary,

as you know downloading the Jforex Dukas data is painstaking. I downloaded eu data last week. From 2007 it took half a day.
But it is worth it. I have and use about 10 different currencies pairs with the data.

I had been using the earlier scripts Birt made, but they dont work on updated versions of Empty4 & you need an active connection to convert the data.

You pretty much have to use his "Tick Data Suit". It works on updated Empty4.
The good thing about this is it is a bit easier to use. It will automaticly place the files in the correct folders for you. And the real spread is working. he wants $110 +++ for it. There is a demo for 7 days.

Is it any good? Hell yes! it gives the most realistic Empty4 backtesting. I have run backtests and compared it to forward trading ea's and the trades match up fairly consistently. I have read alot of ppl disagree with this, but i find it works excellently. Of course it will never be perfect as different brokers have different data. But it gives realistic signals.

garyfritz · Post by **garyfritz** » Fri Jun 08, 2012 3:03 pm

Hm. That's aimed at Empty4 backtesting, which I avoid whenever possible. I don't trust it. I use Tradestation for my serious backtests. Unfortunately TS doesn't handle multi-symbol strategies so I want some ability to backtest in Empty4, possibly with a modified indicator instead of the dodgy System Tester.

So I'm not that interested in getting best-possible tick data into Empty4, not for $100 + ongoing costs. I'd be satisfied with just getting my M1 TS data into Empty4.

hedgeitall · Post by **hedgeitall** » Sat Jun 09, 2012 2:27 am

garyfritz wrote:Hm. That's aimed at Empty4 backtesting, which I avoid whenever possible. I don't trust it. I use Tradestation for my serious backtests. Unfortunately TS doesn't handle multi-symbol strategies so I want some ability to backtest in Empty4, possibly with a modified indicator instead of the dodgy System Tester.

So I'm not that interested in getting best-possible tick data into Empty4, not for $100 + ongoing costs. I'd be satisfied with just getting my M1 TS data into Empty4.

excuse my ignorance, but I dont understand what you want?
You want the ability to backtest in Empty4. But you do not want to use the system tester? You want to use Empty4 because you want to use multi-symbol strategies(which the Empty4 tester does not do)?
Your first line of your first post reads

I want to get some fairly high-resolution historical data

Now you are not interested in "best-possible tick data"

Do you want to test mql4 ea's, with multicurrencies? That would require a new strategy tester? Is such a thing already available?

garyfritz · Post by **garyfritz** » Sat Jun 09, 2012 1:15 pm

"fairly high-resolution" is intentionally not "best-possible." M1 data is fine for what I have in mind, and in fact I could not make use of tick data for what I want to do.

Yes, I want to backtest multi-symbol strategies, with ST cannot do. If you look at my "Simple Shelley Backtester" thread you'll see I've been experimenting with building backtest abilities into **indicators** -- which DOES allow multi-symbol testing.

hedgeitall · Post by **hedgeitall** » Mon Jun 11, 2012 1:23 pm

ok, that makes more sense now.

Have you thought of using Jforex for testing? You can use multiple pairs in the one strategy.
Not many ppl use it, but it is more flexible then Empty4 in alot of ways.

check this out.
http://www.dukascopy.eu/swiss/english/f ... iew=unread

garyfritz · Post by **garyfritz** » Mon Jun 11, 2012 2:22 pm

I looked at Jforex very briefly. If I remember right, you are limited to using the Dukas data, but that's a minor issue. Dukaskopy doesn't accept US clients but I believe you can use their Jforex technology with their partners, such as Alpari-US or FXDD.

A bigger problem is having to re-implement it in Jforex. Given the rate Steve cranks out EAs

it would be impossible for a normal human to keep up with him. Plus it's hard to guarantee that the Jforex implementation actually catches all the behavior nuances of the original. I liked the indicator approach because it meant I could (in theory anyway) turn one of his EAs into a backtesting indicator with just a few minutes' work.

If I wanted to re-implement something, I'd probably do it in Tradestation, which I'm much more familiar with. Khalid can backtest Tradestation code on multi-symbol strategies on MultiCharts.

Post by **NeoTrader** » Sat Jun 23, 2012 4:07 pm

hi gary,

regarding your original question and the amount of data included in your import...did you set your max bars in history to the max (2147483647) Empty4->Options->Charts->Max Bars in History...
and you could also do this with Max bars in Chart. just fill the field with 999999999999999 it will then automatically set it to the max.

only a thought...but i forget it also sometimes..

happy weekend,

NeoTrader

How to get historical data?

How to get historical data?

Re: How to get historical data?

Re: How to get historical data?

Re: How to get historical data?

Re: How to get historical data?

Re: How to get historical data?

Re: How to get historical data?

Re: How to get historical data?

Re: How to get historical data?

Re: How to get historical data?