stevehopwoodforex.com • How to get historical data?

Author:

garyfritz [ Fri Mar 08, 2013 8:02 pm ]

Post subject:

Re: How to get historical data?

Empty4 is REEEEAAAALLLYY fussy about the input data format, especially the date/time stamp. If it doesn't like your format, it fails with no error message to give you a clue what's wrong.

I wrote an indicator to dump data from Tradestation (below) and had a heck of a time getting Empty4 to accept it. I finally got it to accept data with this format:

2012.07.16 15:05,78.84600,78.86500,78.84300,78.86400

I.e. YYYY.MM.DD hh:mm,open,high,low,close

You must use that exact format: "." as date separators, 2-digit month/day/hour/minute, space between date and time, etc. Massage your data into that format and then you should be able to import it with Tools -> History Center -> Import.

Here's the TS indicator I used to dump the data. It dumps the data in proper Empty4 format in the file <SymbolName>.csv in the directory you specify.

It uses the FastFileAppend() function, which if I remember right isn't a standard TS feature. It's in an add-on DLL that I've attached. Place the DLL in your <TS install dir>\Program directory.

Code: Select all

[LegacyColorValue = true]; 

inputs: Dir("C:\temp\TSdata\"), Decpts(5);
vars: Filename("x"), BarTime(0), DDate(0), Append(false);

Once begin
  Filename = Dir & GetSymbolName & ".csv";
  FileDelete(Filename);
  //Condition1 = FastFileAppend(Filename,"Date,Time,Open,High,Low,Close,Volume,OpenInt" & newline);
  Append = true;
  end;

// Empty4 timestamp is at START of bar, TS is at END.  So subtract a minute.
// Category = 12 --> FX, which ends its last bar of the day 1 minute early
DDate = Date;
If (Category = 12 and Time = SessionEndTime(1,1))  
	Then BarTime = Time
	Else begin
		BarTime = Time[1];
		DDate = Date[1];
		end;
If (Append)
	then Condition1 = FastFileAppend(Filename, NumToStr(Year(DDate)+1900,0) & "." & RightStr(NumToStr(Month(DDate)+100,0),2) & "." & RightStr(NumToStr(DayOfMonth(DDate)+100,0),2) & " " 
			& RightStr(NumToStr(intportion(BarTime/100)+10000,0),2) & ":" & RightStr(NumToStr(BarTime+10000,0),2) & "," & NumToStr(Open,Decpts) & "," & NumToStr(High,Decpts) & "," & NumToStr(Low,Decpts) & "," & NumToStr(Close,Decpts)
			& newline);

If LastBarOnChart then begin
	Condition1 = FastFileFlush(Filename);
	Append = false;
	end;

stevehopwoodforex.com https://www.stevehopwoodforex.com/phpBB3/ Print view

How to get historical data? https://www.stevehopwoodforex.com/phpBB3/viewtopic.php?t=611	Page 3 of 3

Author:	garyfritz [ Fri Nov 09, 2012 3:59 pm ]
Post subject:	Re: How to get historical data?
Forward test is better, backtest is for raw qualification only. Forward testing is just "backtesting in realtime." If your backtester is accurate (not a safe assumption with Empty4), forward testing has no advantage over backtesting and lots of disadvantages. (Well, forward testing has one advantage. It accurately reflects bid/ask behavior, while backtesting normally only uses the bid.) Forward testing tells you how one particular parameter set performed during one short window of time. Market behavior changes, and a system that made a fortune during a month of forward testing might have lost a fortune in the month before that. I want to see how the system performed over a long period with many different market behaviors. I would never trust a system based on anything less than a year or two of backtest results.

Author:	ratttyo [ Thu Nov 15, 2012 1:10 pm ]
Post subject:	Re: How to get historical data?
Does anyone know where I can get histoical data for Indices and US crude ?

Author:	johnlucas [ Tue Nov 20, 2012 8:42 am ]
Post subject:	Re: How to get historical data?
giailang wrote:You can try download pages 1. http://www.truefx.com/?page=downloads 2. http://www.histdata.com/ Or use downloader e.g. tools of: 1. http://www.tickstory.com/ 2. http://www.strategyquant.com/tickdatadownloader/ IMO, one should care about the Data Provider, regard the data quality. Not all crims share same provider, each provider produces rates with its specific attributes, e.g, points, gaps, lags. That's not mention of some intentional distortions by spread stretching or modifications by the crims. And those chaos differ from other's hence backtest and optimization settings may vary from one to the other crims. Know your crim, let your ea plays with crim's tricks and cheats, that's best way. Forward test is better, backtest is for raw qualification only. Hi everyone, my 2 cents on backtesting data. I have uploaded M1 EURUSD data from Forextester website and run an EA on them. Showed good profitability over a 10 years period. After that, I have run the same EA with same settings over the Empty4 historical "crims" data of 2 different crims. Guess what? The EA was still profitable, but a lot less. Now...I see that Empty4 historical data is crap, but is there the need to use better historical data if, when running live, the EA will operate over the Empty4 crap data? Or, instead, has the use of a better historical data with Empty4 (example: tick data from Dukascopy) any sense when backtesting? I would happily use Tradestation, but easylanguage is not so easy for me at the moment... Thank you and happy trading to all, John.

Author:	garyfritz [ Sat Dec 01, 2012 6:13 am ]
Post subject:	Re: How to get historical data?
johnlucas wrote:I would happily use Tradestation, but easylanguage is not so easy for me at the moment... John, I'm probably biased because I've used EasyLanguage for 14 years, but... EL is so so SOOO much easier than MQL. I just added a feature to an Empty4 indicator I was writing -- a simple smoothing of my indicator values -- and I fought with the @#^%$!! thing for FOUR HOURS before I finally got it working. Mostly. Sometimes. So I tried writing the same thing in TS. The indicator that had taken several days and 300 lines of code in MQL, took 40 lines of code and about half an hour (to code, test, and verify) in TS, starting from scratch. The smoothing feature that took 4 hours (and many lines of code) in MQL, took one line of code and about 10 seconds in TS. Literally. The TS execution model is vastly simpler than Empty4, the backtester is rock-solid and reliable, the data is automatic (I'd like an M5 GBPNZD chart back to 2003, please. Boom, there it is, possibly after 15-20 sec of downloading if you haven't accessed it before.), the user interface is IMHO much better. You want to backtest an M30 strategy to M1 or even tick resolution? No problem. Unfortunately it doesn't have built-in backtesting for multi-symbol systems, but for single-symbol strategies it's very good. (They've added a portfolio-backtesting feature that they bought from a third party, but it's an add-on for $60/month. It doesn't fit smoothly in with the TS interface, and I think you have to backtest each symbol in isolation. So for example, no EAs that watch a bunch of symbols and then open a basket.) They've added the ability to access multiple symbols, in a way kind of similar to the way Empty4 does it, but you can't backtest with that. If you're serious about backtesting, Empty4 is not the tool to use. There are much better backtesters, but most of them (except Zorro) cost $$. TS is a good solution for most things, it's free for forex, and TS is a pretty darn good crim to boot. That's where I've been doing most of my execution.

Author:	johnlucas [ Sat Dec 01, 2012 4:52 pm ]
Post subject:	Re: How to get historical data?
Hi Gary, thank you for your insights, I am quite convinced that TS is the way to go. I know also of other respectable traders that use TS and found its reliability rock solid. May I ask you how did you start with Easylanguage? Did you read a book? (if yes, would you please advise?). Or used only the free resources on the TS website? A last question (for the time being): as far as you know, is there a 3rd party visual developer tool for writing the most part of an EL program through pre-built blocks? thank you and have a nice weekend, John.

Author:	garyfritz [ Sun Dec 02, 2012 3:00 pm ]
Post subject:	Re: How to get historical data?
It's been 14 years (!) but I don't remember reading any books. I think I just learned from examples. There are plenty of example indicators and systems included with TS. Everything written in EL is open and available for reading/learning. There are books but I don't know if any are any good. I'm not aware of any third-party visual developers for EL, but I think the current TS has some drag-and-drop interfaces to generate EL. I've just never looked at it. BTW TS is NOT the only option, but (other than Zorro which is still in development and has plenty of quirks of its own) it's the only one I know of that's free. NinjaTrader is free as long as you don't use it to place orders, so you could use it to develop your systems and then place orders on Empty4. NT uses a C++ language base, I believe. AmiBroker is supposed to be quite good, supports basket trading, walk-forward optimization, &etc, and it's reasonably priced at $200-280. I've seriously considered getting one or the other of these to work around TS's limitations.

Author:	traderduke [ Thu Mar 07, 2013 7:30 pm ]
Post subject:	Re: How to get historical data?
Gary What is the proceedure to get TS Minute data into Empty4. I also use TS for most of my testing when possible especially for renko! Ray garyfritz wrote:It's been 14 years (!) but I don't remember reading any books. I think I just learned from examples. There are plenty of example indicators and systems included with TS. Everything written in EL is open and available for reading/learning. There are books but I don't know if any are any good. I'm not aware of any third-party visual developers for EL, but I think the current TS has some drag-and-drop interfaces to generate EL. I've just never looked at it. BTW TS is NOT the only option, but (other than Zorro which is still in development and has plenty of quirks of its own) it's the only one I know of that's free. NinjaTrader is free as long as you don't use it to place orders, so you could use it to develop your systems and then place orders on Empty4. NT uses a C++ language base, I believe. AmiBroker is supposed to be quite good, supports basket trading, walk-forward optimization, &etc, and it's reasonably priced at $200-280. I've seriously considered getting one or the other of these to work around TS's limitations.

All times are UTC	Page 3 of 3
Powered by phpBB® Forum Software © phpBB Limited