stevehopwoodforex.com
https://www.stevehopwoodforex.com/phpBB3/
Print view

Currency strength (like Hanover)
https://www.stevehopwoodforex.com/phpBB3/viewtopic.php?t=503		 Page 4 of 11
Author:  squalou [ Fri May 04, 2012 6:14 pm ]
Post subject:  Re: Currency strength (like Hanover)
Hi Gary,

Been busy these days......

I had a look at your work, and tried the indi.

We should use at least the "SpeedLen" option.
There is no point in using the "naked" mode, i.e. setting SpeedLen to 1 is useless.

It gets veeeery slow when using the "AccelLen" >1 option.
I don't know if the Acceleration option will be of any use though.

I added a "BackBars" option to help speeding up the calculation, we don't need to calc all bars if we are going to invoke it from an EA anyway.
With 100.000 bars on the chart, it would take several MINUTES to display the strengths...
So i limited it to just 1000, which is good enough for recent strength.
10.000 is still acceptable.

I changed some things, the main one being the bar shift used to gather each pair historical data.
We need to be absolutely sure that various pairs data for a given bar "shift" belong to the same point in TIME -- the absolute bar "shift" itself is not important.
If bar 'i' is missing in some pairs, Empty4 will return the "wrong" bar data for that pair compared to the chart's pair;
In order to work around that, iBarShift() shall be used.

Example:

instead of writing
AUDUSD = iClose("AUDUSD",0,i);

one should use the following form:
AUDUSD = iClose("AUDUSD",0,iBarShift("AUDUSD",0,iTime(NULL,0,i)));
to ensure that the bar corresponding to the same point in TIME is returned.

I made that change for all pairs in use.


However, i have not yet found a "decent" fix for the "repainting" issue.
The indicator slowly deviates from its "true" values after a few bars, due to the not shifting the arrays on each new bar, which corrupts the MA calculation.
If you let the indicator run on its own for a few bars, and then force a "reload", it shows its "true" values which are different from what it was showing after the reload. -- try on M1 it is very quick, just after 4 or 5 minutes you can already notice the difference.

The simple but inefficient fix would be to recalc all bars at each new bar (no need to recalc at each tick), but it is very heavy.

The better solution would be to shift all arrays by 1 index at each new bar.
Not very efficient, but already much lighter than recalc-ing the whole thing...

Sq
Author:  garyfritz [ Fri May 04, 2012 8:05 pm ]
Post subject:  Re: Currency strength (like Hanover)
Great stuff! Thanks sq!

Was there a reason you removed my "Pink" switch to turn off individual currencies?
squalou wrote:We should use at least the "SpeedLen" option.
There is no point in using the "naked" mode, i.e. setting SpeedLen to 1 is useless.
I agree. Each currency has a different "native" level. I'm not sure how you would use the raw index values, other than looking at the rate of change of the raw index -- which is what the "Speed" does.
It gets veeeery slow when using the "AccelLen" >1 option.
I don't know if the Acceleration option will be of any use though.
I'm not sure yet myself. I observed that the slope of the Speed-smoothed index seemed to match up pretty well with pairs going up/down. That's OK for visual confirmation, but it's not as good for programmatically doing something with it. So the intent of the "Accel" calculation is to roughly calculate the slope of the "Speed" line. If Accel > 0, that currency is strengthening.

I thought it would work to e.g. buy a JPY basket when the JPY Accel was > 0. I've done a little testing with that -- very messy and painful to do by hand -- and I'm not sure that's going to work after all.

I'm not sure why Accel should slow things down so dramatically, though. It's doing almost exactly the same calculation as the Speed. It should slow it down about exactly the same as Speed.
I changed some things, the main one being the bar shift used to gather each pair historical data.
OK. I see your point. Thanks for the clarification on that.
However, i have not yet found a "decent" fix for the "repainting" issue.
The indicator slowly deviates from its "true" values after a few bars, due to the not shifting the arrays on each new bar, which corrupts the MA calculation.
OK, makes sense. I will change it to calculate the index only after a bar closes, and I'll shift the arrays when I do that. Messy, but it seems to be required to provide current and correct information for iMAOnArray().

Thanks sq!
Gary


PS: QUESTION: I have been thinking about the general problem of repainting (NOT for this specific indicator, with its array-shifting issues). If I understand it correctly, repainting normally happens because an indicator "cheats" and peeks into the future by looking at following bars. When you're calculating bar 0, there is no future to peek into, so you calculate the "correct" value. Then when that bar shifts to 1, 2, etc, Empty4 sometimes returns a value of 2 or 3 in IndicatorCounted(), and the indicator recalcs and redisplays the last 2-3 bars. But that means there is future data available and the indicator can "peek," resulting in different values which replace and repaint over the original correct value.

Do I have that correct?

If so, couldn't you avoid repainting by never accepting IndicatorCounted() values > 1? (You'd have to accept large values of IndicatorCounted(), since that would indicate the entire chart is being replotted.) If you only recalc and redisplay bar 1 -- and that one only once, when it first closes -- then you would never redisplay any historic bar. You would leave the original value on the chart, without repainting. Would that work? It seems too easy, so I'm guessing there's some bizarre Empty4 behavior that breaks it...
Author:  squalou [ Fri May 04, 2012 8:52 pm ]
Post subject:  Re: Currency strength (like Hanover)
Was there a reason you removed my "Pink" switch to turn off individual currencies?
Wouldn't setting the color to "NONE" be enough to "remove" the currency ?
I'm not sure yet myself. I observed that the slope of the Speed-smoothed index seemed to match up pretty well with pairs going up/down. That's OK for visual confirmation, but it's not as good for programmatically doing something with it. So the intent of the "Accel" calculation is to roughly calculate the slope of the "Speed" line. If Accel > 0, that currency is strengthening.

I thought it would work to e.g. buy a JPY basket when the JPY Accel was > 0. I've done a little testing with that -- very messy and painful to do by hand -- and I'm not sure that's going to work after all.
we can keep it to try it out and see if it is any useful.
I'm not sure why Accel should slow things down so dramatically, though. It's doing almost exactly the same calculation as the Speed. It should slow it down about exactly the same as Speed.
that's right, it adds about the same time.
With 100.000 bars, it just seemed like an eternity :)

I tried using MODE_SMA instead of MODE_EMA, and that seems to be twice faster already...
without changing the results very much.

OK, makes sense. I will change it to calculate the index only after a bar closes, and I'll shift the arrays when I do that. Messy, but it seems to be required to provide current and correct information for iMAOnArray().
way to go!
PS: QUESTION: I have been thinking about the general problem of repainting (NOT for this specific indicator, with its array-shifting issues). If I understand it correctly, repainting normally happens because an indicator "cheats" and peeks into the future by looking at following bars. When you're calculating bar 0, there is no future to peek into, so you calculate the "correct" value. Then when that bar shifts to 1, 2, etc, Empty4 sometimes returns a value of 2 or 3 in IndicatorCounted(), and the indicator recalcs and redisplays the last 2-3 bars. But that means there is future data available and the indicator can "peek," resulting in different values which replace and repaint over the original correct value.

Do I have that correct?
that's about right i think.
TMA, Snake, SolarWind, and similar, are typical repainters that use "future" bars to calc the "current" bar.
Other indicators recalc all or part of the past bars at each tick, and can therefore repaint eventhough they don't really use "future" bars.
i-regr is one of them (a polynomial regression curve).

If so, couldn't you avoid repainting by never accepting IndicatorCounted() values > 1? (You'd have to accept large values of IndicatorCounted(), since that would indicate the entire chart is being replotted.) If you only recalc and redisplay bar 1 -- and that one only once, when it first closes -- then you would never redisplay any historic bar. You would leave the original value on the chart, without repainting. Would that work? It seems too easy, so I'm guessing there's some bizarre Empty4 behavior that breaks it...
I guess this could do it.
But remains the problem of disconnection or platform shutdown, or laptop suspension, which will probably give you IndicatorCounted() values > 1, and therefore would produce gaps in the display if you only calc bar[1].


Sq
Author:  garyfritz [ Fri May 04, 2012 9:36 pm ]
Post subject:  Re: Currency strength (like Hanover)
squalou wrote:Wouldn't setting the color to "NONE" be enough to "remove" the currency ?
Huh. I thought I tested that and NONE was equivalent to Black. Whoops. :oops:
I tried using MODE_SMA instead of MODE_EMA, and that seems to be twice faster already...
without changing the results very much.
Hm. On all other platforms I've used, which go directly from "first bar" to "last bar" without any "let's display a few bars out of order" behavior, EMA is always much faster. You carry the results forward from one bar to the next, and it's a simple/quick calculation to update the EMA to the current bar -- whereas the SMA has to loop through adding up N bars, etc. But who knows what Empty4 is doing internally...

I prefer EMA over SMA anyway. EMA weights recent values more heavily, which is what you'd want. Plus if you have a big spike value, SMA has a big change when that spike drops out of the SMA window, but EMA just gradually decays it into the past.
If so, couldn't you avoid repainting by never accepting IndicatorCounted() values > 1?
But remains the problem of disconnection or platform shutdown, or laptop suspension, which will probably give you IndicatorCounted() values > 1, and therefore would produce gaps in the display if you only calc bar[1].
Hm. True. You could only re-calc bar 0, and then re-calc the entire chart if you got IndicatorCounted() > 1, but that chart-recalc would result in calculating the "repainted" values. Darn.

It's possible to write a MTF indicator (and probably other repainting indicators?) so you calculate each bar's value "correctly," even on history bars. I did it for the first indicator I ever wrote. (I seem to pick interesting challenges for my first indicators. :D) It worked fine on history but I didn't understand the weird Empty4 realtime model so I never got it working for live data. (I didn't realize you couldn't carry values from bar to bar.) That's a pain but it does work properly for history. Could probably be done correctly for realtime too but I'll worry about that later. :) I don't think this indicator should have a problem with values changing and repainting, as long as I shift the arrays properly.
Author:  SteveHopwood [ Fri May 04, 2012 9:55 pm ]
Post subject:  Re: Currency strength (like Hanover)
Hehe. Tell you what guys, reading this make me think I should be a little less hard on the coders of custom indi's. It also confirms my decision to avoid getting involved with coding them. :lol:

Up-an-at-em, tigers. :lol:

:D
Author:  squalou [ Sat May 05, 2012 7:30 am ]
Post subject:  Re: Currency strength (like Hanover)
garyfritz wrote:
I tried using MODE_SMA instead of MODE_EMA, and that seems to be twice faster already...
without changing the results very much.
Hm. On all other platforms I've used, which go directly from "first bar" to "last bar" without any "let's display a few bars out of order" behavior, EMA is always much faster. You carry the results forward from one bar to the next, and it's a simple/quick calculation to update the EMA to the current bar -- whereas the SMA has to loop through adding up N bars, etc. But who knows what Empty4 is doing internally...
that is true when calculating forward on non volatile data, and i suppose (?) that's what Empty4 does in its internal MA calculations.
But iMAOnArray() has to go thru the whole array at every call because it doesn't control its contents.
And in that case SMA is faster than EMA because SMA simply sums-up values and makes 1 final division, whereas EMA does 2 multiplies and 1 sum for each value
( EMA[today] = alpha * close + (1-alpha) * EMA[yesterday] , where alpha = 2/(1+period) ).
I prefer EMA over SMA anyway. EMA weights recent values more heavily, which is what you'd want. Plus if you have a big spike value, SMA has a big change when that spike drops out of the SMA window, but EMA just gradually decays it into the past.
Agreed;
I don't think this indicator should have a problem with values changing and repainting, as long as I shift the arrays properly.
yep!

Sq
Author:  garyfritz [ Sat May 05, 2012 3:22 pm ]
Post subject:  Re: Currency strength (like Hanover)
squalou wrote:But iMAOnArray() has to go thru the whole array at every call because it doesn't control its contents.
And it may be even worse than that. Currently I pass 0 to the "total" parameter, which specifies "the number of items to be counted, 0 means the whole array." To be totally accurate you should do that, since EMA includes the entire history in its value. Whereas SMA only has to look at the last N values. I'll set the "total" param to something like 2 * N so it provides plenty of history to the EMA without running the loop through the entire array. (At least I assume that's what the "total" param does!!)
Author:  garyfritz [ Sat May 05, 2012 9:38 pm ]
Post subject:  Re: Currency strength (like Hanover)
garyfritz wrote:
squalou wrote:Wouldn't setting the color to "NONE" be enough to "remove" the currency ?
Huh. I thought I tested that and NONE was equivalent to Black. Whoops. :oops:
Turns out that's right, sort of. Sometimes. If I set one of the colors to NONE, that line disappears. But then if I replot the indicator for any of a number of reasons, e.g. changing another currency's color, the lines all re-appear, and they're black. So I put back the check to only call SetIndexBuffer() on a currency's buffer if its color != NONE.

Except... when I first set the color to NONE, MQL says it's -1, the currency color is == CLR_NONE, and the NONE currencies don't display. Then if I change one color, all the OTHER "NONE" colors spontaenously change to -16777216 !? Then the lines start displaying again, in black. If I use Pink, it works. :?: :?: So I'm going back to Pink for now and I'll worry about it later.

I realized, though, that if I don't want to display an index, and I don't call SetIndexBuffer(), then Empty4 won't know it's a buffer and won't automagically shift it when Bar 0 closes. But that shouldn't matter because I'm not displaying it anyway, and I don't access it in any other calculations.

sq, when bar 0 closes and becomes bar 1, I need to shift all the arrays I use for MAs -- e.g. AUDindex[], AUDspeed[], AUDsmooth[]. What about the pair arrays? I load those up with e.g. "AUDUSD = iClose("AUDUSD",0,iBarShift("AUDUSD",0,iTime(NULL,0,i)));". So they should line up with the Bar data already. But when bar 0 closes, the Bar data shifts, and AUDUSD[] doesn't. The IndicatorCounted() value is probably only going to have me calling iClose() one one or two bars. So I should shift the AUDUSD[] pair array too, right?

I'll get this coded this weekend -- I think the changes from this point are pretty simple (famous last words?? :D) -- but I can't test it until Sunday night so I won't release anything until at least then.


PS: BTW I was thinking about how this really simple concept has exploded into hundreds of lines of code and lots of CPU-gobbling work to calculate the averages and maintain the history. Then I realized that about 98% of that is just dealing with the strange Empty4 indicator behavior. Calculating the raw index values is very simple.

Maybe, if you wanted to use the currency strength in an EA and NOT in an indicator, you might not have to drag along that extra 98%. Let me ping Steve and ask him exactly how EAs behave.
Author:  SteveHopwood [ Sat May 05, 2012 10:31 pm ]
Post subject:  Re: Currency strength (like Hanover)
garyfritz wrote:Maybe, if you wanted to use the currency strength in an EA and NOT in an indicator, you might not have to drag along that extra 98%. Let me ping Steve and ask him exactly how EAs behave.
Hehe. The more I watch the likes of you, TIG and sq bat this sort of stuff backwards and forwards, the more I realise how limited is my understanding.

Having said that, here is my understanding of how EA's interrogate indis:
  • - declare a variable to hold the result of the call
    - the call consists of the symbol, timeframe, name of the indi, list of external paramaters - the indi call has to include all of those presented by the indi - and two further paramaters. The first is the buffer that contains (sometimes) values calculated by the indi; the last one is the shift relative to the current candle.
So, in this call that reads Bollinger Bands:
BbUpper = iBands(NULL, 0, BbPeriod, BbDeviation, 0, PRICE_OPEN, MODE_UPPER, shift);

BbPeriod and BbDeviation are user choices via extern inputs in the EA.

MODE_UPPER is the buffer I am interrogating here; it contains the value of the BB upper line.

So, when you are creating an indi that you intend to be used by an EA, you need to make sure that the values the indi calculates are stored within a buffer. If you are populating 8 buffers, they start indexing at 0, so my call would read:

Code: Select all

GaryVal0 = iCustom(NULL, timeframe, "Gary's indi", extern's, buffer0, shift);
through to

Code: Select all

GaryVal7 = iCustom(NULL, timeframe, "Gary's indi", extern's, buffer7, shift);
Obviously, it is a help to the ea coder if the list of indi extern's is as short as possible. The more externs there are, the better the chance of something going wrong. Hanover started exporting data created by his RS indi to an excel file because the list of external inputs in the indi is so huge that coding them in an ea is next to impossible.

Does this make sense? The whishkeysh has been good tonight, so I may be drivelling to international standardshsh. hic

:D
Author:  garyfritz [ Sat May 05, 2012 10:57 pm ]
Post subject:  Re: Currency strength (like Hanover)
Nonono, you misunderstood me. I don't want you to interrogate the indi. If you do that, you bring along all the overhead and CPU effort required to recalculate-recalculate-recalculate everything any time it pops up and asks you to redisplay some particular bar.

I'm saying, dump about 98% of the baggage that is only required because of the indicator execution model. I'm starting to understand indicators but I have no idea how EAs execute. **IF** the EA processes the chart history in linear order, from oldest to newest, without any of the semi-random "oh I want you to display bar 3 now" malarkey that indicators do, then an EA can calculate the currency strengths INSIDE THE EA with very little effort. It should take somewhere under 100 lines of code (maybe a dozen lines of code, replicated for each of 8 currencies), and it would be very low-overhead to calculate.

The key question is if the EA goes straight from "oldest" to "newest" without any re-display / re-calculate / random-order stuff.... ??
All times are UTC Page 4 of 11

Powered by phpBB® Forum Software © phpBB Limited