Bob and Jem and Shelley's Pivot Basket auto-trader

[DING]

Please help,why my all pair that show "waiting for a pivot cross" all week?

[garyfritz]

My BJS is hanging when I try to apply it, but... have you looked at the charts?? Most pairs have been trending hard, and they haven't been crossing the pivot. BJS waits for the market to pull back beyond the daily pivot, then come back. That hasn't happened all week for most pairs.

Steve or anybody, any idea why it's hanging? I ran 1j all week with no problems. Somehow in the last day or two it got removed from my chart -- not sure how/why that happened. When I add it back -- either 1j or 1k -- it hangs Empty4.

The first time I applied 1j, it showed a message "Checking for enough bars on AUDCAD" or similar, and then hung. Since then it hangs without any identifying message as soon as I launch Empty4. I had to delete the expert from the profiles\default\*.chr file in order to rescue the Empty4.

EDIT: No wait, after I removed it from the *.chr file, then I was able to re-apply 1k and it doesn't hang. Weird.

EDIT2: And after running a while, I notice the terminal.exe has started fully burning one core of my CPU. Empty4 isn't hung but it's spinning somewhere. Something's odd...

[SteveHopwood]

EDIT2: And after running a while, I notice the terminal.exe has started fully burning one core of my CPU. Empty4 isn't hung but it's spinning somewhere. Something's odd...

Do you think it might be the endless loop thingy? A tick sends the bot into the loop, then another tick comes so I assume this sends the bot into a fresh loop whilst leaving the other one unclosed.

Is it possible that this could cause memory problems? These would not show up during the rapid development phase because we are always updating the bot, which requires an ea restart. Could it be that beig left unattended for long periods causes memory corruption?

[garyfritz]

No, this happened instantly after restarting Empty4. Multiple times.

My understanding of the start() behavior is that Empty4 calls start() on one tick, and does not return until it's finished processing that tick. If another tick comes in during the first start()'s execution, it doesn't "stack" on top of the first start() -- the second tick just gets unceremoniously dropped.

If I stop EAs or remove BJS, the CPU usage drops to near zero. Apply the EA again and Empty4 (slowly, over a minute or so) climbs back up to 100% of one core. Well, wait. This time when I tried it, it got up to 9% (this is an 8-core box so 12.5% is 100% of one core) and has since dropped down to 4-5%.

The strangest thing is that I didn't notice it doing this earlier this week. The only change I can think of is that a process for my printer occasionally goes berserk and gobbles up several GB of RAM. I noticed this morning it had done so, so I killed it. So possibly it did that overnight, and after having "enough" RAM all week, Empty4 suddenly found itself in a squeezed environment. But I killed the rogue process and freed up tons of RAM and BJS is still munching a bit more CPU than it should. Not hanging Empty4 like it was before, but...

*shrug* Beats me. I doubt we'll figure it out discussing it here. If I have any brilliant insights I'll let you know.

[garyfritz]

I just noticed something else. The markets are closed now, but I have EndlessLoop = true so it's still going through the loop twice a second. Empty4 is still burning one core.

The other interesting thing is memory use. Task Manager says it's using 212 MB of RAM, vs. 90MB for another Empty4 I have running. More interestingly, it's getting about 50,000 page faults per second!!

Page Faults mean it's trying to access some memory location that isn't in RAM right now. Normally Windows will keep frequently-accessed stuff in RAM, and "swap out" to disk stuff that isn't used often. If you try to access something that's been swapped out to disk, it gets a Page Fault, and Windows swaps it back in to RAM.

Usually the PF count for a process stays fairly low, unless you're really short on memory. Then Windows busily swaps stuff in & out all the time, since you're trying to access more stuff than you have room for in memory so something's always getting shuffled off to the disk.

But I have 8GB RAM in this laptop, and currently at least 2GB of it is unused. I can't imagine why Empty4 would be swapping constantly like this. Nothing else is -- Empty4 is swapping 10x more than everything else combined. I especially can't imagine why turning off EAs would cause the CPU use and swapping to stop instantly. (Turning off EndlessLoop also stops the thrashing, since the market is closed and no ticks are coming in.)

Unfortunately I can't test to see if any other EAs do this, since other EAs need ticks to do anything. I don't know of any other EAs that use EndlessLoop.

BTW I was running BJS on an Cowboy IBFX Empty4. I just stopped it and started it on an MBTrading Empty4. That terminal.exe instantly doubled its RAM usage, increasing its memory space by 80MB. So far, though, it's not doing the swapping or the CPU usage, even though EndlessLoop = true. !?!?

I tried exiting the Cowboy IBFX Empty4, restarting it, and running the Baluda EA on it. Almost no RAM increase. Run BJS, and wham! Instantly the memory use goes up 60MB. Something fishy is going on with BJS.

Computer folks, can you guess why Empty4 would be thrashing with 2GB free RAM, and only when BJS is running?? I can't find any obvious array-bounds errors or anything like that...

[NeoTrader]

hi Gary,

for your ticker Problem on weekends...have you tried mt4Ticker ?

This is a neat little tool that simulates ticks and is very useful on the weekends..

happy weekend,

NeoTrader

-

[mobthehop]

I use this little script....

Nice weekend

Mop

[SteveHopwood]

I just noticed something else. The markets are closed now, but I have EndlessLoop = true so it's still going through the loop twice a second. Empty4 is still burning one core.

The other interesting thing is memory use. Task Manager says it's using 212 MB of RAM, vs. 90MB for another Empty4 I have running. More interestingly, it's getting about 50,000 page faults per second!!

Page Faults mean it's trying to access some memory location that isn't in RAM right now. Normally Windows will keep frequently-accessed stuff in RAM, and "swap out" to disk stuff that isn't used often. If you try to access something that's been swapped out to disk, it gets a Page Fault, and Windows swaps it back in to RAM.

Usually the PF count for a process stays fairly low, unless you're really short on memory. Then Windows busily swaps stuff in & out all the time, since you're trying to access more stuff than you have room for in memory so something's always getting shuffled off to the disk.

But I have 8GB RAM in this laptop, and currently at least 2GB of it is unused. I can't imagine why Empty4 would be swapping constantly like this. Nothing else is -- Empty4 is swapping 10x more than everything else combined. I especially can't imagine why turning off EAs would cause the CPU use and swapping to stop instantly. (Turning off EndlessLoop also stops the thrashing, since the market is closed and no ticks are coming in.)

Unfortunately I can't test to see if any other EAs do this, since other EAs need ticks to do anything. I don't know of any other EAs that use EndlessLoop.

BTW I was running BJS on an Cowboy IBFX Empty4. I just stopped it and started it on an MBTrading Empty4. That terminal.exe instantly doubled its RAM usage, increasing its memory space by 80MB. So far, though, it's not doing the swapping or the CPU usage, even though EndlessLoop = true. !?!?

I tried exiting the Cowboy IBFX Empty4, restarting it, and running the Baluda EA on it. Almost no RAM increase. Run BJS, and wham! Instantly the memory use goes up 60MB. Something fishy is going on with BJS.

Computer folks, can you guess why Empty4 would be thrashing with 2GB free RAM, and only when BJS is running?? I can't find any obvious array-bounds errors or anything like that...

All this gives me a couple of ideas.

I can add code that will do the endless loop thingy only if there is a basket of open trades; we only need the endless loop because of the speed with which a basket upl changes.

Second idea is to turn off the endless loop and use one of the fake tick senders to generate regular ticks.

[Lifesys]

We may have stack issues with numerous embedded loops here. Normally there would be one time loop eg.
if (BarsTime != Time[0])
{
BarsTime = Time[0];
etc. to fire on each new bar, but we have -
while (x == 0)
{
..
} // loop
with embedded
if (OldM1BarTime != iTime(NULL, PERIOD_M1, 0) )
{
OldM1BarTime = iTime(NULL, PERIOD_M1, 0);
ReadTable();
}
Not sure why this is necessary since we want to check on closed M1 candle for pivot cross not transients, so why not use this M1 candle change as encompassing loop. What l I am saying is I would not do it this way with embedded loops which can create memory stack problems. Other applications fire OK with a single new candle loop.
Just some thoughts!
Paul

[garyfritz]

Hm, Paul, I don't see a likely stack problem there. It's just a different way of getting periodic updates. Do you see a specific issue that might cause stack growth?

Steve, the "only endless-loop while basket is open" is a good idea -- just make sure you still get into start() occasionally so you can OPEN a new basket.

The endless-loop isn't the problem, though, it's just exposing the problem. What on earth in start() or below is causing an instant 60-80MB memory load? Why was it getting constant page faults? (And now it's not??) Very odd...