How do you assess your strategy's success?

[rebeljedi]

Here I would like to gather our experts' thoughts on how we assess the success of our strategies - be it back testing or forward testing.

In Empty4 Strategy Tester, some of us look at its various components to determine our strategies' success and ability to withstand losses.
I look at the following:
1. %Modelling quality
2. %Profit trades
3. %Loss trades
4. Average profit & loss trades
5. Maximum consecutive wins & losses
6. Equity Graph

What about you?

[rebeljedi]

What do you think about the use of Expectancy formula for your strategies?

Expectancy = (W% x Ave W) - (L% x Ave L)
where:
Percentage of winning trades - W%.
Percentage of losing trades - L%.
Average gain on a winning trade - Ave W.
Average loss of a losing trade - Ave L.

http://www.tradingmarkets.com/.site/for ... -80805.cfm

[rebeljedi]

Then, there is myfxbook - which statistics do you use to make your decision whether your strategy works?
Equity Growth Graph
Trades
Profitability
Avg Win & Loss
Avg Trade Length
Z-Score

[garyfritz]

I don't trust Empty4's Strategy Tester. For many/most systems it's unreliable and often wildly optimistic. I consider its results close to worthless. (IMHO that's why there is such an emphasis on forward testing in the Empty4 community. It's the only thing you can trust with Empty4. I've never seen this in any other trading-platform community.)

So unfortunately I don't think there IS any good way to test systems in Empty4. The backtester is unreliable, and forward testing is insufficient. I'm not going to spend a month forward-testing something just to find out how it performed in that particular month with those particular settings. I want to see how it performed over YEARS, including major events like the 2008 financial crunch. And I want to be able to test and adjust the system so it can respond to past history better. Forward testing can't tell you that. You need a good and reliable backtester.

When I backtest with other platforms (such as Tradestation or Wealth-Lab) I pay attention to a number of things:
* Sharpe ratio -- high Sharpes tend to equate to "good profits with not many ulcers."
* Visual evaluation of the equity curve (basically a discretionary "Sharpe ratio") -- does this look like an equity curve I could be comfortable trading?
* See where the system stumbled, and see if I can figure out why.
* Average trade size -- I don't want to trade something with such a small average trade that a small shift in slippage or costs, or a small shift in market behavior, could turn it into a loser.
* Profitability -- no sense wasting time on it if it only makes $10 a year. Does it make enough profit to make it worth the effort?
* Win%, but mostly just because I don't like trading really low-% systems

One of the most crucial things I look at is Out-Of-Sample performance (OOS). It doesn't do a bit of good to optimize an amazing-looking system in your backtest (the In-Sample test), if it blows up as soon as you give it market data it hasn't seen before (the OOS test). I'll test and tune it on a long period of IS data, and then backtest it on a separate OOS period. If the OOS results are good -- say, no more than a 50% dropoff from the IS optimization period -- that indicates the system holds up well on unseen data. Which is critical, since that's what you trade.

The Gold Standard for backtesting is Walk-Forward Analysis: tune on IS period 1, trade forward on OOS period 1. Move the IS period forward in time and tune on IS 2, trade forward on OOS 2. Repeat to cover your whole testing history. Then stitch together the OOS periods to see what the OOS behavior would have been throughout your whole history. WFA closely simulates the way I would actually trade the system. If it looks good in a long WFA analysis, I'm very comfortable that the system should continue to hold up as I trade it in realtime.

Expectancy: this is a valuable and useful statistic, but realize it's just the average trade size. You get the same result with TotalNetProfit / #trades. You definitely want it to be positive, but it doesn't tell you anything about how well or how consistently the system trades. You could have a system that had 1 winner for $1000 and 500 losers for $1 each, and it would show an Expectancy of $1. According to that, you should "Expect" to earn $1 per trade, on average. But with this system, you might have to take 500 losers before you hit a win! Sharpe ratio and profitability tell you a LOT more than the Expectancy does.

I don't use myfxbook, but most of its stats are similar to what you'll get from any good backtester.

[jb68]

I agree Empty4 is crap. I'e been teach about 1 year about how to use unreliable system in order to make more reliable systems. If you think in measurements history we started with 10-20% error to achive systems with 0.000001 error.

First thing about Empty4 is to know its weakness. So far I discovered it use current market variables for back-test. That means we have to do a back-test on the same time frame as the strategy.

Other important weakness is speed. At start() will use current ASK but at the time of placing order is possible the ASK to be different. In a 2 stage ECN this become very annoying. So if in back-test you will get always the exact amount while in real life you will get 1-5% variations for order open price, SL, TP.

I'm trying to get at least 1000 trades including 2007 - 2008.
Aside from W%, L%, Ave W, Ave L then I'm looking at Profit/DD report and equity curve because this is related with Trader's happiness.

I always do tests using a fixed amount, and look for a close to linear equity curve grow. If everything is okay I do test with geometrical grow to see how the equity graph is coming along.
I'm not a fan of Martingale (including its variants) where the bets are based on hoping the outcome will go in your favor.