Accuracy (Inaccuracy) of Backtests???

[Silent0ne]

Here is a response I got from IBFX when I emailed them concerning my inability to download historical data. I wanted to share it here since we are placing alot of weight on backtest results.

I did try to call the "direct" (it's direct to IBFX, not direct to the CSR) line below and got the whole automated spiel. Most of the options were live trade related so I figured a low priority would be placed on my call.



"Thank you for contacting IBFX. We appreciate your business. Depending on the type of chart you are trying to run on your back test the information will not be accurate.

Because your information isn't very accurate if you could contact us directly then go ahead and do that and we would be happy to assist you further.

Thank you for contacting IBFX


Ben Sorensen
Client Services Represenative
ben.sorensen@ibfx.com
Direct: 801-930- 6948
Toll Free: 866-468-3739, ext
Fax: 1-212-884-0609



3165 E Millrock Dr, Ste 200
Salt Lake City, UT 84121"

[Earl]

Here is a response I got from IBFX when I emailed them concerning my inability to download historical data. I wanted to share it here since we are placing alot of weight on backtest results.

I did try to call the "direct" (it's direct to IBFX, not direct to the CSR) line below and got the whole automated spiel. Most of the options were live trade related so I figured a low priority would be placed on my call.



"Thank you for contacting IBFX. We appreciate your business. Depending on the type of chart you are trying to run on your back test the information will not be accurate.

Because your information isn't very accurate if you could contact us directly then go ahead and do that and we would be happy to assist you further.

Thank you for contacting IBFX


Ben Sorensen
Client Services Represenative
ben.sorensen@ibfx.com
Direct: 801-930- 6948
Toll Free: 866-468-3739, ext
Fax: 1-212-884-0609



3165 E Millrock Dr, Ste 200
Salt Lake City, UT 84121"

most of 2008 should be pretty good...but as you go back farther there isn't enough info...I back tested all of 2005 2006 2007 with like 100 trades for a year...not enough tick data

[Silent0ne]

OK, makes sense. So how far back are you typically backtesting? I know that September/ October were not typical months as far as market behavior goes (or will that type of volatility become the norm?) so am I better off just forward testing on a demo?

[^eagle^]

OK, makes sense. So how far back are you typically backtesting? I know that September/ October were not typical months as far as market behavior goes (or will that type of volatility become the norm?) so am I better off just forward testing on a demo?


The problem with back testing is that past results are no gurantee of future success. You can backtest the hell out of something to get it to look good but it may fail miserably down the road because all patterns happen pretty much randomly. They are subject to the fundamentals.

the patterns themselves are fairly consistent. Its thier place in time which is random.

example uptrend..doubletop uptrend consolidation downtrend cosoildation doublebottom


or

downtrend uptrend consolidation (long time) uptrend doubletop

this is why FAPs fails and fails badly when it does.


so if we split up the trades to minimize risk and direct the trade direction and peel off the bad trades you would have a super EA.

Back testing is great as a guide to find out if an EA will actually function but forward testing is the only way to figure out the bugs in the system and optimize it long term.

[Earl]

The problem with back testing is that past results are no gurantee of future success. You can backtest the hell out of something to get it to look good but it may fail miserably down the road because all patterns happen pretty much randomly. They are subject to the fundamentals.

the patterns themselves are fairly consistent. Its thier place in time which is random.

example uptrend..doubletop uptrend consolidation downtrend cosoildation doublebottom


or

downtrend uptrend consolidation (long time) uptrend doubletop

this is why FAPs fails and fails badly when it does.


so if we split up the trades to minimize risk and direct the trade direction and peel off the bad trades you would have a super EA.

Back testing is great as a guide to find out if an EA will actually function but forward testing is the only way to figure out the bugs in the system and optimize it long term.Well said!!!!:D
random patterns...lol they could repeat every 5 mins 5 hours 5 days 5 months 5 years.... who knows when it'll happen again....

you just have to work with what you got...and a little lady lucky doesn't hurt

[Silent0ne]

The problem with back testing is that past results are no gurantee of future success. You can backtest the hell out of something to get it to look good but it may fail miserably down the road because all patterns happen pretty much randomly. They are subject to the fundamentals.

the patterns themselves are fairly consistent. Its thier place in time which is random.

example uptrend..doubletop uptrend consolidation downtrend cosoildation doublebottom


or

downtrend uptrend consolidation (long time) uptrend doubletop

this is why FAPs fails and fails badly when it does.


so if we split up the trades to minimize risk and direct the trade direction and peel off the bad trades you would have a super EA.

Back testing is great as a guide to find out if an EA will actually function but forward testing is the only way to figure out the bugs in the system and optimize it long term.



Thanks ^eagle^ for confirming my thoughts and feelings on backtesting (though much more eloquently than I would have put it), I feel a little less dumb every day,:D