Thanks for the lead on that website RM, I'll check it out.
Zest, one very very important comment about back-testing - don't forget the back-application - this is absolutely critical and imo is really the proof in the pudding in terms of whether a strategy is over-optimised in back-testing.
I structure my back-testing / back-application as follows:
Back-test sample = minimum 18 month and up to 3 years data
Back-test sample is where I do all my testing of markets & optimisation of variables
Back-application sample = typically 18 months, but most importantly, either 18 months some time before the back-test sample period or 18 months after. I strongly prefer after which means more recent than the back-test sample, because that is likely to be closer to current market conditions than the back-test sample (but of course, there's no guarantee). I do not do any optimisation of variables or testing of various markets on the back-application sample. I test the optimised strategy & portfolio of markets that I selected from the back-testing as a finished package on the back-application sample.
If the outcome of the back-application period is consistent with the outcome of the selected back-test paramaters, then that's validation that the selected back-test parameters are robust enough to withstand changes in market conditions. However if you got really great results in the back-test after optimisation and poor results in the back-application, that's the sign that there has been over (or under) optimisation.
So whether you throw 2 variables into the optimiser or 10 (gadzooks!) ultmately the back-application (and eventually demo and live trading) will tell you whether you have over-optimised or not (apologies rabbiting on if people know all this).
Specifically to your questions:
I believe option 2 would result in a more robust solution than option 1. It will take more thought however into which variables you pair. This goes back to my previous comment in my post to Rick - the variables and their values have to hav real world meaning and in the option 2 scenario I think that's a good way of testing pairs of variables that would be considered natural trade-offs.
And I think you have hit the nail on the head in your last paragraph, one of the fundamentally natural trade-offs in the optimisation process is robustness vs optimisation which comes down to less variables (simpler and less optimised) vs more variables (more complicated and more optimised). In other words, the natural trade-off here is that the simpler strategy will tend to have more similar performance in good and bad market conditions while a more optimised strategy will tend to do really well in good market conditions (that it has been optimised for) and really badly in unfavourable market conditions. I would expect that the simpler strategy will end up having a smoother equity curve than the more optimised one, but the final equity return at whatever point in time will really come down to how much of the time the market is "good" and "bad", which of course can't be predicted, and so the more conservative approach would be to follow the simpler strategy for more moderate but more consitent performance.
Cheers, Sharks
