Curve fitting, and how to recognise it

How it happens

Adjust a threshold, the backtest improves, keep the change. Each individual decision is defensible and the cumulative effect is a system shaped entirely by the noise in one sample of history. No single step is the mistake.

Automated optimisation makes it efficient rather than causing it. Sweeping thousands of parameter combinations guarantees that some will look excellent by chance, and selecting the best is a highly effective way to find the luckiest combination rather than the best rule.

The signals

• ·A large gap between in-sample and out-of-sample performance. Some degradation is normal; a strategy that was outstanding and becomes break-even was fitted.
• ·Parameter instability — the optimal setting jumping around between adjacent test windows.
• ·Results that collapse when a parameter is nudged slightly. A genuine edge has a plateau around it, not a spike.
• ·Profit factor above 2.0 on a short sample. Possible, but assume fitting until demonstrated otherwise.
• ·A large gap between replay trade frequency and live trade frequency. Our DOUBLE-TAP strategy fired 233 times in replay and never once live.

A worked example from our own system

An earlier version scaled the reward-to-risk ratio by market regime — 1.6 in mixed conditions, higher in trends. It improved in-sample results, which is exactly what a fitted parameter does.

Validating against an out-of-sample period showed the scaling was fitting noise. We removed the feature and used a single 2.0 ratio everywhere, and the system got better by getting simpler. That is the shape of a genuine finding: the change that helps is usually a deletion.