What is out-of-sample testing?

What it means

The procedure is to split the available history, develop and refine on one part, then test once on the other. Testing on the held-out data more than once destroys its value — each look leaks information, and after a few iterations the out-of-sample period has quietly become part of the training set.

Why it matters

A large gap between in-sample and out-of-sample performance is the signature of overfitting. Some degradation is normal and expected; a strategy that was outstanding in-sample and merely breaks even out-of-sample was fitted, not discovered.

What this changes in practice

Our shipping configuration was developed on January to July 2026 and then tested on August to December 2025, unseen. It returned +$154 and was positive in three months of five — so the profit generalised, but "green every month" did not. That distinction is recorded in the strategy documentation rather than being quietly dropped.