GLOSSARY · STATISTICS
What is statistical significance?
Statistical significance is the judgement that an observed result is unlikely to have arisen by chance alone, given the sample size and the size of the effect.
What it means
In trading the question is whether a strategy has an edge or merely had a good run. With enough strategies tested against enough data, some will look excellent purely by chance — this is not a hypothetical, it is arithmetic, and it is why testing many ideas requires more evidence per idea, not less.
Why it matters
The standard defence is to hold data back. Fit and refine on one period, then test once on a period never used during development. A result that survives that is not proof, but it is dramatically better evidence than a result that has never left the data it was born in.
What this changes in practice
We ran exactly this check. The shipping configuration was developed on January to July 2026 and then tested on August to December 2025, which it had never seen. It returned +$154 and was positive in three months of five — profitable, but not the "green every month" the in-sample period suggested. That gap is what out-of-sample testing exists to reveal.
Related terms
- Out-of-sample testingOut-of-sample testing evaluates a strategy on data deliberately withheld during development, to …
- OverfittingOverfitting is tuning a strategy so closely to historical data that it captures the noise of tha…
- Sample sizeSample size is the number of trades a performance figure is computed from, and it determines how…
- Walk-forward analysisWalk-forward analysis repeatedly fits a strategy on a rolling window of data and tests it on the…
Educational information only, not financial advice. Trading leveraged products carries substantial risk of loss. Last updated 2026-08-11.