“A non-significant result proves there is no effect.”
Not quite.It may reflect a small effect, noisy data, or a test with insufficient power.
Research & Systems
Validation Theory: Backtest Power and Multiple Testing
A convincing backtest needs power against realistic alternatives and control of data-mining false discoveries.
Mental model
A backtest must distinguish genuine predictive evidence from patterns found by repeated searching.
Backtest power is the probability of detecting a true effect; low power makes failures inconclusive.
Multiple testing without correction inflates the chance of at least one ‘significant’ but spurious result.
Explore the mechanism
Would the backtest detect a realistic failure, and how many alternative ideas were searched before the reported result was selected?
Backtests have power and a multiple-testing problem. With many candidate strategies, a few will look “significant” by luck. Adjusting for this reduces false discoveries but also makes it harder for true signals to pass.
Numbers
Target family-wise error ≈ 5.0% (via Bonferroni)
FWE (nominal α) ≈ 92.3%
FWE (Bonferroni α_B) ≈ 4.9%
True signals: 3, detected (nominal / Bonf) ≈ 3 / 1
Noise strategies flagged (nominal / Bonf) ≈ 2 / 0
Interpretation
As N grows, a fixed α cuts more and more of the noise strategies below the blue line: the family-wise chance of at least one false discovery climbs towards 100%. Bonferroni shrinks α to keep that error near the 5% target, but some genuine signals (green) now fall above the threshold and are missed.
Robust validation means combining multiplicity control with other defences: out-of-sample tests, hold-out periods, economic priors and independent replication. The goal is not zero false discoveries, but a transparent and controlled trade-off between power and reliability.
CONSEQUENCE PROBE
How does repeated testing change the evidence?
one hypothesis→chosen threshold→observed result→bounded error rate
Interpretable evidencePower and false-positive risk can be discussed against a test fixed in advance.
UNDERSTAND
What to notice
Robust validation combines pre-specification, out-of-sample testing, and multiple-testing control procedures.
Where it appears
- validating trading strategies
- testing risk-model exceptions
- controlling research-selection bias
Where the intuition breaks
What this view leaves out
- Corrections for multiple testing reduce false discoveries but cannot repair leakage, nonstationarity, poor hypotheses, or an unrepresentative sample.
Common mistakes
Where this concept sits
Follow the strongest prerequisites, applications, extensions, and contrasts.
Connections for Validation Theory: Backtest Power and Multiple Testing
Validation Theory: Backtest Power and Multiple TestingYOU ARE HERE
Foundation ofModel Calibration and ValidationValidation theory provides statistical discipline for assessing whether calibrated models actually perform out of sample.
SupportsMonte Carlo Model ValidationMonte Carlo validation relies on statistical reasoning about uncertainty, confidence and the power of diagnostics.
ValidatesValue at Risk (VaR)Risk models such as VaR require backtesting, and test power matters when interpreting apparent model adequacy.
Foundation of · Model Calibration and Validation
Validation theory provides statistical discipline for assessing whether calibrated models actually perform out of sample.
Explore concept →