Data Dredging — search long enough, and you'll find something

Dr. R. Düsing · Osnabrück University
Data Dredging & the Spotlight Paradox — Help
What's this about

Anyone who searches a large dataset for significant relationships ("data dredging," "p-hacking," "fishing expedition") is bound to find some — even if no true effect exists at all. Every test at α = 5% has a 5% chance of a false alarm. With many tests, at least one hit becomes almost certain.

Multiple comparisons

For m independent tests under the null hypothesis, the probability of at least one false alarm is:

P(≥ 1 false positive) = 1 − (1 − α)^m

At α = .05 and m = 20, that's already 64%. On average, α·m false alarms are expected. Bonferroni corrects for this by tightening the threshold: α' = α/m. HARKing (Hypothesizing After Results are Known): a plausible-sounding theory is invented after the fact for a random hit.

Stepwise regression (Freedman's paradox)

Regressing an outcome on many pure noise predictors and keeping the "significant" ones yields a model with an impressive R² — one that collapses on new data. The selection process adapted to random patterns (Freedman 1983). This mode shows R² in the training set vs. in fresh data.

What to do?

Set hypotheses in advance (preregistration), correct for multiple testing (Bonferroni, FDR/Benjamini-Hochberg), separate exploratory from confirmatory analyses, and replicate findings on independent data. "Significant" is no substitute for "replicated."

References

Freedman, D. A. (1983). A note on screening regression equations. The American Statistician, 37(2), 152–155. · Simmons, Nelson & Simonsohn (2011). False-positive psychology. Psychological Science, 22(11). · Gelman & Loken (2014). The garden of forking paths.

📋 The Fishing Expedition
You're looking for correlates of life satisfaction and test dozens of variables (chocolate, zodiac sign, shoe size …) — all in truth independent. With enough tests, some light up red: "significant." Question: are these discoveries? Turn the m dial and see what the formula says.
① Test Tally
Tests run
—
Significant (p<α)
—
"random hits"
Expected by chance
—
α · m
P(≥1 false alarm)
—
1−(1−α)m
—
② All Tests (red = "significant")
③ HARKing — Turning Noise into a Story
📋 Stepwise Regression on Pure Noise
One outcome and k noise predictors — none has a real effect. Stepwise selection still keeps the ones that randomly look "significant" in the training set; together they yield a model with an impressive R². The litmus test: apply the same predictors to a fresh sample — does anything survive? Guess first.
① Model Fit: Training Set  vs.  New Data
Training set
(selection happened here)
—
Fresh sample
(same predictors, new data)
—
—
② Every "Selected" Predictor — and What Survives
Concepts
Multiple comparisons
Every test at α=.05 has a 5% false-alarm chance. For m tests: P(≥1)=1−(1−α)m. Already over 50% at m=14. "Search long enough, and you'll find something."
HARKing
Hypothesizing After the Results are Known: first the random result, then the "theory" to explain it. Sounds plausible but is circular — the hypothesis was born from exactly the data it's supposed to support.
Freedman's paradox
Stepwise variable selection on noise yields "significant" predictors and a high R² — which vanishes on new data. Overfitting to random patterns (Freedman 1983).
Remedies
Preregister, correct for multiple testing (Bonferroni/FDR), keep exploratory ≠ confirmatory, and replicate on independent data. The same trap lurks in modification indices in structural equation models (SEM), when a model is adjusted post hoc, data-driven rather than theory-driven. → Lindley's Paradox