Law of Small Numbers — why the "record towns" are tiny

Dr. R. Düsing · Osnabrück University
Law of Small Numbers — Help
What's this about

A rate (here: proportion of left-handers) is exactly the same in every town — say, 10%. But the measured proportion fluctuates randomly from town to town. The key question: which towns have the highest and lowest rate? Answer: almost always the small ones — at both ends.

The mechanism

The standard error of a proportion shrinks with the square root of sample size:

SE = √( p·(1−p) / n )

Small towns (small n) have a large SE → their measured values scatter widely around the true rate. Large towns sit almost exactly at p. That's why both the top performers and the bottom performers come from small towns — not because anything special happens there, but because small samples fluctuate more.

The funnel plot

Rate (y) against population (x, logarithmic). The "funnel" is the ±2-SE boundary: wide on the left (small towns), narrow on the right (large towns). The points fill the funnel — the most extreme ones sit far to the left. This exact shape is called a funnel plot and is also used to detect publication bias.

Famous example (Kahneman)

In US data, the counties with the lowest kidney cancer rate have one thing in common: they're rural and sparsely populated. Startling — until you see that the counties with the highest rate are just as small. Pure sampling variation, no rural-living effect.

What to do?

Rankings of rates that ignore n are misleading. Solution: a funnel plot instead of a bare ranking, shrinkage (pulling small units' values toward the overall mean, → regression to the mean), or Bayesian/multilevel models with partial pooling.

References

Tversky, A. & Kahneman, D. (1971). Belief in the law of small numbers. Psychological Bulletin, 76(2), 105–110. · Kahneman, D. (2011). Thinking, Fast and Slow, Ch. 10.

📋 Headline from the Statistics Office
"Town of Podunk Hollow has the highest share of left-handers in the country!" Sounds like a discovery. Except the proportion of left-handers is the same everywhere (~10%, pure chance per person). Question first: are the record towns (highest and lowest rate) more likely to be large or small towns? Check the funnel plot and the ranking.
① Metrics
True rate (everywhere)
—
Highest measured rate
—
—
Lowest measured rate
—
—
Mean pop. of the 10 extremes
—
vs. overall mean —
—
② Funnel Plot — Rate vs. Population
each point = one town · x = population (log) · y = measured rate · funnel = ±2 SE around the true rate
On the left (small towns), the funnel is wide — that's where the extremes sit. On the right (large towns), everything hugs the true rate.
③ Ranking — Who Stands Out?
🔺 Highest rate
🔻 Lowest rate
🏙 Largest towns
Concepts
Law of small numbers
We expect small samples to be as "well-behaved" as large ones — but they aren't. Small n fluctuates a lot (SE ∝ 1/√n), so small units show up at both extremes.
The mechanism: standard error
The standard error of a proportion shrinks with the square root of sample size: SE = √( p·(1−p) / n ) Small towns (small n) have a large SE → their measured rate scatters widely around the true rate p. Large towns (large n) sit almost exactly at p. This is the real reason both the highest and the lowest measured rate come from small towns — not because anything special happens there, but because small samples fluctuate more randomly.
Why both ends?
The spread is symmetric: a small town can randomly land very high or very low. That's why both the top performers and the bottom performers are small — a telltale pattern.
Funnel plot
Value against sample size, with a ±2-SE funnel. Points outside the funnel are genuine outliers; the funnel shape itself exposes pure size effects. Also standard for detecting bias in meta-analyses.
Remedy: shrinkage
Pull small units' values toward the overall mean (empirical Bayes / multilevel) — extreme random outliers get dampened. That's regression to the mean in action. → Regression to the Mean