What actually is a p-value? p = the probability of obtaining a result at least as extreme as the one observed, if the null hypothesis H₀ (no true difference) were true. That's a very specific, conditional statement about data — and yet it's constantly confused with entirely different statements (more on that below). It's best understood not through the formula but through repetition: what would happen if we could run the same study under H₀ over and over?
📋 Example — learning program & memory performance
A new learning program is supposed to improve memory performance. Control group (classic learning, n=25) vs. intervention group (new program, n=25), test score 0–150 points. Observed: Mcontrol=100, Mintervention=107, SD=12 (pooled) → t(48)=2.06, p=.045. Just barely significant at α=.05 — but what does that actually mean?
① Repeated sampling under H₀: what the p-value actually counts
Imagine the learning program truly had no effect (H₀ holds). Each click below draws two new, fictitious groups from the same population and computes their t-value. Watch how often a result at least as extreme as our t=2.06 occurs purely by chance.
x-axis: simulated t-value · curve = theoretical t-distribution under H₀ (df=48) · red area = "at least as extreme as our t=2.06" (this is our p-value) · gray dotted line = α=.05 boundary (t=±2.01) for comparison — nearly coincides here, but conceptually different
No sample drawn yet.
Convergence toward the p-value
As the number of draws grows, the simulated proportion "at least as extreme" approaches the formally computed p=.045. That is the definition — no interpretation, no additional assumption about the null hypothesis itself. p is a property of the data under H₀, not a property of H₀.
Important for the chart: the red area starts at our own observed t=2.06 — not at the α=.05 boundary (gray dotted line, t=±2.01). These are two different things that just happen to nearly coincide here: t=±2.01 is the fixed boundary beyond which any result counts as "significant"; t=2.06 is our actual result, and the area beyond exactly this value is, by definition, our p-value. If our t had been, say, 3.0, the red area would sit further out (smaller p-value) — the α boundary would stay at ±2.01.
Important for the chart: the red area starts at our own observed t=2.06 — not at the α=.05 boundary (gray dotted line, t=±2.01). These are two different things that just happen to nearly coincide here: t=±2.01 is the fixed boundary beyond which any result counts as "significant"; t=2.06 is our actual result, and the area beyond exactly this value is, by definition, our p-value. If our t had been, say, 3.0, the red area would sit further out (smaller p-value) — the α boundary would stay at ±2.01.
Take-home message: In general, no matter where you set the "critical" boundary — .05, .045, or any other value: if H₀ holds, then upon repeated sampling exactly the corresponding proportion of all samples will, by chance, have a p-value that small or smaller (more extreme). See the distribution of p-values further below (③ The shape of the p-value distribution).
② The p-value dance: how stable is a single p-value?
Now the reverse: there's a real, fixed effect — but every replication draws new, random samples. How much does the p-value jump from replication to replication, even though the true effect never changes?
Top: each dot is one replication (p-value, 0–1) · bottom: histogram of all p-values so far, same x-axis · green curve = theoretically expected density for the current d and n (flat at d=0, otherwise left-skewed) · dashed line = α=.05
No replication drawn yet.
③ The shape of the p-value distribution
The green curve in the lower panel shows the theoretically expected density of p-values for the currently set true effect d and n — not simulated, but computed exactly. At d=0 (H₀ holds exactly) it is flat, with constant density 1 across the entire interval [0,1] — a uniform distribution, regardless of how large n is. That's plausible once you recall what a p-value is by definition — under H₀ every p-value between 0 and 1 is equally likely, and if you set the significance boundary α at 5%, then by construction exactly 5% of all p-values fall below it (that's the false-positive rate, nothing more and nothing less). As you increase d, the curve becomes increasingly left-skewed: small p-values become more likely, large ones rarer — and the larger d (or n), the steeper. The reverse is the basis of p-curve analysis: if H₀ does not hold across a series of studies, significantly more than 5% of published p-values should fall below .05 and the distribution should be left-skewed rather than flat. If H₀ doesn't hold, p-values just below α should for exactly that reason be rare, with values noticeably smaller than that — a method for distinguishing real effects from pure p-hacking (Simonsohn, Nelson & Simmons, 2014). The blue/red bars are the actual replications drawn; click "automatically" a few times and watch the histogram converge on the green curve.
Myths vs. facts
Myth"p is the probability that H₀ is true."
Factp = P(data | H₀), not P(H₀ | data). These two conditional probabilities coincide only in exceptional cases — a confusion that becomes especially visible in Bayesian statistics.
Myth"p<.05 means: the effect is practically meaningful."
Factp depends on effect size and sample size. A tiny, irrelevant effect becomes significant almost every time given enough data.
—
Myth"1−p is the probability that the finding will replicate."
FactSee Section ② above — the p-value dance directly shows how little a single p-value says about a replication (see also the flashcard "A p-value is not a replicability measure" below).
Myth"Not significant means: there is no effect."
FactAbsence of evidence is not evidence of absence. Often there simply isn't enough power to detect a real effect — an n.s. result can equally mean "no effect" or "too little data to see it."
There's a deeper reason on top of that: the p-value is computed under the assumption that H₀ is true — H₀ is the fixed premise of the entire procedure, not its outcome. A test that presupposes H₀ from the outset cannot, structurally, provide evidence for H₀ — at most it can fail to reject it. Anyone who actually wants to quantify how strongly the data speak for H₀ (rather than merely "not against it") needs a different tool, e.g. a Bayes factor.
→ Bayes Thinking Lab · → Power & Sample Size
→ Bayes Thinking Lab · → Power & Sample Size
Myth"A smaller p-value means a bigger effect."
Factp conflates effect size and sample size — two studies with the same p can have completely different effect sizes:
Concepts & context
Fisher understood the p-value as a graded measure of evidence against H₀. Neyman & Pearson independently developed a decision scheme with pre-fixed error rates α and β. Today's practice blends both approaches into an imprecise hybrid — a major source of many misunderstandings.
α is fixed, p is random
α is set before the study as a fixed error rate for the decision rule. p is computed from the data and fluctuates from sample to sample (see Section ②). Confusing the two leads to exactly the myths above.
A p-value is not a replicability measure
At realistic power (e.g. 50–70 %) it's normal for a barely significant replication to be immediately followed by one that isn't — even though the true effect stays constant. A single p-value says almost nothing about how a second attempt would turn out. This is known as the "dance of the p-values" (Geoff Cumming) — directly experienceable in Section ② above.
One-sided vs. two-sided testing
A one-sided test has more power in the expected direction, but completely ignores effects in the opposite direction. The direction must be fixed before data collection — switching afterward depending on the result is a form of p-hacking.
What to report instead of just p?
Current guidelines (including the 2016 ASA statement) recommend reporting effect size and confidence interval alongside p — p alone says nothing about the size or precision of an effect.
When p isn't enough: Bayes
Bayes factors directly answer the question many mistakenly read into p: how strongly do the data speak for H₁ over H₀? → Lindley's Paradox shows a case where the two answers diverge. Often even more informative than a single Bayes factor is Bayesian parameter estimation: instead of just weighing H₀ vs. H₁ against each other, it delivers a full posterior distribution of the effect itself — including uncertainty, with no detour through a null hypothesis at all. → Bayes Thinking Lab
Related tools
Multiple testing and its link to the "number of chances" for a random hit: → Data Dredging. How sample size and effect size jointly determine power: → Power & Sample Size.