Lord's Paradox — ANCOVA vs. Difference Score

Dr. R. Düsing · Osnabrück University

Help — Lord's Paradox

What does this tool show?

Lord's paradox (1967) is the observation that two correct statistical analyses — the difference-score method and ANCOVA — can lead to opposite conclusions. The paradox doesn't arise from an error, but because the two methods answer different causal questions.

The running example

A university compares two courses: Course A (standard instruction) and Course B (intensive course with tutorials). Students are tested in math at the start (pre) and end (post) of the semester. Course B has a higher average starting level — stronger students choose the intensive course.

The two analyses

Difference score (post − pre): how large is the mean improvement per course? If both courses improve by the same amount, the difference-score analysis finds no effect.

ANCOVA (post ~ course + pre): at the same starting level — does one course end up ahead? Because Course B starts at a higher level and regression to the mean dampens this, Course B looks better in the ANCOVA comparison.

The key formula — derivation

Both analyses start from the same four summary numbers: the pre- and post-test means per course (ȳpre,A, ȳpost,A, ȳpre,B, ȳpost,B). A small rewrite shows exactly why the two methods diverge — no new assumption, just algebra.

Step 1 — rewrite δDiff: with Δ = mean(post − pre), δDiff = ΔB − ΔA is the same as:

δ_Diff = (ȳ_post,B − ȳ_pre,B) − (ȳ_post,A − ȳ_pre,A) = (ȳ_post,B − ȳ_post,A) − (ȳ_pre,B − ȳ_pre,A) = (ȳ_post,B − ȳ_post,A) − Δ_base

So δDiff subtracts the full starting-level difference Δbase from the raw post-test difference — it implicitly assumes that one point of pre-test lead translates 1:1 into one point of post-test lead (a slope of 1).

Step 2 — δANCOVA makes the same correction, but with an estimated rather than an assumed slope:

δ_ANCOVA = (ȳ_post,B − ȳ_post,A) − b_w · Δ_base

Instead of subtracting Δbase in full, ANCOVA weights it by the estimated within-group slope bw — how much a point of pre-test lead actually translates into a point of post-test lead, on average.

Step 3 — subtract the two formulas (the raw post-test difference cancels out exactly):

δ_ANCOVA − δ_Diff = [(ȳ_post,B−ȳ_post,A) − b_w·Δ_base] − [(ȳ_post,B−ȳ_post,A) − Δ_base] = (1 − b_w) · Δ_base ⟹ δ_ANCOVA = δ_Diff + (1 − b_w) · Δ_base

The correction term (1 − bw) · Δbase vanishes exactly when either bw = 1 (no regression to the mean) or Δbase = 0 (no starting-level difference, as is usually the case in an RCT).

What exactly is bw — and why ≈ ρ?

bw ("within-group slope", pooled) is the slope you get when you regress post on pre separately within each course and pool the two slopes across courses — exactly the slope that an ANCOVA with a common-slope assumption (Post ~ Pre + Course, no interaction) estimates internally.

For a bivariate normal distribution, the general formula for the regression slope of Y on X is:

b = ρ · (σ_Y / σ_X)

Here Y = post, X = pre, so bw = ρ · σpost/σpre. If the spread barely changes from pre- to post-test (σpost ≈ σpre — exactly how this tool simulates it, and realistic in many test-retest settings), this simplifies to bw ≈ ρ. That's why the ρ slider on the left acts almost directly on how far the two methods diverge.

The intuition: a difference score is ANCOVA with a fixed slope

Comparing steps 1 and 2 reveals the core of the paradox: difference scores are mathematically nothing other than ANCOVA with bw not estimated, but stubbornly fixed at 1. Whether that's justified depends on the design — in an RCT, Δbase is already ≈ 0, so fixing the slope has essentially no consequence; in an observational study with a genuine starting-level difference, a wrongly fixed slope can account for the entire bias you incur.

Controls

Δ starting level: difference in the pre-test (Course B minus Course A). True effect: additional improvement from the intensive course (beyond the semester's general improvement). ρ (pre-post corr.): the closer to 1, the more strongly the two methods agree. Scenarios: four preset situations showing different aspects of the paradox.

Best practice: which tool when?

RCT: ANCOVA is preferable (higher power, random baseline differences get corrected). Observational study: the choice depends on the estimand — which causal question is to be answered? A DAG is needed. Both methods can be legitimate — for different questions.

References

Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 68(5), 304–305.
Vickers, A. J. & Altman, D. G. (2001). Statistics notes: Analysing controlled trials with baseline and follow up measurements. BMJ, 323(7321), 1123–1124.
Senn, S. (2006). Change from baseline and analysis of covariance revisited. Statistics in Medicine, 25(24), 4334–4344.
Pearl, J. (2016). Lord's Paradox Revisited – (Oh Lord! Kumbaya!). Journal of Causal Inference, 4(2).

📋 Example — Remedial Program Study, Math Class
Two courses at a university: 100 students in Course A (standard) and 100 in Course B (intensive). Course B students start out with, on average, a 10-point lead. Both courses improve by about the same amount — and yet two analyses give different answers.
Results of the Two Analyses
Difference-Score Analysis
δDiff = Δ̄B − Δ̄A · t-test
ΔA̅ = —   ΔB̅ = —
δDiff = —
t(—) = —, p = —
—
ANCOVA
Post ~ Pre + Course · regression
bw = —   δANC = —
SE = —
t(—) = —, p = —
—
Visualization
The Mathematical Relationship
Both methods estimate the same group difference — just computed differently (step ①). The trick: δANCOVA can be rewritten algebraically as δDiff plus a correction term (step ②) — this explains exactly how large the gap between them is and what it depends on (step ③). The full derivation is in the Help.
① The Two Analyses — Each on Its Own
δDiff = ΔB − ΔA  where Δ = mean(post − pre) per course
δANCOVA = (ȳpost,B − ȳpost,A) − bw · Δbase
② Rewritten in Terms of Each Other — Why They Diverge
δANCOVA = δDiff + (1 − bw ) · Δbase
— = — + (1 − — ) · — = —
→ The two methods differ here by — points — that is the size of the paradox for the current settings.
③ The Two Ingredients of the Correction Term
δDiff — Difference of the mean changes: ΔB − ΔA
δANCOVA — Group coefficient from Post ~ Pre + Course
bw — Within-group slope of post on pre — here ≈ ρ, since σpost ≈ σpre (details in the Help)
Δbase — Starting-level difference: ȳpre,B − ȳpre,A
The correction term (1 − bw) · Δbase vanishes only if one of its two factors is zero: bw = 1 (no regression to the mean) or Δbase = 0 (no baseline difference). In an RCT the latter holds approximately by randomization — which is why the two methods usually agree there.
Two Questions — Two Answers
The difference score asks:
"Did Course B improve more on average than Course A?"

Estimates the difference in the mean changes: ΔB̄ − ΔĀ.

The question is marginal: it compares group means — without conditioning on the starting level. Individual differences in the starting point are implicitly cancelled out by taking the difference.
ANCOVA asks:
"At the same starting level — does Course B end up ahead?"

Estimates the group difference at a fixed pre value: E[Post|Course=B, Pre=y] − E[Post|Course=A, Pre=y].

The question is conditional: it compares people with an identical starting value. That makes sense if you want to know whether the course makes a difference within a given performance level.
DAG — Causal Structure
ρ δ (effect) Selection? (obs. study) Course G Pre-Test Y0 Post-Test Y1 Ability U (latent, unobs.) Treatment ANCOVA: adjusted Outcome
Green arc (G → Y1): causal effect δ of the course on the exam result.   Blue arrow (Y0 → Y1): regression to the mean (strength ρ).   Gray dashed: latent ability U influences both measurement occasions.   Red dashed (G → Y0): in observational studies, stronger students choose the intensive course — this arrow is absent in an RCT.
Best Practice: Which Tool When?
Prefer ANCOVA when:
Randomized experiment (RCT) — baseline differences are random; ANCOVA notably increases statistical power without introducing bias. It's the modern standard (Vickers & Altman, 2001, EMA guidelines).

Observational study with a clearly defined confounder — when Y₀ (pre) is the only relevant covariate and G is not caused by Y₀ (i.e. G ⊥ Y₀ | U in the DAG).

The question is: "What would the effect be if we had equal starting conditions?"
Prefer the difference score when:
Groups are defined by their starting level — e.g. "high- vs. low-performing students." Here the pre value is the group-defining trait. ANCOVA would adjust for exactly that trait, thereby canceling out the very group difference that motivated the intervention — that's logically circular.

Individual change is what's of interest — when you want to know whether a person or group developed, regardless of starting point.

U (ability) is the main confounder — difference scores implicitly eliminate time-stable confounders (fixed-effects logic).
Modern takeaway (Pearl, 2016, Senn, 2006): Lord's paradox is not an error — it's a mirror. The two methods answer different causal questions. Best practice today: pin down the causal question first, draw a DAG, then choose the method. In RCTs, ANCOVA is nearly always preferable. In observational studies, the answer depends on the estimand.
Concepts
What is Lord's paradox?
Frederic Lord (1967) showed that two statisticians with the same dataset — both computing correctly — can reach opposite conclusions. Statistician A uses difference scores and finds no group effect. Statistician B uses ANCOVA and finds a significant effect. Both are right — because they're answering different questions.
Regression to the mean as the cause
The mechanism behind the paradox is regression to the mean: whoever was extremely high or low on the pre-test moves toward the group mean on the post-test. Course B has a higher mean — its members regress toward a different center than Course A's. ANCOVA "sees" this difference at a fixed pre value; difference scores don't.
The estimand decides
An estimand is the precise definition of the quantity to be estimated. ATE (average treatment effect) asks about the average effect in the population — difference scores come close to this. CATE (conditional ATE) asks about the effect at a fixed Y₀ — ANCOVA estimates exactly that. Without a predefined estimand, the choice of method is arbitrary.
RCT: ANCOVA almost always better
In randomized experiments, ANCOVA is statistically superior to a plain t-test on difference scores: it reduces residual error by explaining variance through the pre value. The estimated group differences are unbiased and more precise. Vickers & Altman (2001, BMJ) recommend ANCOVA as the standard for clinical trials. Difference scores are only equally good when ρ = 1 or there's no baseline variance.
Observational studies: DAG first
In observational studies, ANCOVA on Y₀ (pre) can backfire: if Y₀ is a mediator of the G effect (G → Y₀ → Y₁), we adjust along the causal path and underestimate the effect. If Y₀ is a collider (G → Y₀ ← U → Y₁), adjusting opens a backdoor path. A DAG clarifies which case applies.
You already know δDiff — from other tools
Treat pre/post as two time points and Course A/B as the control/treatment group, and δDiff is exactly the difference-in-differences estimator. It's also the Group×Time interaction coefficient in a 2×2 repeated-measures ANOVA — and the coefficient of a dummy-coded group variable in a regression of (post − pre) on course. Three names for the same number, depending on your field.

→ Difference-in-Differences · → ANOVA (Dummy vs. Effect Coding)