Running Example
n10 patients (objects) — e.g. recorded therapy sessions
k3 therapists (raters) — rate each patient on a clinical scale
σ²_Bvariance between patients — the portion we want to measure
σ²_Rrater bias (strictness/leniency) — systematic but avoidable error
σ²_Erandom residual error — cannot be explained away
How reliable is the assessment — and does it matter whether we count rater bias as error (agreement, ICC(2)) or correct it out (consistency, ICC(3))? And does reliability increase if we average multiple raters?
I
Data Pattern — Raw Data and Variance Sources
Switches between two perspectives: objects (how much do raters scatter per object?)
and raters (how consistently does each rater score?).
Each color = one rater · Dashed = true value (μi) ·
Vertical scatter per object shows σ²E + σ²R ·
Parallel offset of all of one rater's points = rater bias (σ²R)
II
ICC Formulas with Current Values
All six forms after Shrout & Fleiss (1979). The mean squares from the ANOVA table (sidebar)
are plugged directly into the formulas. Numbers update live with the sliders.
| ICC(1) k different raters Agreement |
ICC(2) same k raters, bias = error Agreement |
ICC(3) same k raters, bias adjusted Consistency |
|
|---|---|---|---|
| Reliability of the average rater (?,1) |
|||
| Reliability of the averaged ratings (?,k) |
III
Estimation Task — Train Your Intuition
The slider values are hidden. Estimate the displayed ICC from the data-pattern plot alone.
Try to recognize the variance structure — how widely do the rater points scatter per object?
How different are the objects from one another?
Task 1 of 6
Estimate: how high is ICC(?) for this dataset?
My estimate:
0.50
IV
Concepts to Take Away
What Does the ICC Measure?
The intraclass correlation is a reliability expressed as a variance share: what portion of the total variance stems from genuine differences between the objects (σ²_B) — and not from rater bias (σ²_R) or random error (σ²_E)? ICC = 1 means perfect agreement, ICC = 0 means the ratings carry no information about the objects.
The Three Models (Shrout & Fleiss)
Model 1: each object is rated by different, randomly chosen raters. Model 2: the same raters rate all objects and are treated as a random sample of all possible raters → the result generalizes (agreement). Model 3: the same raters are the only ones of interest (fixed) → consistency only. The model choice follows the study design, not the desired result.
Agreement vs. Consistency
Does systematic rater bias (strictness/leniency, σ²_R) count as error or not? Agreement counts it as error:
ICC(2,1) = σ²_B / (σ²_B + σ²_R + σ²_E). Consistency corrects it out: ICC(3,1) = σ²_B / (σ²_B + σ²_E). With large σ²_R, ICC(2) falls noticeably below ICC(3). Rule of thumb: if you need absolute values (a diagnostic cutoff), use agreement; if only the rank order matters, consistency is enough.Single Rater vs. Averaging (k)
ICC(_,1) describes the reliability of one rater, ICC(_,k) that of the mean of k raters. Averaging reduces the random error → reliability rises (same logic as the Spearman-Brown test-lengthening formula): ICC(3,k) = k·σ²_B / (k·σ²_B + σ²_E). More raters raise reliability because the random error (σ²_E) partly cancels out on averaging. A systematic rater bias (σ²_R), however, is unaffected — it does not vanish through averaging and continues to pull down the agreement ICC.Variance Components & ANOVA
ANOVA doesn't yield the σ² components directly, but via the expected values of the mean squares (MQ):
E[MQ_zw] = σ²_E + k·σ²_B, E[MQ_rat] = σ²_E + n·σ²_R, E[MQ_res] = σ²_E. Substituting and solving gives the ICC formulas. The sliders set the true σ²; each sample deviates due to sampling error and converges only for large n·k.Context & Related Tools
The ICC is the continuous sibling of inter-rater agreement: there κ/Fleiss/Gwet for categorical judgments, here variance decomposition for metric ones. As a variance share, it is at the same time a reliability coefficient in the sense of Classical Test Theory. → Inter-Rater Agreement · → CTT Foundations