DIF — Differential Item Functioning

Differential Item Functioning (DIF) is present when an item is differentially difficult for people from different groups — e.g. by gender, background, language, or age — even though these people have the same latent ability θ. DIF is thus an item-level phenomenon: it describes whether the item itself is fair, independent of whether groups differ in their mean ability.

The concept is central to test fairness analysis: a test can be harder for a group on average because that group has, on average, lower ability (that is not bias). A test is unfair when an item is harder for a group than expected — given equal ability (that is DIF).

Impact: the reference group and the focal group differ in their mean ability θ. This can be legitimate — e.g. when groups have genuinely acquired different amounts of knowledge. Impact alone does not mean bias.

DIF: an item is harder (or easier) for the focal group even after conditioning on equal ability θ. This can point to irrelevant item characteristics: culture-specific prior knowledge, linguistic complexity, or context (e.g. sports, occupations) more familiar to one group.

In the tool: change the group distribution (μ_focal) without changing the DIF parameters — impact changes, DIF stays zero. Change Δb or Δa without changing the group distributions — DIF changes, impact stays the same.

With uniform DIF, the focal group's difficulty parameter b is systematically raised (or lowered): b_foc = b_ref + Δb. The ICC shape stays the same — the curves run parallel, without crossing. The focal group is disadvantaged (or favored) uniformly across all θ levels.

Typical example: a vocabulary item that requires language-specific knowledge. People without the corresponding language background have a lower probability of solving it despite equal logical ability.

With non-uniform DIF, discrimination a additionally differs between groups. This causes the ICCs to cross: the item favors one group at low θ values and the other at high θ values. Non-uniform DIF is harder to interpret and detect, because it cancels out across the θ range.

Mantel-Haenszel (MH) (Mantel & Haenszel, 1959): non-parametric. People are grouped into strata by total score (as a proxy for θ). Within each stratum, a 2×2 χ² test for group × response is computed and aggregated across strata. The MH odds ratio α_MH estimates the common effect. The ETS D-DIF scale (Holland & Thayer, 1988) is computed from α_MH: D-DIF = −2.35 · ln(α_MH). ETS classification: |D-DIF| < 1.0 → A (negligible), 1.0–1.5 → B (moderate), > 1.5 → C (substantial).

Logistic regression: item response (0/1) is regressed on total score, group, and their interaction. A significant main effect of group indicates uniform DIF; a significant interaction term indicates non-uniform DIF. Advantage: both DIF types are captured in one analysis.

IRT-based methods: item parameters are estimated separately for both groups and compared. Two common approaches are Lord's χ² test (tests whether the entire parameter vector [a, b] differs significantly) and Raju's (1988) area measure (quantifies the difference between the ICC curves as an area).

Both measures are based on the difference curve D(θ) = ICC_ref(θ) − ICC_foc(θ). This curve shows, for each θ value, who is favored — and by how much.

Signed Area (SA) = ∫ D(θ) dθ = ∫ [ICC_ref(θ) − ICC_foc(θ)] dθ
The signed integral: areas above the zero line (reference favored) count positive, areas below (focal favored) negative. SA > 0 means the reference group is overall at an advantage. SA = 0 can still coincide with strong DIF — namely when the ICCs cross (non-uniform DIF): then positive and negative areas cancel out, even though both groups are disadvantaged at different θ locations.

Unsigned Area (UA) = ∫ |D(θ)| dθ = ∫ |ICC_ref(θ) − ICC_foc(θ)| dθ
The integral of the absolute difference. Positive and negative regions do not cancel out — UA always measures the total amount of DIF, regardless of direction. That's why UA is the more robust measure and forms the basis for DIF classification:
UA < 0.05 → class A (negligible) · 0.05–0.10 → class B (moderate) · > 0.10 → class C (substantial, revision recommended).

When is SA ≠ 0 but small, even though UA is large? Exactly with non-uniform DIF (Δa ≠ 0, crossing ICCs): the item favors one group at low θ values and the other at high θ values — both areas are large, but they cancel out in SA. UA reveals this case, SA does not. That is the core difference between the two measures, and the reason UA is used for classification.

The statistical problem with DIF analyses: one needs a set of "fair" items to anchor the ability scale (anchor items). If the anchor item set itself contains DIF items, all estimates are biased. The solution is iterative purification: first estimate with all items, then remove items with DIF from the anchor and repeat the analysis until the anchor stabilizes. In R: mirt::DIF(), difR::difMH(), or lordif.

DIF does not automatically mean an item must be removed from the test. The following steps are sensible: (1) Substantive review: is there a substantive reason for the DIF? Language-specific knowledge, a culturally more familiar context, gender- or background-specific experience? (2) If the reason is irrelevant to the construct: revise or remove the item. (3) If the reason is substantively justified (e.g. occupational prior experience that belongs to the construct): DIF can be legitimate. (4) With substantial DIF and no substantive explanation, the item is considered biased and should not be used in the test.

In R: mirt (mirt::DIF), difR (difMH, difLord, difRaju), lordif for logistic regression, TAM for Rasch-based DIF.

Mantel, N. & Haenszel, W. (1959). Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute, 22(4), 719–748.
Holland, P. W. & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test Validity. Erlbaum.
Raju, N. S. (1988). The area between two item characteristic curves. Psychometrika, 53(4), 495–502.
de Ayala, R. J. (2022). The Theory and Practice of Item Response Theory (2nd ed.). Guilford Press.
Tutz, G. (2025). A Short Guide to Item Response Theory Models. Springer.
Petersen, I. T. (2024). Principles of Psychological Assessment: With Applied Examples in R. Chapman & Hall/CRC.