Diagnostic Validity — Test & Criterion

A depression questionnaire (PHQ-9, z-standardized: X) is compared with the result of a clinical interview (criterion Y). The correlation r=0.50 describes the validity. Both cut-off values split the scatterplot into four quadrants — TP, TN, FP, FN. The central utility metric is PPV = TP/(TP+FP): of everyone who tests positive — how many are actually affected?

The correlation r determines how tight the relationship is between test (X) and criterion (Y). At r=0: the point cloud is circular — no diagnostic utility. At r=1: all points on a line — perfect prediction.

In the example: r=0.50 means the PHQ-9 explains about 25% of the variance in the criterion (r²=0.25). Good diagnostic tests typically have r=0.40–0.70.

The vertical line (X = test cut-off) and horizontal line (Y = criterion cut-off) define four groups:

In the example: moving the test cut-off to the left raises sensitivity (more TP), but also FP. The cut-off lines can be dragged directly on the canvas.

Sensitivity and specificity describe how well the test itself works — prevalence-independent, a property of the test. In everyday clinical and diagnostic practice, though, a different question is central: what does this specific result tell me about this specific person? PPV and NPV answer that question.

PPV (positive predictive value) = TP/(TP+FP): if someone tests positive — how likely is it that a condition is actually present? NPV (negative predictive value) = TN/(TN+FN): if someone tests negative — how certain can one be that no condition is present?

Both values depend strongly on the base rate (prevalence). For rare conditions, PPV stays low even with a good test — the denominator is full of unaffected people, a small proportion of whom are still false-positive. In the example: if depression prevalence is 10% instead of 50%, PPV drops drastically at the same sens and spec.

Sensitivity = TP/(TP+FN) and specificity = TN/(TN+FP) are the foundation: prevalence-independent and transferable across populations. Via LR, they determine how strongly a test result changes the pre-test probability.

LR+ = Sens/(1−Spec), LR− = (1−Sens)/Spec: prevalence-independent measures of result strength; directly usable to convert pre- into post-test probability (Bayes formula).

In psychological assessment, alongside the primary quality criteria (objectivity, reliability, validity), secondary quality criteria are also required — among them utility. It asks: is it worth using this test in this situation — does it improve decisions relative to the status quo?

PPV and NPV are the direct operationalizations of utility in the diagnostic context: a test is useful if a positive finding substantially raises the probability of a condition (high PPV) and a negative finding substantially lowers it (high NPV). Validity r is a necessary but not sufficient condition — base rate and cut-off determine how much utility r delivers under real conditions.

As r rises, the point clouds move apart — sensitivity and specificity improve simultaneously, and so do PPV and NPV (at the same base rate). As r falls, sensitivity and specificity become a zero-sum game: any improvement in one comes at the cost of the other — and PPV/NPV stagnate or worsen depending on the cut-off choice.

For selection contexts (personnel, therapy slots, support programs), the Taylor-Russell tables (1939) formalize the relationship between utility, validity, and boundary conditions. They connect three quantities:

• Validity r — how strongly does the test predict the criterion?
• Base rate BR — how many would succeed/qualify even without the test?
• Selection ratio SR — what proportion of applicants is selected?

From these three quantities the success rate emerges: the proportion of those selected who are actually qualified. This shows when a test with moderate validity (r=0.30) has substantial utility under favorable conditions (low SR, medium BR) — and when a test with high validity delivers little (e.g. when almost everyone would be selected anyway). The interactive Taylor-Russell tool displays this success rate as a table and a nomogram with a target crosshair.

Success rate = PPV — mathematically the same thing, different language: in clinical diagnostics, the proportion of true positives among everyone who tests positive is called PPV; in aptitude assessment and personnel psychology, the proportion of qualified people among everyone selected is called the success rate. The formula is identical: TP/(TP+FP). The term changes with the field of application — the concept stays the same.

Taylor, H. C. & Russell, J. T. (1939). The relationship of validity coefficients to the practical effectiveness of tests in selection. Journal of Applied Psychology, 23(5), 565–578.