When a machine-learning model makes consequential decisions about people — who gets bail, who gets a loan — “is it fair?” turns out to be a question with several precise, mutually incompatible mathematical answers. This lecture is about what those answers are, why you can’t have all of them at once, and why even picking one is philosophically fraught.
Around 2015 the newsroom ProPublica obtained criminal-justice data from Broward County, Florida (which is unusually open with its data). At issue was COMPAS — Correctional Offender Management Profiling for Alternative Sanctions — a tool used in many probation departments. You feed in answers to a long questionnaire about a defendant (charges, prior record, whether offenses involved violence, and so on — memorably including “does the offender have a criminal alias?”), and it outputs risk scores: probability of re-offending (“recidivating”), of a violent offense, of failure to appear.
The scores drive real decisions — e.g. whether to grant bail. Race was not an input. Yet ProPublica found that, broken down by race, the system’s scores and error patterns differed between groups — in particular, Black defendants who did not go on to be rearrested were more often flagged high-risk. The company replied that the scores were equally predictive across groups. Remarkably, both were right — and understanding how is the whole point.
Each of these is a reasonable-sounding definition. They are different quantities.
You generally cannot satisfy more than one of these at once — unless the base rates are equal across groups. If the groups genuinely reoffend at different rates, calibration, false-positive parity, and accuracy parity mathematically cannot all hold.
Here’s the intuition, made concrete. Suppose a risk tool sorts people perfectly into “low risk” (actually reoffend 10% of the time) and “high risk” (80%), and we flag every high-risk person. It can’t see the future for an individual — it only knows their risk bucket. Now give two groups different base rates: group A is 60% high-risk, group B is 50%.
metrics <- function(frac_high, p_lo = 0.10, p_hi = 0.80) {
frac_low <- 1 - frac_high
# Predict 1 for the high-risk bucket, 0 for the low-risk bucket.
# Expected cell proportions (per person):
TP <- frac_high * p_hi # high-risk & reoffend -> predicted 1, correct
FP <- frac_high * (1 - p_hi) # high-risk & not -> predicted 1, wrong
FN <- frac_low * p_lo # low-risk & reoffend -> predicted 0, wrong
TN <- frac_low * (1 - p_lo) # low-risk & not -> predicted 0, correct
c(
flagged_rate = TP + FP, # for demographic parity
error_rate = FP + FN, # for accuracy parity
FPR = FP / (FP + TN), # for false-positive parity
PPV = TP / (TP + FP) # calibration-ish: is a flag equally trustworthy?
)
}
rbind(groupA = metrics(0.60),
groupB = metrics(0.50))## flagged_rate error_rate FPR PPV
## groupA 0.6 0.16 0.2500000 0.8
## groupB 0.5 0.15 0.1818182 0.8Look at the two right-hand columns. PPV is identical across the groups (a flag is exactly as trustworthy either way — the tool is “calibrated”), yet the false-positive rate is not (an innocent person in the higher-base-rate group is flagged more often). Equalizing FPR would force you to deliberately be more wrong about one group — which then breaks calibration and accuracy parity. That trade-off is not a bug in COMPAS; it’s a theorem, and it bites the moment base rates differ. In the real Broward data they do differ (partly because the group populations differ in age structure, and crime skews heavily young).
There’s a principled argument that calibration is the “right” target. Suppose you attach a cost to each true/false positive/negative — a false positive jails someone who wouldn’t have reoffended; a false negative releases someone who does. Then there’s a theorem (Corbett-Davies et al.) that the decision rule minimizing total expected cost is always a single threshold on the true probability: flag everyone whose \(P(Y=1\mid X)\) exceeds some cutoff. The intuition is trivial once stated — if you must jail some fixed number of people, jail the ones most likely to reoffend. And a threshold rule needs the score to mean the same thing for everyone — i.e. calibration.
So calibration is optimal if you accept that framework. Whether you should is exactly what’s contested.
The tempting fix — drop the protected attribute — doesn’t deliver fairness. In the real COMPAS data, at the same score, women reoffend less than men. So a sex-blind score of 5 overstates a woman’s risk relative to a man’s; using it for bail treats her worse than her actual risk warrants. Dropping the attribute didn’t remove its influence — it just hid it, and the proxies (prior record, etc.) carry it anyway. (Same structure as sex-blind car-insurance pricing.)
Every criterion above assumes \(Y\) measures what we care about. It often doesn’t.
One response is to insist on simple, transparent rules (“use only number of prior arrests”). That buys interpretability, sometimes at a real accuracy cost — though for COMPAS specifically, the score is largely driven by priors, age, and sex anyway, so transparency would lose little.
A more philosophically satisfying idea: fairness should be about a counterfactual — would the decision change if we intervened to switch only the person’s protected attribute, holding their own choices fixed? This is the \(\mathrm{do}\)-operator from the causal inference notes, not plain conditioning.
Non-controversial example. Your age influences which major you pick, which determines which math course you take, which affects your physics grade:
Age ──► Major ──► Math course ──► Physics gradeConditioning, physics grades do differ by age — but only through a channel you chose. The counterfactual “hold the major fixed, flip the age” leaves the grade unchanged. That feels fair: age mattered only via your own major choice. (This is not anti-classification, which would just refuse to look at age.)
The COMPAS-style version (from Kusner et al.’s law-school example): unobserved knowledge drives GPA, LSAT, and first-year grades; a protected attribute may also feed into GPA and LSAT (unequal access to tutoring, etc.). Counterfactual fairness says: predict from knowledge, and don’t use anything downstream of a protected characteristic (here GPA and LSAT). The catch: applied strictly, “drop everything downstream of a protected attribute” collapses to essentially demographic parity, and it requires you to (a) declare which pathways are “fair” and (b) know the true causal graph — which you never do. So causal fairness is, in this view, a compelling ideal that’s rarely implementable in practice.
There’s no criterion that’s simply “correct.” Practical advice: