---
title: "Fairness in Machine Learning"
output:
  html_document: default
  pdf_document: default
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE, message = FALSE, warning = FALSE)
```

When a machine-learning model makes consequential decisions about people —
who gets bail, who gets a loan — "is it *fair*?" turns out to be a question with
several precise, **mutually incompatible** mathematical answers. This lecture is
about what those answers are, why you can't have all of them at once, and why even
picking one is philosophically fraught.

## The COMPAS story

Around 2015 the newsroom **ProPublica** obtained criminal-justice data from
Broward County, Florida (which is unusually open with its data). At issue was
**COMPAS** — *Correctional Offender Management Profiling for Alternative
Sanctions* — a tool used in many probation departments. You feed in answers to a
long questionnaire about a defendant (charges, prior record, whether offenses
involved violence, and so on — memorably including "does the offender have a
criminal alias?"), and it outputs risk scores: probability of re-offending
("recidivating"), of a violent offense, of failure to appear.

The scores drive real decisions — e.g. whether to grant bail. **Race was not an
input.** Yet ProPublica found that, broken down by race, the system's scores and
error patterns differed between groups — in particular, Black defendants who did
not go on to be rearrested were more often flagged high-risk. The company replied
that the scores were *equally predictive* across groups. Remarkably, **both were
right** — and understanding how is the whole point.

## Setup and notation

- $C$ — the **classifier's** output (e.g. $1$ = "predicted to reoffend").
- $Y$ — the **ground truth**: was the person actually rearrested? (Note already:
  *rearrested*, not *reoffended* — they aren't the same thing. More below.)
- $D$ — the **group** (race, sex, age...): the axis we're checking for disparity.
- $X$ — other features used for prediction.

## Five things "fair" could mean

Each of these is a reasonable-sounding definition. They are **different
quantities.**

1. **Demographic parity.** The prediction rate is equal across groups:
   $P(C = 1 \mid D = 0) = P(C = 1 \mid D = 1)$. The same fraction of each group is
   flagged.
2. **False-positive parity** (a.k.a. *equal opportunity*). Among people who did
   *not* reoffend, the flag rate is equal across groups:
   $P(C = 1 \mid Y = 0, D = 0) = P(C = 1 \mid Y = 0, D = 1)$. "Of the people who
   didn't do it, how often are we wrong about them?" — the criterion the ProPublica
   story turned on.
3. **Accuracy parity.** The classifier is equally accurate in each group:
   $P(C = Y \mid D = 0) = P(C = Y \mid D = 1)$.
4. **Calibration.** A score means the same thing regardless of group: among
   everyone assigned score $s$, the chance of $Y = 1$ is the same across groups.
   "A 7 out of 10 is a 7 out of 10, whoever you are." (This is the sense in which
   COMPAS *was* fair.)
5. **Anti-classification.** Don't use the protected attribute (or its proxies) as
   an input at all. This is also a *legal* concept: two people identical except for
   a protected characteristic should get the same output.

## The impossibility result

**You generally cannot satisfy more than one of these at once — unless the base
rates are equal across groups.** If the groups genuinely reoffend at different
rates, calibration, false-positive parity, and accuracy parity mathematically
*cannot* all hold.

Here's the intuition, made concrete. Suppose a risk tool sorts people *perfectly*
into "low risk" (actually reoffend 10% of the time) and "high risk" (80%), and we
flag every high-risk person. It can't see the future for an individual — it only
knows their risk bucket. Now give two groups **different base rates**: group A is
60% high-risk, group B is 50%.

```{r impossibility}
metrics <- function(frac_high, p_lo = 0.10, p_hi = 0.80) {
  frac_low <- 1 - frac_high
  # Predict 1 for the high-risk bucket, 0 for the low-risk bucket.
  # Expected cell proportions (per person):
  TP <- frac_high * p_hi          # high-risk & reoffend  -> predicted 1, correct
  FP <- frac_high * (1 - p_hi)    # high-risk & not       -> predicted 1, wrong
  FN <- frac_low  * p_lo          # low-risk  & reoffend  -> predicted 0, wrong
  TN <- frac_low  * (1 - p_lo)    # low-risk  & not       -> predicted 0, correct
  c(
    flagged_rate = TP + FP,               # for demographic parity
    error_rate   = FP + FN,               # for accuracy parity
    FPR          = FP / (FP + TN),         # for false-positive parity
    PPV          = TP / (TP + FP)          # calibration-ish: is a flag equally trustworthy?
  )
}

rbind(groupA = metrics(0.60),
      groupB = metrics(0.50))
```

Look at the two right-hand columns. **PPV is identical** across the groups (a flag
is exactly as trustworthy either way — the tool is "calibrated"), yet the
**false-positive rate is not** (an innocent person in the higher-base-rate group
is flagged more often). Equalizing FPR would force you to *deliberately be more
wrong* about one group — which then breaks calibration and accuracy parity. That
trade-off is not a bug in COMPAS; it's a theorem, and it bites the moment base
rates differ. In the real Broward data they do differ (partly because the group
populations differ in age structure, and crime skews heavily young).

## If calibration is optimal, why not just use it?

There's a principled argument that calibration is the "right" target. Suppose you
attach a **cost** to each true/false positive/negative — a false positive jails
someone who wouldn't have reoffended; a false negative releases someone who does.
Then there's a theorem (Corbett-Davies et al.) that the decision rule minimizing
total expected cost is always a **single threshold on the true probability**: flag
everyone whose $P(Y=1\mid X)$ exceeds some cutoff. The intuition is trivial once
stated — if you must jail some fixed number of people, jail the ones most likely
to reoffend. And a threshold rule needs the score to *mean* the same thing for
everyone — i.e. **calibration**.

So calibration is optimal *if* you accept that framework. Whether you should is
exactly what's contested.

## Why "just don't use race" (anti-classification) fails

The tempting fix — drop the protected attribute — doesn't deliver fairness. In the
real COMPAS data, at the *same* score, women reoffend *less* than men. So a
sex-blind score of 5 overstates a woman's risk relative to a man's; using it for
bail treats her *worse* than her actual risk warrants. Dropping the attribute
didn't remove its influence — it just hid it, and the proxies (prior record, etc.)
carry it anyway. (Same structure as sex-blind car-insurance pricing.)

## Deeper problems: the labels themselves

Every criterion above assumes $Y$ measures what we care about. It often doesn't.

- **Label bias.** We observe *rearrest*, not *reoffense*. If some neighborhoods
  are policed more heavily, their residents are rearrested more for the *same*
  underlying behavior. A model calibrated to arrests is calibrated to a biased
  proxy.
- **Feedback loops.** **Redlining** — blanket denial of mortgages in
  predominantly Black neighborhoods — may once have looked like "statistical
  discrimination," but its systemic effect was to make those neighborhoods poorer,
  which then justifies more denials. Predictive policing can loop the same way:
  predict crime → send police → detect more crime → predict more.
- **No counterfactuals.** The training data only records what happened to people
  under the decisions actually made. You never see what *would* have happened had
  you released someone you jailed.
- **Distribution shift.** If the world changes after training, yesterday's
  calibrated model is today's biased one.
- **Externalities.** A rule can be optimal for each individual decision yet harmful
  to society in aggregate.

One response is to insist on **simple, transparent** rules ("use only number of
prior arrests"). That buys interpretability, sometimes at a real accuracy cost —
though for COMPAS specifically, the score is largely driven by priors, age, and sex
anyway, so transparency would lose little.

## Causal / counterfactual fairness

A more philosophically satisfying idea: fairness should be about a
**counterfactual** — would the decision change if we *intervened* to switch only
the person's protected attribute, holding their own choices fixed? This is the
$\mathrm{do}$-operator from the [causal inference](causal_inference.html) notes, not
plain conditioning.

Non-controversial example. Your **age** influences which **major** you pick, which
determines which **math course** you take, which affects your **physics grade**:

```
Age ──► Major ──► Math course ──► Physics grade
```

Conditioning, physics grades *do* differ by age — but only *through a channel you
chose*. The counterfactual "hold the major fixed, flip the age" leaves the grade
unchanged. That feels fair: age mattered only via your own major choice. (This is
**not** anti-classification, which would just refuse to look at age.)

The COMPAS-style version (from Kusner et al.'s law-school example): unobserved
**knowledge** drives GPA, LSAT, and first-year grades; a protected attribute may
*also* feed into GPA and LSAT (unequal access to tutoring, etc.). Counterfactual
fairness says: predict from `knowledge`, and **don't use anything downstream of a
protected characteristic** (here GPA and LSAT). The catch: applied strictly, "drop
everything downstream of a protected attribute" collapses to essentially
**demographic parity**, *and* it requires you to (a) declare which pathways are
"fair" and (b) know the true causal graph — which you never do. So causal fairness
is, in this view, a compelling *ideal* that's rarely implementable in practice.

## What to actually do

There's no criterion that's simply "correct." Practical advice:

- **Measure several criteria** and report them. You know in advance at most one
  can hold exactly — the useful information is *which*, and by how much.
- Every criterion is attackable (its incompatibility with the others is an argument
  against any single one), and all of them are only as good as the labels and
  inputs, which may be biased.
- **Model cards** (Mitchell et al.) are the emerging standard: a short document
  accompanying a model that states its intended use, and reports performance and
  the fairness metrics above, broken down by group.

## Summary

- "Fair" has several precise meanings — demographic parity, false-positive parity,
  accuracy parity, calibration, anti-classification — and they are different
  quantities.
- **When base rates differ across groups, you cannot satisfy them all** (a
  theorem). COMPAS was calibrated *and* had unequal false-positive rates — both
  true at once.
- **Dropping the protected attribute doesn't fix it**; proxies carry the signal,
  and a group-blind score can mistreat a group.
- The labels are often biased proxies (rearrest ≠ reoffense), there are feedback
  loops and missing counterfactuals, so even calibration is on shaky ground.
- **Causal/counterfactual fairness** is the most principled framing but needs a
  causal graph you don't have, and tends to collapse to demographic parity.
- Measure multiple criteria, report them (model cards), and be honest about the
  trade-offs.

## References

- Angwin et al. (2016), ProPublica, "Machine Bias" — the COMPAS story.
- Kleinberg, Mullainathan & Raghavan (2016); Chouldechova (2017) — the impossibility
  theorems.
- Corbett-Davies et al. (2017), "Algorithmic Decision Making and the Cost of
  Fairness" — the threshold/utility argument for calibration.
- Kusner et al. (2017), "Counterfactual Fairness" — the causal framing and
  law-school example.
- Mitchell et al. (2019), "Model Cards for Model Reporting."
- Connects to [causal inference](causal_inference.html) (the $\mathrm{do}$-operator)
  and to [logistic regression](logistic_regression.html) (FPR/FNR/PPV/NPV).
