This topic is built around a deck called “The Truth About Linear Regression” — material based on Cosma Shalizi’s textbook (which is free online if you are interested).
There’s an XKCD comic that captures the spirit: “I used to think correlation implied causation. Then I took a statistics class. Now I don’t.” “Sounds like the class helped.” “Well, maybe.”
So here is Shalizi’s framing. These are lies about linear regression — things people commonly believe that aren’t true.
Not necessarily. That is correlation/causation. Discussed in detail in the correlation-vs-causation topic.
Also not necessarily. We will see this with collinearity below, and with errors in variables.
Also not always true. Again, collinearity is one mechanism.
Suppose you have two predictor variables: someone’s weight and someone’s height. We are predicting GPA:
\[\widehat{\text{GPA}} = a_0 + a_1 \cdot \text{height} + a_2 \cdot \text{weight}.\]
Weight and height are correlated — taller people tend to weigh more, and vice versa. Now we estimate \(a_1\) and \(a_2\).
The problem: if height and weight are highly correlated, \(a_1\) and \(a_2\) are basically not going to tell you much. You can push \(a_1\) up and \(a_2\) down (or the other way), and the prediction is going to be kind of the same, because weight and height are essentially the same variable on different scales.
So the value of \(a_1\) doesn’t really tell you about how GPA is related to height. There is no unique answer — there is a trade-off between \(a_1\) and \(a_2\).
As a result, you cannot say:
“Increase the height by 1 cm and GPA rises by \(a_1\).”
Even if \(a_1\) happens to be, say, 0.1. The reason: \(a_1\) could be lower, \(a_2\) could be higher, and the data would still be fit.
You are trying to predict GPA from height and weight. You see that \(a_1 > 0\) and \(a_2 > 0\). You might be tempted to say height and weight are positively associated with GPA in some interesting way.
But there might be a lurking variable that is the actual cause. For example, early-childhood nutrition. Good nutrition → you grow taller and have appropriate weight; also good nutrition → better cognitive development → higher GPA. There is no direct effect of height or weight on GPA — both height-and-weight and GPA are downstream of nutrition.
People often say that the amount of ice cream consumed in a day is correlated with the number of drownings in that day. This is true.
The lurking variable: the weather. Nice weather → people go to the beach → more ice cream eaten AND more swimming → more drownings.
If you actually want a model, you can include the weather as a predictor. Then conditional on the weather, the spurious correlation between ice cream and drownings disappears.
Ronald Fisher — same Fisher as in “fishers exact test” and frequentist statistics in general — was famously unconvinced to his final days that smoking causes lung cancer. He said that yes, smoking and lung cancer are correlated, but: maybe genetics causes some people to both smoke and have lung cancer. Or maybe people who are getting ill (independently) feel sad and want to smoke. Common-cause arguments.
He was probably wrong. (There are also claims he was paid by tobacco companies; I don’t know.) But the structure of the argument is logically valid — for any correlation, there is always the possibility of a common cause.
It is widely reported that there is a wage gap between men and women. If you just regress:
\[\text{wage} = a_0 + a_1 \cdot I_{\text{male}}\]
\(a_1\) comes out significantly positive — men, on average, have higher wages, in Canada and the US and many countries.
Now you ask: are there lurking variables? And do you want to include them?
gave_birth\[\text{wage} = a_0 + a_1 \cdot I_{\text{male}} + a_2 \cdot I_{\text{gave\_birth}}.\]
What happens: \(a_2\) becomes negative and \(a_1\) becomes much smaller. Women who don’t give birth have wages very close to male wages. The story: giving birth → child-care responsibilities → maternity leave → career suffers → lower wages.
The question becomes: which model is “right”? Is the original gender wage gap explained “by” having given birth?
field_of_employmentIf you add field of employment, a lot of the remaining gap gets attributed to field choice. Female-dominated fields historically have lower wages.
This is more a matter of interpretation than of math. Math tells you what the coefficients are; it doesn’t tell you what the causal structure is, and even less how to interpret it. You could control for the name of every individual and have zero gap — which would obviously be meaningless.
Another example, more recent: there was a US Supreme Court lawsuit against Harvard alleging that Asian applicants had better grades but lower interview scores, and consequently needed higher grades for the same admission probability. The counter: the interview score is measuring a real thing.
If you read the statistician briefs for both sides, you have highly qualified people arguing for completely different interpretations — and that kind of indicates that statistics alone is not enough to decide, because you can use statistics to “prove” basically whatever, depending on what you control for.
Input variables are measured imperfectly. For example, family income — you never know the family income; you know what the family says on a survey, which is probably inaccurate.
Imprecise measurement of an input tends to make its coefficient smaller in magnitude (toward zero). Intuition: imagine the input is pure noise (measured infinitely imprecisely). Then the only thing the model can do is multiply it by zero and ignore it.
If your sample size is large enough, all the coefficients are going to be significant — you can reject zero for everything — just because your standard errors get tiny.
Even a coefficient of 0.000001 can be statistically significant if you have a billion rows. But the effect size is meaningless.
There is one more important pattern: the effect of one variable depends on the value of another. We add an interaction term.
(The lecturer is vegetarian, but the example is illustrative.)
The data:
| avocado | bacon | deliciousness |
|---------|-------|---------------|
| no | no | 1 |
| no | yes | 3 |
| yes | no | 3 |
| yes | yes | 9 |The claim: adding avocado has a different effect depending on whether the sandwich has bacon.
Our initial model:
\[\text{deliciousness} = a_0 + a_1 \cdot I_{\text{avocado}} + a_2 \cdot I_{\text{bacon}}.\]
From this, every time you add avocado, you add \(a_1\) to deliciousness, no matter what. That’s the whole point of an additive model. So this formula can never reproduce the data (2 vs 6 for adding avocado).
Add a third term:
\[\text{deliciousness} = a_0 + a_1 \cdot I_{\text{avocado}} + a_2 \cdot I_{\text{bacon}} + a_{12} \cdot I_{\text{avocado}} \cdot I_{\text{bacon}}.\]
The product term is 1 only when both avocado and bacon are 1.
Solve for the coefficients from the four data points:
So the interaction coefficient is \(a_{12} = 4\), and the model fits the data exactly.
If the effect of \(x_1\) on \(y\) is different depending on the value of \(x_2\), you can add the term \(x_1 x_2\) to the regression. That is called an interaction:
lm(deliciousness ~ avocado * bacon, data = df)In R, the * in the formula means “include both main effects and the interaction.” avocado:bacon would be just the interaction.
Add it when you have a theoretical reason to expect that the effect of one variable depends on another. Then test whether \(a_{12} = 0\) — if you reject, that’s evidence the interaction is real.
Don’t just throw interactions in everywhere; you’ll end up with multiple-comparisons issues again, and lots of trivially-significant interaction terms that don’t mean anything.
To summarize this whole “truth about linear regression” topic:
Step 5 — testing your hypothesis — is just one step. Most of the work is in 1–4.