This is a review of probability — definitions, conditional probability, independence, Bayes’ rule, and the law of total probability. The Greek goddess of chance, by the way, is Tyche (well, the statue; not the actual goddess).
One confusing thing is that we write the probability of an event happening, and sometimes write it as \(P(H)\). More correctly, you should write it as the probability of the random variable taking some value: \(P(X = H)\), the probability that the outcome of the coin toss equals heads.
We will use capital letters for the random variable (e.g., \(X\) for the outcome of a coin toss) and lowercase for the specific value (\(x\) for heads or tails).
Say \(X_1\) is the outcome of the first coin toss and \(X_2\) is the outcome of the second coin toss. Each of these could be heads or tails.
Example. Express the event “no coin came up heads”:
\[X_1 \neq H \text{ and } X_2 \neq H\]
or equivalently \(X_1 = T\) and \(X_2 = T\).
If the coin is fair, what is the probability? It is \(1/4\). If they are independent:
\[P(X_1 \neq H, X_2 \neq H) = P(X_1 \neq H) \cdot P(X_2 \neq H) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}.\]
The conditional probability \(P(A \mid B)\) is the probability that event \(A\) happened given that we know event \(B\) happened.
Example. We have two random variables for a U of T student: \(H\) is their height, \(S\) is their favorite sport.
\[P(H > 6\text{ ft} \mid S = \text{basketball})\]
reads: out of all the people whose favorite sport is basketball, what is the probability that their height is greater than 6 feet. Or you could go the other way: \(P(S = \text{tennis} \mid H > 6\text{ ft})\).
For discrete variables this just works the obvious way: take all the people whose favorite sport is basketball, compute the proportion of those whose height is greater than 6 feet.
If we want to write the math definition, conditional probability is just defined by the formula:
\[P(A \mid B) = \frac{P(A, B)}{P(B)}.\]
If you read a lot of textbooks, what they say is: actually the definition of conditional probability is just the formula. There is an interpretation of what it means, but the official definition is the formula.
Suppose we toss a coin and say heads is \(0\) and tails is \(1\). Let \(X_1\) be the first toss and \(X_2\) the second.
Q1. \(P(X_1 + X_2 = 2)\) for a fair coin?
The only way \(X_1 + X_2 = 2\) is if \(X_1 = 1\) and \(X_2 = 1\). So:
\[P(X_1 + X_2 = 2) = P(X_1 = 1)\, P(X_2 = 1) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}.\]
Q2. \(P(X_1 + X_2 = 2 \mid X_1 = 1)\)?
We already know \(X_1 = 1\). Now we are tossing \(X_2\); either \(X_2 = 0\) or \(X_2 = 1\), each with probability \(1/2\). So the answer is \(1/2\).
Another way to see it: the population of possible outcomes is \((0,0)\), \((0,1)\), \((1,0)\), \((1,1)\). Conditioning on \(X_1 = 1\) rules out the first two. Out of the remaining two, half have sum equal to 2.
This is a famous related problem that ties everything together nicely.
Q1. Mr. Jones has two children. The older child is a girl. What is the probability that both children are girls?
Q2. Mr. Smith has two children. At least one of them is a boy. What is the probability that both children are boys?
The kind of fun here is that the answers are different.
Let us list out all the possibilities for the children: \(\{(B,B), (B,G), (G,B), (G,G)\}\).
Q1. “Older child is a girl” leaves us with \((G,B)\) and \((G,G)\). Of these, half have both girls. So \(P = 1/2\).
Q2. “At least one is a boy” leaves us with \((B,B)\), \((B,G)\), \((G,B)\). Of these, one out of three has both boys. So \(P = 1/3\).
Isn’t that cool? There is no neat resolution — you can read about the different assumptions; it depends on how you read the problem, but this is the usual way that people read it.
Mathematically, two random variables are independent if:
\[P(X_1 = x_1, X_2 = x_2) = P(X_1 = x_1) \cdot P(X_2 = x_2)\]
for all \(x_1\), \(x_2\). Equivalently:
\[P(X_1 \mid X_2) = P(X_1).\]
The intuition is: information about \(X_2\) doesn’t affect your beliefs about what \(X_1\) might be. We are not going to do a lot of math in this course; the mathematical definition is just this.
Bayes’ rule:
\[P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}.\]
One way of thinking about it is that this is just the mathematical definition of conditional probability:
\[P(A \mid B) = \frac{P(A, B)}{P(B)} = \frac{P(B \mid A)\, P(A)}{P(B)}.\]
The formula is what it is. The point of writing it this way is that often the right-hand side is easier to compute than the left-hand side.
\[P(B) = \sum_i P(B \mid A_i)\, P(A_i)\]
where the \(A_i\) partition the sample space.
We are saying: \(B\) could happen when \(A = A_1\), it could happen when \(A = A_2\), and so on. The law of total probability says I can compute \(P(B)\) as a weighted sum of conditional probabilities — weighted by the probability of each \(A_i\).
A common reason to write it on the same slide as Bayes’ rule is that you can substitute it into the denominator:
\[P(A_k \mid B) = \frac{P(B \mid A_k)\, P(A_k)}{\sum_i P(B \mid A_i)\, P(A_i)}.\]
Everything on the right side is conditional probabilities of \(B\) given various things. That is going to be useful when we compute posteriors.