---
title: "Probability Review"
output:
  html_document: default
  pdf_document: default
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

This is a review of probability — definitions, conditional probability, independence, Bayes' rule, and the law of total probability. The Greek goddess of chance, by the way, is Tyche (well, the statue; not the actual goddess).

## Random variables and events

- A **random variable** is a quantity that you observe. For example, if I had a coin and tossed it, the outcome of the toss is a random variable. The color of chalk you would choose when you pick one of my pens is a random variable.
- An **event** is the observation of the value of a random variable (or more generally, of a function of the values of random variables). For example, an event would be "the number of chalk pieces I have in my right hand equals two".

### Notation

One confusing thing is that we write the probability of an event happening, and sometimes write it as $P(H)$. More correctly, you should write it as the probability of the random variable taking some value: $P(X = H)$, the probability that the outcome of the coin toss equals heads.

We will use **capital letters for the random variable** (e.g., $X$ for the outcome of a coin toss) and **lowercase for the specific value** ($x$ for heads or tails).

## Two random variables

Say $X_1$ is the outcome of the first coin toss and $X_2$ is the outcome of the second coin toss. Each of these could be heads or tails.

**Example.** Express the event "no coin came up heads":

$$X_1 \neq H \text{ and } X_2 \neq H$$

or equivalently $X_1 = T$ and $X_2 = T$.

If the coin is fair, what is the probability? It is $1/4$. If they are independent:

$$P(X_1 \neq H, X_2 \neq H) = P(X_1 \neq H) \cdot P(X_2 \neq H) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}.$$

## Conditional probability

The **conditional probability** $P(A \mid B)$ is the probability that event $A$ happened given that we know event $B$ happened.

Example. We have two random variables for a U of T student: $H$ is their height, $S$ is their favorite sport.

$$P(H > 6\text{ ft} \mid S = \text{basketball})$$

reads: out of all the people whose favorite sport is basketball, what is the probability that their height is greater than 6 feet. Or you could go the other way: $P(S = \text{tennis} \mid H > 6\text{ ft})$.

For discrete variables this just works the obvious way: take all the people whose favorite sport is basketball, compute the proportion of those whose height is greater than 6 feet.

If we want to write the math definition, conditional probability is just defined by the formula:

$$P(A \mid B) = \frac{P(A, B)}{P(B)}.$$

If you read a lot of textbooks, what they say is: actually the definition of conditional probability is just the formula. There is an interpretation of what it means, but the official definition is the formula.

## Practice: sums of coin tosses

Suppose we toss a coin and say heads is $0$ and tails is $1$. Let $X_1$ be the first toss and $X_2$ the second.

**Q1.** $P(X_1 + X_2 = 2)$ for a fair coin?

The only way $X_1 + X_2 = 2$ is if $X_1 = 1$ and $X_2 = 1$. So:

$$P(X_1 + X_2 = 2) = P(X_1 = 1)\, P(X_2 = 1) = \frac{1}{2} \cdot \frac{1}{2} = \frac{1}{4}.$$

**Q2.** $P(X_1 + X_2 = 2 \mid X_1 = 1)$?

We already know $X_1 = 1$. Now we are tossing $X_2$; either $X_2 = 0$ or $X_2 = 1$, each with probability $1/2$. So the answer is $1/2$.

Another way to see it: the population of possible outcomes is $(0,0)$, $(0,1)$, $(1,0)$, $(1,1)$. Conditioning on $X_1 = 1$ rules out the first two. Out of the remaining two, half have sum equal to 2.

## The boy-and-girl paradox

This is a famous related problem that ties everything together nicely.

**Q1.** Mr. Jones has two children. The older child is a girl. What is the probability that both children are girls?

**Q2.** Mr. Smith has two children. At least one of them is a boy. What is the probability that both children are boys?

The kind of fun here is that the answers are different.

Let us list out all the possibilities for the children: $\{(B,B), (B,G), (G,B), (G,G)\}$.

**Q1.** "Older child is a girl" leaves us with $(G,B)$ and $(G,G)$. Of these, half have both girls. So $P = 1/2$.

**Q2.** "At least one is a boy" leaves us with $(B,B)$, $(B,G)$, $(G,B)$. Of these, one out of three has both boys. So $P = 1/3$.

Isn't that cool? There is no neat resolution — you can read about the different assumptions; it depends on how you read the problem, but this is the usual way that people read it.

## Independence

Mathematically, two random variables are independent if:

$$P(X_1 = x_1, X_2 = x_2) = P(X_1 = x_1) \cdot P(X_2 = x_2)$$

for all $x_1$, $x_2$. Equivalently:

$$P(X_1 \mid X_2) = P(X_1).$$

The intuition is: information about $X_2$ doesn't affect your beliefs about what $X_1$ might be. We are not going to do a lot of math in this course; the mathematical definition is just this.

## Bayes' rule

Bayes' rule:

$$P(A \mid B) = \frac{P(B \mid A)\, P(A)}{P(B)}.$$

One way of thinking about it is that this is just the mathematical definition of conditional probability:

$$P(A \mid B) = \frac{P(A, B)}{P(B)} = \frac{P(B \mid A)\, P(A)}{P(B)}.$$

The formula is what it is. The point of writing it this way is that often the right-hand side is easier to compute than the left-hand side.

## Law of total probability

$$P(B) = \sum_i P(B \mid A_i)\, P(A_i)$$

where the $A_i$ partition the sample space.

We are saying: $B$ could happen when $A = A_1$, it could happen when $A = A_2$, and so on. The law of total probability says I can compute $P(B)$ as a weighted sum of conditional probabilities — weighted by the probability of each $A_i$.

A common reason to write it on the same slide as Bayes' rule is that you can substitute it into the denominator:

$$P(A_k \mid B) = \frac{P(B \mid A_k)\, P(A_k)}{\sum_i P(B \mid A_i)\, P(A_i)}.$$

Everything on the right side is conditional probabilities of $B$ given various things. That is going to be useful when we compute posteriors.