Binomial distribution

Null hypothesis: *the probability of success is \(p\)

P-value: If the null hypothesis is true, how often would we observe the outcome that we do?

Outcome example: HTHHTHHHHH We summarize the outcome and say we got 8 heads in 10 trial. This summary is called a statistic. Another summary is that out of 10 trials, we got success 80% of the time. 80% is another statistic.

Computing the p-value

What is the probability of observing an outcome that’s as extreme as 80%, assuming the null hypothesis that \(p = 0.6\) is true?

0 1 2 3 4 5 6 7 8 9 10

pbinom(q = 4, size = 10, prob = 0.6) + (1 - pbinom(q = 7, size = 10, prob = 0.6))
## [1] 0.3335284

If the null hypothesis were \(p = 0.5\), the calculation would be

pbinom(q = 2, size = 10, prob = 0.5) + (1 - pbinom(q = 7, size = 10, prob = 0.5))
## [1] 0.109375

Now, the probability of observing 2 or fewer heads is the same as the probability of observing 2 or fewer tails, if heads and tails are equally likely. So we can simplify things and compute

2 * (1 - pbinom(q = 7, size = 10, prob = 0.5))
## [1] 0.109375

In fact, we could use

2 * pbinom(q = 7, size = 10, prob = 0.5, lower.tail = F)
## [1] 0.109375

The “lower tail” is to the left of the q, and the “upper tail” is to the right.

Power

“Underpowered study” – too little data is collected to gather evidence against the null hypothesis, even if the null is false

Suppose we toss a coin 100 times. Our null hypothesis is that the coin is fair, but actually, the probability of heads is \(0.51\). Could we ever tell?

OBSERVATION: 45
EXPECTATION: 50

EXPECTATION - abs(EXPECTATION-OBSERVATION) -- lower extreme value
EXPECTATION + abs(EXPECTATION-OBSERVATION) -- higher extreme value

Prob(fewer than 46 heads)  (same as fewer than 46 tails)

Prob(fewer than 46 heads) + Prob(fewer than 46 tails)

= 
2 * Prob(fewer than 46 heads)
p.val.r <- function(n.tosses, prob){
  sample1 <- rbinom(n = 1, size = n.tosses, prob = prob)
  p.val <- 2 * pbinom(q = 0.5*n.tosses - abs(0.5*n.tosses-sample1), size = n.tosses, prob = 0.5 )
}

my.exp <- data.frame(p.val = replicate(10000, p.val.r(100, .51)))
ggplot(my.exp) + geom_histogram(mapping = aes(x = p.val), binwidth = 0.05)

This looks about the same as when the null hypothesis is exactly true.

my.exp <- data.frame(p.val = replicate(10000, p.val.r(100, .50)))
ggplot(my.exp) + geom_histogram(mapping = aes(x = p.val), binwidth = 0.05)

But what if we toss the coin 10,000 times?

my.exp <- data.frame(p.val = replicate(10000, p.val.r(10000, .51)))
ggplot(my.exp) + geom_histogram(mapping = aes(x = p.val))
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

It looks like the data is now not consistent with the null hypothesis. Let’s make sure that if the null hypothesis is exactly true, things would look different.

my.exp <- data.frame(p.val = replicate(10000, p.val.r(10000, .50)))
ggplot(my.exp) + geom_histogram(mapping = aes(x = p.val))
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.

This illustrates the fact that the p-value is not the probability of the null hypothesis’s being true (it either is or isn’t, and usually it isn’t). Rather, the p-value measures how consistent the data is with the null hypothesis.

A lot of the time, a small sample will be consistent with the hull hypothesis just because it’s small.

Normal approximations

We know that for large enough size (number of trials) and prob close enough to 0.5, the Binomial distribution can be approximated using the normal distribution.

For example,

If the true probability of heads is 0.6 and the number of trials is 1000, the probability of q heads or fewer can be approximated using pnorm(q = q, mean = 0.6*1000, sd = sqrt(1000*0.6*(1-0.6))

Another way of putting that is that we know that the statistic “Number of heads” is distributed with \(N(p\times n, n \times p\times (1-p))\).

We can compute the p-value using this knowledge.

Normal distribution

The normal distribution \(N(\mu, \sigma^2)\) generates real numbers with mean \(\mu\) and standard deviation \(\sigma\). If we have just one observation in the sample, the observation is the statistic. With a null hypothesis \(\mu = 5\) and \(\sigma = 1\), we can compute the p-value for the observation \(x_1 = 10\) using

pnorm(q = 0, mean = 5, sd = 1) + (1-pnorm(q = 10, mean = 5, sd = 1))
## [1] 5.733031e-07

Or we could use

2 * (1-pnorm(q = 10, mean = 5, sd = 1))
## [1] 5.733031e-07

Multiple observations

For multiple observations, the statistic we can use is the mean of the sample. We know that the mean of the sample \(\bar{x}\) is distributed according to

\(N(\mu, \sigma^2/n)\)

Normal distribution, unknown standard deviation

If we have multiple observations, and the standard deviation is unknown, we cannot use the idea above. But we know that we can compute

\(T = \frac{\bar{X}-\mu}{S/\sqrt{n}}\)

We know that the t-statistic is t-distributed with \(n-1\) degrees of freedom.

Two samples from normal distributions

If we have two samples, we don’t have a null hypothesis about the the mean. Instead, we have a null hypothesis about the difference between the means. We know that the t-statistic

\(T = \frac{\bar{X_1}-\bar{X_2}}{\sqrt{S_1^2/n_1 + S_2^2/n_2}}\) is t-distributed, with \(\nu\) degrees of freedom.