---
title: "Week 2 Lecture 2"
output: html_document
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```


## Using `tidyverse`/`dplyr` to wrangle data

```{r message = F}
library(tidyverse)
library(babynames)
```

### `filter`

The `filter` function is used to select rows from a data frame. The following are all equivalent. They are different ways of selecting rows for which the `year` is 1880 **and** `sex` is `"F"`.


`filter(babynames, year == 1880, sex == "F")`

`idx <- (babynames$year == 1880) & (babynames$sex == "F")`

`babynames[idx, ]`

`babynames[(babynames$year == 1880) & (babynames$sex == "F"), ]`

`babynames %>% filter(year == 1880 & sex == "F")`

`babynames %>% filter(year == 1880, sex == "F")`

The last way is probably preferable, though different people might have different preferences.

### `arrange`

`arrange` is used to produce a data frame sorted in whatever way you want to specify. See what happens when you run

`View(babynames %>% arrange(name))`

The resultant data frame is sorted alphabetically by `name`. If you want to sort things in descending order, use

`View(babynames %>% arrange(desc(name))`

Note that, similarly to `filter` and other functions, this does not change `babynames`: it just produces a new data frame with the same contents, sorted in the way we specify. You can also sort by `year`, and then also sort by `name`, within the same year. Try running and viewing the following:

`babynames %>% arrange(year, name)`

### `rename`

The following produces a data frame with the same contents as `babynames`, but with the `n` column now named `number`:

`babynames %>% rename(number = n)`

### `select`

`select` is used to produce a data frame only with the specified columns (the columns are in the order that you specify):

`babynames %>% select(year, sex, number=n)`

(Note that we also renamed the column `n` to `number`).

The previous line is equivalent to:

`babynames %>% rename(number = n) %>% select(year, sex, number)`


### `mutate`

Let's try to estimate the total number of recorded female newborns in a given year. First, let's do it with an example. If there were 7065 Marys born in the year, and Marys constitute 0.07 of all newboarn female babies, then we can use the following to find the total number of female newborns $N$:

$0.07N = 7065$, so $N = 7065/0.07 \approx 100929$.


We can use `mutate` to compute a new column:

```{r}
b.total <- babynames %>% mutate(total_by_year = round(n/prop))
```

Here, we compute `n/prop` (working within the data frame `babynames`). This means we divided the corresponding entries in `n` (the total number of babies with a particular name) and `prop` (the proportion that the babies with that name constitue within their sex). We then rounded the numbers, since the total population should always be an integer.

We stored the resultant data frame in the variable `b.total`. Let's take a look at some of the rows of `b.total`:

```{r}
b.total
```

### `distinct`

`dplyr`'s `distinct` is similar to the function `unique`, which we already saw, but it operates on data frames. It takes in a data frame, and returns all the distinct rows. That is, duplicate rows are not included in the returned data frame.

Let's try this:

```{r}
b.total <- babynames %>% mutate(total_by_year = round(n/prop))
b.total %>% select(sex, year, total_by_year) %>% distinct
```

What can you conclude from this?

We can also give arguments to `distinct`. (For precise language fans: an *argument* is a value we pass to the function; a parameter is the *variable* we assign the argument to).

```{r}
babynames %>% distinct(year)
```

```{r}
babynames %>% distinct(name, year)
```

There are as many distinct combination of (name, year) as there are rows in `babynames`! (Why?)


### `group_by` and `summarize` 

Without `group_by`, the function `summarize` simply computes a function of a column. For example:

```{r}
babynames  %>% summarize(total=sum(n))
```

This produces a new data frame, which contains the sum of the column `n`.

The function `group_by` groups the rows of the data frame by the values in the specified columns, for the purposes of using `summarize`. For example, consider the following:

`babynames %>% group_by(year, sex) %>% summarize(total=sum(n))`

This groups the rows of `babynames` into
        rows with year==1880, sex=="F"
        ...
        rows with year==1881, sex=="M"
        ...
        ...
        ...
        rows with year==2000, sex=="F"
        ...

Then, when we apply `summarize`, we compute the sum of the column `n` in each of the groups of rows. So we get as many rows as we have different (year, sex) combinations.

We can compute several different outputs:

```{r}
babynames %>% summarize(mean=mean(n), median=median(n))
```

Recall:

The mean is  the sum divided by the count

The median is the number x such that half the set is smaller or equal than x and half the set is larger or equal to x         

### `group_by` and `mutate`

Here is an example of using `mutate` togerther with `group_by`:

```{r}
a <- babynames %>% group_by(year, sex) %>% 
                   mutate(prop.of.total = n/sum(n))
a[1:10, ]
```

When computing `n/sum(n)`, we compute `sum(n)` for each group of rows -- the total number of boys or girls in a given year. So for each row, we divide the entry in `n` by the  approximate sum total of the relevant population (e.g. the number of boys registered in 1880).

