---
title: "Scraping Data with a Coding Agent"
output:
  html_document: default
  pdf_document: default
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

For the course project, one option is to scrape a unique data set. Reddit is a particularly nice source because it has multiple sub-forums about basically any topic, so you can come up with a question and there will be data for it. This topic walks through the workflow.

## Tools

- **Reddit archive APIs.** [Pushshift](https://pushshift.io) used to be the go-to one; it has been down for a while. The active alternative as of writing is [Arctic Shift](https://arctic-shift.photon-reddit.com) and similar mirrors.
- **A coding agent.** VS Code has a built-in coding agent; `claude code` is also good (pretty limited on the free tier). Codex has a student-credit program — Google "Codex student version", upload a picture of your T-card, and you get approved within a few days. With approval you can use it for free.

## Workflow

In VS Code, do **File → Open Folder** and select an empty folder for the project. By default the coding agent is only allowed to mess around with the folder that's open; it cannot go elsewhere on your computer. That is the right safety level — it is OK to tell the agent it can run inside that folder, and it cannot do damage outside.

Then ask the agent to fetch data. A typical first prompt would be something like:

> Download the most recent posts from r/datascience using Arctic Shift. Save the result to a JSON file.

It will figure out the URL, fetch the data, save the file. Open the file in the editor (press **Alt+Z** to wrap long lines so you can see what is in there).

## Doing an analysis

Once you have data, ask for an analysis. For example:

> Write a Python script that finds the 10 most frequent words used in the posts you downloaded.

The agent will create a script, run it, and report. You should audit:

- It might silently remove stop words. The default behaviour of common NLP libraries is to drop "the", "a", "and", etc. — and the agent may do that without telling you. If you don't notice and then report the top 10 words in your paper as if they were the raw top 10, that is a problem.
- The agent will probably do the right thing, but you should know what it actually did. Read the script. If it imports `stopwords`, it removed them.

If you want to compare:

> Re-run the analysis without removing stop words.

That can give you the *actual* top 10 (which will be "the", "a", "to", etc.). At the end of the day **you** have to know what is going on — when you write the report, you should know whether stop words were removed.

## A worked example: drowning in r/boating

Suppose we want to study people's worries about drowning. Boating is a subreddit. Workflow:

```
1. Use Arctic Shift to download ~1000 posts from r/boating.
2. Come up with a list of drowning-related keywords (let the agent suggest some).
3. Count how many posts have at least one of those keywords.
```

When we actually did this in class, it came out to roughly 7 in 10,000 — pretty rare. Obviously a silly analysis (we made it up on the spot), but it illustrates the shape of an end-to-end project: scrape a unique data set; ask a question that turns into a count or a ratio or a model; report.

## Other data sources

- **Wikipedia** is free and (relatively) well-explored.
- **Statistics Canada** has lots of free numerical data. Just downloading one table isn't enough — that data set already exists — but if you download several tables and combine them in some new way, that can be an interesting project.
- **Reddit** via Arctic Shift, if you take a couple of days to scrape carefully (the API will rate-limit you if you go too fast).
- **Twitter** is no longer practical for academic scraping — you have to pay Elon a couple thousand bucks and it is not clear what you get for it. Skip.

## A note on running the agent

These coding agents are usually OK at this kind of scraping task, but:

- **Audit the output.** Modern agents are not going to *make things up* ordinarily, but they sometimes do things you did not ask for (like stop-word removal). You have to read the code and verify.
- **You can be rude to the agent.** That is fine; it might be rude back. Mostly tell it "no, I want this; redo it."
- **Treat it as a fast intern.** It writes the boilerplate; you read and check.