r/place

Architecture of r/place

Requirements

• 1000x1000
• Clients in sync with current state of board, so lots of gets!!
• 100,000 simultaneous users
• One tile/5min/user = 333 puts/sec
• ... other requirements regarding not effecting their systems, open api

Backend: Cassandra attempt

• Cassandra: Single row with (x, y): {'timestamp': epochms, 'author': user_name, 'color': color}
• Problem: read 1,000,000 columns takes 30 seconds

Backend Redis + Cassandra

• Store board in redis using a bitfield of 1 million 4 bit integers. Each 4 bit integer = 1 cells color.
• Redis gives random access to bits in bitfield.
• Using 2d array organization... address(x,y)= x + 1000y
• So could transfer entire board state by sending the compact bitfield.
• Still store sequence of updates in Cassandra, so can view tile history.
• Reading board takes 100ms.
• Problem: Could swamp redis read bandwidth.
• Solution: Use fastly to cache the whole board. Fastly is a content delivery network (CDN). ... Fastly's delivers its CDN service from key access points to the Internet called "points of presence" (POPs). Fastly places POPs where their connectivity to the Internet reduces network transit time when delivering content to end-users. Fastly's 33 POP: cache full board state redis expire state in 1 second used: stale-while-revalidate cache header so a request for expired state results in old state.
• Result: 33 read requests/second from redis.

Connection to Clients

• Complete board state: Fastly CDN
• Updates to board state: Websockets to reddit via their reddit-service-websockets Websockets service supports 100,000 simultaneous conections.

Communication Flow

1. Establish connection for updates first, then request full board. When board state arrives, can apply the stream of updates.
2. Request full board
3. Updates
4. See How We Built r/place (api)

References