A look into what happened behind the scenes during the longest outage in Roblox history.
Sources:
https://blog.roblox.com/2022/01/roblox-return-to-service-10-28-10-31-2021/
https://www.hashicorp.com/resources/how-we-used-the-hashistack-to-transform-the-world-of-roblox
https://roblox.fandom.com/wiki/2021_Roblox_outage/
https://roblox.fandom.com/wiki/Timeline_of_Roblox_history/2016#August_2016
https://news.ycombinator.com/item?id=30013919
https://raft.github.io/
http://www.lmdb.tech/media/20130329-devox-MDB.pdf
http://www.lmdb.tech/doc/
https://db.cs.cmu.edu/mmap-cidr2022/
https://www.youtube.com/watch?v=HDOipdFPbB4
Chapters:
0:00 Intro
0:33 HashiStack Explanation
4:47 Outage Investigation
8:20 Root Causes Found
11:30 Return to Service
12:19 Slow Leaders
15:56 Resolution
Corrections:
- At 9:44, the default unbuffered channel in Go does not hold any items and has a buffer size of 0. Sends to such a channel are blocked until another goroutine is ready to receive the value. The illustration in the video shows a **buffered channel of size 1** however the overall point still stands.
Music Credits:
- Firecracker by LEMMiNO (https://www.youtube.com/watch?v=ulfoU2MziOc)
- Impact Prelude by Kevin MacLeod
- We're Finally Landing by Home
Twitter: https://twitter.com/kevinfaang/
Instagram: https://instagram.com/kevinfaang_yt/