Cash App migrated 400TB of data to PlanetScale's cloud(planetscale.com) |
Cash App migrated 400TB of data to PlanetScale's cloud(planetscale.com) |
Fortunately, PlanetScale runs a well-maintained fork ourselves, so we're very used to taking custom changes and getting them deployed. In this case, we asked the Block team for all of their changes and went through them one by one to see what was needed to pull into our fork.
By the time we did the migration, we made sure that any behavior wouldn't be different where it mattered.
> At peak times, Cash App's database handles approximately 3–4 million queries per second (QPS) across 400 shards, totaling around 400TiB of data.
400TiB represents not a lot of this data. If each query stored only 1 byte, this would only be 4 years worth of this data.
If duplicated, or processed and the results stored, that would add up, too.
With current 22TB magnetic disks it's less than 20 disks, they would fit into a single machine (4u, likely).
The Storinator XL60 from 45drives (https://www.45drives.com/products/storinator-xl60-configurat...) can hold 60 disks, for (advertised) ~1.4PB of data.
(btw i've learned about 45drives through Linus Tech Tips channel so I think it's obligatory to say LTTSTORE.COM - for the meme)
Even a full 60 disk server filled with magnetic disks would provide less I/O capacity for running a relational database than a single EBS volume.
It's might not look like a lot of data if you're talking about storing media files, but it's quite a bit of relational data to be queried in single-digit milliseconds at-scale.
And by the way, it was just posted today that you can get 60TB pci gen5 SSDs from Micron: https://news.ycombinator.com/item?id=42122434 : you can still fit all that dataset in a single machine and provide all the iops you need. You'd need just 7 of those.
So yeah, 400TB of data is not much data nowadays.
60 drives in 6U is a crap ton of weight.
Because then you need three times the rack space. What you don't spend one-time for the hardware you'll spend extra every month for rack space, connectivity, cooling, power etc.
You might use that budget for redundancy, for example.