Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage(blog.zachbjornson.com) |
Feeding data to 1000 CPUs – comparison of S3, Google, Azure storage(blog.zachbjornson.com) |
edit: Gbps not GBps
If you want to saturate network bandwidth with S3 that's the one tool I know that can do it.
I either missed it or OP didn't specify how many instances they was using at once to run their benchmark, but the more instances they used, the worse it will be per node.
This did not seem to be accounted for.
EDIT: OP says below it was from one instance, so what I said doesn't apply to this writeup.
Google Cloud Storage does not limit read or write throughput with the exception of our "Nearline" product (and even Nearline's limiting can be suspended for additional cost, a feature called "On-Demand I/O").
(Note that I have done some testing from AWS Lambda, where we had 1k lambda jobs all pulling down files from S3 at once. That's a bit harder to benchmark...)
It sounds like that wouldn't have been a factor, except for the cap you seem to have discovered on Amazon that you called out.
My only suggestion then is you may want to make it explicit that you ran the benchmarks from a single instance.
They explicitly mention the RPS per account limit in that doc, which is related.
Is this a limit that is hit anywhere near the 150GB discussed in this article, or is it something that you hit only if you are Netflix? We have TB in S3 and have not observed any limit other than EC2 instance bandwidth.
I'm not sure why anyone would build a server with less than 96GB on it these days, so its not at all unreasonable. Now your service provider my jerk you around but you can run two racks of machines (48 machines) in a data center with specs like that for about $25K/month (including dual gigabit network pipes to your favorite IP transit provider) So it isn't even all that huge of an investment.
[1] Consider your typical 'memcached' type service where data is named as a function of IP and offset.
The colors used for S3 and Azure Storage in the graphs are very near indistiguishable to me, as I have moderate red-green colorblindness. It's easier to tell apart on the bar graphs, since the patches of color are much larger, although I still have to work at it, and use the hints of the labels, but on the line graphs, it's basically impossible to tell apart. A darker shade of green would solve the problem for me personally, but I'm not all that bad a case, nor an expert on the best shades to pick for general color-blindness accessibility.
Just something to think about when presenting data like this.
The bigger point in the article is that these exact "take processing to the data" architectures operate exceedingly well on S3, GCS, Azure.
And, as a biased observer, these architectures operate on GCS the best due to great performance measured in the article, quick VM standup times, low VM prices, and per-minute billing.
EMR-on-S3 is the "copy the data to the processing nodes" variety.
Spinning up a cluster of VMs and use 10 seconds and they charge you min. 1 hour seems expensive to me.
These tests were done over a year ago so bandwidth limitations on EC2 may have changed since.
This testing was with https://github.com/rlmcpherson/s3gof3r
It's a private connection to AWS services including S3. You'd use the same URL as it's a routing basically. No idea if VPC endpoints would be better than IG though. P.S. Just tested and I get about half of the latency on VPC endpoint.
EFS supports NFSv4 so it should avoid being as routinely limited by server round-trip latency as NFSv3 tends to be but it'd be nice to see how well that works in practice.
1. Enhanced Networking (SRIOV) only works in a VPC and not in EC2-Classic.
2. I think the 4x instances don't support 10Gb ethernet. If that is the case, it would also be instructive to test the 8x instances on S3.
For some very application (Hadoop) specific tests of Enhanced Networking, please take a look at https://www.qubole.com/blog/product/hadoop-enhanced-networki...
The throughput correlated directly with how much RAM we allocated to the Lambda function (which presumably means we were sharing the VM with fewer other jobs).
512 MB RAM, 19.5 MB/s
768 MB RAM, 29.8 MB/s
1024 MB RAM, 38.4 MB/s
1536 MB RAM, 43.7 MB/s
Note that this also used the node.js AWS SDK, which is slower to download files than some other APIs.
I'm not doubting a custom build will give him much greater bandwidth. I just doubt the workload has to be "extremely" spiky to make the cloud cost-effective.
Of course, he's going to get billed for 10m or 1hr minimum (Google or Amazon), so that's assuming he can amortize his startup across multiple jobs.
The total amount of data is 150 GB; that would easily fit into memory on a single powerful 2-socket server with 20 cores and would then run in less than 15 minutes. The hardware required to do that will cost you ~ $6000 from Dell; assuming a system lifetime of five years and assuming (like you do) that you can amortize across multiple jobs, the cost is roughly the same as from the cloud, about $0.036 per analysis.
I'm fairly certain that, in the end, it's not more expensive for the customer to just buy a server to run the analysis on.
Edit: I see OP says 80% of the time is spent reading data into memory, at about 100 MB/s. Add $500 worth of SSD to the example server I outlined, and we can cut the application runtime by >70%, making the dedicated hardware significantly cheaper.
You have to know how to read their docs. :) This is basically code for, "there is a default limit here that you have to get raised if you want to go above it".
>Amazon S3 scales to support very high request rates. If your request rate grows steadily, Amazon S3 automatically partitions your buckets as needed to support higher request rates. However, if you expect a rapid increase in the request rate for a bucket to more than 300 PUT/LIST/DELETE requests per second or more than 800 GET requests per second, we recommend that you open a support case to prepare for the workload and avoid any temporary limits on your request rate. To open a support case, go to Contact Us.
So this looks like an auto scaling issue. It states "S3 automatically scales to support higher request rates". However, if we know that a bucket is going to need to scale dramatically, we can request, in advance, that the S3 team pre-scales it.
I'm sure there is an account limit, but to run 1000 cpu's already requires requesting an increase in the account's EC2 instance limit. Are you saying that a team trying to access 150Gb of files, or to make 1000 RPS, as the article documents, will hit that limit? From your experience, how big is this hard limit? Is it Netflix scale or is it GB or TB?
We've considered a few kinds of platforms (AWS spot fleet/GCE autoscaled preemptible VMs, AWS Lambda, bare metal hosting, even Beowulf clusters), and while bare metal has its benefits as you've pointed out, at our current stage it doesn't make sense for us financially.
I omitted from the blog post that we don't rely exclusively on object storage services because its performance is relatively low. We cache files on compute nodes so we avoid that "80% of time is spent reading data" a lot of the time.
(Re: Netflix, in qaq's other comment, I don't have a hard number for this, but I thought a typical AWS data center is only under 20-30% load at any given time.)