Reining in the thundering herd: Getting to 80% CPU utilization with Django(blog.clubhouse.com) |
Reining in the thundering herd: Getting to 80% CPU utilization with Django(blog.clubhouse.com) |
And the solution to this problem is to slowly, rate-limited, bring the service back online, rather than letting the whole thundering herd go through the door immediately.
Your problem is a real problem though. Where I worked, we would call that backlog, and we would manage it with 'floodgates' ... When the system is broken, close the gates, and you need to open them slowly.
In an ideal world, your system would self-regulate from dead to live, shedding load as necessary, but always making headway. But sometimes a little help is needed to avoid the feedback loop of timed out client requests that still get processed on the server keeping the server in overload.
I think the article just used this phrase to describe something else. (Great article otherwise).
The short version is that when you have multiple processes waiting on listening sockets and a connection arrives, they all get woken up and scheduled to run, but only one will pick up the connection, and the rest have to go back to sleep. These futile wakeups can be a huge waste of CPU, so on systems without accept() scalability fixes, or with more tricky server configurations, the web server puts a lock around accept() to ensure only one process is woken up at a time.
The term (and the fix) dates back to the performance improvement work on Apache 1.3 in the mid-1990s.
That... doesn't have much to do with the thundering herd problem. It also doesn't make much sense as a concept on its own merits -- say you come in to work and your inbox is full enough for three inboxes. Does that fact, in itself, mean that you decide you're done for the day? No, it just means you have a much longer queue to work through than usual.
The thundering herd problem refers to what happens when (1) a bunch of agents come to you for something while you're busy; (2) you tell them all "I'm busy, go away and come back later"; and (3) the come-back-later time you give to each of them is identical, so they all come back simultaneously.
And that's exactly what's happening here, except that instead of giving each worker thread a come-back-later time when it asks for work, you're receiving work, sending out individual messages to every worker saying "hey, I'm not busy anymore, come back RIGHT NOW and get some more work", and then rejecting all but one of the thundering herd that shows up. The reason the Gunicorn docs and the uWSGI docs both refer to this as a "thundering herd" problem is that it's a near-perfect match for the problem prototype. The only difference is that, instead of giving out identical come-back-later times to worker threads as they ask you for work, you tell them to wait for a notification that includes a come-back-later time, and then when you get one piece of work you fire off that notification separately to every sleeping thread, including identical come-back-later times in each one.
If my SLA is 24 hour response time, and the inbox is FIFO, and I can't drop old messages, I'm most likely not hitting the SLA. If they all came in overnight, I'll hit the SLA for day 1, but I will be busy all of day 2 and 3 and never respond on time. If after day 1, I get a days worth of messages every day, I'll never catch up.
One solution is to use a soft starter which slow brings the motor up to speed.
If anybody is interested, I've packaged both as Docker containers:
HAProxy queuing/load shedding: https://hub.docker.com/r/luhn/spillway
nginx request buffering: https://hub.docker.com/r/luhn/gunicorn-proxy
* It does have an http_buffer_request option, but this only buffers the first 8kB (?) of the request.
Works amazingly well! We run our python API tier at 80% target CPU utilization.
2. our load balancer buffers requests as well
> In fact, with some app-servers (e.g. most Ruby/Rack servers, most Python servers, ...) the recommended setup is to put a fully buffering webserver in front. Due to it's design, HAProxy can not fill this role in all cases with arbitrarily large requests.
A year ago I was evaluating recent version of HAProxy as buffering web server and successfully run slowloris attack against it. Thus switching from NGINX is not a straightforward operation and your blog post should mention http-buffer-request option and slow client problem.
Of the 3 main languages for web dev these days - Python, PHP and Javascript - I like Python the most. But it is scary how slow the default runtime, CPython, is. Compared to PHP and Javascript, it crawls like a snake.
Pypy could be a solution as it seems to be about 6x faster on average.
Is anybody here using Pypy for Django?
Did Clubhouse document somewhere if they are using CPython or Pypy?
I am glad I am not the only one. I've had so many issues with setting up sockets, both with gevent and uWSGI, only to be left even more confused after reading the documentation.
At a guess, it's probably most loved by people picking old school simple architectures that aren't the sort of thing that goes viral.
I have been through this journey, we eventually migrated to Golang and it saved a ton of money and firefighting time. Unfortunately, python community hasnt been able to remove GIL, it has its benefits (especially for single threaded programs), but I believe the cost (lack of concurrent abstractions. async/await doesn't cut it) far outweigh it.
Apart from what the article mentions, other low hanging fruits worth exploring are
[1] Moving under PyPy (this should give some perf for free)
[2] Bifurcate metadata and streaming if not already. All the django CRUD stuff could be one service, but the actual streaming should be separated to another service altogether.
If not for the troubles they experienced with their hosting provider and managing deployments / cutting over traffic, it possibly could have been the cheaper option to just keep horizontally scaling vs putting in the time to investigate these issues. I'd also love to see some actual latency graphs, what's the P90 like at 25% CPU usage with a simple Gunicorn / gevent setup?
I am also wondering on 144 Workers, on 96 vCPU which is not 96 CPU Core but 96 CPU thread. So effectively 144 Workers on 48 CPU Core possibly running at sub 3Ghz Clock Speed. But it seems they got it to work out in the end. ( May be at the expense of latency )
And on top of that Nginx Plus is also expensive as hell.
I pay for apps, its not a healthy attotude
* use uWSGI (read the docs, so many options...)
* use HAProxy, so very very good
* scale python apps by using processes.
The opportunity cost of spending time figuring out why only 29 workers are receiving requests over adding new features that generate more revenue, seems like a quick decision.
Personally, I just start off with that now in the first place, the development load isn't any greater and the solutions that are out there are quite good.
What I'm talking about is just pointing at something like AppEngine, Cloud Functions, etc... (or whatever solution AWS has that is similar) and being done with it. I'm talking about not running your own infrastructure, at all. Let AWS and Google be your devops so that you can focus on building features.
It is ridiculous people brag about it.
Guys, if you have budget maybe I can help you up this by couple orders of magnitude.
They’re all web-capable and blow the doors off PHP, Python, etc.
Good article though! I’ve dealt with these exact issues and they can be very frustrating.
Usually engineering blogs exists to show that there are fun stuff to do in a company. But here it just seems they have no idea, what they are doing. Which is fine, I'm classifying myself in the same category.
Reading the article I don't feel like they have solved their issue, they just created more future problems
In gunicorn, `sync` mode does exhibit a rather pathological connection churn, because it does not support keep-alive. Generally, most load balancing layers already will do connection pooling to the upstream, meaning, your gunicorn processes won't really be accepting much connections after they've "warmed up". This doesn't apply in sync mode unfortunately :(. Connection churn can waste CPU.
Another thing to also note is that if you have 150 worker processes, but your load balancer only allows 50 connections per upstream, chances are 100 of your processes will be sitting there idle.
Something just doesn't feel quite right here.
EDIT: I do see mention of `gthread` worker - so you might be already able to support http-keepalives. If this is the case, then you should really have no big thundering herd problem after the LB establishes connections to all the workers.
Sounds like an app like clubhouse might have lots of small, fast responses (like direct messaging), where very little of the response time is spent in application code. Does your API happen to do a lot of CPU-intensive stuff in application code?
When using something like Golang, I have apps doing normal CRUD-ish queries at 10k QPS, on 32c/64g machines. For most web apps, 10k QPS is much more than they will ever see, and the fact that it is all done in a single process means you could do really cool things with in-memory datastructures.
Instead, every single web app is written as a distributed system, when almost none of them need to be, if they were written on a platform that didn't eat all of their resources.
People don't use python because they want performance. People use python because of productivity, frameworks, libraries, documentation, resources and ecosystems. Most projects don't even need 10k qps, but instead most projects do need an ORM, a migrations system, authentication, sessions, etc. Python has bottle tested tools and frameworks for this.
I've been told off in code review for using Python's concurrent.futures.ThreadPoolExecutor to run some http requests (making the code finish N times faster, in a context where latency mattered) "because it's hard to reason about".
I specifically said "market share", not "best" or "favorite".
https://www.wappalyzer.com/technologies/programming-language...
You can compile it to JS or to Webassembly. But you can do that with every language.
Now, maybe they could have fixed that issue instead, but going from 29 to 58 workers is easy, it's not the same going to 29,000 to 58,000. And 1000 hosts vs 500 is a non-trivial cost.
one process per container, easy peasy
This is what they did, but because they didn't need to schedule other jobs on the same machine, kubernetes or even docker would be overkill.
In this case, simple VM orchestration seems like a fine solution.
One process per container and multiprocessing is a huge lift most of the time. I’ve done it but it can be a mess because you don’t really have as much a handle on containers than subprocesses because you can only poke them at a distance through the control plane.
All of those would affect the answer, and would preclude being able to guarantee "up this by couple orders of magnitude"
And no, it does not require any special tricks. It is regular Java / WebFlux / REST / MongoDB backend service.
CPUs can do really a lot and if your node processes 16 requests per second on a multi-core machine then you are using billions of clock cycles and gigabytes of possible transfer to memory for a single request. Something is not quite right...
It seemed to be all about how to extract the most performance from the lemon they had to deal with.
I found the linked reference really informative too: https://rachelbythebay.com/w/2020/03/07/costly/
Per my experience most applications that mostly serve documents from databases should be able to take on at least 10k requests per second on a single node. this is 600k requests per minute on one node, compared to their 1M per 1000 nodes.
This is what I am typically getting from a simple setup with Java, WebFlux and MongoDB with a little bit of experience on what stupid things not to do but without spending much time fine tuning anything.
I think bragging about performance improvements when your design and architecture is already completely broken is at the very least embarrassing.
> poor hindsight from original developer (co-founder)
Well, you have a choice of technologies to write your application in, why chose one that sucks so much when there are so many others that suck less?
It is not poor choice, it is lack of competency.
You are co-founder and want your product to succeed? Don't do stupid shit like choosing stack that already makes reaching your goal very hard.
Speed of development is far more important than optimizing CPU usage.
You can fake your way to fast responses with good caching, but there's not really many ways to fake having the best features.
It isn't any macros. Not sure what you're talking about.
> all the usual footguns of vanilla JS still apply
Yeah that does suck but fortunately ESLint and Typescript have options to prevent most of them. If you use Deno they're enabled by default.
> including many performance optimizations that could be made in a statically typed runtime
Also true, but we're comparing it to PHP and Python.
Do you mean multiprocessing inside the containers? Or are you managing multiprocessing child procs by forking into a container somehow? If the latter, I'd be really interested to learn how to do that; I didn't think it was possible, and it would be super useful for some of what I work on.
One app pool with one worker x number of cores
Wrapping it around a container makes no difference
Way to miss the point.
At the end of the day efficiency is ultimately a business problem and not a technical problem and is rarely the thing that tips a project (Clubhouse in the article) from being profitable to being unprofitable. It's usually an investing question - I have X engineer-months to spend. I can cut costs by Y by optimizing stuff or get Z more profit by building a feature. I will choose to optimize stuff if and only if Y>Z as it returns more.
Clubhouse's major costs are probably bandwidth and engineer time rather than servers. That is to say, even if efficiency was infinity for compute (i.e. server costs magically went to zero) it would probably not change Clubhouse's business proposition that much.
More to the point, I think you are uncharitable at best when you say elsewhere that other frameworks and languages won't require more development work. These frameworks (and the choice of language being implicit in that) are specifically designed to reduce development work. Let's examine for example garbage collection. Garbage collection is undeniably more wasteful than other solutions to memory management, absolutely. But would you really argue that garbage collection does nothing to reduce development time? I find that extremely hard to believe, empirically and subjectively having written programs in many environments including bare metal, reference counted or otherwise semi-managed and garbage collected languages. And so it goes with all of the choices these frameworks like Django and Rails take. And it's getting better with time as things like JRuby are developed, inefficiencies in Rails or Django are removed, etc.
> So many options meant plenty of levers to twist around, but the lack of clear documentation meant that we were frequently left guessing the true intention of a given flag.
And then reading your link, they complain >inside the docs< that the docs aren't complete. I have no idea what to believe anymore :D
Strange read.
Adding capacity is also simple to say, but not always simple to do. And there can be a large difference between the capacity needed to handle a cold start at peak vs the capacity needed for peak under regular operations.
https://httpd.apache.org/docs/2.4/mod/mpm_common.html#listen...
This lays out what I'm trying to achieve: https://aws.amazon.com/builders-library/using-load-shedding-...
Javascript
Python
Java
PHP
C#
but you wouldn't be thinking about instance sizes,
how many processes per instance and
wondering if you're hitting kernel limits with all the issues coming up
One app pool with one worker x number of cores
The "blame our co-founder for the choice" bit is exactly what that graph about the cost of defects vs how early they are fixed is talking about.
If they had just picked Go or Java right at the start they wouldn't have had to expend all this engineering effort to get to a still-not-very-good solution.
Another silly thing:
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
"Serverless" scales infinitely due to its simpler request/response lifecycle.
For large PHP setups it is usually the number of database connections that is the limiting factor, however that is why historically the replicated MySQL databases was such a good fit for PHP, thus only creating a limit for writes on the master.
Django also has in-memory caches, for example for templates which can be extremely slow (seconds) and CPU intensive to render. So you really don't want to have AWS or Google restart your application on AppEngine whenever they feel like it.
I've run Django on AWS Lambda in a a scenario that scaled between 25-250 calls per second depending on time of day (for a runtime of 5-30 sec). Moving Django's bootstrapping so it would stay warm across calls was very easy.
gunicorn has an option --max-requests to restart every X requests but unless you have unfixable memory leaks there is no reason to do this.
Nginx can't directly run WSGI applications, you can do it with Nginx Unit which also never restarts processes.
a) you get to fire the devops person, which saves $150k+ a year.
b) you add appropriate caching layers in front of everything.
c) you spend time adding features, which generate revenue.
I've done all of this before at scale. This whole case study was written about work I did [1]. Two devs, 3 months to release, first year was $80m gross revenue on $500/month cloud bills. Infinite scalability, zero devops.
You are deluded or extremely short-sighted if you believe you can actually fire the devops guy. From my experience, the more you stray away from the conventional "dedicated server" paradigm the more you need a devops guy and you are in a very precarious position if you do fire him and something goes wrong.
The job of the cofounder is to create a thing that people want, which has nothing to do with performance. The first goal is capturing lightning in a bottle with social products. Performance doesn’t matter until the lightning is there, and 99%+ of the time you never have to worry about performance, because you don’t get the lightning. So, probably the correct choice is leveraging the tech stack that gives you the best shot at capturing the lightning. Django seemed to help!
Velocity of development is priority #1 and having something that needs to be scaled is a monumental achievement.
The job of the cofounder is also to anticipate possible risks.
And building your company on an astronomically inefficient technology sounds like a huge risk to me.
Those 1000s of servers are probably a very significant cost with such small technical staff. Just by choosing the right technology for the problem, most of that cost could have been avoided.
Django has nothing special in it that would allow building applications faster than in a lot other frameworks that are also much more efficient.
So it is just a matter of simple choice.
Nobody expects people to write webapps in C++ or Rust. Just don't choose technology that is famous for being inefficient.
This may hurt you but the truth is scaling and software engineering is highly commoditised. That’s the whole point of being in the valley. You can hire people for such things and forget about it.
Clubhouse is not a tech company. They don’t have to care about being the best at infra
It is the decision to choose it to run load that will require 1000s of servers when it could be handled with 5-10 servers in another technology without more development effort.
CPU is much cheaper for scaling a business.
For pretty much every modern programming language, IO is the bottleneck over everything else.
To save you some time, there are practically no metrics in which I think PHP beats another programming language other than maturity, and even then, not really.
It's also useful to set this threshold to prevent long-lived connections to services/datastores not used by every request from accumulating and consuming resources on those services.
Well, it is. It is a fact.
https://rachelbythebay.com/w/2020/03/07/costly/
> Clubhouse is not a tech company.
When you spin 1000s of nodes you need some tech competency.
Or in other words, if it blew one day and there would be a link to writeup on HN, people would be asking "They had 1000s of servers and nobody competent to maintain it?"
Additionally, your thought of having my company held hostage by a single devops person is terrifying. Now you need two of them, which is even more expensive.
It is a great way to bootstrap a company by saving on a salary (or two) that can honestly be engineered out for a lot of SASS businesses. It worked super well for us... and calling someone who did $80m in the first year deluded seems well, rude.
But, if you start off designing systems that scale on their own, you are much better prepared for when you do get some fast growth than dealing with hiring a good devops person (which is extremely hard, as they say.. all the good ones are taken).
At the end of the day, the actual elephant in the room is that django was the wrong choice. You end up having to go through a lot of contortions to make things work, as evidenced by the blog post. The architecture doesn't make things easy to spin up quickly... which creates a lot of bottlenecks. There are better cloud-based solutions.
https://www.techempower.com/benchmarks/#section=data-r20&hw=...
Java wins as expected, but a typical setup with Spring versus the typical top PHP frameworks isn't blowing the doors off. Typical Python + Django is far behind, as someone pointed out.
However what we can see in the diagrams is that ORM layers, regardless of language, are more expensive than what most people realize, even for a compiled language like Java.
Why PHP wins is because it is fast enough, compared to other dynamic languages, but is a better fit for web development than Java or other compiled languages.
If Python weren’t fast “enough”, would it be used to power so many successful web backends?
I haven’t coded spring for a few years now, but I was thinking about the traditional spring setup that most use and that is comparable.
You can of course use Python successfully, my argument is that it is easier with PHP, not that it is not possible with Python.
It is a similar argument compared with Java, it is easier with PHP than Java in a web context. Java has other benefit thats fits better for web services IMHO, general higher performance is one.