How We Found 7 TiB of Memory Just Sitting Around(render.com) |
How We Found 7 TiB of Memory Just Sitting Around(render.com) |
This is anecdotal, but if my experiences aren't unique then there is a lot of lack of reasonable in DevOps.
DevOps has - ever since it's originally well meaning inception (by Netflix iirc?) - been implemented across our industry as an effective cost cutting measure, forcing devs that didn't see it as their job to also handle it.
Which consequently means they're not interfacing with it whatsoever. They do as little as they can get away with, which inevitably means things are being done with borderline malicious compliance... Or just complete incompetence.
I'm not even sure I'd blame these devs in particular. The devs just saw it as a quick bonus generator for the MBA in charge of this rebranding while offloading more responsibilities in their shoulders.
DevOps made total sense in the work culture where this concept was conceived - Netflix was well known at that point to only ever employ senior Devs. However, in the context of the average 9-5 dev, which often knows a lot less then even some enthusiastic Jrs... Let's just say that it's incredibly dicey wherever it's successful in practice.
Sadly devs are incentivized by that and going towards the cloud might be a fun story. Given the environment I hope they scrap the effort sooner rather than later, buy some Oxide systems for the people who need to iterate faster than the usual process of getting a VM and replace/reuse the 10% of the company occupied with the cloud (mind you: no real workload runs there yet...) to actually improve local processes...
I wonder if msft simply cut dev salaries by 50% in the 90s, would it have had any measurable effect on windows quality by today
At this point, I wonder if instead of relying on daemonsets, you just gave every namespace a vector instance that was responsible for that namespace and pods within. ElasticSearch or whatever you pipe logging data to might not be happy with all those TCP connections.
Just my SRE brain thoughts.
There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried. Sadly I think flame graphs made profiling more accessible to the unmotivated but didn’t actually improve overall results.
If you see a database query that takes 1 hour to run, and only touches a few gb of data, you should be thinking "Well nvme bandwidth is multiple gigabytes per second, why can't it run in 1 second or less?"
The idea that anyone would accept a request to a website taking longer than 30ms, (the time it takes for a game to render it's entire world including both the CPU and GPU parts at 60fps) is insane, and nobody should really accept it, but we commonly do.
The sympathy is also needed. Problems aren't found when people don't care, or consider the current performance acceptable.
> There’s about a factor of 3 improvement that can be made to most code after the profiler has given up. That probably means there are better profilers than could be written, but in 20 years of having them I’ve only seen 2 that tried.
It's hard for profilers to identify slowdowns that are due to the architecture. Making the function do less work to get its result feels different from determining that the function's result is unnecessary.
I'm curious, what're the profilers you know of that tried to be better? I have a little homebrew game engine with an integrated profiler that I'm always looking for ideas to make more effective.
These kind of resource explosions are something I see all the time in k8s clusters. The general advice is to always try and keep pressure off the k8s API, and the consequence is that one must be very minimal and tactical with the operators one installs, and then engage in many hours of work trying to fine tune each operator to run efficiently (e.g. Grafana, whose default helm settings do not use the recommended log indexing algorithm, and which needs to be tweaked to get an appropriate set of read vs. write pods for your situation).
Again, I recognize there is a tradeoff here - the simplicity and openness of the k8s API is what has led to a flourish of new operators, which really has allowed one to run "their own cloud". But there is definitely a cost. I don't know what the solution is, and I'm curious to hear from people who have other views of it, or use other solutions to k8s which offer a different set of tradeoffs.
Aren't they supposed to use watch/long polling?
There were recent changes to the NodeJS Prometheus client that eliminates tag names from the keys used for storing the tag cardinality for metrics. The memory savings wasn’t reported but the cpu savings for recording data points was over 1/3. And about twice that when applied to the aggregation logic.
Lookups are rarely O(1), even in hash tables.
I wonder if there’s a general solution for keeping names concise without triggering transposition or reading comprehension errors. And what the space complexity is of such an algorithm.
> keeping names concise without triggering transposition or reading comprehension errors.
Code that doesn’t work for developers first will soon cease to work for anyone. Plus how do you look up a uuid for a set of tags? What’s your perfect hash plan to make sure you don’t misattribute stats to the wrong place?
UUIDs are entirely opaque and difficult to tell apart consistently.
Vector is a daemonset, because it needs to tail the log files on each node. A single vector per namespace might not reside on the nodes that each pod is on.
We run Vector as Daemonset as well but we don't have a ton of namespaces. Render sounds like they have a ton of namespaces running maybe one or two pods since their customers are much smaller. This is probably much more niche setup then many users of Kubernetes.
The common element between attempts is new visualizations. And like drawing a projection of an object in a mechanical engineering drawing, there is no one projection that contains the entire description of the problem. You need to present several and let brain synthesize the data missing in each individual projection into an accurate model.
The sandwich view hides invocation count, which is one of the biggest things you need to look at for that remaining 3x.
Also you need to think about budgets. Which is something game designers do and the rest of us ignore. Do I want 10% of overall processing time to be spent accessing reloadable config? Reporting stats? If the answer is no then we need to look at that, even if data retrieval is currently 40% of overall response time and we are trying to get from 2 seconds to 200 ms.
That means config and stats have a budget of 20ms each and you will never hit 200ms if someone doesn’t look at them. So you can pretend like they don’t exist until you get all the other tent poles chopped and then surprise pikachu face when you’ve already painted them into a corner with your other changes.
When we have a lot of shit that all needs to get done, you want to get to transparency, look at the pile and figure out how to do it all effectively. Combine errands and spread the stressful bits out over time. None of the tools and none of the literature supports this exercise, and in fact most of the literature is actively hostile to this exercise. Which is why you should read a certain level of reproval or even contempt in my writing about optimization. It’s very much intended.
Most advice on writing fast code has not materially changed for a time period where the number of calculations we do has increased by 5 orders of magnitude. In every other domain, we re-evaluate our solutions at each order of magnitude. We have marched past ignorant and into insane at this point. We are broken and we have been broken for twenty years.
And they're a "Cloud Application Platform" meaning they manage deploys and infrastructure for other people. Their website says "Click, click, done." which is cool and quick and all, but to me it's kind of crazy an organization that should be really engineering focused and mature, doesn't immediately notice 1.2TB being used and tries to figure out why, when 120GB ended up being sufficient.
It gives much more of a "We're a startup, we're learning as we're running" vibe which again, cool and all, but hardly what people should use for hosting their own stuff on.
Does that frame things differently? There's are times in your product lifecycle where you doing want your developers looking at things like this, and a time when you do
You know why? I am not saying that what you said does not make sense. Of course it does make sense, financially so. But! You the manager one day come to me and my team and say "How could we allow to have 7TB of unused memory sitting around and we paid for it?!" and we'll then have multiple follow-up meetings where we'll be scolded and "trained" how to avoid things like this. We'll get sent articles and told to improve.
And believe me when I tell you, _all_ the techies in these meetings want to roll their eyes through it all. Because many of them likely asked "Can I take a closer look at our infra, it seems expensive and we can potentially optimise it?" and were said no by managers like yourself.
As an engineer you just can't win. So I don't blame myself or any other techie who sometimes goes cowboy mode to find such problems without asking.
Finally, "my contribution for the month" is technological work and nothing else. If I wanted to be a cofounder or have a seat in the board so I have fiduciary duty, I would have said so. It's your job as a manager to put this barrier between stakeholders and front soldiers so the latter can do their thing without disruption, so the organisation can succeed.
Are you doing your job well?
All of which have gotten perhaps an order of magnitude worse in the time since I started on this theory.
My first job out of college, I got handed the slowest machine they had. The app was already half done and was dogshit slow even with small data sets. I was embarrassed to think my name would be associated with it. The UI painted so slowly I could watch the individual lines paint on my screen.
My friend and I in college had made homework into a game of seeing who could make their homework assignment run faster or using less memory. Such as calculating the Fibonacci of 100, or 1000. So I just started applying those skills and learning new ones.
For weeks I evaluated improvements to the code by saying “one Mississippi, two Mississippi”. Then how many syllables I got through. Then the stopwatch function on my watch. No profilers, no benchmarking tools, just code review.
And that’s how my first specialization became optimization.
The service dashboards already existed, all I had to do was a bit of load testing and read the graphs.
It's not too much extra work to make sure you're scaling efficiently.
Did you accidentally respond to the wrong comment? Because if anything you're giving another example of "most devs not wanting to interface with ops, hence letting it slide until someone bothers to pick up their slack"...
This is how customers end up with too-expensive Rube Goldberg machines.
You have to take some interest in how your code will run in production, even if you don't personally "operate" it.
The app should write records to a database? Fine. Here's where you configure the connection. The app in production is slow because the database server is weak? Not my problem, talk to your DBA.
The app should expose an HTTP endpoint for liveness probes? Fine. It's served from the path you specified. Your reused it for an external outage check, and that's reporting the service is down because the route timed out due to your ops team screwing up the reverse proxy? Literally not my problem, I could not care less.
Okay, so, what is the DBA to do? Double the server capacity to "see if that helps"?
It didn't, and now the opex of the single most expensive cloud server is 2x what it was and is starting to dwarf everything else... combined.
Maybe it's "just" a bad query. Which one? Under what circumstances? Is it supposed to be doing that much work because that's what the app needs, or is it an error that it's sucking down a gigabyte of data every few minutes?
How is the DBA to know what the usecases are?
The best tools that solve these runtime performance are modern APM tools like Azure App Insights, Open Telemetry, or the like.
Some of these products can be injected into precompiled apps using "codeless attach" methods, and this works... okay at best.
So SysOps takes your code, layers on an APM, sees a long list of potential issues... and the developers "don't care" because they think that this is a SysOps thing.
But if the developer takes an interest and is an involved party, then they can integrate the APM software development kit, "enrich" the logged data, log user names, internal business metadata, etc... They log on to the APM web portal and investigate how their app is running in production, with real-world users instead of synthetic tests, with real data, with "noisy neighbours", and all that.
Now if Bob's queries are slowing down the entire platform, it's a trivial matter to track this down and fix Bob's custom report SQL query that is sucking down SELECT * FROM "MassiveReportView" and killing the entire server.
Troubleshooting, performance, security, etc... are all end-to-end things. Nobody can work in isolation and expect a good end result.
If you put that responsibility on the developer--meaning you expect the dev to diagnose an issue that they introduced in the first place--what kind of result do you think you're going to get?
Layering these demands takes away from the overall quality of the application in my experience. You want an app developer to learn all about Prometheus so the app can have an endpoint with all these custom metrics, okay, and you want structured logging and expect the dev to learn how to use Kibana effectively? All that's a huge cognitive burden that eats a slice of the same pie (their brains) as domain knowledge, language & runtime knowledge, etc.
Get maybe one app developer to specialize, get maybe one app developer to cross-train with ops or monitoring even. But leave most of us out of it.
When you flip that expectation of developer involvement in operations, it exposes how unreasonable that arrangement is. Hey, DBA, the app is sucking up resources. Why don't you crack open an IDE and write a patch for it? What do you mean you don't know Go, what do you mean you don't use Git? Every DBA should know how to attach a debugger to a remote process, shouldn't they?
It's just exploitative. Or at least that's been my experience, so there's my bias.
https://en.wikipedia.org/wiki/Speed_of_light
Just as an example, round trip delay from where I rent to the local backbone is about 14mS alone, and the average for a webserver is 53mS. Just as a simple echo reply. (I picked it because I'd hoped that was in Redmond or some nearby datacenter, but it looks more likely to be in a cheaper labor area.)
However it's only the bloated ECMAScript (javascript) trash web of today that makes a website take longer than ~1 second to load on a modern PC. Plain old HTML, images on a reasonable diet, and some script elements only for interactive things can scream.
mtr -bzw microsoft.com
6. AS7922 be-36131-cs03.seattle.wa.ibone.comcast.net (2001:558:3:942::1) 0.0% 10 12.9 13.9 11.5 18.7 2.6
7. AS7922 be-2311-pe11.seattle.wa.ibone.comcast.net (2001:558:3:3a::2) 0.0% 10 11.8 13.3 10.6 17.2 2.4
8. AS7922 2001:559:0:80::101e 0.0% 10 15.2 20.7 10.7 60.0 17.3
9. AS8075 ae25-0.icr02.mwh01.ntwk.msn.net (2a01:111:2000:2:8000::b9a) 0.0% 10 41.1 23.7 14.8 41.9 10.4
10. AS8075 be140.ibr03.mwh01.ntwk.msn.net (2603:1060:0:12::f18e) 0.0% 10 53.1 53.1 50.2 57.4 2.1
11. AS8075 2603:1060:0:10::f536 0.0% 10 82.1 55.7 50.5 82.1 9.7
12. AS8075 2603:1060:0:10::f3b1 0.0% 10 54.4 96.6 50.4 147.4 32.5
13. AS8075 2603:1060:0:10::f51a 0.0% 10 49.7 55.3 49.7 78.4 8.3
14. AS8075 2a01:111:201:f200::d9d 0.0% 10 52.7 53.2 50.2 58.1 2.7
15. AS8075 2a01:111:2000:6::4a51 0.0% 10 49.4 51.6 49.4 54.1 1.7
20. AS8075 2603:1030:b:3::152 0.0% 10 50.7 53.4 49.2 60.7 4.2Uber could run the complete global rider/driver flow from a single server.
It doesn't, in part because all of those individual trips earn $1 or more each, so it's perfectly acceptable to the business to be more more inefficient and use hundreds of servers for this task.
Similarly, a small website taking 150ms to render the page only matters if the lost productivity costs less than the engineering time to fix it, and even then, only makes sense if that engineering time isn't more productively used to add features or reliability.
Microservices try to fix that. But then you need bin packing so microservices beget kubernetes.
I always liked Shaw’s “The reasonable man adapts himself to the world: the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man.”
That adage only applies to people with resources and connections, not the average programmer who can't afford to lose a job.
Billing, serving assets like map tiles, etc. not included.
Some key things to understand:
* The scale of Uber is not that high. A big city surely has < 10,000 drivers simultaneously, probably less than 1,000.
* The driver and rider phones participate in the state keeping. They send updates every 4 seconds, but they only have to be online to start a trip. Both mobiles cache a trip log that gets uploaded when network is available.
* Since driver/rider send updates every 4 seconds, and since you don't need to be online to continue or end a trip, you don't even need an active spare for the server. A hot spare can rebuild the world state in 4 seconds. State for a rider and driver is just a few bytes each for id, position and status.
* Since you'll have the rider and driver trip logs from their phones, you don't necessarily have to log the ride server side either. Its also OK to lose a little data on the server. You can use UDP.
Don't forget that in the olden times, all the taxis in a city like New York were dispatched by humans. All the police in the city were dispatched by humans. You can replace a building of dispatchers with a good server and mobile hardware working together.
What the internet will tell me is that uber has 4500 distinct services, which is more services than there are counties in the US.
The reality is most of those requests (now) get mixed in with a firehose of traffic, and could be served much faster than 16ms if that is all that was going on. But it’s never all that is going on.