Why Twitter didn’t go down: From a real Twitter SRE

Why Twitter didn’t go down: From a real Twitter SRE(matthewtejo.substack.com)

1009 points by mtejo 3 years ago | 1351 comments

From this operations engineer's perspective, there are only 3 main things that bring a site down: new code, disk space, and 'outages'. If you don't push new code, your apps will be pretty stable. If you don't run out of disk space, your apps will keep running. And if your network/power/etc doesn't mysteriously disappear, your apps will keep running. And running, and running, and running.

The biggest thing that brings down a site is changes. Typically code changes, but also schema/data changes, infra/network/config changes, etc. As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. The trick is to design it to be as immutable and simple as possible.

There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site.

The last thing off the top of my head that will absolutely bring a site down over time, is expired certs. If, for any reason at all, a cert fails to be regenerated (say, your etcd certs, or some weird one-off tool underpinning everything that somebody has to remember to regen every 360 days), they will expire, and it will be a very fun day at the office. Over a long enough period of time, your web server's TLS version will be obsoleted in new browser versions, and nobody will be able to load it.

mjr00 3 years ago | |

It's crazy to think about, but many people who use and build software today, including HN readers/commenters, are young enough to have only been exposed to the SaaS, cloud-first era, where software built with microservices deployed from CI/CD systems multiple times per day is just the way things are done.

You're totally right; if you don't make changes to the software, it's unlikely to spontaneously stop working, especially after that first 6-12 months of "hardening" where bugs are found and patched.

Many people working in tech have never been exposed to a piece of software which isn't being constantly changed in small increments and forced upon end users. People are assuming that software is inherently unstable simply because they never use anything that isn't a "cloud service".

This probably comes off as "old man yells at cloud" but I'm not trying to bash cloud here. The cloud/SaaS approach has a ton of advantages for both consumers and businesses. But the average tech person in their 20s vastly underestimates how stable software can be when you aren't constantly pushing new features.

Lutger 3 years ago | | |

Absolutely. I remember we build a unimaginably brittle application many years ago, I think it was running on Windows XP and glued together a complex system with COM calls into this single page webapp even before react was a thing. It was build on a very small budget, serving the core business of a very tiny company.

Like maybe 8 years later I found out it was still humming along happily, without really even a sysadmin attending to it, on a single workstation using consumer hardware, servicing the company that had grown tenfold in size.

It blew my mind it still just worked all these years.

dfghbbjiiugb 3 years ago | | |

Your broad point is obviously correct (most outages are caused by code or config changes) but there are still classes of failures that can happen without any real changes, like various performance degradations (maybe your table grows too large) or occasional catastrophic failures from things like disk space or id overflow or something.

There's also the stability of third party systems: forced deprecations, security EOL, etc. The cert expiration stuff people have been mentioning is in this category too. I wouldn't be surprised if something does slip through the cracks at Twitter in the next 4 or 6mo.

Tade0 3 years ago | | |

I remember how it was over a decade ago and one hallmark of such systems was that they were easy to exploit.

My friend in college would just go into Wordpress admin panels and the like by using common exploits because nobody updated PHP on their VPSes back then.

As someone who spent most of their career to date as a front-end developer I learned that as long as they have the budget, stakeholders are insatiable. It's just that ten years ago most of their ideas were either technically not feasible or very expensive.

Nowadays browsers are much more capable, so the pressure to produce more features is much greater.

To our own peril, we can do much more now.

mgkimsal 3 years ago | | |

The other side of that is browsers. Even if you don’t change your code, the platform people are running your code in changes, automatically in many cases. New JS or CSS behavior in next safari or chrome? You need to patch/push to accommodate running environments that are outside your control.

mrazomor 3 years ago | | |

The ecosystem changed since then. Now updates of your binary's environment are more frequent, and often enforced. How often did you update or patch your Windows 98 or NT?

Today, it's false to assume that fire and forget releasing will work even for standalone Windows binaries.

shapefrog 3 years ago | | |

> The cloud/SaaS approach

You dont have to push constant updates for a cloud / SaaS product - many chose to - but ultimately you dont have to.

A year of 'no new features' should be something customers and vendor alike benefit from.

drdrey 3 years ago | |

Another thing we noticed at Netflix was that after services didn’t get pushed for a while (weeks), performance started degrading because of things like undiscovered memory leaks, threads leaks, disks filling up. You wouldn’t notice during normal operations because of regular autoscaling and code pushes, but code freezes tended to reveal these issues.

TranquilMarmot 3 years ago | | |

We used to have a horribly written node process that was running in a Mesos cluster (using Marathon). It had a memory leak and would start to fill up memory after about a week of running, depending on what customers were doing and if they were hitting it enough.

The solution, rather than investing time in fixing the memory leak, was to add a cron job that would kill/reset the process every three days. This was easier and more foolproof than adding any sort of intelligent monitoring around it. I think an engineer added the cron job in the middle of the night after getting paged, and it stuck around forever... at least for the 6 years I was there, and it was still running when I left.

We couldn't fix the leak because the team that made it had been let go and we were understaffed, so nobody had the time to go and learn how it worked to fix it. It wasn't a critical enough piece of infrastructure to rewrite, but it was needed for a few features that we had.

lcw 3 years ago | | |

Agreed, one of the craziest bugs I had to deal with was we had a distributed system using lots of infrastructure. Said distributed system started having trouble communicating with random nodes and sub-systems. I spent 3 hard days finding a Linux kernel bug where the ARP cache was not removing least recently accessed network addresses. Normally, this wouldn't be a big deal for a typical network because few networks would fill up the default arp cache size. That was even true for ours except that we would slowly add and remove infrastructure over the course of a couple months until eventually the ARP cache would fill and remove the random network devices... It wasn't even our distributed application code... Some bugs take time to manifest themselves in very creative ways.

polio 3 years ago | | |

If resource leaks became a serious issue I imagine they could buy time by restarting. I'm curious what the causes were for code freezes. At Meta they would freeze around Thanksgiving and NYE because of unusually high traffic.

__bjoernd 3 years ago | | |

I once debugged a kernel memory leak in an internal module that manifested after around 6 years of (physical) server uptime. There are surprises lurking very far down the road.

r3trohack3r 3 years ago | | |

We joked about adding this to the NodeQuark platform:

    // Fix Slow Memory Leaks
    setTimeout(() => process.exit(1), 1000 * 60 * 60 * 24)

bandrami 3 years ago | | |

Back in the Pleistocene I worked in a ColdFusion shop (USG was all CF back then and we were contractors) and we had two guys whose job was to bounce stacks when performance fell under some defined level.

evanelias 3 years ago | |

> you don't run out of disk space (from logs for example)

For a social media / user-generated content application, the macro storage concerns are a lot more important than the micro ones. By this I mean, care more about overall fleet-wide capacity for product DBs and media storage, instead of caring about a single server filling up its disk with logs.

With UGC applications, product data just grows and grows, forever, never shrinking. Even if the app becomes less popular over time, the data set will still keep growing -- just more slowly than before.

Even if your database infrastructure has fully automated sharding, with bare metal hosting you still need to keep doing capacity planning and acquiring new database hardware. If no one is doing this, it's game over, there's simply nowhere to store new tweets (or new photos, or whichever infra tier runs out of hardware first...)

Staffing problems in other eng areas can exacerbate this. For example, if automated bot detection becomes inadequate, bot posting volume goes way up and takes up an increasing amount of storage space.

threeseed 3 years ago | |

> absolutely bring a site down over time, is expired certs

From today's Casey Newton's newsletter:

In early December, a number of Twitter’s security certificates are set to expire — particularly those that power various back-end functions of the site. (“Certs,” as they are usually called, serve to reassure users that the website they are visiting is authentic. Without proper certs, a modern web browser will refuse to establish the connection or warn users not to visit the site). Failure to renew these certs could make Twitter inaccessible for most users for some period of time.

We’re told by some members of Twitter’s engineering team that the people responsible for renewing these certs have largely resigned — raising concerns that Twitter’s site could go down without the people on hand to bring it back. Others have told us that the renewal process is largely automated, and such a failure is highly unlikely. But the issue keeps coming up in conversations we have with current and former employees.

kibibyte 3 years ago | | |

I can imagine both cases being true, that the renewal process is automated and that certs won't get renewed because institutional knowledge has left the door. Where I'm at, service-to-service TLS certificates (the bulk of our certs) are automatically rotated by our deploy systems. But there are always the edge cases: the certificates manually created a long time ago (predating any standardized monitoring systems) with long expiry dates, and certificates for systems that simply can't run off the standard infrastructure. Sometimes, they'll bring down systems with low SLOs; other times, they'll block all internal development.

edanm 3 years ago | |

> There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc.

In my experience as a data engineer, unusual states are one of the leading causes of issues, at least after something is built for the first time. You can spend half a year running into weird corner cases like "this thing we assumed had to always be a number apparently can arbitrarily get filled in with a string, now everything is broken."

Also, conditions changing causing code changes is the norm, not the exception, definitely in the beginning but also often later. Most services aren't written and done - they evolve as user needs evolve and the world evolves.

TranquilMarmot 3 years ago | |

> As long as nothing changes, and you don't run out of disk space (from logs for example), things stay working pretty much just fine. > ... > There are other things that can bring a site down, like security issues, or bugs triggered by unusual states, too much traffic, etc. But generally speaking those things are rare and don't bring down an entire site.

Aren't these changes inevitable, though? There is no such thing as bug free code.

Another thing that forces consistent code changes is compliance reasons- any time a 0-day is discovered or some library we're using comes out with a critical fix, we would have to go update things that hadn't been touched sometimes in years.

At my last job, I spent a significant amount of time just re-learning how to update and deploy services that somebody who left the company years ago wrote, usually with little-to-no documentation. And yes, things broke when we would deploy the service anew, but we were beholden to government agencies to make the changes or else lose our certifications to do business with them.

Eventually, Twitter will have to push code changes, if only to patch security vulnerabilities. Just waiting for another Heartbleed to come around...

smaudet 3 years ago | | |

Software never goes stale, it's the environment around it which stales.

Something from the 70s works perfectly fine, except it can't run on anything bare any longer, and the hard drives etc. have all long since failed or their PSU capacitors have blown....so Twitter will absolutely rot, how fast depends on several factors.

I personally suspect the infrastructure used to build Twitter will rot faster than Twitter itself, and of course the largest most dramatic source of rot is the power required to run it - several large communities have abandoned it already, making it less much less relevant, meaning the funding for it will also dry up, meaning more wasted cpu cycles and the like.

Thats of course assuming its left in some sort of limbo, it doesnt sound like thats the case with the current management, its only a matter of time before it topples over from shitty low-rate contractor code. Honestly, the app worked like so much hot garbage already, I could see it falling over itself and imploding with a couple poorly placed loops...

grayfaced 3 years ago | |

But real world conditions can force code changes. For example, a region abandons daylight savings time or a court order on copyright infringement. Someone unqualified working a system they are unfamiliar with could blow it up. Losing that knowledge of how the system works is a risk.

threeseed 3 years ago | | |

> But real world conditions can force code changes

Security fixes.

ithkuil 3 years ago | |

An example where something that correlates with time can reveal pre-existing bugs long after the system was chugging along just fine: counter limits/overflows.

Simple example: you have a DB with a table with an auto incrementing table. You chose a small integer type for the primary key and after years this just worked fine, you finally saturate that integer type you can no longer insert rows in the table. Imagine now this has cascading effects in other systems that depend on this database indirectly and you end up with an "outage"

btbuildem 3 years ago | |

> The biggest thing that brings down a site is changes

Absolutely agreed. In that vein, there is such a thing as too much automation. Sometimes, build chains are set up to always pull in the newest and the freshest -- and given the staggering number of dependencies software generally has, this might mean, small changes all the time. Even when your code does not change, it can eventually break.

It's been my experience that a notable part of software development (in the cloud age, anyway) is about keeping up with all the small incremental changes. It takes bodies to keep up with this churn, bodies which twitter now does not have.

It'll be interesting to keep observing this. So far it's been a testament to the teams that built it and set up the infra -- it keeps running, despite a monkey loose in a server room. It's very impressive.

sleight42 3 years ago | |

"Outages": this is an enormous ellipsis.

* Power outages and general acts of God

* Resource utilization

How do your databases perform when their CPUs are near capacity? Or disks? Or I/O? I've seen Postgres do some "weird s%$#": where query times don't go exponential but they go hockey stick.

* Fan-out and fan-in

These can peg CPU, RAM, I/O. Peg any one of these and you're in trouble. Even run close to capacity for any one of these and you're liable to experience heisenbugs. Troublesome fan-out and fan-in can sometimes be a result of...

* Unintended consequences

The engineering decision made months or years ago may have been perfectly legitimate for the knowledge available at the time. However, we live in a volatile, uncertain, complex, and ambiguous (VUCA) world; conditions change. If your inputs deviate qualitatively or quantitatively significantly, you risk resource utilization issues or, possibly, good ol' fashioned application errors.

"No battle plan survives contact with the enemy." -- Stormin' Norman

Same with software systems. They're living entities that can only maintain homeostasis so long as their environment remains predictable within norms. Deviate far enough from that and boom.

dymk 3 years ago | |

Any sort of cached object expiring might bring the servers down. Who knows when the Death TTL will come?

coldcode 3 years ago | |

I worked as an engineer for a very large non tech company (but used a lot of tech, both bought and in-house). We had 100s of teams supporting services, internal apps (web and mobile), external apps (web and mobile), and connections to vendors plus a huge infrastructure in the real world that interconnected to all of this. One time someone changed something in a single data center (I vaguely remember some kind of DNS or routing update) and every single system worldwide failed in a short time. Even after the issue was resolved, it took most of a day and hundreds of people to successfully restart everything, all while our actual business had to continue without pissing off all of our customers. The triage was brutal as to what mattered most.

You can't do this without a lot of people. Sure you could pare it down, maybe improve some architecture, but without a ton of people involved who understand the systems and how they connect, when things might go south they may never return.

radu_floricica 3 years ago | |

I have an old project I gave up on - haven't touched it, done any code changes or maintenance in... almost a decade? At least a stubborn client is still using it, successfully. And it's not an old guy in a living room, but an honest small sized company that has this software as the core of its operations.

So yeah, I totally agree with you. No code changes = long life.

nebula8804 3 years ago | | |

You should be proud. I hope that one day some software I write can serve people for that long.

lr4444lr 3 years ago | |

You didn't mention data scale. Just because the disks have room, doesn't mean the data access patterns in perfectly stable code will perform well at continual multiples if old data isn't somehow moved to colder storage.

vonmoltke 3 years ago | |

> There are other things that can bring a site down, like [...] too much traffic[.] But generally speaking those things are rare and don't bring down an entire site.

I agree with your assessment, but I do want to highlight that this condition is not rare for Twitter. Load is very spiky, sometimes during predictable periods (e.g., the World Cup, New Year's Eve) and sometimes during unpredictable periods (e.g., Queen Elizabeth II's death, the January 6th US Capitol attack). It isn't going to cause a total site failure (anymore), but it can degrade user experience in subtle or not-so-subtle ways.

An aside on the "anymore", there was a time when the entire site did go down due to high-traffic events. A lot of the complication in the infrastructure was built to add resiliency and scalability to the backend services to allow Twitter to handle these events more gracefully. That resiliency is going to help keep the services up even if maintenance is understaffed and behind a learning curve.

mmcnl 3 years ago | |

Sorry for hijacking your expertise, but why no mention of memory leaks? In my experience they can cause really weird bugs not obvious at first, and are difficult to reproduce, i.e. triggered by edge cases that happen infrequently. Or are you assuming services automatically restart when memory is depleted?

throwaway892238 3 years ago | | |

It depends how well the service was "operationalized":

1) Best case: Monitoring of the service checks for service degradation outside of a sliding window. In this case, more than X percent of responses are not 2xx or 3xx. After a given time period (say, 30 minutes of this) the service can be restarted automatically. This allows you to auto-heal the service for any given "degradation" coming from that service itself. (This does not detect upstream degradation, of course, so everything upstream needs its own monitoring and autohealing, which is difficult to figure out, because it might be specific to this one service. The development/product team needs to put more thought into this in order to properly detect it, or use something like chaos engineering to see the problem and design a solution)

2) If you have a health check on the service (that actually queries the service, not just hits a static /healthcheck endpoint that always returns 200 OK), and a memory leak has caused the service to stop responding (but not die), the failed health check can trigger an automatic service restart.

3) The memory leak makes the process run out of memory and die, and the service is automatically restarted.

4) Ghetto engineering: Restart the service every few days or N requests. This extremely dumb method works very well, until you get so much traffic that it starts dying well before the restart, and you notice that your service just happens to go down on regular intervals for no reason.

5) The failed health check (if it exists) is not set up to trigger a restart, so when the service stops responding due to memory leak (but doesn't exit) the service just sits there broken.

6) Worst case: Nothing is configured to restart the service at all, so it just sits there broken.

If you do the best practice and put dynamic monitoring, a health check, and automatic restart in place, the service will self-heal in the face of memory leaks.

mschuster91 3 years ago | |

> If, for any reason at all, a cert fails to be regenerated (say, your etcd certs, or some weird one-off tool underpinning everything that somebody has to remember to regen every 360 days), they will expire, and it will be a very fun day at the office. Over a long enough period of time, your web server's TLS version will be obsoleted in new browser versions, and nobody will be able to load it.

At least for expired certs, most people have learned the hard way just how bad that is, and either implemented automated renewal (thank to heavens for cert-manager, LetsEncrypt, AWS ACM and friends) or where that doesn't work (MS AD...) monitoring.

marstall 3 years ago | |

I'll add one: when usage scales beyond anticipated levels. then that code that is "good enough" will no longer be, and serious intervention may be required - by senior engineers with history.

enominezerum 3 years ago | |

Takes me back to the first broken mess of an environment I worked in. Change freezes were a day of life and lasted and, magically, nothing would break during that time.

Now, those change freezes even extended to preventative maintenance, one of the dual PSUs in a core switch went bad and we couldn't get an exception to replace it... for 6 months. We got an exception when the second one went down and we had to move a few connections to its still alive mate.

itsoktocry 3 years ago | |

>The biggest thing that brings down a site is changes.

Well, Elon is talking about a massive amount of changes coming down the pipe, so I guess we'll see how that goes!

illiac786 3 years ago | |

I think that without code push they won’t be able to maintain compatibility. With updated APIs from third parties, new hardware, new encryption requirements from clients or browsers etc. It’s a slow descent into chaos indeed.

city41 3 years ago | |

A browser update is a form of "new code". It's rare, but having to work around newly introduced browser bugs does happen.

ranguna 3 years ago | |

And vulnerabilities.

resonious 3 years ago |

> This left a lot wondering what exactly was going on with all those engineers and made it seem like it was all just bloat.

I was partly expecting the rest of the article to explain to me why exactly it wasn't just bloat. But it goes on talking about this 1~3-person cache SRE team that built solid infra automation that's really resilient to both hardware and software failures. If anything, the article might actually persuade me that it was all bloat.

poulsbohemian 3 years ago |

I did SRE consulting work for a phase of my career... as the author points out, these systems are scaled out and resilient, but what happens next is entropy. Team sizes shrink, everything starts to be viewed through a cost cutting / savings lens, overtaxed staff start ignoring problems or the long-term view because they are in firefighting mode, it becomes hard to attract new talent because the perception is "the good times are over." Things start to become brittle and/or not get the attention needed, contractors are brought in because they are cheaper and/or bits get outsourced to the cheapest bidder... the professional care and attention like the author clearly brought just starts to shift over time. Consultants like me are brought in to diagnose what's wrong - the good staff could write our briefs, they know what's going on - and generally we slap a band-aid on the problem because management just wants to squeeze whatever value they can out of the assets rather than actually improve anything.

Dave3of5 3 years ago |

I don't think it will fail on a technical level. As this article says lots of engineering has gone in to make the thing pretty resilient. I would also say that there are still enough engineers who work there who can figure out what's gone wrong and "turn it on and off again" or w/e makes it splutter back to life.

In terms of changes to the Platform ditto. It's not difficult to make these changes that a team of 100's of devs who are not 100% aware of what's there already can't figure it out. I've taken over systems that I knew very little of and were pretty big (not as big as twitter) and I managed for years to make changes without drastically breaking stuff. In any event if they do break stuff they will be able to fix what they have broken.

No, the real failure here is the massive debt burden and the fact that there is no way that twitter can ever service that amount of debt. Note that before EM took over it was ticking along with a relatively small loss. If they had cut headcount by maybe 10% they would have been break even easily. There is no way that's possible with $4million of interest per day. They have to radically change the way they monetise the platform to get to that level. I don't think they will ever get there and Musk will sell off at a bargain basement price at some point in the future to pay back the debt.

swellguy 3 years ago |

I think the real question is: Twitter grew 3x on the headcount front with a flat stock price over the course of less than 5 years. What exactly where these thousands of employees actually doing and why did the previous CEO think what they were doing was worth hiring them for? That's just basic accountability from a stock holder or employee perspective. That's apparently a ton of money being wasted on nothing at all.

ulkesh 3 years ago |

All this does is point out that smart people worked at Twitter who may now no longer work there, whether on their own accord, or due to Elon’s bulldogging tactics.

Elon thinks he knows what he’s doing, but what he is going to be left with are people who are willing to work hard by his standards, but not necessarily smart.

The simple truth is Elon knows nothing about the actual work involved in tech. He knows words or elicits help from others on what to say that sounds like tech speak (RPCs!), but when it comes to being truly knowledgeable in this space, he is losing his most valuable assets because of his amazingly poor managerial and ownership style.

I know there are a lot of Elon fans on this site, and will disagree with all of this; but his abilities have not at all been proven. Yes, he knows how to spend money to claim credit for technical advances, but until he actually has his hands dirty in the muck of the hard work of tech, he will always be a glorified self-promoter with no substance.

And Twitter will suffer for it.

raldi 3 years ago |

The most helpful thing to reflect on in these Twitter operational discussions is the difference between homeostasis and evolution.

You can get rid of 80% of the work force and the existing homeostasis systems will keep things running smoothly despite known day-to-day chaos.

Where you’re really going to run into trouble is inventing responses to novel chaos and gradually changing times.

mtejo 3 years ago |

Hey, sorry for the new account, i just like to try my best to keep my online identity separate. this for better or for worse has my real name on it. Hope this is interesting!

cloudking 3 years ago | |

Nice work! Does Twitter runs their own DCs or hosted somewhere else?

devoutsalsa 3 years ago | | |

As of three years ago, Twitter had 2 to 3 data centers, and was moving some stuff to GCP. Not sure if current state of things.

mtejo 3 years ago | | |

Thanks! Yeah the other guy is right

leftcenterright 3 years ago | |

Kudos for nice work!

What did you make of Mudge's report regarding resiliency of data-centers?

> Insufficient data center redundancy, 59 without a plan to cold-boot or recover from even minor overlapping data center failure, raising the risk of a brief outage to that of a catastrophic and existential risk for Twitter's survival.

- https://techpolicy.press/wp-content/uploads/2022/08/whistleb...

mtejo 3 years ago | | |

Thanks! There was a lot in that and I didn't follow it closely. Honestly that always confused me about the dc failing prep. Like I get that it would be a good idea to have a documented plan but at the same time, we did lose a whole DC and figured out how to recover(not that it was that easy) Im not sure having a cold boot prep plan would have helped that much. The things that failed that I dealt with personally during that event, I don't know if I would have foreseen. The site went on afterwards so it didn't seem to be existential.

ilaksh 3 years ago | |

So you still work there? Or did you quit recently?

mtejo 3 years ago | | |

I left over the summer

zekrioca 3 years ago | |

Weren’t Twitter moving out of Mesos to K8s?

mtejo 3 years ago | | |

Yeah there was a few teams trying to make it happen

robswc 3 years ago |

It was if I was in two realities the other day. I did experience a few bugs but twitter never went "down" for me, despite everyone claiming that 1) it would and 2) it already did.

All 10 of the "top trends" for me were about how twitter was dead/dying and armchair experts on reddit said it was only a matter of hours at this point.

Maybe it was close (in reference to external factors not implied in your very insightful post!) but its amazing how confident people are with opinions on events they have 0 insight into. Everyone knows how to solve the war in Ukraine or world hunger but how rare is the "consensus" (in terms of up-votes or popularity) right... just something that got me thinking. Thanks again for this article. I always love seeing details of tech ops!

hairofadog 3 years ago |

Regardless of what’s up with the tech stack, I find it difficult to want to be a part of whatever this is:

https://mobile.twitter.com/elonmusk/status/15945006557246095... (screenshot: https://raw.githubusercontent.com/aboxwithrocksinit/test-buc...)

If this is the new town square, you can forward my mail to a cabin in the woods.

Havoc 3 years ago | |

That is questionable even by Musk standards

Maybe the guy is really losing it

throwawaylinux 3 years ago | |

I think that's pretty hilarious. It has caught the ire of a lot of furious unhinged puritans too, which has actually been the funniest thing about it for me.

hairofadog 3 years ago | | |

It doesn’t bother me because it’s lascivious, it bothers me because it’s the comedy equivalent of fart noises. It was one thing when he was just a user of Twitter, but now he’s in charge, and it seems like a tone is being set that’s not really the vibe I’m into. Whatever floats your boat, I guess.

HyperSane 3 years ago | | |

It would be funny for Musk to share privately with friends. It is a deeply strange and inappropriate thing for the CEO of a billion dollar company to post publicly. Something is deeply wrong with Musk's decision making process.

memish 3 years ago | | |

That post really did bring the church ladies out. Puritanism runs deep.

Operative0198 3 years ago | |

You do know that you can block Elon's account.

reverius42 3 years ago | | |

If you have to block the account of the CEO of the service you are using, perhaps you should just stop using that service?

tux2bsd 3 years ago | | |

but hairofadog needs a reason to be hysterical.

dpedu 3 years ago | |

What's objectionable about this?

RickTM 3 years ago | |

bold strategy on space karen’s part, marketing twitter as the next thing not to put your dick in

emodendroket 3 years ago |

I think people's expectations became so exaggerated it was inevitable they wouldn't be lived up to. I'm sure Twitter will experience degradation from the drastic cost-cutting, but it was never going to happen overnight and I'm not sure why news outlets were saying that (except that their sources were employees with slightly inflated senses of their own importance, which we're all sometimes guilty of). And people became really invested in the idea that a site cannot possibly stay up without dedicated SREs, as though tons of tech sites (including big names like Amazon) don't just devolve this work to their on-call rotations.

danielodievich 3 years ago |

Excellent article. However!, software, like everything, is subject to laws of physics. Entropy always wins in the end. No matter how good the original engineering and planning, without maintenance it will all fall apart soon enough.

areoform 3 years ago |

There's a pattern of diverging expectations here, one is the non-technical/naïve one,

    - Twitter is going to go down tomorrow and it's all over. RIP.

The second is,

    - Twitter is going to experience a failure cascade over time.

The third is,

    - It's all going to be fine.

I suspect that the real question is, how many individual wires can break before the cable holding the suspended platform snaps?

I am not that good of a developer, but watching Twitter I can't help but be reminded of Arecibo, except at a larger, more abstract scale. There was no single massive event that caused the failure, rather a series of factors and events, tiny cables breaking that eventually leads to a failure cascade that then causes the suspended platform to crash.

From what I can tell, in the past week or so,

    - Twitter's copyright system failed

    - Two Factor Authentication broke down (it seems to be back up?)

    - (anecdata) Tweets have been loading sporadically for me and other people, sometimes we try to open a tweet and it says that it doesn't exist. Happens more frequently with new/recent tweets.

    - (unconfirmed) Twitter's managed account backend is behaving "strangely." For e.g., "One of my campaign managers logged in last week and found all our paused creatives from the past 6 years had been reactivated." from https://www.teamblind.com/post/i-told-my-team-to-pause-our-750kmonth-twitter-ads-budget-last-week-4dnbo1Ft ———— Friends have told me other similar stories

Are these failures symptomatic of a larger problem, or are they well-isolated parts misbehaving? Can Twitter even experience a failure cascade like Arecibo? Can that be paused/stopped?

I am asking this question because I don't know. And I'd like to develop a better mental model to understand what happens next.

shagymoe 3 years ago |

When I was learning programming, I "wrote" an internal app where a supervisor input some information in a form, it connected to an oracle database and then printed out an outbound manifest for a truck full of car parts.

I put "wrote" in quotes because what I had actually done was install Apache and PHP Nuke on a Windows NT server and then modified an existing form page to do what I wanted.

I wrote that application in 2002 and never had a problem with it. I never restarted Apache or the server or did any maintenance. I didn't even upgrade Windows NT if I'm being honest. Windows NT became unsupported in something like 2006 and I left the company in 2010.

I received a call in 2018 from the last person still working there who knew who I was. It had finally fallen over and they wanted to know if I could help. That was the first time it ever had an issue.

skc 3 years ago |

Twitter pretty much passed it's first real test with flying colors since Elon took over.

The world cup is on and the site didn't collapse.

That is a huge win and while I don't believe Twitter will ever be profitable, I think Elon will be feeling rather smug right now with his skeleton staff.

zxcvbn4038 3 years ago |

Now that Elon has cleaned house, when is Twitter hiring again? It might be a good opportunity to get in on the new ground floor.

pcthrowaway 3 years ago | |

Who sees the foundation of a high-rise starting to crack and thinks, "perfect time to get in on the ground floor"?

Ground floor is exactly where you don't want to be when the structure collapses under its own weight.

Rather than bring in additional specialists to reinforce it, Musk has partially evacuated the building at least.

zxcvbn4038 3 years ago | | |

It’s Elon, don’t think he going to take a 44 billion hit on this, he is going to make his (and his co-financiers) money back.

piskerpan 3 years ago | |

Why would anyone want to work in a dumpster fire?

justahuman74 3 years ago | | |

I've wondered if the hiring funnel will be so lean that the TC may be forced high

UberFly 3 years ago | |

The new HARDCORE ground floor.

inawarminister 3 years ago | |

I've sent Elon an email asking for internship, though sadly I can't move to Sanfran and live in the office floor for 12 weeks like @geohot can. Still...

justahuman74 3 years ago | |

How would stock grants work, cause I doubt he'll IPO twitter? Or would it just be netflix-style cash offers

didip 3 years ago |

I am mostly curious about the Mesos layer itself.

Mesos is dead. So you need in-house expertise to patch it without being able to leverage community knowledge.

Does Twitter retain enough people to manage Mesos?

thakoppno 3 years ago | |

The Mesos detail struck me as a risk for the reason you stated.

Also, the article seems to suggest Twitter only has two datacenters. That seems surprising for the global reach of the company. Perhaps there are other smaller datacenters that are not prepared to handle the entirety of the site’s traffic.

My current thinking is there’s time to figure out how to operate the current system before it runs into issues that would render it degraded for a prolonged period of time. I noticed TLS certs have already rotated for instance. That was my best guess for simple thing that could fail if managed poorly.

mtejo 3 years ago | |

The company was moving off it. I wish I could find a twitter eng blog about it. Interesting there's a bunch of not directly from twitter sources about the decision though.

wyclif 3 years ago | |

I was surprised to hear that a cutting edge social media app uses Mesos. Why did they choose that over other options?

rickette 3 years ago | | |

Twitter didn't start yesterday. There was a time when Mesos was all the hotness and k8s was this new thing that looked promising but wasn't nearly production ready.

Apple is also a big Mesos user, but also moving to k8s.

__warlord__ 3 years ago | | |

IIRC Mesos was an internal tool (at Twiiter) that got released under the Apache umbrella later on.

Then Mesosphere (a company) wanted to bring it to the enterprise market but at the time was competing with Kubernetes... and we all know who won.

ggm 3 years ago |

The article is good, and informative but a little odd in one respect. It uses air-quotes to introduce concepts like "rack" and "shelf" in a DC, but does O(logn) notation mid-flow.

If you don't know what a rack is, how are you meant to know what the Order of scaling function means? Thats a highly computer sciences specific notation, and if you grok O(n) you know what a rack, a host, a DC is.

mtejo 3 years ago | |

Hah fair point, I was trying to write for a more general audience. Part of blogging for me is to improve my written communication skills. Appreciate this!

ggm 3 years ago | | |

Something like "scales linearly" or "scales exponentially" probably covers it, for whatever case it was (I forget)

really good article btw. really enjoyed reading it.

might pay to flag when systems you talk about are twitter-internal or are open-source. People love that kind of thing. "wow: twitter uses the gnu C compiler" type thing.

housecarpenter 3 years ago | |

I get your point that both of these are specialized notations/terminologies, but it's entirely possible for somebody to understand algorithmic complexity without knowing what "a rack, a host, a DC" is, there's no inherent connection between the two. As somebody whose interests tend to be more theoretical than practical, myself prior to reading the article would be an example.

ggm 3 years ago | | |

As the OP said "I was trying to write for a more general audience."

irrational 3 years ago |

Very interesting article. But the constant misuse of “to” for “too” kept throwing me out of reading mode. I wonder why my brain does that instead of reading past it since I know what is actually meant.

blamazon 3 years ago | |

Interesting, I didn't notice despite reading the article start too finish. The brain is so mysterious!

stoniejohnson 3 years ago | | |

Irony!

smegsicle 3 years ago | |

> "to many" - 5 matches

he did get it right one time though

mtejo 3 years ago | | |

haha I'll go fix that. sorry :(

mradek 3 years ago |

Tangent but I do hope that Musks trim down results in orgs that have less “executives” and a layered cake of a org structure, and more autonomous small teams that execute on shared overarching initiatives.

I really don’t understand why so many tech companies have like 8 layers of engineering levels. If the argument is that you need more money so more levels, just have a bigger band. Don’t chase titles they don’t mean shit. Not to mention the management stacks that seem to just hang out in meetings and take pvt. I haven’t worked at a proper startup but I’ve been on projects where a dozen or so people rebuilt apps used by tens of millions of people in a few months, or launched completely new applications for bigger companies.

Now that I work in a tech company (not big tech, but still a multibillion dollar corp) I’ve noticed that since IPO we have added a ton of bureaucracy whereas back in the day we were small teams building completely new and at times complex features. Literally was in a meeting earlier this month where people were patting each other on the back because we added a single attribute to a table. I’m obviously reducing the entire initiative to a small thing but it kind of explains all we had to do. It’s soul crushing but with the economy the way it is I must deal with it. Hell I’m down to show up next Monday and work for Elon if he wants a go getter. At least I’ll get to DO stuff. Or any other startup in SF if they are hiring.

wnevets 3 years ago |

I know it became a meme but did people actually think it would just crash overnight? Aren't a lot of people on hacker news in tech?

uluyol 3 years ago |

I wonder how big Twitter's infra costs are (capex). Based on the article, they need to build capacity > 2x peak loaf to tolerate one of the DCs failing. But if they instead had 4 and still only tolerated 1 failure, they'd only need capacity > 1.33x peak load.

This doesn't account for the extra overhead associated with extra DCs, but it seems like there's opportunities for major effeciency wins.

bigtones 3 years ago | |

Well Elon is planning on shutting down one of their three data centers, the Sacramento data center - so they must have a lot of extra capacity.

https://www.datacenterdynamics.com/en/news/report-elon-musk-...

midoridensha 3 years ago | | |

Go Elon! Shutting down a datacenter will save a lot of energy. I applaud this move. (Though I suppose they could just sell it to some other company.)

Now he just needs to shut down the other 2.

hot_gril 3 years ago |

You can usually do more with less if the remaining few are up to the challenge. It will come at some cost, like having to "move fast and break things" more. There can even be a positive side if the previous situation had a lot of friction due to "too many cooks," as I've experienced on several teams. What's more worrisome is the alleged 80hr work weeks. Yes, two SWEs on a team aren't as good as one SWE working double hours, since SWE-time is known not to scale that way. But burning people out, aside from being unethical, is risky when nobody is ready to fill the spot.

Musk has a point, but it feels more like he's doing this out of financial desperation.

I'll bet it starts going down a bit more often, nothing too severe. Bigger issue might be inability to roll out new features, especially if they dig a horrible tech debt hole. Maybe someone has more details, I haven't kept up, but it seems like the $8 verification rollout got botched because they cut corners on actually verifying the subscribers.

justahuman74 3 years ago | |

> cut corners on actually verifying the subscribers

Did they actually verify anything other than the ability to pay $8? It seems wild to me that they thought it would work out just fine

blisterpeanuts 3 years ago | | |

Feels like one of those "ideas that sounded good in the conference room" that turns out to have unintended consequences.

Hopefully they can fix this and have an effective way to verify identity... or else just give it up and go back to the old, creaky, somewhat effective method since people are used to it.

ineedasername 3 years ago |

I wasn't really expecting a large failure quite yet. I'm guessing there are enough people left with secondary or tertiary knowledge of systems to keep things ticking over for now. The question is how long that state of affairs can continue before some combination of worker burnout or deferred maintenance will stretch things too thin. Then, all-hands emergency work to get things running could contribute even more to burnout -> resignations etc.

They could stay ahead of things if they hire into areas related to infra maintenance that got hit the hardest, before issues reach the point of cascading failure. Or maybe put everything not critical to keeping the site up & running on hold as remaining devs get a bit of cross training using the limited people w/ institutional knowledge that remain as trainers.

For myself, I don't see much value in Twitter, in terms of net social value. It's format seems pretty much designed for only the most surface level discussions, part of what I believe leads to some of its toxicity: it's simply too hard to have conversations complex enough to invite enough discourse for that to tip the productive/toxic ratio a bit more positive.

So I've been rooting for Twitter to fail for years. I'm not rooting against Musk: I like his other businesses and at least a portion of their ongoing success is tied to his persona. (At least before the Twitter stuff: His actions there may impact his relationship with major banks that his other businesses will rely on, and Musk & JP Morgan, a bank that wasn't included in the deal & therefore won't be hurt by it, is already on somewhat negative terms with Musk. The twitter deal has added a few more to that list, and other banks have undoubtedly taken note. Some bank will always finance the regular sorts of things any large corporation need for him, but they're all going to be pricing in some additional risk. That's not really a big deal, it's just that I think Musk's persona has previously been a net positive for his companies and now it's lost at least a little bit of that.)

yieldcrv 3 years ago |

I don't think it's that crazy. Private Equity and Wall Street doesn't have pressure for resilience, it has pressure for churn. As long as revenue and spending... exists... things are fine. Ideally they are numbers that grow. This doesn't lend itself to clearing out a development backlog, or engineers doing the most important things, it lends itself to rapid iteration and justifications for the iterations.

So now that it is owned by someone that doesn't need churn and just needs to reduce cost, people can focus on discreet resilience factors, just like any small tightly held software operation. Many of Twitter's pet projects go away, the ad sales relationship have to get re-evaluated too, but the core consumer product that everyone sees can be made resilient and operate cheaper.

a-dub 3 years ago |

i think i prefer a culture where one is rewarded for automating their job.

moves like musk's are bad for the industry. they produce more systems with "job security" built in.

TravHatesMe 3 years ago | |

> more systems with "job security" built in.

How common is this?

AtlasBarfed 3 years ago |

Twitter is a lot of drama, but ultimately this is a common private equity play:

1) take private

2) fire/layoff until things break, patch up / rehire with cheaper labor. Repeat as necessary

3) spit shine/repackage what's left with a theoretically more appealing balance sheet.

4) resell to another sucker...uh, buyer, or take public again.

Fundamentally Musk bought 44 billion for 5 billion of annual revenue, and presumably 5 billion in costs. Unlikely to add revenue in twitter's model, he's cutting costs.

Honestly, at this point, is there revolutionary technology that is needed to keep the lights on at twitter? Do you need graphQL gods and SREs that would ace any Amazon raising the bar and master of silicon valley interviewing?

Nah. He'll honestly probably outsource a ton of the upkeep.

It's ugly, but the Twitter board and shareholders took their money and ran, and abandoned the product and the workforce. They could have backed out and let Elon Musk off the hook of his dumb contract with them, but they just wanted the money, and sold to someone that just wants to get his money out of it too.

dehrmann 3 years ago | |

There are a few key differences. PE tries not to massively overpay, it'll put more effort into targeted layoffs, and it'll maintain enough of a "business as usual" posture to not scare away advertisers. Edit: they also put in a CEO whose full-time job is being CEO of that company.

But yes, this is very bad execution of the PE playbook.

midoridensha 3 years ago | | |

Elon thinks he's brilliant at everything he tries his hand at, so he thought he'd jump into the private-equity arena and show everyone how brilliant he is.

usrusr 3 years ago | |

4) who'd that sucker be? How much hubris would it take to sell investors on the idea of salvaging a failing company that "the guy who killed the internal combustion engine" and "the guy who made rockets land on their tail" couldn't make profitable? If it comes to selling off a failure, Musk's reputation will be quite a dealbreaker. The days of whatever happens, if all else fails we can still sell to Yahoo are long gone. Perhaps China might be interested, for obvious reasons, but surely not after Twitter has gone down fighting, fading away into irrelevancy in the process.

AtlasBarfed 3 years ago | | |

If Elon holds the line on advertising and drops costs such that Twitter turns two billion in profit, they yeah the narrative in the business circles will be that Elon trimmed the fat and turned twitter into a viable property and someone with cash will buy it. I mean, Microsoft paid 8 billion for github! They bought Skype for 8.5 billion (probably like 12-20 billion these days), and THAT was after ebay bought skype for 2.6 billion (and none of the source code rights) in what was considered a vast overpay.

kaba0 3 years ago | |

I mean, they got an offer for roughly a 10x deal. At that point it was in the best interest of shareholders to sell. Especially that it was yet to turn a profit.

Like, I like my car, but would probably sell it for an order of magnitude its worth.

npn 3 years ago | |

Chinese developers are famous for making apps serving 1.4 billion people, with at most 1/3 salary of US, and the willingness to do 996 a whole year around.

I would be surprised that no company in the west take advanced of that.

AtlasBarfed 3 years ago | | |

The mythical man month exposed a wealth of problems in software development, only some of which have been addressed in the last 44 years (MMM was 1979, right?)

2000 miles and a language barrier do not help things.

kaba0 3 years ago | | |

1/3 the salary may well give you a better life quality in China, though.

redeyedtreefrog 3 years ago |

SpaceX and Tesla made affordable, reusable rockets, and affordable, high performance electric cars with a long range. Those things have obvious value, but required solving big engineering challenges to make them a reality. Musk doesn't seem to have any clear goal with twitter. There are vague ideas about making it subscription-funded rather than advertising-funded, and about having less censorship, but neither are things the majority of people want.

I imagine the site will mostly continue to more-or-less work, despite all the layoffs. They still have thousands of staff. The network effect of Twitter is so big that people will continue to use it even if fail whales become more common. Others suggest that it will soon crash and burn, or that Musk will get bored and sell it for a few billion. Or that Musk is a genius who will make some sort of amazing Twitter 2.0 that does for social media what Tesla did for electric cars. But without any appealing long term vision, and with an owner who bought it to satisfy their ego rather than with any real plan, the reality may be more boring. I imagine it will just languish for many years, with occasional manufactured drama, and occasional downtime, but no real innovation. Maybe to be eventually supplanted by something else in 10 years or so.

mattbrewsbytes 3 years ago | |

> Or that Musk is a genius who will make some sort of amazing Twitter 2.0

Based on texts exposed via the lawsuit about the purchase, I don't think this is the case. I don't think he, or his advisors, understand that the product at twitter (and other social media) is content moderation. You can have a vision of whatever type of content or pricing scheme you want but without solid content moderation you will lose advertisers, gain lawsuits (people were posting movies the other day) and lose users because the "feed" becomes a muddied mess. Users are only really the product when you can moderate their content to have profit via advertisers.

bushbaba 3 years ago |

If there’s no code changes, reliability is greatly increased. Probably 99% of all outages and incidents are from a bad code change.

Within most companies you don’t want to pause innovation.

PeterStuer 3 years ago |

In every software company of over 50 employees I have seen up close it would be fairly to identify 50% of people that had minimal to negative contribution.

That said, it takes some time to get to know the real workings of a company, knowledge required to select the right 50%.

That said, afaict the current mo is keep the lights on for the current offering while a new team builds a new product for that enormous user base that offers something much more profitable.

YokoZar 3 years ago |

> drove some large cost savings projects

Having worked as SRE on efficiency projects for BigCo, it is not at all uncommon to recover more than your lifetime salary in company savings with only a few months work or even less. The scale of things is so immense that even a slightly better handling of things can lead to outsize returns.

Laying off someone like that, rather than putting them on full-time efficiency work, is an obvious waste.

andy_ppp 3 years ago |

There seems to be a load of people in this thread making statements on both sides that a) the article doesn’t prove there were too many people at twitter b) the article implies there were too many people at Twitter c) Twitter is a simple app why so complicated etc.

I would say this; Twitter at this scale is extremely complicated. We don’t have enough information to know what all these roles were doing and how many were truly superfluous. The article gives an interesting insight into something extremely complicated- it might be a slice repeated in a lot of areas. I’m of the opinion that you can do a lot more with less but there’s no need to send emails to everyone in the organisation being an arsehole and essentially accusing them of being lazy, then rehiring people who never worked there just for the meme.

Anyway Musk will blow up some rockets, put enormous pressure on people to fix them, and come up smelling of roses again even if he treats a load of people like crap along the way. His fans will say he’s a genius and his enemies will say he’s a dickhead. And so it goes.

raspberry1337 3 years ago |

Musk makes leftists lose their shit and all rational functioning, even on Hacker News; that is my reflection.

socialismisok 3 years ago | |

Not just leftists. His fans are equally out of touch in the other direction.

tzm 3 years ago |

Looks like 28% of Twitter employees took the offer for 3 months of severance.. not the rumored 75% or 80%.

yalogin 3 years ago |

The reason why Twitter hasn’t crashed is it’s well written and well orchestrated. Once a bug comes in and crashes something that is when the chaos starts. It’s almost a guarantee that the current crop do not know how to fix the bug. It will be interesting to see how they handle that.

trollied 3 years ago | |

> It’s almost a guarantee that the current crop do not know how to fix the bug

Seeing lots of comments like this. Why do you feel it’s ok to say that, when you don’t actually work there or have intimate knowledge of who is currently working there? The pure speculation and nonsense surrounding Twitter at the moment is plain awful.

yalogin 3 years ago | | |

It’s just statistical probability. The number of employees left is small and the amount of code is huge. It’s not a huge leap to say that a bug is unlikely to be in the area of expertise of the folks that are left. I would say it’s the probability is greater than 50%. You should notice the probability is not proportional to the employees still remaining. That is because I am guessing some good folks are still there.

Marazan 3 years ago |

Once thing I've seen that I think is overlooked in the whole "is Twitter's tech stack inefficient" chat is the nature of the service.

A lot of the "Twitter could be run off my laptop" style comments seem to come from people who run, effectively, Read-Only services. They might serve data at thousands of queries per second but the data itself is _slow_. It is video files or music streams or other data that updates infrequently or, if there is dynamic data involved, it comes from a very small number of fixed sources.

Twitter deals with thousands of posts per second that are subject to huge fluctuations in the density _and_ those posts have to be disseminated to the millions of users. Twitter is a two sided problem.

jiggawatts 3 years ago | |

I've heard this elsewhere, and when Twitter was originally founded in 2006 it wouldn't have been a particularly hard problem.

Twitter processes roughly 10K tweets per second. Even if you bloat out the text quite a lot to account for encoding overheads, metadata, etc, etc... and assume that each one is 10KB, then this is just 1 Gbps. A single NIC on an old server.

Okay, I get it, Twitter needs a lot of data too. Lists of users, etc...

Twitter has 450 million monthly active users, which sounds like a lot, but even if there's 1MB of profile data per user such as who they are following, that's just 400 TB.

That's... not that much these days. A large-but-not-enormous database cluster.

Sure, there's "historical" data, but that can be compressed and put on cheap cold storage, like S3 or whatever.

Give me a few million annually as an opex budget, a small team of decent developers, and I can guarantee you I could whip up a cloud-hosted service that can process tweets at Twitter's scale.

Obviously, what I can't replicate is the much larger set of tools and systems behind the scenes that are used for moderation, analytics, ad sales, etc...

Marazan 3 years ago | | |

That's looking at Twitter from the other end, as a Write-Only service with non-realtime requirements on the read layer.

Twitter is both sides. Tweets are replicated out in milliseconds. Twitter search is lightning fast, comprehensiveband powerful. o Old tweets are not just glaciered into S3!

csomar 3 years ago |

Did the OP confirm Elon theory that most of the stuff is not needed?

> For four of those years I was the sole SRE for the Cache team. There was a few before me, and the whole team I worked with, where a bunch came and went. But for four years I was the one responsible for automation, reliability and operations in the team. I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it. (There might be only one or two other people)

If you only need one person for the caching department (which is, as I understand, is critical as it delivers most of the data); then maybe you need a handful other dozen engineers and there you have a functional Twitter.

That or the OP is full of himself. Kinda like Musk?

vba616 3 years ago | |

I can believe that a dozen talented engineers could in principle suffice for Twitter.

But who believes those 12 engineers still work there? The author of this specific item is in fact not there any more.

And a lot of other people are needed to bring in revenue, don't you think? Nobody is paying for a beautiful caching system.

It's like if I doubled my weight in the last ten years. Half of me is bloat, and yet, there is no possibility bisection will improve my health.

mapme 3 years ago | |

One SRE, many SWE. Also have fun asking someone to be permanently oncall with one person on the team.

The cache clusters size are also described here for anyone who wants a good technical read over speculation. https://www.usenix.org/system/files/osdi20-yang.pdf

csomar 3 years ago | | |

The OP claims he did the implementation (so he was the software engineer too?):

> I designed and implemented most of the tools that are keeping it running so I think I’m qualified to talk about it.

okdood64 3 years ago | | |

SWEs can share the oncall rotation with SRE.

dsq 3 years ago |

Its possible to teach a horse to eat less and less each day. They just have this unfortunate tendency to die just after you succeed in weaning them off food altogether.

A horse, of course, is not same as a reduction in force.

Engineers don't eat hay, neither does a horse have a 401(k).

kar1181 3 years ago |

For all the hand wringing about keeping the lights on, Twitter more than doubled its headcount in 3 n bit years. I'm not sure anyone knows how much of a skeleton crew is required to keep things ticking along anymore given that sort of hiring binge.

rubiquity 3 years ago |

Great post. Thanks for the insight on how mature Twitter’s Cache infrastructure tooling is. Having done similar work at a different place where you’re exposed to the realities of racks and racks of machines I could appreciate the evolution of the stack and knocking off problems one by one.

I suspected that the hopes and fears of Twitter’s quick demise were overstated and shared[0] some of those thoughts last week, on Twitter of course.

Also hi from another infrastructure person in SD!

0 - https://twitter.com/bitsandhops/status/1593637241527578625

mkl95 3 years ago |

I have seen this happen twice at a smaller scale. Turnover is too high quantitatively and qualitatively to survive by hiring new people, so Musk's "solution" will probably be to oursource a lot of stuff to some consultancy firm. Twitter will almost inevitably rot within 5 years. But don't expect anything noticeable overnight.

Edit: also the reason Musk wants to shut down all those microservices has nothing to do with money, it's that he doesn't have enough manpower to maintain them. If Twitter is like most non FAANG out there those "microservices" are coupled with each other, so good luck.

dboreham 3 years ago |

...because it's only been a week and it takes longer for things to rot badly?

jdminhbg 3 years ago | |

I think this post is aimed at non-technical "RIP Twitter" posters from last week who are surprised it didn't fall apart in 24 hours.

npn 3 years ago | | |

There was a long twitter thread from some one proclaimed 10+ years SRE listing a lot of things that could bring the server down. Lot of them are pretty far-fetched and outdated.

Too bad I did not bother to save the link.

randyrand 3 years ago | |

servers don’t really rot if they don’t rely on 3rd party cloud software.

Linux doesn’t upgrade itself.

the web is very standards based.

Hacker news has pretty much been running for decades on the same software hardware stack

kaba0 3 years ago | | |

I’m fairly sure HN had outages as well.

treme 3 years ago |

turns out CEO with exceptional track record & success in 2 different industries is some what competent.

I'm pretty surprised to see the occasional redditesque "Elon is an idiot" rhetoric on hn tho.

bearmode 3 years ago | |

In spite of him, not because of him.

superdug 3 years ago |

I will just wait for the bug the breaks the camels back here.

There will be a necessary breaking change that there will be no support for that will cascade into the downtime the media is demanding at the moment.

textbookrental 3 years ago |

I guess I am just a classic HN reader, so I wonder why Twitter needed so many people.

Also, I find it pretty interesting how this is playing out so far.

I'm starting to think that Mr. Enron Musk is going to pull this off.

Kye 3 years ago |

>> "How the cache’s keep running"

It's a little thing, but it's the little things that cascade into big things that require teams of people to fix. Here, we see this person who's quite proud of his automations making the simple, but harmless error of using a possessive instead of a plural. Humans are good at reading through this sort of thing.

Computers are not. Somewhere deep in all his scripts is a misplaced ', `, or ’ waiting to break something.

ellen364 3 years ago |

Linear extrapolation seems to be a common theme in the comments. People take a product they know and then essentially say “Twitter only has X times as many users, surely they only need X times as many engineers. The other employees were excess!”

I haven’t worked on Twitter-scale products. But I do wonder if it’s safe to draw a straight line between services that have middling and large scales. Things don’t necessarily work the same at the extremes.

SMAAART 3 years ago |

> if you have a cluster of servers where serving 1000 requests a second might cost $1000, you can instead use the cache to store the responses and serve it from that cache server instead. Then you would have a small cluster for $100 and a cheap and large cache cluster of servers maybe for another $100. The numbers are just examples to illustrate the point.

Reddit needs to implement this if they want to get to the next level.

pie314isi 3 years ago |

So, this? https://www.youtube.com/watch?v=3ztFaYvgDAg

rvba 3 years ago |

> When I joined the team the first project I had was to swap old machines that were being retired for new machines. There were no tools or automation to do this, I was given a spreadsheet with server names.

Writing a program to store a list of servers to be swapped instead of keeping them in a spreadsheet sounds a bit like buying a brewery when you want to drink 1 beer. Program used by a team of one sounds like over-engineering.

ojosilva 3 years ago | |

This spreadsheet was, from the OPs narrative, the master-list used by the whole operations team. An operations team that is now reduced to "ashes" - therefore not a bad idea at all to have your server inventory centralized.

AtNightWeCode 3 years ago |

Anybody working in the biz knows about technical debt. At Twitter that will skyrocket now. My firm believe and experience tells me that keeping everything somewhat up to date over time always pays off.

The problem is not that things will break. Things break ALL the time. The problem is more about avoiding cascading effects and the time it takes to fix stuff.

bawolff 3 years ago |

I think it would be more surprising if it did go down immediately.

Presumably in the turmoil nobody is deploying new code. Sure shit happens, but i imagine that twitter is mature enough that they aren't having weekly critical incidents, especially during times where nothing is changing.

I still think twitter is going to have some disaster, but give it a few weeks.

clayhacks 3 years ago |

He says at the end his automation allowed him to automate more. A great positive feedback loop. I see the inverse is also true. Where having no automation makes automating new things harder. Without enough engineers I think they will hit some issue and just not have the bandwidth to fix it in a reasonable amount of time

dchia 3 years ago |

Why do people assume that things would break so easily? I'm pretty sure Twitter (ex)team did tons of scalability testing with loads that are 10x or even 100x. Plus random 100x to 1000x spikes to account for breaking news. And all of these are done automatically, constantly, and consistently over the years.

aryehof 3 years ago |

There will always be those that think that Twitter can be written by a single programmer in a day or so. How hard can it be for such a simple idea after all, given that there are hundred of tutorials producing a version in just about every language just a search away.

Keep it running? One part-time should surely do it?

csmpltn 3 years ago |

Why Twitter didn’t go down, you're asking. Short answer? H1Bs: https://twitter.com/elonmusk/status/1593899029531803649/phot...

new2yc 3 years ago |

Someone please post this as a win for open source, nsidb. I don't know how

https://lichess.org/blog/Y3u1mRAAACIApBVn/settlement-reached...

Existenceblinks 3 years ago |

Well, debugging a company in prod mode like this is adventurous. Just like we remove code to see if things break, and put it back piece by piece until it's not. That's how some of us get rid of bloat. (Yes, I'm aware of orange and apple comparison)

mesozoic 3 years ago |

As an SRE I know when people aren't working things rarely break (not ever but much more rare)

karol 3 years ago |

I don't understand why would a mature platform just fall down without its engineering team. There are surely a lot of small fires but 80/20 rule with regards to triaging should keep the lights on when properly applied.

Speaking as an experienced single tech founder.

Nokinside 3 years ago |

More than 50% of large distributed system downtime is caused by configuration errors, failed software updates, or network attacks.

With configuration error, or failed software update everything can go down at once, or there are cascades of restarts that kill the system.

notyourday 3 years ago |

News at 11: people who waste 90% of their work time on HN while collecting a-la FAANG salaries because they are used to bullshitting their management and getting away with it are terrified of the management that can look at the code.

tiffanyh 3 years ago |

Fail Whale.

I think people just miss seeing the fail-whale graphic.

https://www.techopedia.com/definition/1987/fail-whale

bdcravens 3 years ago |

I wonder if the Twitter situation may be the event that causes the industry to push back on large technical organization sizes.

I wonder how many reading this are starting to feel nervous about their own roles.

newbieuser 3 years ago |

Considering the maintenance cost of a twitter-scale site, it is inevitable that even a very small-scale problem they will experience will become much larger with the snowball effect.

b99andla 3 years ago |

Why would twitter go down due to staff layoffs? The stuff I build would keep running for years without my involvement. Do you mean that other systems are hand cranked?

squegles 3 years ago |

Not sure when OP left, but Twitter has 3 datacenters now.

mtejo 3 years ago | |

Over the summer, Yeah there is three but I didn't think the 3rd was really running yet. It was kind of just there with some small things dripping in.

dekhn 3 years ago |

The article didn't address what happens when Mesos goes down. How much of the team running the underlying cluster infrastructure remains?

nokeya 3 years ago |

So, from 7500 people only 1 (ONE) person was responsible for cache, one of the most crucial parts of the service?

octodog 3 years ago |

Thanks for the write-up OP! Appreciate it.

totorovirus 3 years ago |

If Twitter survives and people shut up their mouths complaining after few months we'll see who was right.

EagnaIonat 3 years ago | |

It's already breaking down. HW/SW aside, a lot of the services had teams monitoring them, which don't exist anymore.

Some of the side effects of that:

- Trends are not working correctly.

- Copyright reporting is not working.

- Appealing flagged tweets isn't being responded to.

It's only a matter of time before these get abused with no one to fix them.

NaturalPhallacy 3 years ago | | |

>- Trends are not working correctly.

"Trends" were subject to Twitter's "trends blacklist" before; something that leaked a few years ago. Maybe they're working correctly now that they're unencumbered. Can you describe how they're not working now?

>- Appealing flagged tweets isn't being responded to.

Mine was responded to in a handful of hours. Much faster than I was expecting.

>It's only a matter of time before these get abused with no one to fix them.

"the walls are closing in" - https://www.youtube.com/watch?v=rLEchPZm318

ReptileMan 3 years ago | |

And then the other big tech companies will take note. I personally know of highly compensated teams in VMware that do almost nothing.

eshork 3 years ago |

I read ~60% of your post, and noticed it repeats, so I stopped reading. Not sure if you are aware

Giorgi 3 years ago |

It did not go down because huge companies don't usually go due to reddit woke culture.

npn 3 years ago |

Let don't kid ourselves here, even if Twitter crashes and needs a few weeks or even a few months to recover it doesn't matter. People who leaved will comeback again, after a year everything will be like nothing ever happened.

And even if Twitter lost 50% of it users and revenues, it is still more profitable than have to pay the salary of 7000 people.

kaba0 3 years ago | |

More profitable than what? It was and is losing money, plus now it also has some serious debt on top. With staff that actually know their shit (and a sane CEO), it may be possible to turn it profitable, but I don’t think it will happen as things currently stand.

npn 3 years ago | | |

7000 people x $300k = $2.1B And that's salary alone. The operation cost to serve all those people is a lot more.

Twitter revenue last year was around $5B.

uwagar 3 years ago |

automation is indeed key. IRC networks run with no obvious employees and salarys and been running fine since the dawn of the internet. i see Twitter as a centralised, walled IRC. the # was pinched from IRC if u remember.

kaba0 3 years ago | |

An IRC network server can serve what, a few hundred thousands of concurrent users’ text messages on a single PC.

Twitter has around 200 million daily user, peaking up to a billion, running on their own server farm. They are not remotely comparable, especially if we note that read-write dbs are notoriously hard to scale.

uwagar 3 years ago | | |

thank you, i was trying to be provocative to learn more.

otoh, to get an automation going is like climbing mount everest the first time? very hard but makes it easier to scale subsequent attempts?

reacharavindh 3 years ago |

Has anybody suggested the idea of rewriting twitter in Rust yet to Elon? Memory safe, needing fewer servers(hence less $), and the story sells itself to someone like Elon who jumps on such impulses...

^ is a snark. I'm not suggesting that they should..

nigamanth 3 years ago |

Twitter won't go down that easily. Besides, Elon Musk hasn't bought it for no reason. I'm sure that he has his plans with Twitter to implement new features and whatnot to restore it.

Till now, he's fired a lot of engineers probably because:

a) he doesn't need so many. b) he thinks he can do the job himself.

randomsearch 3 years ago |

Is it essential for Twitter to run their own datacentres?

lolsoftware 3 years ago |

I'm curious how/if the ostensible decline in active users is affecting things, and to what degree.

user_ 3 years ago | |

> Twitter added 1.6M daily active users this past week, another all-time high

https://twitter.com/elonmusk/status/1594865247323660290

datalopers 3 years ago | | |

Interesting how this summer all we heard from Musk was how Twitter was overflowing with bots and spam, but now that he runs the place he sure spends a lot of time bragging about all-time high "user" counts.

acdha 3 years ago | | |

Want to bet that’s not unconnected to sacking the people who tried to block bots?

kaba0 3 years ago | | |

Catastrophe tourism. Also, contrary to Elon’s shitty graph with cut off y-axis (can we stop doing that?), that’s like a 1% change around a major American event.

dehrmann 3 years ago | | |

Louis Rossmann said something like "there's a difference between people showing up to laugh at you vs. laugh with you."

throwaway5371 3 years ago |

is twitter going to be profitable if it can run with 10% of the workforce?

eBombzor 3 years ago |

Was this a thing people actually believed? Other than the typical room temperature iq Twitter user.

NaturalPhallacy 3 years ago |

All the people saying twitter is now dying remind me of "the walls closing in" on Trump, who is running for president in 2024: https://www.youtube.com/watch?v=rLEchPZm318

fragmede 3 years ago |

The "why hasn’t the site gone down" stuff is the wrong conversation. The engineers built a train engine, and this robber baron held up the train and forced the engineers off. Inertia will keep the train going for a good long while because they did such a good job of building the train.

At some point the engin’s fluids will run out and need topping off. There's a warning system though so someone'll get notified before the engine blows up, so it's entirely possible this hardcore crew of H1Bs pulls it off. (Great job replacing the SSL cert!) At some point the train’s boiler will need retrofitting. A new team could successfully replace the boiler, but doing it while the engine keeps running just isn't an easy job.

The handful of devs and SREs at Twitter will do their damnedest to keep the train, err, site running. They might even succeed. That doesn’t prove the site was over-engineered, it means they built a damn fine train engined that successfully pulled the whole train up and over the mountain pass.

Twitter wasn't run like a lean start-up type business. Because wasn’t one. Society needs better than that. The vulture capitalism mindset is what's wrong with this county. What's the barest minimum I can pay people to work for me? Not; what can I pay them to let them live prosperous lives but what's the bare, subsistence minimum.

America needs a middle class and these people were part of it. You don’t get a middle class by injudiciously paring things down to a not-even-skeleton crew that’s going to be worked till they burn out, only to get fired.

Twitter was only $14 million from profitability in Q2, and any idiot can randomly fire people to find that much in that large an organization, but cutting costs like that is to misunderstand the situation entirely. Just like starving yourself isn't healthy and won’t improve your self image, randomly cutting costs like that saves money, but doesn’t result in a healthy business.

fatneckbeardz 3 years ago |

ever notice that the truly competent people can explain things in really simple language?

yuppie_scum 3 years ago |

I hope they consider moving to Kubernetes from Mesos.

rubiquity 3 years ago | |

Why

wankle 3 years ago |

I can't help but feel it might not be legal to post all those details.

mk_stjames 3 years ago | |

Out of all the comments here, I can't believe this is the only one that shares my sentiment. I'm not a software engineer, but in my area of engineering if I ever revealed anything publicly as detailed as this kind of architecture; for instance, the failure modes effect analysis framework use at a certain aerospace company... or details of the simulation models involving the crash analysis at a certain car company, so soon after departing one of those companies, I'd expect a letter from a lawyer within a week. In most places I've worked I don't think I could even post anything on the internet invoking the name of the company unless it was a repost of some public information the company itself had previously released. Do the vast majority of people here on HN/in Silicon Valley not have 5-year NDA's? Or do they and they just not care?

asutekku 3 years ago | |

Well twitter (allegedly) does not have a legal team anymore so ¯\_(ツ)_/¯

HeyLaughingBoy 3 years ago | |

What law do you think is being broken?

encryptluks2 3 years ago | | |

Breach of contract possibly. Usually companies don't want you revealing their internal infrastructure publicly unless you get permission. Who knows though.

randyrand 3 years ago | |

given that it’s showing Twitter in good light, it would be super silly to pursue anything IMO, even if it were illegal.

What would be a realistic fine in terms of damages (in $$)?

You could argue its actually negative damages.

HeavyStorm 3 years ago |

Objectively, the article proves that there was bloat since all things are automated.

tyiz 3 years ago |

She mentioned you: https://youtu.be/g-voQsFY6SE

She was assistant to the gender health support board.

zapdrive 3 years ago |

I know most of my comment is going to be redundant, but I need to say it in order to point to it an year or two from now as an "I told you so!"

Twitter is going to be here for a long long time. It's not going to suddenly shut down, it's not going to slowly decline, there is not going to be mass abandonment.

Elon might be cocky, but at the end of the day he is a successful businessman. He didn't just sent out that loyalty pledge email out of cockyness. He really wanted to kick out everyone who doesn't believe in him and his beliefs. And he sent that email well after understanding how Twitter runs and putting his loyal people from Tesla ans SpaceX in charge of operations.

The woke crowd needs to get out of denial and start coping with the fact that Twitter is not the bastion of unopposed woke ideology anymore.

kaba0 3 years ago | |

He doesn’t understand shit, you don’t fire that many people before even having as crude of an understanding as was seen on that whiteboard. Also, people working on embedded systems for rockets know jack-shit about a system serving petabytes of data.

faizmokhtar 3 years ago | |

What kind of woke ideology exactly? Sorry not familiar the current climate in USA.

acknorabotr 3 years ago | | |

It's a form of progressive ideology that is largely based on symbolic actions and relies on mechanisms like shame and shunning, struggle sessions, etc.

So instead of actually doing any sort of real tangible action to solve problems, you mostly spend your time denouncing your enemies, including the ones in your own ranks that are not pure enough.

It's very much associated with the left-wing parties in many Western countries, who have for the most part traded their historical defense of working classes for this new brand of performative progressive politics.

linuxhansl 3 years ago | | |

Definition of woke: alert to injustice in society, especially racism

Somehow, some folks are now saying this is bad.

fractalb 3 years ago |

Twitter will soon experience the negative effect of the mass layoffs and die slowly. If it survives, please don't ascribe it to Elon's management skills. Don't forget the employees who are stretching themselves there to rescue Twitter.