Twitter was down(twitter.com) |
Twitter was down(twitter.com) |
I won't argue that firing all the engineers so swiftly wasn't a mistake (I mean I _could_, but don't want to dig myself any deeper than necessary in a single post). What I will say though, is that this gives me hints as to the quality of engineering that went on at Twitter.
I feel like some stellar Rube Goldberging went into solving problems that could have used simpler, if not _less CV boosting_, solutions. I've been at places like this. The engineers have fun building convoluted "solutions" for the sake of saying they worked on cool-tech-x. And surprise, it needs endless maintenance.
I've become accustomed to the fact that it's beyond clear that pre-Musk engineers pushed changes to the servers live without testing them first, and didn't even bother doing an incremental roll-out either. Either they had no engineering culture, or that culture was gutted long before he arrived, because Twitter had problems with frightening regularity.
Before it is all '#hugops' and 'best wishes to the engineers at Twitter' when it had downtime before the acquisition. But as you can see, they all love to hate it because a person called Elon R. Musk owns the blue bird site.
Just like when everyone here thought that Facebook, Instagram, WhatsApp all collapsed and went down for hours it was declared as the 'end of times' the 'death of Meta', etc, with it's network effect still unaffected by that.
The same thing is happening for Twitter as even when it goes down, it's network effect of 220M+ users is still sitting there waiting for it to get back up.
for reference: I see absolutely none of the things anybody is saying they are seeing as errors. Twitter works exactly as it ever has for me.
The latter is a kind of anecdotal evidence that does have no value in discussions like this. The linked post is from the official Twitter account stating that things are broken and it is for many people. If you don't see it this could be due to many conditions on why it might seem working on your device/client/account. This does not lead to any logical conclusion that these information might be wrong.
At Twitter's scale with the number of API users they had, I can't imagine how they'd filter the signal out of the noise.
This is very much a "someone would have had to know we're using our own API to avoid this disaster" scenario, which is an institutional knowledge problem. The kind of problem that gets hard to solve when XX% of your workforce suddenly left to seek new opportunities.
0 engineers to prevent this from happening
I can access it from my mobile browser.
Farewell?
https://twitter.com/CaseyExplosion/status/101855884867059302...
This is "Elon Musk is deliberately mandating bad software engineering practices". I will decline to speculate whether that is out of malice or incompetence.
That 467 error code suggests to me something else is going on and Twitter is emitting the wrong error message (467 isn't a standard HTTP, so I'm wondering if something lower-level emitted a 467 error, it was misinterpreted as an HTTP status, and the error string chooser logic is giving that API access error message due to fall-through).
I'd never used tweetdeck prior to now. Yet, it just now loaded fine for me. Once it loaded, I clicked a button that said something like, "setup some fields", and then tweets were displayed. What's not working? I'm just a free user of twitter.
Or... it was broken for normal users and is now fixed.
The newspaper said there was a fire at this building? Yet, it looks fine to me now?
"Disable them and see what happens."
Obviously, that's paranoid thinking. But the results are the same in many cases.
A market for lemons developing. A reinforcement of the strategy for others. And more stressful things in our lives.
If people are resentful, I understand. I hope they can find peaceful communities and safety wherever they go. I'm sure there will be political or technical reforms to accomplish that.
I hope people remember in the end that there were beautiful and good people in different spaces on the Internet, including Twitter.
I hope I can choose my own story.
Good to know that I can rely on H/N to tell me what's really going on... and it's not "just me".
One reason the site was stable pre-Musk is that no one made any changes to it. For better or worse, it's now owned by someone who actually uses the product and has ideas for how to develop it.
From a business perspective does it matter that Twitter is down for a moment? It's not a plane or medical machine. It's not blocking anyone from doing anything useful. Everyone are just going to check & post when it is up again.
It also doesn't mean that Twitter employees have a bad work experience like most here assume. Maybe they've got a permission to iterate fast, even if it risks bringing things down, and everyone are having a jolly good time, skipping on the boring chores ensuring everything will go smoothly usually takes.
Is down too, at this moment.
What a colossal piece of shit. Everything there is information-free and radicalized. The only maybe useful thing is the result of some game, that I assume it's soccer. (But British soccer, why would I care about British soccer? The rest of the page is localized into Brazil.)
Looks like it's already being/has been fixed. Still, pretty impressive level of brokenness for a short time.
help.twitter.com, api.twitter.com, developer.twitter.com etc.
Before, it was "healthy for everyone" when Twitter went down (search hn.algolia.com for pre-Elon Twitter downtime threads)
Now it's an episode of Real Housewives just like any other Elon thread on HN.
I'm probably not the ideal person to write this down, there was so much stuff going on, some people probably made a whole blog-article series on this. This is just a few of the things that I'm able to remember right now.. and with everything on hn here it gives you some ideas and things to think about. Hope that Helps.
... mostly around the hope they can jump ship before the Titanic pulls them under.
The only cases I can think of for staying:
- Unfortunate H-1Bs tied to the job. (We really need to make it easier for these folks to switch jobs and keep their visa!)
- A feeling one needs the job and can't easily get work elsewhere. Perhaps living paycheck to paycheck, haven't interviewed in awhile, imposter syndrome... (If you're worried you'd not be able to get hired, instead ask why you weren't fired.)
Seems to be related to a query param in my case (if I remove the ?lang=en, the page loads properly).
* Not to mention the SSL certificate appears to be expired too.
———-
EDIT-01: Appears there’s a status page just for the API:
EDIT-02: Wow, Twitter is using Twitter to post status updates:
- https://twitter.com/TwitterSupport/status/163279294226274713...
Love that detail. Just the icing on the top that says "everybody who was vaguely aware of this particular infrastructure is gone."
(Assuming this is more of a the site’s not down unless the CTO says it is services.)
How odd.
"Some parts of Twitter may not be working as expected right now. We made an internal change that had some unintended consequences. We’re working on this now and will share an update when it’s fixed."
https://twitter.com/TwitterSupport/status/163279294226274713...
We felt bad for the engineers that are still there, and were actually surprised it's stayed up this long, with all the knowledge that walked out the door. Basic stuff like "where are our servers actually located".
I guess the folks that left loved Twitter enough to leave behind good notes, because even the most well architected systems still need some institutional knowledge when they go down.
[0] e.g. Elon: https://twitter.com/elonmusk
[1] https://twitter.com/elonmusk/status/1632574090216103937
I guess there's still one aspect of the company that Musk hasn't "fixed" yet...
https://www.technologyreview.com/2022/11/08/1062886/heres-ho...
But there's one more gotcha, a company can't just keep 10% - 40% and expect the service to still run, because it's rotating and oncalls can always reach out to other people on the team.
Twitter not only fired indiscriminately, with perverse pattern (more capable to find another job leave first), and didn't even keep the minimum to keep the lights on.
From a software engineering perspective, it's doomed to fail, maybe it can retract to a smaller footprint (abandoning features/stacks) and still work, but that would also be pretty hard. That, and I don't think anyone not desperate will apply for a job there in the foreseeable future.
https://twitter.com/emollick/status/1632788892599586816
> The Twitter outage seems to be around a critical piece of infrastructure: the Twitter link shortener.
> Every link on Twitter has been shortened to a t.co link. Only Twitter’s systems know where the link goes
> If these systems ever go down, its link rot at scale
Only if one's mental model is that t.co links were ever permanent.
They're privately-owned obfuscator links. Unless someone in the wild was building a mirror mapping, they were always doomed and never to be trusted; they knocked the "U" right out of the "URL."
Just a steady degradation of service.
Lots of people were waiting with baited breath for Twitter to fail the first week Musk bought it, but most of the failure conditions had been automated away...
Instead the reliability loss creep happened more slowly. The people that understood the edge conditions in the system were fired. Then new changes need implemented and the old, now not understood systems, were ripped out and new pieces put in. And lo and behold they have tons of failure conditions and edge cases that can knock out the system.
Even if your programming team wrote immaculate documentation on all these edge conditions, it can take an immense amount of time to both read them, and then fully understand them, and when you have a micromanager breathing down your neck for changes 10 minutes ago this is what happens.
I don't spend enough time on Reddit for that to really resonate, but the badges and icons flying all over the place and talk of Twitter Gold (or whatever they're calling it) do feel very Reddit-esque these days.
I barely use the app anymore and half the time stuff isn't loading at all or is broken in obvious, overt ways.
Just ... stunning.
I still don't understand why every fifth post in my feed is an Elon tweet, I don't even follow him. I could block him but I find it interesting to see how much of his content is pushed toward me who hasn't asked for it.
I'm not a delicate flower when it comes to offense so I'd just roll my eyes when the first two kinds of tweets popped up. But seeing a grown man repeatedly trying and failing so hard at being a poster is a bit much to take.
Given all the random things Elon has cut from Twitter, it wouldn't surprise me if that included a staging environment.
1. The site would crash and burn the next day.
2. Nothing would change. All those engineers were anyways just sitting around.
3. There would be no immediate impact (servers can run by themselves after all), but the site would slowly degrade over time as institutional knowledge around maintenance, upkeep and all the various system quirks was gone.
We are now seeing #3 play out in front of us.
Just top 3 Mastodon instances(mastodon.social, pawoo.net, mstdn.jp) collectively has > 2mil users, same level as Twitter in 2008. Misskey, an ActivityPub/Mastodon compatible server software, now has >10K users. Nostr isn't much, Bluesky is still invite only and had just went past 2k registered, but those are all available options for direct substitutes to Twitter in case anyone still in need of a legacy twitterform social media.
You really think 'pawoo.net' and 'mstdn.jp' are fabulous advertisements for Mastodon of over 2M users on those instances, unless you think CP and loli culture there is a great thing since it is totally illegal in hundreds of countries?
Secondly, both pawoo.net and mstdn.jp instances are owned by a single company called Mask Network [0], which essentially means it is not community owned and is under central control of a company that can do whatever it pleases, even if the users on the platform dislike it.
Finally you missed out another large Mastodon instance after mastodon.social which is braag.et which promotes CP and illegal content. So it seems that the instances with the larger amount of users also allows CP and illegal garbage on it.
It's quite hilarious to believe that Mastodon with these examples remotely has a chance against Twitter as an alternative.
[0] https://masknetwork.medium.com/mask-network-acquires-pawoo-n...
Yes.
My understanding at this point is that, firstly, instances that do not federate with these, and those cheap “CP” accusations, has no relevance in this discussion, because nothing comes out of those.
Secondly, it seems completely plausible to me at this time, that a central Mastodon instance will become the core of a Western European social media landscape, in the way this wartime Russian Federation maintains VKontakte, or that Communist China maintains Sina Weibo. And Misskey could be another such for another region, each contributing to slow segregation of the Internet at religious-social-political interfaces.
Such violent responses that you have expressed against the names of .jp and Pawoo is plausibly an indication of those developments.
I remain astounded by how much their product has declined (even pre-Musk acquisition) from their glory days of being so light weight that they were used to organize protests when governments would clamp down on internet access.
Maybe fetching through the API is working better?
I assume they do it that way to get around all the rate limiting and threat of having access revoked that the official API comes with.
This person missed their calling at the State Department.
1. https://www.entrepreneur.com/business-news/twitter-lays-off-...
> But in a sign of just how deep Elon Musk’s cuts to the company have been, only one site reliability engineer has been staffed on the project, we’re told. On Monday, the engineer made a “bad configuration change” that “basically broke the Twitter API,” according to a current employee.
https://www.platformer.news/p/how-a-single-engineer-brought-...
It appears that when Twitter goes down, its engineers were indispensable but when it's up, it's because the engineers foresaw this.
Given those factors, how long it stays up must be an interesting combination of how indispensable the engineers made themselves vs how much foresight they had.
But at the scale and complexity, eventually you hit an unforeseen error that requires some institutional knowledge to fix, and it's doubtful they still have that, unless the ex-engineers loved the product so much they are willing to take phone calls from the people that are left and point them in the right direction.
I don't think we landed on a time frame. I'm actually surprised it stayed up this long.
The truth is, they often do. Especially when there are changes happening. I once took a week off and we went down on day 2. Other times, I'm convinced I could've gone 6 months without anything breaking.
At Twitter's scale, I'm sure there are systems where everything is automated and self-healing. But there are probably other systems that are more fragile. And then there's the eternal race against scale; eventually a system with no maintenance will get buried under the weight of its own data or the scope of its usage. That always requires some work.
I use the app for a couple hours a day (according to my iPhone weekly reports), and have barely noticed any degradation since "the takeover". (Notwithstanding the obvious outage happening currently).
There's been a few minor glitches for sure, but then what large service is immune? Facebook, Exchange, AWS, etc. have all had notable outages this past year, and the Apple Services I regularly use (Apple Music, Siri, etc.) "fail" more often for me in daily use.
Maybe you're trying to do something more advanced than me, or are using a less reliable platform somehow, but generally I find all the "Twitter is so dead" talk being confirmation bias from folks that really want it to fail in light of recent changes.
In my ideal world, Twitter - or something similar - succeeds. I don't care much about the politics of the ownership because that's generally going to be an iffy subject with anything coming from the valley.
But to regularly use it and not see things breaking here and there every day (not even including the times when the API is borked or it's only recommending Elon Tweets in your feed or a person's timeline won't load no matter what, not being able to send a tweet, etc.) ... I have to imagine you're very lucky. There was about 11 hours when people had exceeded their daily tweet quota by trying to send a single tweet. You don't remember stuff like that? It happens multiple times a week in the last few months.
I'm certainly not doing anything advanced. I'm mostly a casual user.
Hiring for twitter is going to be quite expensive, nobody is taking this up without huge payoff and probably upfront given Elon pre disposition not to honor severance etc
Like.. wokeness was annoying but the trash heap of craziness it is today is 5 steps backward. My feed is nothing but conspiratorial nonsense.
If twitter was “tilted left” before, today it seems to have capsized rightward.
I honestly can't believe that not only did he welcome back actual Nazis that were previously banned, but he sold them checkmarks that let them have outsized influence.
#2 is what hardcore Elon fans said.
#3 was always the most likely prediction and is what any actual tech engineer should have said.
- https://www.theatlantic.com/technology/archive/2015/01/the-s...
There's a great 2015 essay by Peter Seibel who was tech lead on their Engineering Effectiveness team:
https://gigamonkeys.com/flowers
I'd guess almost nobody from that era remains at Musk-Twitter.
The fact that things start to fail after that infrastructure has been put on place and running find for a while is noteworthy.
Maybe it's because I use an Android, but I've always found the experience terrible. I get notifications, but when I click them and they don't take me anywhere; the back button is completely broken; mentions randomly don't produce any alerts; it resets me to the algorithmic timeline every few days; replies often fail to load; the spam problem is completely out of control and most adverts are irrelevant crypto-spam; search never finds what I wanted; it's constantly putting tweets on my timeline from people I don't follow; and the whole app seems optimised to produce as much hatred and outrage as possible. Nowhere on the planet is as toxic as a Twitter thread, and yet the media and political sphere collectively decided that reporting on and catering to Twitter spats is the most important aspect of their career.
Frankly, if Musk destroys Twitter, it might be the best thing he could do for society. I may be misremembering, but I thought that prior to Musk buying Twitter, everyone was decrying it as toxic and lots of people were suggesting the government should step in to reduce its influence. Now it seems to be imploding by itself, yet everyone's upset.
Well we're not even a year in and these "small issues" seem to be repeatedly occurring. Let's get a little further out with some more stability before we call them justifiable lol
What you're describing is the loss of institutional knowledge with a fantasy tacked on the side.
1) There's no reason to expect those employees would come back, even at rates far higher than Musk is likely willing to pay. Everything indicates Musk has destroyed a lot of goodwill with his shambolic layoffs, and getting people to come back is one of the situations where you need goodwill.
You're basically talking about gig-work on a shitty boss's terms. Only desperate people would play that game, and the people Musk would need probably aren't the desperate ones.
2) Any employees that come back are going to be rusty and lack the institutional of what happened after they left, so their effectiveness at immediate fire-fighting will be greatly reduced.
3) If there's a "massive catastrophe" at Twitter, do you think they even know who to call back? I wouldn't be surprised if they've lost a lot of the institutional knowledge about who knew what, at this point.
I'm sure there are some people who would do that, but those opportunities become very expensive-to-impossible when you burn bridges the way certain new management has. People hold personal value on things like respect and principle that can make them behave "irrationally" in naive analyses like this.
Not by bringing people in during a fire and pointing into the smoke and saying "fix that!"
Elon Musk has a rock-solid reputation of paying his bills, after all.
It's possible Twitter didn't need to be spending all the money they were to be maintaining five nines or whatever, given the shift in product focus (humans and not a general/universal message bus - bots can't @-mention anymore, etc).
Given that they are unprofitable, if I were a shareholder I would be upset if I did not see experiments being run to see how much cost can be cut and where. I doubt people will abandon the platform en masse over a few minutes of downtime here and there. They will lose 100% of users and be down forever if they don't stop the bleeding, however.
Twitter will never get to $44bn market cap again (much less a valuation with a decent IRR) by cutting costs alone. They need to grow way more than they need to cut costs.
If any of those experiments lead to less growth (which I’d argue is true of a broken Tweetdeck), then they are wholly not worth it.
... wait, they absolutely were, though? I remember at the time people saying that Twitter was doomed for this very reason, by analogy to Friendster, an early MySpace-ish/Facebook-ish thing which was never able to stay up.
It was a pretty common belief at the time that unreliable Twitter would be replaced by Pownce/Google Buzz/something entirely different.
It could be because Twitter was ascendant at the time as you described.
And this thing he was complaining about back in May, he completely f'd it up post-purchase:
https://www.makeuseof.com/how-to-fix-twitter-feed-elon-musk/
A revenue collapse of 40% with a surge in fake verified profiles does not make Twitter better than ever. It is crazy to see a sudden sharp 40% revenue decline being described as better than ever.
They are serving the cert for "*.twimg.com" here.
It's really been kind of miserable to use the app and I only open it once or twice a day. It had issues before but never all at once like this
Either he shuts it down at the point or he has to spend a lot of money .
IMO More likely he shuts down and blame any range of external factors were conspiring to take him down rather than accept he decisions were wrong
https://www.statista.com/statistics/272140/employees-of-twit...
What will sink Twitter is not %99.8 uptime. It's those network effects fading away.
For example, if people stopped using MySpace, Facebook, etc it was not because of their availability.
Again, I'm not saying this is cool. Im just saying if that extra %0.2 downtime (in short term) costs you 80% of your work force costs, a business manager may consider that acceptable.
Easy access to information and articles about breaking news events.
Community.
A feed of interesting articles and discussions.
Enjoyment gained from following quickly evolving memes, discussions, and jokes.
A way to verbalize incidental thoughts, and the pleasure of crafting those thoughts into small nuggets to share with my friends.
Of course there are many negative aspects to Twitter, but those are the things I found value in.
I still use it to follow certain artists but if that subcommunity moves I'll follow them elsewhere.
The only issues I have with shorteners is that they obfuscate what you’re accessing (so if it changes, it’s harder to know) and they’re a privacy issue
> Twitter insiders: We can't protect users from trolling under Musk
> Twitter insiders have told the BBC that the company is no longer able to protect users from trolling, state-co-ordinated disinformation and child sexual exploitation, following lay-offs and changes under owner Elon Musk. [1]
For some reason, the headline decided to focus on "trolling" over an increase in child sexual assault and state-co-ordinated disinformation...
Sounds like the people who say all the petty things they dislike about someone on the internet and then finishes with "he's a pedo" to add gravitas.(yes elon does fall into this category)
Perhaps the takeaway is that we should get our shit together. The opposite of r/antiwork.
No need to pull out terms like gaslighting to justify creating more drama.
A 1 hour downtime is nothing.
If it’s nothing, why is Musk on Twitter saying the whole codebase is brittle for no reason, and needs a complete rewrite? Sounds like a pretty big deal to me.
As far as the drama comment, you’re free to disengage from this thread at any time. No need to continually interject how against the drama you are. Just leave if you find it distasteful.
It is totally relevant since not only you just mentioned both of those instances, but they also have over a million users there which CP and loli culture is pervasive on those instances and hiding the problem under the rug only makes Mastodon a far worse place, given that the majority of those instances is full of that CP content which is completely illegal in lots of countries and is hardly appealing to normal users.
> Secondly, it seems completely plausible to me at this time, that a central Mastodon instance will become the core of a Western European social media landscape...
Right, eventually defeating the whole point of federation with more re-centralization and co-ordinated instance-level banning and you advocating a centralized government controlled social networks being much better. What could possibly go wrong? /s
> Such violent responses that you have expressed against the names of .jp and Pawoo is plausibly an indication of those developments.
What "violent responses"? The basic facts about Mastodon may indeed hurt, but you advertising those two instances that have over a million users using it for loli and illegal CP content doesn't put Mastodon in a good light and makes it completely unsuitable as a 'direct substitute' to Twitter.
I say they can totally beep beep beep about that. I don't advocate violence and/or harm, but that's the Internet. If that isn't appealing to normal users, the Web 2.5.x and social media in their entirety are not for normal users, which it probably is a correct observation given recent reports of mental health issues skyrocketing. But I doubt Web3 or later will be any "better".
>> Such violent responses > What "violent responses"?
.jp, Pawoo, Misskey at this moment are kids running around. I would think something very similar can be said about *chan, though personally they creep me out. Either way, it's not a horde of groomers conspiring over Tor. When a bunch of smart people actively tries to reframe and crush kids running around as "dark web illegal CP sharing activities left unsolved in un-enlightened countries" that's a violent response.
With the continuous growth of streaming and video social medias and looking at what are trending there, I doubt the adoption of so-called "completely illegal in lots of countries and is hardly appealing to normal users" content will even slow down. It will completely normalize, and if it did not in the EU, the EU is going to isolate itself leading to further fragmentation of the 'net scape.
> and you advocating a centralized government controlled social networks being much better. What could possibly go wrong? /s
Oh, forgot this part. I think it's within possibility that more municipal level governments start hosting ActivityPub servers, irrelevant to whether it's idealistically ideal, in case there will be significant numbers of entities that can't migrate to Insta but actually needed non-Twitter-Twitter.
Who mentioned or is talking about Web3 here?
> .jp, Pawoo, Misskey at this moment are kids running around. I would think something very similar can be said about *chan, though personally they creep me out. Either way, it's not a horde of groomers conspiring over Tor. When a bunch of smart people actively tries to reframe and crush kids running around as "dark web illegal CP sharing activities left unsolved in un-enlightened countries" that's a violent response.
It takes me ONE second to link to open evidence of CP and loli content on the TWO original instances YOU promoted. If you're going to mention another instance; I might as well mention 'baarag.net' again. Since that is an equally large Mastodon instance in the millions and is one that you know that is not 'kid' friendly which the content hosted there is unsuitable and even illegal in the majority parts of the world and you know that.
> With the continuous growth of streaming and video social medias and looking at what are trending there, I doubt the adoption of so-called "completely illegal in lots of countries and is hardly appealing to normal users" content will even slow down. It will completely normalize, and if it did not in the EU, the EU is going to isolate itself leading to further fragmentation of the 'net scape.
By allowing loli and CP content for all normal users? You do know that there is a reason why the majority of countries that have CP laws have made it illegal?
> Oh, forgot this part. I think it's within possibility that more municipal level governments start hosting ActivityPub servers, irrelevant to whether it's idealistically ideal, in case there will be significant numbers of entities that can't migrate to Insta but actually needed non-Twitter-Twitter.
Governments go where the majority of normal user voter base is and that is on centralized social networks like Facebook, Twitter, Instagram and TikTok. They don't follow where techies, furries or loli and CP fans are hiding.
It was arguably even a fair price when he offered it (acquisition always have premiums), the market just turned.
Brittle code means it's hard to change, and causes issues when changed, but other than that it's stable when running.
And I have every right to point out the drama here. People here do somewhat have the right to create drama, but it is distasteful and unprofessional and I will continue to call it out.
Glad for you to admit there is drama here though :)
If they had frozen features and left the existing policies in tact, I suspect we would have a dramatically different narrative about the layoffs. If brief interruptions like this are the worst that happens when you cut engineering to the bone, it's a good argument that is Twitter was indeed wildly overstaffed.
Instead, though, we have a company in crisis due to its mismanagement of other areas, so we're primed to view stuff like this through the lens of that broader failure.
Overstaffed in order to maintain Twitter as a static service that never ships new things, sure.
I guess they've been able to ship some things that the old Twitter had already implemented and/or a/b tested. But I'm not sure those count. Meanwhile people have been paying in advance for Twitter Blue features that were promised 3 months ago.
What was the reason when Twitter went down pre-Elon?
I really don't see the reason for this artificial drama. Do other "X service is down" threads go this way?
People say Elon is dramatic, but this thread is honestly ridiculous and way more dramatic than anything I've seen him post.
I don't interpret a lot of "drama" as you put it, but interest. Many observers here are in this field, and follow the "chaos engineering" discipline. Some of them use tools like "chaos monkey" that simulates a metaphorical monkey running through your server room turning off random things, to see how well your resilience systems cope. It's a rare and greatly interesting sight to such practitioners to get to see what happens when the monkey "disconnecting the more sensitive server racks" is a more literal one.
To use a tennis metaphor, good players minimize unforced errors and recover quickly from forced errors.
This is a very clear unforced error that could likely have been prevented by just waiting to roll out the new feature.
To extend the tennis metaphor, it would be like Serena Williams losing a set on 50 double faults. Sure, she's lost other sets before, but it would be notable for her to lose in such a unique way, even if she still went on to win the match.
That's why people are talking about this, it's a very weird way for a site to fail and it's interesting how it happened.
Musk's cost-cutting may yet pan out from a business perspective, but it seems to be a pretty risky move.
Further, as a dev of 15 years, I’m not even sure what you mean by “crash” here. Are you saying that Reddit serves exclusively 500s or timeouts for a sustained period of time on a weekly basis? Or that it serves more than zero 500s at some point during a given week? Or that you, personally, encounter an issue of any kind over a week?
So many ways to describe “crashes” that yeah, by some definitions it probably does, but “can’t use the service” such as what Twitter is experiencing, I don’t think that’s accurate.
https://www.reddit.com/r/OutOfTheLoop/comments/2trmnk/why_is...
(See: all the people agreeing)
One of the triggers used to be that if a thread got too many comments it would bring down the website. So sports communities had to start splitting up their game threads during massive events so Reddit wouldn't break.
It definitely used to happen more often, but it was doing it for me yesterday coincidentally.
And stale. When they changed algos in ~2016(?) to try and stop certain subs I swear it got stale and feels like it updates on a cron job once a day anymore.
At least once a week i'm met with the 'trippin' out snoo for a few minutes.
That seems extremely unlikely to me.
This thread is just an hour old btw, not sure when the issue started.
Seems like a huge reaction from HN for an hour long downtime given that Twitter has gone down for longer in the past.
How much to get in on this? Seems like free money if you thought the time frame was that early.
Sure, images and hyperlinks breaking and not paying the Slack bill and Musk having another mental break makes the front page of hacker news, but will a spam and abuse detection pipeline being silently busted for 3 months do that? A reporting backend that was providing advertisers with incorrect data on their ad spend? Some customer-relations workflow that ruined the week of a group of TAMs, and had revenue impact?
Twitter is a big ship. There are more places things can break than anyone, inside or outside it can imagine. But as an outside observer, the only signal that we can act on is 'Has the ship sunk? Or is it floating?'
Observing that binary state answers a question, but not the question of 'Is the ship working well?'
It seems really unlikely to die from a single spectacular failure, where a key bit of infrastructure dies and just cannot be brought back online for any amount of love or money.
A steady drip of annoying but individually minor issues will gradually chase people away: more spam and less relevant recommendations discourage users; bad spend data and fewer account reps make it a lower priority for advertisers. Twitter's main value prop is its network effect, and if a few people leave (or even engage less) every issue, that value will wither away....
There's really no correlation between layoffs and outages, I suspect.
I worked for a different eccentric billionaire early in my career. It was stressful, demanding, and unpredictable as he tried to micromanage everything by himself in short bursts of 5 minutes before moving on to the next thing.
The hook was an idea that it was a famous company and he was a [not quite] famous billionaire and that we were sitting on a once in a lifetime opportunity to build our resumes. He was going to reward us greatly at some future date when everything was working well again, and we’d be able to work anywhere else we wanted after this.
Didn’t really pan out for us as everyone got burned out, the billionaire’s micromanagement and constant product churn diminished the company’s reputation, and he eventually laid most people off in favor of even cheaper foreign labor.
But there'll be one or two people who get cozy with Elon and win the lottery and will write articles on how hard work always pays off (neglecting the other 99.9% of Twitter that gets fired when it all gets shipped to India).
Or we could reduce the hoops & cycle time for legal immigration. The US has an opportunity right now to cement itself as the leader in technology for the next 2-3 decades and stave off the impending population growth crisis if we could find a way to enable people who want to live here to live here.
Be careful what you wish for.
Also, there is no population growth crisis. A decreasing population means less stress on the environment and lower housing costs. Yes, there will be economic effects, but it's hardly catastrophic. Japan is managing just fine.
Yes, there will be economic effects, but it's hardly
catastrophic. Japan is managing just fine.
Of course, you can find a lot of "experts" saying anything. Economics is an area where I haven't got the chops or the hubris to tell shit from shinola.
They become Americans and rise to your level of comfort and compensation. By working here in the US, they increase the wealth, tax base, and culture of our country. Immigration is a good thing. Skilled immigration even more so.
The alternative is that they stay in their home country, and that country grows a tech industry that rivals the US. Those workers will work for even cheaper than in the US, putting an even wider delta on price and creating an incredible arbitrage opportunity for talent.
Much like the automotive industry, foreign competition will drive margin out of our comfortable tech industry that has enjoyed being peerless for decades.
We're only going to see more `TikTok`s and `Spotify`s arise.
At some point, talent won't want to come to the US anymore. That should scare you.
I say all of this as someone who wants everyone to enjoy wealth and prosperity regardless of where they live. I still want opportunity and the ability to capitalize on it to be accessible to any American that would take it. And for that to continue, we should keep growing our talent pool and increasing the scope of what we can achieve together.
"It says here you were terminated for breaking into a vending machine in the lobby and distributing snacks to your colleagues while dressed as Robinhood."
"Yeah it was an act of protest because management threatened to take away our lunch breaks."
"Well... it... sounds plausible I guess."
If they actually make it all the way to the end, maybe being the last man standing there actually does have some perks we don't yet know about?
You forgot the bootlickers, I'm sure not all of them have been fired yet.
They can switch jobs. H-1B doesn't force you to stay at your job.
Clearly not for everyone given the amount of opting out and controversy.
My recollection is that around ~2011 a day without a fail whale was the odd one out.
Record engagement these days is most likely measured in single digit percentage points over average engagement (at most maybe 10-20%), where when you're early and your traffic is spiking, it can be an order of magnitude more traffic than what you're used to seeing. The absolute numbers are larger yes, but from the perspective of "what can my infrastructure / architecture handle", relative changes are far more relevant.
You're talking about a man who has more than once been fined by the SEC for making false statements about public securities. What makes you think that he'd be a more reliable narrator for a private company for which he has no external accountability and a large personal financial interest?
Those numbers have been proven to be incorrect, by people who are posting from locked accounts with 0 followers racking up large nonzero view counts within a few minutes.
I personally don't think that 100% uptime for everyone around the globe is even a worthwhile goal for Twitter. The incremental costs for that last 0.5% aren't worth the incremental benefits.
HA is most necessary for what, journalists? (Who are already antagonized under Musk)
Everyone else can wait the hour.
Even if number of engineers is decisive, we will never stop it by immigration. China + India have more than 8x the population of the US. We would never be able to deprive them of enough engineers via immigration.
As anyone who has operated a software system knows, the #1 reason things break in production is new code, and from a third-party observer with no inside knowledge it certainly seems like more changes are happening at Twitter than ever. This is orthogonal to layoffs, though.
Best source I could find. Currentlydown didn't even register yesterday's outage, and downdetector doesn't share historical data.
It'll have one, maybe two more reads every time you look. Not dozens. One person posted an example that had literally over a hundred alleged "views" within a minute, despite not refreshing their timeline after posting the tweet.