Reddit crashed because of the growing subreddit blackout(theverge.com) |
Reddit crashed because of the growing subreddit blackout(theverge.com) |
So when 1000 subs go private, they potentially had to update hundreds of millions of comments.
Anyone else have a theory?
When you close down all the most popular content, you have to dig deep into the long tail for fresh content.
Also, my guess is that the code for building homepages is not optimized for having a lot of skips due to private Reddits, since most people have probably never been subscribed to a private reddit, or if they have it wasn't for very long, or even if for very long, never more than one or two at at time.
One theory I have seen floated but which seems unlikely to me is that this was some kind of internal sabotage. I could maybe buy this if the protest was about a war or human rights issue or something, but I really don't think any IT pros would be willing to risk jail time for this one.
AKA why you should normalize your data models.
Could also be some kind of fanout query that assumes the top subreddits are open and wastes CPU cycles.
Another possible explanation is that operations on private subreddits are not as optimized as it is expected that they have a small number of subscribers and take up a small amount of total resources. These assumptions would be flipped.
then again, not sure why that messaging would be any better for them than admitting to a DDOS...
> (I used to work as a backend developer at Reddit - I left 6 years ago but I doubt the way things work has changed much)
> I think it's extremely unlikely that this is deliberate. The way that Reddit builds "mixed" subreddit listings (where you see posts from multiple subreddits, like users' front pages) is inefficient and strange, and relies heavily on multiple layers of caches. Having so many subreddits private with their posts inaccessible has never happened before, and is probably causing a bunch of issues with this process
https://tildes.net/~tech/163e/reddit_appears_to_be_down_duri...
> Mr. Rathschmidt added that some apps are more efficient and require significantly fewer A.P.I. calls and that “Apollo is notably less efficient than other third-party apps.”
``` UPDATE subreddits SET is_private = FALSE WHERE is_private = TRUE AND updated_at BETWEEN 2023-06-11 AND 2023-06-12; ```
Bonus: Run it every hour
It's as if their mismanagement has come back to bite them. Mods have been allowed to run the show and ruin communities for years and years.
Very few people are using 3rd party apps but now they are causing inconvenience to millions for their own capricious reasons. I knew that the site was ran by a small powertripping group of mods but now that they make it so obvious it has kind of chepened the whole medium in my mind
Calling bullshit on that one. If they had expected stability issues, and knew exactly which subreddits will go private and when, how is it possible they had so much "anticipated issues"?
I mean, honestly that's probably what I would have done if I were still in charge.
Not too much of a stretch, IMO.
A fantastic Schadenfreude generator, regardless.
What's worse is that the system evolved as a response to the previous similar event, and what broke now is probably not what had broken in the past.
Kinda frustrating design, IMO, speaking as an end-user.
I think this would be a poor move.
Reddit is extremely entitled. They as a company create nothing. The users create everything and do all the work of moderation. Spitting in the eye of your all-volunteer staff on which you depend completely for your livelihood is corporate suicide.
I agree that Reddit won’t be the same after this, but there’s just not a ton of remaining value in the big defaults. They’re already plagued with low value, low effort content, low cardinality content. The true damage that’s being done right now is to the small communities which represent(ed) the best of Reddit. Out of the spotlight but deep in quality content for those that care about the topic. The inherent risk of stability is more certain now with Reddit’s new policies. And I suspect now with many subreddits weighing indefinite shutdowns that many users (perhaps more importantly: the best content creators) will scatter to the wind for greener pastures. Those that are left are given crappy apps, lots of ads, and hollow communities eagerly looking for something better.
What's old is new again, I suppose!
Similar happened on IRC back in the day. All the big channel wars and drama were over the big popular channels for internet points. I certainly participated in my fair share.
Then you had the small channels of folks who were just pretty chill chatting about whatever random topic of interest.
Might be nostalgia, but I still think the best balance of this was the phpbb/vbulletin era. The amount of "reference material" lost behind the walled gardens is crazy.
I’m not sure what the alternative is though. Independent websites? Where are dejected Redditors going to go?
However, I do agree with the blackouts as it is a coordinated action to a larger problem, not some power obsessed individual subreddit with arbitrary rules also applied arbitrarily.
As someone who is close to a reddit mod, they did not know which subs would go private
I’m sure someone somewhere at Reddit was surprised by sub X or Y going private, but if they were surprised at half the largest subs that went private, then that would be as sad as it is amusing.
Edit: typo
You get an immense amount of power with very little accountability except to the community. Put that in the wrong hands and you can destroy a community incredibly quickly.
Or, put power in the hands of a dilettante, and you get overrun with spam and insults very fast.
And any sub with serious activity will have power mods submitting applications for take-over so they can commercialize it.
It would also be a huge invitation for the displaced the rest to make life unbearable for those new mods and the handful of admins overseeing them. Reddit as a community would make for one hell of an intellectual DDOS.
To borrow from another comment buried below: I think the company cares more about the appearance of "business as usual" in the face of an IPO than the opinions of the users.
In their eyes, this is all nothing more than a tempest in a teapot; a small pothole on the road to public investor dollars.
The only thing that I'll be able to guarantee is that you'll really not like me as a mod.
After that, they'll definitely reclaim the shuttered subreddits with new moderators.
If they take over these subs, you just radicalise the users, and you now have to find new moderators for these subs. In a worst case scenario, users actually start migrating to another platform.
The adage "there's no such thing as bad press" could definitely apply, given how broadly covered this is (I mean, hell, I saw it show up on NPR).
Re-hosts of that site have the figure even higher.
Yeah, this morning I noticed only one sub that I follow was actually private. I was going to mention it here, but I forgot which one it was - and now I can't find the one that was blacked out.
Looking at the google trends, it was a big spike today compared to the classic daily workload: https://trends.google.com/trends/explore?date=now%207-d&q=%2...
It coincides exactly with when Down Detector shows the Reddit report spike... https://downdetector.com/status/reddit/
I'm gonna go ahead and say my theory is probably right.
Anyway, it turns out the primary (toxic) subreddit for this condition is taking part in the strike, so a lot of users have come over to my alternative subreddit, and they are glad to have found it because they were getting fed up with the toxicity of the other place.
Unfortunately I'm sure it will be back to normal in a couple of days.
I'm currently using kbin. Waiting for Tildes to come out of beta.
I think it's more like there are a small number of users who are mods on a huge number of subs, but there are other mods that focus more directly on the actual moderation of those subs.
I also don't see some subreddits listed there that are part of the strike.
But that was 13 years ago, no idea how it would do today. :)
so you're saying, we just have to knock it out once, and it'll be gone forever? that makes things easier ...
There's an unconfirmed but strong possibility that Ghislaine Maxwell was one of the largest power mods on Reddit. https://kirbysommers.substack.com/p/evidence-that-reddit-use...
All of this in the post-Fed rate hike world where money isn't free. Besides I feel it's important to remember that Reddit has been talking about an IPO for many years, and I doubt it's going to happen now.
it depends on volunteers to function at all. Posters and commenters are volunteer too, which is something reddit oft seems to forget.
[The Mythical Man Month has entered the chat]: Nine mothers, all hands on deck, should be able to gestate a baby in 1 month.
Do you have a more detailed pitch sheet? :)
> intentionally accesses a protected computer without authorization, and as a result of such conduct, recklessly causes damage;
I would think it's reasonably easy to establish mens rea when someone switches their sub from public to private and back every few hours, even if manually done because what else would they want to do but take down Reddit? Not to mention the punishes much harshly "conspiring" to commit such a crime. Careful there.
IANAL, of course.
Reddit mods are not accessing a protected computer without authorization.
In their eyes, this is all nothing more than a tempest in a teapot; a small pothole on the road to public investor dollars.
If they do that, forget IPO, there will be no more reddit at all.
right now they already missed the IPO point years ago, today for a company to have a successful IPO they need more than good MAU numbers, and this API change was never going to get to the profitability numbers they need for a good IPO
Their IPO dreams were fucked long before today
Having /r/Pics down permanently because of a protest would be a problem, but a poorly moderated /r/Pics would not. The quality would just go down, and Reddit would get a good justification for locking it down.
"No, /r/Pics wasn't shut down because of the protest, it was shut down because there were too many law-breaking posts. And you know how tough we have to be on those kinds of posts."
At least, that's my take on how most investors would look at it. Investors love companies who focus on profitability, regardless of their long term perspective.
1. A reddit request was made in 2018 when you were banned. This was manually approved by the admins: https://www.reddit.com/r/redditrequest/comments/8m8cdm/reque...
2. The new mod heavily restricted posts, seemingly for three years. They declined to reply to messages for nine months, until a new Reddit request happened on Jan 3 2021. A comment in that thread said "there hasn't been a new post in ages!", confirming your replacement had locked the place down: https://www.reddit.com/r/redditrequest/comments/kpf083/reque...
3. Sometime between Jan 3 and Jan 19th, the request was approved. Likely closer to the 19th, the day the new mod posted: https://www.reddit.com/r/radioastronomy/comments/l0j6qj/new_...
Sub seems alright now, but after a three year long false start with an inactive mod who locked the place down.
Further, it took up to 16 days to handle the request. For a very small sub. And I've seen reddit requests take much longer.
Now reddit would have to, at a stroke, replace at least 8,000 of their most active subs, all at once, and make sure they don't appoint people who are insane, or draconian, or power hungry, or in it for their own profit, etc.
I don't see this working. You may have some knowledge of /r/radioastronomy in those periods which contradicts what I wrote, but the public record suggests the transition wouldn't bode well for mass replacing every single moderator at once.
92% of reddit is down right now.
It's a very cynical move, but you've laid out in exquisite detail why it's their ONLY move, other than walking back their policy. While the company as a whole would benefit from that walkback, I think Mr. Huffman himself would probably lose his job.
A reddit admin just posted this:
> We also want to reiterate that we respect your decisions to do what’s best for your community, and will do what we can to ensure you're safe while doing so. However, we do expect that these decisions have been made through consensus, and not via unilateral action. We ask that you strive to ensure that your moderator team is aligned on community decision-making – regardless of what decisions are being made. If you believe that your community or another community is being subject to decisions made by a sole moderator without buy-in from the broader mod team, you can let us know via the Moderator Code of Conduct form above.
They’re hoping to have some lower level mods who disagree with going private turn to the admins. I doubt it would work but might reopen a few subreddits.
https://www.reddit.com/r/ModSupport/comments/147ysr6/moderat...
You seem to believe a low quality /r/pics would bring in traffic, even more weirdly you seem to believe the admins shutting down /r/pics because of illegal content, is better for profitability then it being shut down because of a protest, not sure how that works....
Traffic, and users drive profitablity, High Quality User Generated content drives that traffic and attracts users
If reddit loses high quality user generated content, there is no traffic, and thus there is nothing to sell to be profitable, no ads, not API, nothing
it would not matter if /r/pics was shutdown in protest, or shutdown because of poor moderation, it is still shut down, and thus no profits from it.
investors love companies that are profitable, especially right now when the cost of capital is very high. "focusing on profitability" is not the same as delivering profitability. Investors right now want results, not power points about how they project the results being
Having your sub be unmoderated is against reddit TOS, whereas taking a sub private is not yet against the official TOS.
Process and technicalities like this matter a lot and can effect a platforms actions.
But if neither make headway, just going unmoderated is the logical next step. Yeah, reddit could take over moderating the subreddits. They can also do that if they're set private. But they don't want to. They outsourced that to volunteer moderators so they could get free labor. If they want to run their own moderation, and submit their own content, the reddit corporation can have fun with that.
Really shouldn't be anyone's idea of a good time.
I'd also worry that if Reddit moved in that direction, the mods would feel like they didn't have control over their own subs and bail. I don't believe that Reddit can afford that, especially given how many subs and what %age of the user base we're talking about here.
I guess that depends on how many people are willing to risk prison just because they're mad at Reddit. Or on what automatic moderation capabilities Reddit has. Not a moderator, so I honestly don't know how much work it does or doesn't take. They probably count on volunteer mods to filter spam and whatnot, but for legal compliance? Highly doubtful.
I'm not a mod, but I have several friends who are, and you wouldn't believe the sheer amount of gross nonsense they have to sift through. Even ignoring the relatively banal death threats, there are always questionable elements trying to sell drugs (both Rx and other), running scams, offering malware, and worse.