Apollo Back end just made public(old.reddit.com) |
Apollo Back end just made public(old.reddit.com) |
A gentle reminder that the whole history is here for show. This is very vulnerable — you can see the times I was in a rush, or when I took my time. You’ll find my mistakes. Either way, all I ask is please be kind. I know there are 50 things I could have done better, but this is a product of the reality at the time — a balance between something that works and something that was cost effective for Christian.
User rapnie mentioned the code in the repository doesn't have a license [0]. You might want to review that.
Some of the 'code' running massively profitable departments in finance are spreadsheets, on shared drives, with thousands of linked cells thought up by a brilliant employee who left the company five or six years in the past.
I wish I was kidding.
Thanks for how great you kept apollo.
Show HN is always really close to to that ideal. One of the reasons I like this place.
Christian is very visible in the community, but I want to say thank you to you too.
Thank you for all the work you did! This made a lot of our experiences great.
Do you think Christian would be willing to open source the app (under a non-commercial license) and allow us to build it with our own Reddit API keys?
That way we could still use Apollo, but pay for it ourselves
I doubt reddit had any actual issues other than it impacting profits and no optimization to Apollo would fix it.
The TLDR here is that we only ever requested new notifications (give me anything that came after this thing I know about), which funnily enough then broke for some users, because if a reply you got notified about gets deleted and you use it for reference, Reddit will never return anything new to you, so I had to go and write a 2nd service that runs much more infrequently but does some checks on each account to make sure this isn’t the case.
If Apollo's users (or a good percentage of them) moved over to an alternative platform, that would be poetic justice, at least.
To go even further, all the app devs should get together and create a new platform or pick one Reddit alternative and point all their apps at it.
That way it would be even more likely to get critical mass quickly. 3 weeks should be enough time to come up with a coordinated plan. Don't let Reddit push you out -- push them out and let your users stick with the app they love.
Reddit is only its users, nothing else. They don't own the users' loyalty -- the apps do. Take advantage of that and teach them a lesson by pointing the apps and users elsewhere!
Most would hate to see what a sausage gets made out of, but nearly everyone will eat that same sausage.
Especially when you're solo or on a small team, technical debt is actually something you can lean into. In this domain, refactoring is cheap and cutting corners can help you get where you need to go faster.
It's a too common mistake to think that because technical debt is a tar pit that kills many large products that this is universally true. The real trick is lowering the tolerance for flaws as the project grows and matures.
Sample of latest commit messages:
batch size wants to be smaller for less locking time, not higher
increase scheduler batch size
bump locks again
make pool sizes customized
increase prefetch
[skip render] remove old redis from infra
increase poolsize a bit
bring new redis up to line in blueprints
reduce pool size
bump pg client count
tweak pool sizes and consumer count
double the amount of consumers
small tweaks
[ci skip] update pgbouncer github url
clean allow lists
add new redis instances
make connections to the correct redis url envs
add new env variables
spam prevention
update reddit http client
feat(ops): add distributed tracing (#109)
The way I took Reddit's comments were there were things that could be optimised to reduce costs. I've not seen a codebase yet where that isn't true.
Sadly, in the hands of Redditors who think you can host an API on AWS for $1 per million requests comments like come across as "Oh they did shoddy work".
What I think is we're seeing production quality code like the majority of us have running.
A payload of 25 comments vs a payload of 100, takes less time to generate and deliver.
So sometimes, to optimize his users experience, he might request 25 to show something quick to the user and then do a request for 100 and append it.
He also mentioned that he’s being compared to other apps, _not_ to the API limit (which he was under anyway).
So, can it be less?, of course. Heck, he could make it 1/minute or even 1/day. But his quota was 60/min. So why not use the 60?.
As far as the app goes, an average of ~330 reqs/user/day is pretty good, considering they built the platform budgeting for 86,400.
Apollo saved my Reddit use after they bought and killed Alien Blue. I have gotten so much out of and out so much into the more obscure technology subreddits. It’ll be missed.
I appreciate all your work. For what it’s worth, I’ve never had server-side issues with Apollo (that I’ve noticed anyway!)
Edit: typo
IMHO the gesture of releasing the code shows how much integrity there is behind the devs, and I could care less if there's not-best-practice in there.
I am waiting on delivery of an iPad pro, my first apple device in 10 years. And, even though I hate reddit, I was excited to check out Apollo, if for no other reason than to see what a great ios app looks like.
I know that reimplementing reddit API from scratch will take more than a few weeks. But, I'm sure loyal Apollo users will be happy with a much scaled-down version. Maybe, to begin with you put the app in Read Only mode, and point it to a reddit archive. Then work on making one sub read/write again and expand from there.
I mean, even just a good reddit archive browser has TONS of value. I'd happily pay a small sum for an awesome reddit archive explorer.
"Chatgpt, build the back end for this API: $(cat reddit_api_docs)"
How was the experience overall with this tool? If you have experience with Flyway, how does it compare?
Also, thank you for making this code available, but would it be possible for you and Christian to choose a license for it so people can use it? I don't think which license matters all that much, personally, but I can also understand wanting to keep it proprietary, either way, thank you again for all your (and the rest of the Apollo teams') work.
The one fun thing I learned was about databases. Given enough records, a query planner might decide to start doing things very differently and you will get woken up at 3am with a CPU usage alert. That was a fun one to debug for sure…
Thank you.
Would love to have a decent client for this site…
Can't see it being anything other than a PR disaster for them, unless they change their stance.
Interestingly, this seems to just be a proof of concept, and the repo is archived. While the developer of the Apollo front end is getting screwed in this whole debacle, this seems to me like an opportunity. He could literally build an alternate Reddit back end (ok, pay someone to do it, if he doesn't want to do it himself), switch Apollo to use that API instead, and get an instant network of hundreds of thousands of Apollo users. A mass migration from Reddit to a new Apollo back end seems like an amazing way to start a new social news website.
The short answer: You must have an interface you control, because app store review times mean there is no "hot fix".
When something upstream of the client breaks - you are DAYS (assume 24 to 48 hours for review, after you fix the bug) from being able to get a new client in front of users, and several weeks out from having all of your users (all being some accepted percentage - I use ~95% for my releases) actually update to the latest version.
If that "something upstream" is you - you can usually plan for it, release the updated client ahead of time, get it rolled out, then pull support for the old feature.
If that "something upstream" is reddit... They break things, and they might (but probably won't) inform you ahead of time. You will have many, many more outages.
Also a place to pull in any access keys needed for other services if they connect to so it doesn't sit in the client code.
1. Those aren’t consistently scheduled 2. They have an extremely tight deadline (and sometimes Reddit can be a bit slow)
Making it server side means we controlled that experience better.
I’d be happy to try and convince my 100k community to Apollo and if the UX is similar for web and mobile then I’m not sure switching costs would be that high during the blackout.
I’d help fund it but all my money is wrapped up in my trash cooperative.
They've succeeded in killing their main competitor (in terms of ad share) and could even now walk back the api charges for any smaller operators and leave Apollo dead and avoid the biggest backlash.
It's scummy as hell and I won't want to continue to contributing to reddit, but I can't help but feel they've been shrewd in getting what they wanted.
Sync is similar. I have a one-time purchase for "ad free" Sync. That stops me seeing adverts I would see in the official app for a fraction of the cost of reddit premium.
Sure, it doesn't have ads in today, but it could, so there is opportunity cost.
No one said they were open-sourcing it. It's just published for information/transparency.
Anything I read on the website has been met with extreme skepticism since it very well could be an astroturfer.
It's money. Reddit wants astroturfers, it grows their platform and helps sell ads.
This will be a spectacular shitshow to watch.
https://www.theverge.com/2023/6/8/23754780/reddit-api-update...
You were correct
The average internet user by the way, is statistically not you reading this, doesn't really care too much about APIs and finds the current Reddit client just fine really and as a recent registration, doesn't understand that the developer community basically built the interest in reddit and without third-party moderation tools, clients and a community that deeply cared about their special website, flawed though it was.
All platforms go through this cycle, but Reddit felt special to me and it hurts to find that wasn't true.
It's not that I haven't known for a long time now that reddit isn't the site it used to be, it's that this episode in particular has put that in such focus I can't ignore it.
So now I feel a sense of loss for the reddit of old, even if it hasn't really been such for a long time anyway.
Reddit itself will carry on fine, and perhaps get even stronger with a tighter control over content. Efforts to make a federated alternative are doomed.
But I’m not an investor, maybe Reddit has some numbers that make this make sense, or something. Or maybe they are just hoping to rip off some investors who don’t know how the site works.
Even today I can’t find answers on Google to my daily inquiries, but will find it on Reddit.
If there is an internet worth building, it’s not on ads.
But we all knew that from the start.
Here’s to those working on the federated web, maybe the outcome can be different.
This is not always true and there’s an important correction to this: all VC-funded platforms go through this, especially monetized by ads.
There are examples of privately owned platforms with subscriptions that are doing great and show no sign of quality decline. Telegram most notably being one of those.
This chase for constant high returns and also the focus on an audience very ortogonal to your actual users (ads buyers) are what seriously misaligns the incentives of users and platform owners.
There is nothing inevitable about that.
The reality is that Reddit hasn't ceased growing since then. The same thing happened with YouTube. YouTube pushed its content to switch from filthyfrank to Jimmy Kimmel/Fallon. Advertisers are not interested in us, early-adopters, mostly young and middle-aged western males. They want kids and soccer-moms: they're less likely to install an ad-blocker, and have a much wider range of products for which they're the target audience.
I think we need to understand that we were never the target audience.
Reddit, like YouTube, will do more than fine. The early-adopters just won't be part of it.
My hope is that we will go back to independently ran niche-community bulletin-boards, like in the 2000s with phpBB, instead of piggy-backing on some other new for-profit corporate walled-garden.
[0] https://m.youtube.com/watch?v=K3Qzzggn--s&pp=ygUESm9qaQ%3D%3...
[1] Seriously, I’m not actually going to post the video. Im sure you can find it if you really want to. My kids aren’t allowed to watch Blippi though.
If these people exist and they’re the majority, why kill 3rd party apps to mess with a minority?
Reddit has centralized what would otherwise be hundreds (thousands?) of obscure, splintered and isolated communities and forums. What could possibly compete with that without having already reached critical mass and become an obvious competitor to Reddit?
(Although at little personal cost, since I don't normally go to reddit directly, but I paid just to stop the inline ads that I would see when I would, as often happens, end up on reddit after googling something with kagi.com or bing.com.)
This always makes me chuckle. Kaging something may not be mainstream yet but Kagi-fu sounds better then Google-fu!
This will have outsized implications for the 1% of users who actually post the content the other 99% come to see in a way that Twitter (and its 280-character comment limitation) does not.
It has. As somewhat of a Twitter addict, I'm finding it much easier not to log on these days, because the experience and the conversations have really hollowed out. I'm also no longer seriously considering using Twitter for promoting any projects or anything of the like going forward.
Reddie relies more on genuine content and less personal brand (as you are subbed to subreddits and not public users, mostly)
Typically it comes out of my mouth in a more moderate stance, like "sorry, I don't use Twitter anymore. Any chance you could link a better forum?"
Reddit has been more or less re-inventing itself for their asinine IPO aspirations, and in doing so it has moved to closed source code, redesigned their interface, adjusted content policies to court advertisers, and now they're finally going after the crowd of enthusiasts who depend on features that Reddit has failed and/or declined to implement.
This all reeks of venture capitalist sabotage and it's the very thing that ultimately killed off Digg.
UPDATE: Oh I see, they did the kill-the-API-to-kill-third-party-clients thing before too, long before the current brouhaha. OK, never mind.
I find tech/power-user crowds like that on HN tend to dramatically overestimate the proportion of a userbase they actually make up.
Just like it did Twitter.
The entire public reddit database of posts, comments, and users could be imported as a starting point.
He says he has no desire to do so as he wants to develop products, not manage a product.
https://old.reddit.com/r/apolloapp/comments/144f6xm/apollo_w...
>I've received so many messages of kind people offering to work with me to build a competitor to Reddit, and while I'm very flattered, that's not something I'm interested in doing. I'm a product guy, I like building fun apps for people to use, and I'm just not personally interested in something more managerial.
>These last several months have also been incredibly exhausting and mentally draining, I don't have it in me to engage in something so enormous.
Not everyone wants to throw away their soul for money ;)
* a giant headache
* a totally different skill set from building an app
Anyway the code is o̶p̶e̶n̶ ̶s̶o̶u̶r̶c̶e̶ probably going to be open source soon and his company is probably not going to keep working on it now that Reddit has basically wiped out their business model, so we may as well ask why don’t you or I fork the code and then create the social media site?
Sure the server costs for the API costs alone aren't 1.2m a month for Reddit. But when you add in all the things that need to operate. Databases, storage, data transfer, etc. It would realistically be more than 1.2m in server costs. That's without paying people to build and maintain it.
I can't think of much else that could have done a better job at putting me off from ever touching Lemmy than some of the descriptions of federations it "recommended".
Also, by being open about how many people were federating in each community, it also made it clear that even the biggest server was smaller than most discords.
- incredible moderation and respectful community
- you can't sort by controversial
- non-profit, no investors can corrupt it
- beautifully simple UI, better than old.reddit on mobile
- fast and not decentralized
- no images just text
- open source
- The site is the main mobile interface, not an app
- No limits to logged-out browsing
- Completely functional for browsing without javascript
- Zero third-party scripts/assets during normal use
- Uses modern versions of simple, reliable, "boring" technology
If we blackout long enough and then figure out how to cooperate to create an actual community alternative in the blackout period, it could work
Not easy and will take real leader with 80 hour a week work ethic for a year, and probably a few million dollars to start.
However it would be an amazing groundbreaking feat
I think it's doable, but we need a leader to step up.
I should probably just say "recycling cooperative"
It’s just people talking around a topic and helping each other. Subreddit mods doing everything they can to keep their area clean for people reading. I’ve been on Reddit over for over a decade and this is the power behind it. Just wanting to help and bring good member.
Thank you Andre. For your work and commitment to Christian, Apollo, and the community.
If there’s an issue with the cost to process that many calls I’m much more suspicious of inefficiency on the API side than the client. Those numbers seem very reasonable.
I don't think Reddit cares about the number of requests. They just want paid. They weren't caring before when it was free.
Surprising that Reddits communication on their own platform is so bad, or maybe it's not...
Thank you for the update!
Not to sound too pessimistic, but most of the mods of the bigger subreddits aren’t going to give up either, since it’s a part of their life, and not really their battle.
I really hope I am wrong though, since it’s about time we have something new and shiny.
I'm not saying I approve the practice. I'm saying that it makes sense from their perspective.
Also, no company's success should ever depend on alienating its users and slandering longtime contributors.
the parallel for traditional companies is choosing to bust unions, and a month before you IPO is an absolutely terrible time to choose to bust your union unless you're absolutely 100% sure you're immune to a "strike".
in this case it's not just picking a fight with powerful stakeholders (like mods) but also turning off features that a ton of users interact with the platform via, and trying to force everyone off third-party clients onto the new experience and native app. In this model the users are a stakeholder that you're picking a fight with at the same time.
the best time to do it is 20 years ago, the second best time is today, unless today is a month before you IPO.
They may be selling that ad space for pennies on the dollar... But they are managing to fill it.
When I first started coding (some 15 years ago) I wrote the ugliest, buggiest and "owasp-ridden-est" PHP spaghetti code. Turns out that code went viral and was used by millions of users for a few years.
Yet, I came to the conclusion that that's the best code I've ever written, for it's been the most used code I released so far.
Real software development is far different than the ideal one.
Good advice for any site. It's not like HN is immune to astroturfing.
I wasn't a big issue to me tbh. Vote fuzzing had always made those counts meaningless.
https://arstechnica.com/information-technology/2012/06/reddi...
talking about manipulation, reddit's current ceo edited users comments back when the bad guys were on business side of it.
https://www.theverge.com/2016/11/23/13739026/reddit-ceo-stev...
I seriously question the motives of people saying this guy is a "moderate." You cannot gaslight America into thinking the far-right is middle of the road.
That part of the community is the part that Reddit probably wants to preserve, and that part could very easily see an AMA that appears to clear everything up and then get mad when their sub moderators go ahead with the blackout anyway.
[0] For example: https://www.reddit.com/r/gaming/comments/142p1ax/comment/jn6...
Wondering if you still have any invites to send my way?
Email in profile. Thanks!
> I also notice that there is no such thing as sort by controversy
> This is one of the things I like about Tildes so far. An option like that is specifically designed for hate. I want to spend my time on these sites talking with other nerds about homelab stuff and niche 30 year old RPGs no one cares about. Not feel the urge to constantly validate myself by looking at comments people are metaphorically laughing at for being ridiculous
https://tildes.net/~tech/15v5/stop_trying_to_make_a_good_soc...
This is very short-sighted, I feel. The controversial comments often have the most interesting discussions, and helps to avoid the issue of the top-voted comment being a popular, but incorrect, view/discussion where all the activity happens.
Maybe I could get used to it overtime, but we'll see if it ever becomes non-invite-only.
1. All rights reserved - the source is visible but not open.
2. AGPL, to ensure that if someone does take this forward, no part of it can be closed source without your agreement.
3. MIT, which is extremely permissive and effectively you don't care what someone does with it, commercial or not.
Depending on your comfort level I would start with the middle option, I don't think people tend to mind you re-licensing more liberally but if a community has formed and you decide to go commercial further down the road it can cause problems.
Gives those who wish to learn and stops from anything taking the future glory.
Just not MIT or BSD, I like these sorts of licenses normally, but it would be a bummer if Reddit got to steal their code after all this bad behavior.
I wouldn’t add a license unless you specifically want to allow people to copy and use this code, otherwise just ensure that the appropriate copyright notice is present. No license is better than a badly written or badly chosen license — no license already means that all rights are reserved and no rights are given.
GPL on server side software doesn't really matter, that is why the AGPL exists
That's maybe too general. There are cases where it does matter.
Maybe "GPL on software-as-a-service code has limited usefulness"
Even then, people do sometimes choose it on purpose for SaaS, where they want to ensure any redistributed code, where the entity doing it makes that choice...is bound by the GPL.
Here is a good comparison of licenses.
https://gist.github.com/nicolasdao/a7adda51f2f185e8d2700e157...
At that point you're likely going to cause more damage to the brand/corp with the half-assed implementation then the value you're providing with said code.
By far our biggest problems are:
1. Leadership at the time just wanted an MVP, and they wanted it yesterday.
2. Leadership now believes "You don't need more people, and rewrites are a waste of time. Look what we've built with a skeleton crew and tight timelines!"
Fellow engineers, beware: if you train your leaders to believe your team can deliver with no budget and unrealistic timelines, they will continue to give you no budget and unrealistic timelines.
That’s at a Fortune 500. It all started with a “well we can do an MVP and get that out the door first.”
It turns out that was a massive mistake.
I’m not saying they’ll do this or that it makes sense. But I’ve definitely done it in much smaller projects.