Disruption with Some GitHub Services – Resolved(githubstatus.com) |
Disruption with Some GitHub Services – Resolved(githubstatus.com) |
It's a shame that this is happening and hopefully Microsoft do the right thing and spin it back off into a private entity with its own leadership and resources.
Also, self-serving: this is why the whole "big company means better" thing is bs. We need to aggressively go back to a culture that rewards and supports independent teams who actually give a damn about what they're building and responsible for. Otherwise, this is just a preview of everything everywhere in a few years.
But that doesn't fit into the "spend gobs of money as fast as possible so you can be a hypothetical feudal lord in a sci-fi dystopia novel you read when you were twelve" line of thinking.
Either 1) infrastructure for paying and non paying customers must be separated, 2) excessive load must be ended by not offering the free tier anymore or 3) they must fix their broken monolith but they are seemingly incapable of doing so.
It took ages for us to get permission and licenses. We only recently finished moving the last repos over and shut down Bitbucket.
If AWS offered a product like GH, I could probably unilaterally start moving to that. Any other alternative would take 3-4 years of meetings, budgets, lawyers, and other such nonsense before we could declare we'd left Github.
So - where do we stand in August 2026?
*EDIT:* No, I don't want anything related to Felon Musk.
While GitHub is imploding itself, maybe you should think about self-hosting.
As an example:
$> gh omens
reckon github's about to go full bin chicken
—-
You can find it here: https://github.com/sandermvanvliet-stack/gh-omens
LOL is this really something Aussie's say? That’s hilarious!
https://chromewebstore.google.com/detail/is-github-down/lcfo...
A VSCode extension:
https://marketplace.visualstudio.com/items?itemName=RuslanRy...
Firefox:
https://github.com/matagus/github-status-checker
Caveat: https://news.ycombinator.com/item?id=49450924
(disclaimer: none of these are recommended by me to use - merely sharing to build on for this discussion)
I'm also not sure why you'd share links to software you don't even recommend yourself? Isn't it better to just not share that then? I think others could use search engine/LLM too if they need whatever, but most generally expect things shared in the comments here to actually at least have been looked at by the person sharing them.
Which surprised me because of the many hundreds of services I rely on regularly that also have status pages.
Uptime reliability is nkt additive, it's multiplicative, sometimes even logarithmic.
Downtime of 99% and 99% is 98.1%. I hope almost all those services are non-prod related.
It could be a new Microsoft feature !
Probably just a coincidence that Github started to have issues after beginning their move to Azure at the end of last year.
I wonder if it was down that much, if users would get credits the way we pay when we use the services - its kind of ridiculous for a critical service to be down that much and all we do is "ah okay, its just github". Like, as if that was normal to be down that much...
Retries continue until the message is transmitted or the sender gives
up; the give-up time generally needs to be at least 4-5 days.
And now they're blaming their upstream vendor! Embarrassing stuff to be writing on a public page.
Seems like a weird thing to post on a status page. Shouldn't this have happened automatically and therefore precluded the need to inform users of it?
If the problem is scalability, just rate limit git commands on free accounts already. Nobody realistically need to push multiple times per minute, and that alone is bound to trickle down to anything that triggers on commits and pushes.
There may be non technical reasons for it, even: legal, or PR, but if they want me to believe the "scalability excuse" they need to convince me they're trying.
This is why it's hard to take GitHub seriously. How can a single database cause an outage for everyone? This is amateur stuff. Have they no sharding or partitioning internally? Paying customers should not be impacted in the same way as free ones are.
RDBMS integrity basically requires that one master server is responsible for the whole data set and other servers may replicate from it. And it usually doesn't wait for a quorum of replicas, just for one, because the design is to recover from a hardware failure, not a network partition, although that could be fixed at the cost of increased latency.
> primary failover briefly improved performance but did not fully mitigate, we've throttled inbound traffic and are investigating upstream Vitess issues
Maybe you expect that part of GitHub to have a scale where a single database can't handle it, but evidently that isn't true.
We can criticise them for not splitting up free and paid customers but again, most companies don't do that.
I thought I was cynical enough about goings on at Microsoft / Azure, but apparently not.
Right.
I'd like to think that Azure has improved in a meaningful way in the past half a decade or so... but it has not. Maybe less inexplicable 400 errors in random API calls, I guess.
And yet Microsoft keeps posting record growth. Unbelievable.
I'm told that GitHub has asserted to us that moving to this model means we would not be exposed to github.com outages. It's not at feature parity with github.com though.
We're currently on "GitHub Enterprise Cloud" on github.com and are affected by this outage (even though we use self-hosted runners!), but we're not on "GitHub Enterprise Cloud with data residency" on *.ghe.com, which I understand is/may not be affected by this outage?
I wonder how much egg is on that exec's face.
It’s not the same thing as GitHub Enterprise Cloud hosted on the shared global network on github.com.
https://docs.github.com/en/enterprise-cloud@latest/admin/dat...
The real money is going to go towards, "make this all more reliable".
Their last postmortem made clear that their challenges are operational. Scale puts pressure on operation, but it's not what blocks them from keeping up.
Doubling the operation doubles the operational challenges.
I suspect that rate-limits would fly in the face of a lot of narratives like “Azure is ready for hyperscale”. And getting eyes on Github is probably a part of their Copilot product strategy.
This has been a bad time, though. I'm ready to move back to self-hosting if they can't get it together or we move to GHDR and it's still bad.
Their bin-chicken status is testament to their adaptability, but it’s essentially our fault they’re that way.
> It would be really cool if someone built an extension to show GitHub status live
"These already exist, why wouldn't you search before posting?"
> There exist quite a few extensions to show live GitHub status
"Why would you not post them if you know they exist?"
Or, what actually happened:
> Here are some extensions that might work for you to show live GitHub status
"Why would you recommend extensions to do this?"
Seems to me like, if your goal is to pick apart someone suggesting something, you'll find a way to.
This is not my complaint though, my complain is:
Why would you recommend extensions you haven't tried nor even read about yourself?
The goal isn't to pick apart the message to piss someone off, the goal is prevent someone else than the author from getting hacked because they install some random extension, not understanding what kind of access you're giving others when doing so.
Is all content in HN supposed to be professional and serious, or are humorous comments in fact allowed when discussing a market leading company who has had regular outages for a year and a half? (Yes, I know the actual reasons why GitHub are struggling.)
Overall, I disagree and think it's valid and responsible to share links to apps if I disclaimed that I don't recommend installing their linked applications in an environment that has access to private data.
One valid reason, to share links, is to illustrate that there are more than zero efforts to address this issue - the parent comment issue - through and even more convenient practice than firing up a web browser, following link, waiting for it to load, reviewing the material, to see if GitHub status is red or green today. So this reason attempts to build up the importance of the parent comment's idea, and suggests that legitimate verifiable work is advisable, to continue down that path of making it more convenient for developers have simpler, more human indicators about the reliability of their digital tools.
Pretty simple reason but that's my reason.
Common Name (CN) www.dayswithoutgithubincident.com
Organization (O) <Not Part Of Certificate>
Common Name (CN) YR1
Organization (O) Let's Encrypt
Issued On Monday, August 10, 2026 at 10:01:51 AM
Expires On Sunday, November 8, 2026 at 9:01:50 AM
Common Name
R12
Validity
Not Before
Tue, 10 Feb 2026 17:28:52 GMT
Not After
Mon, 11 May 2026 17:28:51 GMTEdit:
Read that document - why do more people not do the self hosted option with GitHub Enterprise Server ?
This can be implemented in front of any RDBMS as a separate layer. Traditionally they weren't designed for quorum-sensing since it hadn't been invented yet. I'd be surprised if something like pgbouncer couldn't do it.
In the example, in my 20 year career, I've seen bitbucket in use once. I'm the one who usually manages that stuff since I'm an infra eng.
but I always have sense of fear that I might need it someday
Works great for small-medium scale
But GitHub, in its heyday, was a centre. Created a sense of "community"
It has taken a while, but since MS bought it it has been shirking that. I expected honest enshittification, but instead it has been technical collapse
What ever.
IMO we need a federation protocol for Git that can rebuild some sort of "community", but on solid foundations.
So we can find one another on our self hosted instances
I’m not happy with the Actions API. I think Gitlab’s cicd design is much better, and I’m not fighting it all the time when I use it.
sys-admin-ing gitlab can be a bit of a maze of web menus tho.
I used to use Gitlab at work, and small teams would run into so many footguns with CI that we had to throw up guardrails to prevent mistakes. Far too many links to gitlab issues that were not fixed even after >8+ years of being open ended up biting us. With GHA I haven't had that experience, and same for all of my self hosted Forgejo instances. I used to hate using GHA from about ~2018 to 2021, but they've fixed a lot of things I disliked since then.
Maybe I’ve been burned too much by pipeline maintenance (because we didn’t have yaml anchors?) but I rather have builds defined in make /bazel/etc than in yaml. So the only thing the pipeline does is optionally restoring caches, kicking off the build system, uploading PR validation results, and saving cache. Pushing artifact etc is all done from inside the build system.
There is no “setup” like installing packages because we make the build image seperately.
No
> are humorous comments in fact allowed
Yes
Neither is what's going on here. It's reckless to share stuff you haven't even looked at yourself, that's fairly basic thing to care about. Sharing "here's a bunch of random GitHub repositories" isn't humor (I think?) nor would it been different if the text was "more professional and serious", whatever that means.
Why you so mad about someone trying to help others from getting hacked? Install all extensions you want, based on what someone recommend or what you find randomly on YouTube, but let others be educated about what that means for their own security, not sure who that hurts and why you'd wanna stop it.
Yes it's reckless to say that I trust these browser extensions and GitHub repos with my private information.
In this circumstance, and somewhat ironically because of the actual issue at hand of us not being able to trust GitHub when we need to trust GitHub in order to get through our daily triage burdens, but in this circumstance, I wrote that I did not trust these.
So, I'm sorry for the confusion that it's causing, but it's really an indicator, showing that other people are experiencing frustration around the friction between they themselves realizing something is not working, and them resolving the issue after going through a potentially lengthy process to figure out that it actually is their source control and remote developer automation tooling that is failing them.
So I didn't put that conclusion in my comment.
When you two, in this thread, talk about humor on hacker News, I see where you're going. I really should have just put a conclusion in my comment so that it staved off the wandering to help make my comment more comprehensible, more quickly.
Sorry to have blown as much mental energy as it did, obviously resulting in an unwelcome and toxic result.
You could have posted a warning to not actually install random extensions. Instead you came across as questioning why anyone would post links to them at all. The poster was clear they didn't actually recommend the extensions. Sharing stuff is fun, and hardly any software posted to HN is professionally reviewed beforehand anyway.
Sharing stuff we know are good is fun. Sharing just random stuff? No, why would you do that? The whole point is why would you share those extensions if you don't actually recommend them? It's noise, the opposite of why we all come here in the first place.
Do you also just submit whatever to HN as link submissions, without even reading it or checking it yourself?
I didn't say, and I probably should have said, "Use the fact that there are a number of these developer focused IDE/ browser extension tools as motivating information, about the emerging need to support human developers in there daily triage burdens, as is it's arguably a need .. a trend that is worsening."
but I still keep it on my browser, maybe someday I would like to fetch LLM and describe my interest over time
... I intended to indicate more a Quantity concept, the existence of these links is to suggest that there is additional interest and effort out there in the human universe of developers, that this whole GitHub disruption services issue is worsening and we all here on this Hacker News thread are trodding a somewhat well-worn path.
And I apologize for not making that conclusion in my comment so as to stave off spinning out a tangential discussion unnecessarily.