GitHub Incident with Git Operations, Pull Requests and Actions(githubstatus.com) |
GitHub Incident with Git Operations, Pull Requests and Actions(githubstatus.com) |
A few hours ago GH pages just would not publish at all.. and the status page showed as fine, I guess it's now cascaded into a proper event.
GitHub is never going to improve and at this point. You might as well say that the AI chatbots, Tay and Copilot are maintaining the platform and are running it into the ground. No CEO of GitHub to go to as well.
It is time to self host instead of using something as broken as GitHub as I predicted 6 years ago.
Not many people are interested in providing investment funding a meaningful competitor to GitHub. There are sexier, bigger bets out there than "providing a place for version control software to be hosted". The people who are frustrated enough with GitHub to self-host have the ability to do so. The rest of us just get up from our desks, go for a coffee break/walk, and wait it out, because migrating to self-hosted is a PITA and not a core business competency.
They'll keep having outages, and most will keep paying for it. If they're paying for it despite fewer resources being spent on uptime, that means the company's more valuable as an equities market participant. Line go up.
Premature declaration of Resolution is pretty bad user experience, if I’m being honest…
Google may be more reliable overall, but Gcloud at least tends to be more critical to a business that uses it, so it kind of balances out. I'd much rather have a Github outage than a Gcloud one.
It's not getting better, if anything it's getting worse. Best time to get off GitHub was yesterday, the second-best day to get off is today. Tangled, Codeberg or self-hosted Forgejo (my approach) or Gitea are all good alternative solutions here.
1. Homebrew really, really wants you to host your taps on GitHub. The docs are very GitHub-focused: https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap as are downstream projects like dist: https://axodotdev.github.io/cargo-dist/book/installers/homeb...
2. Terraform and OpenTofu public registries support only GitHub, see e.g. https://developer.hashicorp.com/terraform/registry/modules/p... and https://opentofu.org/docs/language/modules/develop/publish/#...
3. Only GitHub allows you to use free minutes on managed Windows and macOS runners, which are essential for building and signing native Windows and macOS/iOS software. GitLab's are in beta and require a paid plan, Buildkite requires you to pay at least $30/user without any bundled macOS minutes and no hosted Windows runners available. And for small shops, keeping the macOS and Windows runners clean is such a stupid time sink, with the small number of minutes involved it's much more preferable to just start with free managed minutes and plan to pay for managed runner minutes later.
They really should be partitioning their infrastructure so that their paying customers are not in the blast radius of outages from the legions of repos running their 700 test vibecoded regression suite every time they change a config file.
This misses the point in a bad way. You shouldn't have a single deploy funnel to begin with. You should be able to do deploys from multiple places.
However, rather than understand the obvious explanation for it, that reddit and HN's karma systems encourage performative comment behavior with people trying to be epic for the peanut gallery, they make the boneheaded decision to just assume it's an "illusion". It's like Dunning-Krueger but for their theory of mind and general emotional intelligence, with the cause being obvious enough that I don't feel I need to name it.
I hope BlueSky makes a Lemmy-style Reddit alternative, just something to satiate people's social media-ified forum fix.
1: The status site says that pull requests are working,
2: A PR I'm working on right now is missing commits,
3: The contact support page errors out.
I combined it with Zot for an OCI registry but there are quite a few choices there.
I've also noticed a meaningful decline in the quality and reliability of several other major software/apps and services - notably Google maps, Spotify, codex. (Is this because of AI?)
Codex, especially, is a strange one - initially I added it beside Claude code thinking it would actually be a higher quality, faster alternative (even if the model/harness is not quite as good at coding as Claude code/opus), and it seemed that way for the first few weeks. At this point though, codex is completely unusable for me - the contexts from different sessions are polluting/mixing with each other, prompt history is mixing/polluting each other. Codex randomly enters safe sandbox mode or whatever multiple times a day per session - it just starts seeing everything as read-only and loses tool and MCP access. It also randomly gets interrupted all the time for no apparent reason - 'your session has been interrupted' - so much for 'autonomous'. And holy shit codex is sooo slow, I'm guessing there are both server side and client side rate-limits, I have the 20x pro plan and I can't even use it up (I use up my Claude code pro 30x in 2-3 days, though to be fair Anthropic usage limits are arbitrary and low), because it is so slow plus all of the issues and constant random stops/blockers that prevent any kind of long-running autonomous work.
Opened a couple codex issues on GitHub for these things - all closed as duplicates. I look at the existing issues - all closed as resolved.
What a horrendous shitshow.
AI coding. Which can be interpreted one of two ways:
* The generous way, which is that Github is so overloaded with massive volumes of AI codebases and AI-driven automation that they're hitting a scale they never anticipated; or
* The not-so-generous way, which is that Github itself was one of the first companies to push everyone to AI code as much as possible, which has lead to an eventual breakdown of the stability of the system, as the people responsible for it no longer understand how it actually works, leading to production outages once every few days.
And of course, Microsoft is saying that none of this downtime is at all related to them moving everything to Azure, and also at the same time they'll fix all this downtime by finishing moving everything to Azure.
I wouldn't hold my breath here.
The data speaks clearly: https://damrnelson.github.io/github-historical-uptime/
And while I was typing this comment it just blipped again. Atleast this time I know why.
I feel dumb for setting up tailscale like 7 years ago using the github auth, time to figure out how to just move to email on that account.
Unless I've missed something, I don't think you can. I also had to use GitHub auth, as it was the least-bad option of the choices given. I wish they just did email/user + pass + TOTP like regular platforms.
https://tailscale.com/blog/passkeys
Maybe running your own OpenID service would work?
https://openid.net/developers/how-connect-works/
Too much work though. Easier to just switch to Nebula or Netbird and self-host the full stack.
There 4 updates contains webhook mention in 2 of the updates.
"Webhooks is experiencing degraded performance"
I noticed because SSO was down. They don't even report on that as far as I can tell.
Edit: 4 minutes later and I can push again
In the past 2 months I've seen 3+ instances where GH action were not triggered on commit pushes, leading to silent workflow failures, and seeing GH merge queue being stuck for 3+ hours for production product facing repos
Did I break this? Making me insecure about changing anything on github.
A self-hosted enterprise GitLab instance in Azure, on the other hand, has had outages and performance degradation quite a few times.
A GitLab runner is a machine/VM which can get notified of a GitLab CI job, then run the job's container image (downloading it from GitLab's container registry if necessary), then run the code in the yaml file. You get to control your build environment.
It's so weird that GitHub Actions's model is "have one absolutely gigantic container image which contains everything any build could ever need, and if something's missing, the documented answer is to apt install it every run".
Maybe 30 minutes to get it all setup and then I moved on with my life.
Their status page records the outage anywhere between 6 to 17 minutes across the different features. The detailed status page clearly has recorded time of when the errors were escalated, when the fix was applied and when it was assumed that the system has recovered. Its clearly a lot more than 17 minutes, infact its more than an hour.
Unless there is something more to it, these status pages feel like a blatant lie, wondering how long before some one actually sues them cause they are publishing/advertising wrong SLAs.
Another example
> https://www.githubstatus.com/incidents/0rn90wk115q9
> On September 13, 2026, between 08:43 and 10:44 UTC
The status page records the incident outage duration as 1 hour 12 minutes
I'm going to be investigating self hosting, this is ridicolous.
Same as with container registries; prod shouldn't be pulling from Dockerhub.
(about to be hit with rotten vegetables by the purists...)
Lots of things we should do, only a few we actually have time to fix. If I were choosing between "Migrating away from GitHub" and "Adding another way to deploy in case GitHub is down", I'd take advantage and do the first, because you'll end up having to replicate SCM, build infrastructure and so on anyway, why not do it properly?
This misses the business reality in a bad way. A typical outage with something like Github is not a big deal for most businesses. Sure, some nerds get annoyed, but that's about it. The extra effort in maintaining multiple deploy approaches for a non-trivial system simply doesn't make business sense in most cases.
Also, the uptick in odd issues post acquisition pales in comparison to the scaling issues they've had over the past 1-2 years. As a matter of scale, trying to link this back to Azure doesn't really square. If anything they'd potentially be in a worse spot without having access to the resources of a massive public cloud..