Incident with Actions – Resolved(githubstatus.com) |
Incident with Actions – Resolved(githubstatus.com) |
> We are working with our compute provider to alleviate elevated queue times and failures for Actions Jobs running on Hosted Runners in the East US region affecting 10% of runs. Hosted Runners with private networking can fail over to a different Region to mitigate the issue.
> We are investigating elevated queue times on Actions Jobs running on Standard Hosted Runners in East US affecting 10% of runs
Frustrating if you were impacted, but from the comments here you'd think the entire site was down again.
All those "GitHub is dead" and "GitHub will crumble soon" cries are massively exaggerating things. Most users probably won't even notice these "issues" or "downtimes", since they only affect certain parts of the site.
Sure, it's annoying if you're impacted, but that still doesn't spell doom for GitHub as an organization
What I can attest to is that Github has been uncharacteristically flaky this past year. At least for large clients in the EU. Its not that there is outright downtime but if you have an Actions or PR invested team you probably have felt the uptick in troubleshooting these two features in the past 12 months.
And again, its not that these features go completely down, mostly its just "why is this status not being reported" or, "where is the run for this event?" And similar things. Its not that the roof fell off, its just that it is leaky and it rains and this distracts you from actually doing important things.
> … platform activity is surging. There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
> GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.
My outsider perspective is that GitHub teams are having to fight for compute, and since Actions can be billed and timed, it has become the default compute layer for everything. But it makes for a terrible experience as an end-user.
[1] https://liberapay.com/forgejo
Can't even do three 8s properly.
Imagine you run 365 services, and each goes down 1 day a year.
If those are all on the same day, this would report you having 99.7% uptime.
If instead, each service goes down 1 day per year but on different days, this would report you having 0% uptime.
Despite the same actual downtime for any given service.
The truth is somewhere in the middle, that github has run degraded for a significant amount of time.
But I don't think it is fair to take an incident like this one[1], where 5% of requests were incorrectly denied authorisation, and count it the same as you would the whole of github being down.
2. Github is doing some classic big org sneaky things where they don't count degraded service fully. So if github actions is partially down for most people in a away that makes you say "github is down", there's a good chance that microsoft doesn't count that or counts it partially instead.
We're not treading new ground anymore.
Here's a few of the better ones in the last 2 weeks;
https://news.ycombinator.com/item?id=48012022
https://news.ycombinator.com/item?id=48010301
https://news.ycombinator.com/item?id=47924775
Another challenge is that the GitHub Actions paid tier is already very expensive, the quality of service is poor, and they have major security challenges. They could load shed by raising prices, driving customers to other platforms, but they already charge 10x what others charge (https://runs-on.com/pricing/#runner-pricing, https://www.ubicloud.com/docs/about/pricing). Anyone using GitHub Actions at scale would be somewhat price insensitive already.
Unless you're suggesting driving demand to zero...
Incidents spiked recently and I wonder what's repeatedly causing the issue.
Mistakes (preventable or not) can happen, but repeated failures is a systematic issue.
It's the second day in a row now where the main HN post about GitHub downtime has had "GitHub" edited out of the post title.
So what are people who are at github doing right now? Like what do the priorities look like?
Once again a reminder for people to look at codeberg. An Uptime of 84.88% is just not acceptable Github. I don't think that Github can come out of this personally. This problem has gone for too long and has become too large for people to ignore.
Since we're guessing I could also imagine the opposite: stressful sure and heaps of work, but numbers absolutely through the roof, and lots of opportunities, bonus/promo-boosting wins, etc.
For my personal work I did a hard cutover to GitLab last month. The issues import is the most complex part as the default import messes up issue authors.
If everyone is vibecoding, and SaaS plays have no moats anymore, and everyone says they are mad at Github's reliability...why aren't there like 10 viable replacements already?
Why are you still using Github?
We should probably take a break on these. It's probably more newsworthy now when GitHub is "up".
IT seems to be quoting incident reports for the duration of each outage, so there is accountability in terms of being able to verify all the details of what they are counting.
> and why it's sooo different from gh's official status numbers?
Maybe this is counting any period with any service showing any level of issue as a complete fail, and the official numbers are cherry-picking a bit (only counting core services? not counting significant performance issues that the other count does because things were working, just v…e…r…y … s…l…o…w…l…y) or averaging values (so 75% services running at a given time looks ¼ as bad in their figures), the two sets of calculations could be done with a different granularity, …
In other words: lies, damned lies, and statistics!
The only way to know is to know how both are calculated in detail, and that information might not be readily available.
- And yes, GitHub is a massive product with like 50 different huge features. No reasonable person would say you can trivially vibecode that. Vibecoding would still make it easier. I feel this argument is a bit silly, no? "Ah, you can't vibecode GitHub in a weekend? That proves vibecoding was a mirage!" Surely even the most fervent anti-AI skeptic must admit there must be some middle ground between "a mirage" and "can literally replace millions of man-hours of work".
Open-source projects, yes I can sort of get it, you want to be where the contributors are (but there are downsides to that also).
I have yet to see literally anyone say this.
I have yet to even see anyone claim that software can't constitute a moat anymore but I expect that there are people saying that. GitHub has a huge non-software moat in the form of network effects, brand recognition, and good will.
The hardest part isn't making a "forge", it's making money off of making a forge. Getting a sufficiently large number of paying customers.
If GitHub doesn't get their quality issues under control someone probably will manage to breach that moat and take over the market. It's not like there's a lack of competitors (Pre-llm: GitLab, BitBucket, Gitea, Source Hut, etc. Post LLM: Tangled, esrc is promising something any day now. Probably more in both camps that don't come to mind).
I've heard a lot of people say this...including myself after a root beers. I think you just have to look to any time an AI feature is announced and some related companies stock price crumbles. Just google something like "stock price tumbles after anthropic announces" or something like that.
I'm using GitHub for my open source projects as:
1. While GitHub Actions has its issues and doesn't work for everyone, I've found it easy to build and test an IntelliJ plugin against multiple IntelliJ versions.
2. I don't have to pay for and manage the hosting of the git repository.
Because everyone wants the fake internet points (sorry, stars) to mention on their CV.
Because there are already a number of viable alternatives, them not being chosen has nothing to do with AI coding but other factors like market momentum & network effects and familiarity. They are used, just much less so. If there are already good alternatives, why would anyone vibe code a new one any more than they would write a new one manually? Forges are not sexy stuff, and the existence of numerous decent free ones means that you aren't going to be able to sell a new one in any way (paid accounts, stalking/advertising, …) at least not until it has a significant following and that is unlikely to happen because of the reasons above.
People not wanting to use github (or one of the common alternatives that already exist) are more likely to just use git as-is, and other tolls as needed for issue tracking, CI, etc, than to create a new forge.
I still have a GitHub account I actively use.
GitHub outages and stuff don’t really affect me, so I have no great reason to leave. But I have good reason to stay, because that’s where everyone else is already.
Github is struggling because of compute, which comes from everyone vibecoding and triggering actions 10x more.
I can vibecode an alternative, but once i have users, who is going to secure this amount of compute? Compute+talent to manage it(devops isnt vibecoded *yet) is a moat
So I’m working on waza.sh. I have NO intent on making it commercial, nor open-source (unless someone wants to collab on it).
Will it need compute? Yes! For my team only. So a dedicated box at OVH/Hetzner for 30eu is more than sufficient.
Show me the VCs who are willing to fund the marketing effort that would be needed to conquer the network-effects moat, and I'm in.
Stars. Grifters have discovered vibe-coding and the ability to buy GitHub stars, followers, etc.
Rachel famously wrote about this in "Your nines are not my nines"[0].
The truth is though, that some systems depend on others. Actions being down means you don't merge code or release: but you know... git operations being unavailable has the same effect. It's meaningless to separate the two.
So it depends on the framing.
Even worse example is the Travis CI. For more than a year their CI jobs sometimes get stuck or do not start for days, and, surprise-surprise, it's never shown at their status page[1] - always green. We would switch to something else entirely if not the unique offering of PowerPC and SystemZ servers/runners. Apart from that - it's the worst CI service I used so far.
It's not like OP is running out of space in the HN title, either. How is "Incident with Actions" more user-friendly than including "GitHub outage" in the headline?
Furthermore, the subtitle of the OP is "Incident Report for GitHub," so why not just include this if we need to follow the rule so carefully?
Because if we follow carefully:
> If the title includes the name of the site, please take it out, because the site name will be displayed after the link.
This is a personal pet project, just to see what it might look like, how it would work, to satisfy personal curiosity. If it comes to the point that it actually becomes usable for the team (read: actually add value), then it's up to $TEAM to decide whether to use it or not.
I would hate my team to waste company time on such an endeavour. ;-)
Then they came for the three nines And I did not speak out Because I was not paying for Enterprise
Then they came for the two nines And I did not speak out Because the status page said all systems operational
Then they came for the one nine And I did not speak out Because I was a manager, and outages are just extra standup material
Then they came for the coin flip And there was no one left to merge my PR Because Actions was down And so was Pages And so was Codespaces And the status page said all systems operational