Incident with Actions(githubstatus.com) |
Incident with Actions(githubstatus.com) |
I know that frontier models (Astra, Fable, Opus 5.5) at some point always end up writing a test that unnecessarily elongates CI. I've seen everything from literal sleep calls in a test unit to arbitrarily deciding a test needs to download a 100MB file to prove something works. As a engineer, I catch these, but a vibe coder has no idea there's probably hundreds of these in their code making CI take 10-20 minutes. Hell, they probably don't even click the "Actions" tab.
What a mess.
I think that even if you stripped out non-professional vibe coders from the equation - the problem is still there, and will keep compounding.
But yes, I also hammer GitHub a lot harder nowadays. I've had to move all of my private repos to self-hosted bare metal because I chew through 2k minutes in a few days.
Last year Github said they are giving away 11.5B action minutes [1] for public repos for free, that translates to about ~30,000 cores . The explosion in commits, pull requests and action minutes that Vlad the CTO mentioned[2] in the August postmortem is approximately 10x and he also highlighted they added 3 Million cores to their fleet.
Assuming a similar 10X growth in open source; total public repo compute budget is only 10% of the new compute they have added. Only half of Github is on Azure as of April, and Azure itself is much much larger.
Last year numbers were worth $180 Million going by list prices of action minutes [3]. However that would be only $20-30M equivalent typical outlay for a mid-size tech company if they were actually buying 30,000 cores on the cloud ; and definitely much cheaper for Azure's procurement ;
It is a good PR strategy and very good deal for Open Source but is really only a small customer acquisition line item for them and doesn't likely meaningfully impact their compute issues one way or other.
[1] https://github.com/resources/insights/2026-pricing-changes-f...
[2] https://github.blog/news-insights/company-news/the-august-17...
[3] Action minutes are priced more expensive than even buying on-demand compute on the cloud including with Azure;
Those limitations were put in place for a reason, and using the work-around feels dirty. I recognize that all platforms are flawed once you start to do more than they offer, but I feel that with AI its easier to build the plumbing around it.
The question shouldn't be: can you do this, but 'is this the right thing to do'?
For one of my green-field projects that I just vibe-coded with AI's (where Claude, Codex and Gemini critique each other's design and code), a PR could go through many iterations (commits) until every AI approves, and if every commit runs the CI, it'd be very slow. So I eventually come up with a mechanism to only run CI when all reviewers approve. That improves things a lot.
So you can reduce the free resources they consume with the slop.
Nothing about complexity, simply github setup such an easy automated system but AI cares not about whether their commit+push is going to kcik off a whole rebuild.
Same thing happens when I run docker build loops. The AI gives very little shit, unless I tell it, about not busting cache; so it'll sit there for hours making minor changes just to make a 30 minute build.
AI has not concern about how long anything takes, in general, but it if it's just waiting for it to return, it won't get impatient.
https://us.githubstatus.com/posts/details/P7VGB7I
https://au.githubstatus.com/posts/details/PO54BK8
https://eu.githubstatus.com/posts/details/PRESCZY
https://jp.githubstatus.com/posts/details/P0N7ZG5
What's the point of (supposedly) separate and isolated data residency deployments if they all have single point of failure?
It was ridiculously easy to set up, reduced my issues with GitHub significantly and the mini will literally pay for itself 3-4x over this year with the $$$ I'm saving not paying for GH actions.
The most surprising part for me was not the cost savings, but that all my workflow runtimes went down by about 70-80%.
We were thinking about opensourcing our flows.
I don't know why but individual users have to create a single instance runner and mate it with exactly one of their repositories.
Converting from an individual user to a GitHub organization is not simple. The docs certainly make it look that way but I have yet to run into anybody that has done so without some form of catastrophic error that requires support to fix
Those issues affecting other services are not visible anywhere else on the status page, are they?
Meaning your alternative provider is also facing massive outages?
I hear many complaints about github these days due to outages. I am genuinely curious if anyone has switched recently to a competitor for this and gets more uptime?
(I said "believable", not "realistically")
So the real reason GH continues to grow is that betting on GitLab just means that you'll probably run into the same problem at some point, except they aren't a hyperscaler and if even Microsoft has to span their workloads into AWS to scale, what hope can you have with smaller vendors?
The annoyedAtGitHub counter is so far monotonically increasing though.