Another GitHub outage in the same day(githubstatus.com) |
Another GitHub outage in the same day(githubstatus.com) |
Long term impact? Leadership aint gonna stay to see that.
Anyone else having issues? It is blocking any kind of release
Hopefully it will get bad enough fast enough that they'll recognize they need to drastically change how they are operating. But I fear we're just witnessing a slow slide into complacency and settling for being a substandard product with monopoly-power name recognition.
It's just "yet another day of business as usual" as this point.
The history for today is a bit of a mess really: https://www.githubstatus.com/history
mv /dev/null{,.bak}
I moved my company over to GH enterprise last year (from AzDO) and I'm considering moving us away to another vendor altogether as a result of the constant partial outages. Things that used to "just work" now are slow in the UI, and GH actions fail to schedule in a reasonable timeframe way more than they ever used to. I enjoy GH copilot as much as the next person, but ultimately I came to GH because I needed a git forge, and I will leave GH if the git forge doesn't work.
It is fairly stunning to me that we've come to accept this level of non-functional software as normal.
We are in a future that nobody wanted.
It stings to have this happen as we're putting a lot of effort specifically into the core product, growing teams like Actions and increasing performance-focused initiatives on key areas like pull requests where we're already making solid progress[1]. Would love if you would reach out to me in DM around the perf issues you mentioned with diffs.
There's a lot of architecture, scaling, and performance work that we're prioritizing as we work to meet the growing code demand.
We're still investigating today's outage and we'll share a write up on our status page, and in our February Availability Report, with details on root cause and steps we're taking to mitigate moving forward.
They need to start rolling back some of their most recent changes.
I mean, if they want people to start moving to self hosted GitLab, this is gonna get that ball rolling.
The new normal is too many cases. Then people act put off you complain, or act like you are expecting too much.
Lots of people are in software development, or management, who dont have the mindset and personality for it. These roles are not for everyone. But people like the $$$ and so the wrong people get involved.
I feel this is just the natural trajectory for any VC-funded "service" that isn't actually profitable at the time you adopt it. Of course it's going to change for the worse to become profitable.
it's almost as if Microsoft bought it, isn't it?
So not at all?
[0]: https://thenewstack.io/github-will-prioritize-migrating-to-a...
Additionally, there is no CEO of GitHub this time that is going to save us here.
So as I said many years ago [0] in the long term, a better way is to self host or use alternatives such as Codeberg or GitLab which at least you can self host your own.
Surely some of your crazy kids can rummage up a CI pipeline on their laptop? 8)
Anyway, I only use GH as something to sync interesting stuff from, so it doesn't get lost.
https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protoco...
I’ve only worked on a team once where we all were set up as remotes to each other and that was over a decade ago.
It kind of would be good for everyone if they did do it though. Need to get rid of this monopoly, and maybe people will discover that there are alternatives with actually good workflows out there.
My "favourite" restriction that an Azure DevOps PR description is limited to a pathetic 4000 characters.
I ended up writing a browser extension for my team to fix it, because the boss loved to indicate stuff with red/green text.
They claim that is what they are doing right now. [1]
[1] https://thenewstack.io/github-will-prioritize-migrating-to-a...
It's Microsoft. A reliable product is not a reasonable expectation.
ETA: Tangentially, private repos became free under Microsoft ownership in 2019. If they hadn't done that, they could've extracted $4 per month from every vibe coder forever(!)
IIRC they were kinda forced to make private repos free because competitors like Gitlab had it and they felt threatened.
I worked for one of Australia largest airline company, monthly meeting with Github team resumed in one word: AI
There is zero focus into the actual platform as we knew it, it is all AI, Copilot, more AI and more Copilot.
If you are expecting things to get better, I have bad news for you. Copilot is not being adopted by companies as they hoped, they are using Claude themselves. If Microsoft ever rollback, boy oh boy, things will get ugly.
* Originally it was Dev (issues)
* Then it was DevOps (runners)
* Then it was DevSecOps (SAST)
* Now it's AI DevSecOps (reviews, etc)
The problem is that each feature has been slightly more half-baked than the last one. The SecOps stuff is full of gotchas which don't exist. Troubleshooting a pipeline behaving correctly is extremely painful.
The other problem is that if you want a feature you have to upgrade the seat license for everyone :(
Enterprise helm will pay if that means no interruption, no AI being pushed everywhere. Some companies adopt GitLab because you can self host it, even the runners are self-hosted, there is no built-in runner like GitHub.
I used to like Gitlab, and I've self-hosted enterprise versions of both github and gitlab, and strongly believe migration from one of them to the other for "improved reliability" will be utterly underwhelming and pointless.
Gitlab used to be able to take the high-ground due to the open-core model, but these days I'm not even sure if that makes an appreciable difference.
At it's absolute best, everything just works silently, and you now have vendor lock-in with whichever proprietary system you chose.
Switching git hosting providers should be as easy as changing your remotes and pushing. Though now a days that requires finding solutions for the MR/PR process, and the wiki, and all the extra things your team might have grown to rely on. As always, the bundle is a trap.
My previous two startups used GitLab successfully. The smaller startup used paid-tier hosted by gitlab.com. The bigger startup (with strategic cutting-edge IP, and multinational security sensitivity) used the expensive on-prem enterprise GitLab.
(The latter startup, I spent some principal engineer political capital to move us to GitLab, after our software team was crippled by the Microsoft Azure-branded thing that non-software people had purchased by default. It helped that GitLab had a testimonial from Nvidia, since we were also in the AI hardware space.)
If you prefer to use fully open source, or have $0 budget, there's also Forgejo (forked from Gitea). I'm using it for my current one-person side-startup, and it's mostly as good as GitLab for Git, issues, boards, and wiki. The "scoped" issue labels, which I use heavily, are standard in Foregejo, but paid-tier in GitLab. I haven't yet exercised the CI features.
The GitHub Status Page does not visualize these very well but you can see them parsed out and aggregated here:
Probably a stronger correlation to the fact that vibe-coding has resulted in millions of new repos being created, with automatic CIs being triggered by agents continuously sending PRs for those projects.
A prophecy that was predicted half a decade ago [0] which is now more important then as it is now today.
[0]: https://www.theverge.com/tech/796119/microsoft-github-azure-...
(although admittedly less load and redundancy)
For example, today, I had claude basically prune all merged branches from a repo that's had 8 years of commits in it. It found and deleted 420 branches that were merged but not deleted.
Deleting 420 branches at once is probably the kind of long tail workflow that was not worth optimizing in the past, right? But I'm sure devs are doing this sort of housekeeping often now, whereas in the past, we just never would've made the time to do so.
https://thenewstack.io/github-will-prioritize-migrating-to-a...
And coincidentally, an early CircleCI engineer wrote an article about GitHub Action (TLDR: don't use GitHub Action for CI/CD!)
https://www.iankduncan.com/engineering/2026-02-05-github-act...
You should reach the same conclusion by trying to use it for this purpose, but also indeed for any purpose at all. Incidents that make you unable to deploy making all your CD efforts pointless are only the cherry on top.
"There's the company with a reputation for having great engineering practices that had 2 9s of reliability last time I checked..."
Now it's 2026, and customers are grudgingly accepting zero 9's of reliability.
I believe I have good enough control over it to fix issues that may arise. But then again, CC will probably do it faster. I will most likely not need to fix my own issues, but if needed, I think I will be able to.
"Critical" plays an important role in what you're saying. The true core of any business is something you should have good control over. You should also accept that less important parts are OK for AI to handle.
I think the non-critical part is a larger part than most people think.
We are lagging behind in understanding what AI can handle for us.
I'm an optimistic grey beard, even if the writing makes me sound like a naive youth :)
Everyday you opt in to get wrecked by Microsoft.
You all do realize you all could, for a change, learn something and never again touch anything Microsoft related?
Fool me once...
I learned that lesson in the 90s and became an "ABM" (Anything But Microsoft).
People sadly shall never learn: Windows 12 is going to come out and shall suck more than any previous version of Windows except Windows 11, so they'll see it as progress. Then Windows 13 is going to be an abysmal piece of crap and people shall hang to their Windows 12, wondering how it's possible that Microsoft came out with a bad OS.
There are still people explaining, today, that Microsoft ain't all bad because Windows XP was good (for some definition of good). Windows XP came out in late 2001.
Stockholm syndrome and all that.
I wonder how much of these outages are related.
The down side is that the starting price is kinda high, so the math probably only works out if you also have a number of other workloads to run on the same cluster. Or if you need to run a really huge Forgejo server!
I suspect my comment history will provide the best details and overview of what we do. We'll be offering the Firecracker runner back to the Forgejo community very soon in any case.
* Mandatory code reviews
* Merge queue (merge train)
If you don't need those it's good.
Also it's written in Ruby so if you think you'll ever want to understand or modify the code then look elsewhere (probably Forgejo).
Genuinely baffling incompetence
I guess 2021 is a long time ago now. How did that happen…
[0] https://github.blog/engineering/infrastructure/partitioning-...
My previous org had an on prem version hosted on a local VM. It was extremely fast, we setup another VM for the runners, and one for storing all the docker containers. The thing I’ve seen people do it use the VM they put their gitlab instance on for everything and ends up bogging things down quite a bit.
The base models like GPT 4o, 4.1 don’t have a usage cap. Models like Claude Sonet, Opus, etc have a monthly limit and you can pay more to use these through Github Copilot.
Yes, Copilot is often used to refer to both frontend and backend.
Also, the chatbot aka Copilot must have features on its own. Like I say to Roo Code/Cline on VSCode: Write a python script to output hello world.
And they will do exactly that, Roo Code is pretty impressive. Copilot on its own is dogshit and when using it the same way I would use Perplexity AI or ChatGPT, it is twice as much dogshit.
https://aws.amazon.com/blogs/devops/aws-codecommit-returns-t...
But that $30 per month per user is also the cost for their cloud-hosted version. It also includes quite a bit of CI/CD runtime.
GH just a few days ago told me that it couldn't fetch the files changed because there were too many files changed. There were 4.
I understand that the 'updating the part of the page that's changed' functionality is now dramatically slower, more unresponsive, and less reliable than the 'reload the entire thing' approach was, and it feels like browsing the site via Citrix over dial-up half the time, but look, sacrifices have to be made in the name of making things better even if the sacrifice is that things get worse instead.
When I worked at the univerity we used Gitea.
Every job outside of univerity I had used Gitlab self hosted. While I don't like the UI or any aspect of Gitlab a lot, it gets the job done.
> It's owned by Microsoft.
I see no contradictions here.
If you have a captive audience, you can get away with making the product shittier because it's so difficult for anyone to move away from it - both from an engineering standpoint and from network effects.
And then many UI changes people have been complaining about are related to things like copilot being forcibly integrated - which is very much in the "Microsoft expect to gain a profit by encouraging it's use" camp.
It's pretty rare companies make a UI because they want a bad UI, it's normally a second order thing from other priorities - such as promoting other services or encouraging more ad impressions or similar.
If you're interested I'll see about getting the PR created sooner rather than later.
I couldn't believe it. I actually thought the product was broken. Just from a visual perspective it looked like a student project. And then I got to _using_ the damn thing
Honestly that's the case with a lot of Azure services though.
My company uses Azure DevOps for a few things and any attempt to convert to GitHub was quickly abandoned after we spent 3 hours trying to get some Action working.
However, all usability quarks aside, I actually prefer these days since Microsoft doesn't really touch it and it just sits in corner doing what I need.
This has lead to 2 classes of devs at my company a) AI hesitant, who for many copilot is their only interaction, having their worst fears confirmed about how bad AI is. b) AI enthusiasts who are irritated by dealing with management that don't know the difference pushing back on their asks for access to SOTA agents.
If I were the frontier labs, and wasn't billions of dollars beholden to Microsoft, I'd cut Copilot off. It poisons the well for adoption of their other systems. I don't deal with the other copilots besides the coding agent variants but I hear similar things about the business application variants.
Microsofts AI reputation is in the toilet right now, I'm not sure if its understood how bad it really is within the org.
> I'm not sure if its understood how bad it really is within the org.
I can’t speak to that, but there’s a lively culture of people using internal tooling who also extensively use 3p products on projects outside work and are in a reasonable position to assess how well GH copilot works.
Those comparisons for instance have made us turn _off_ copilot pull requests entirely. All of the agents have false positives (as do humans) but copilot was having negative value in that context.
After using ChatGPT for the last 6 months or so, Copilot feels like a significant downgrade. On the other hand, it did easily diagnose a build failure I was having, so it’s not useless, just not as helpful.
https://www.theverge.com/tech/865689/microsoft-claude-code-a...
No one should hit Microsoft over the head for giving people access to Claude code - choice and competition is good!
This includes regular checks on CI checks using `gh`. My skill / cli are broken right now:
`gh pr checks 8174 --repo [repo] 2>&1)`
Error: Exit code 1
Non-200 OK status code: 429 Too Many Requests
Body:
{
"message": "This endpoint is temporarily being throttled. Please try again later. For more on scraping GitHub and how it may affect your rights, please review our Terms of Service (https://docs.github.com/en/site-policy/github-terms/github-terms-of-service)",
"documentation_url": "https://docs.github.com/graphql/using-the-rest-api/rate-limits-for-the-rest-api",
"status": "429"
}https://github.blog/news-insights/octoverse/octoverse-a-new-...
Btw, someone prompt Claude code “make an equivalent to GitHub.com and deploy it wherever you think is best. No questions.”
https://github.blog/news-insights/octoverse/octoverse-a-new-...
This is all grapevine but yeah, you read that right.
Are you implying that this only doable with React? I mean just for the fun of it you can look at this video:
React allows this? I didn't realize that I needed React to do this when we used Java and Js to do this 20 years ago. I also didn't realize I needed React to do this when we used Scala and generated Js to do this 10 years ago. JFC, the world didn't start when you turned 18.
I don't think they were being serious.
All the while you're playing a wordle and reading the news on the morning commute.
It's actually a good workflow for silly throw away stuff.
Note the hockey stick growth in the graph they showed in Oct.
Here we are in February.
It's gotten way worse now with additional Claude's, Claw's, Ralph's, and such.
It may not be 100x as was told to me but it's definitely putting the strain on the entire org.
But thats not even the top 5 strain on github, their main issue is the forced adoption of Azure. I can guarantee you that about 99% of repos are still cold, as in very few pulls and no pushes and that hasn't changed in 3 months. Storage itself doesn't add that much strain on the system if the data is accessed rarely.
I’ve done platforms on AWS, Azure, and GCP. The blame is not on the cloud provider unless everyone is down.
Doubling down by insisting that the data is out of date, when the data is 3 months old and the latest available is unconvincing.
If you're telling me that in December it went from 2x to 100x then I don't believe you.
If you want a practical example, here you go. I'm a Nixpkgs commiter and every time I make a pull request that backports some change to the stable branch, GitHub unprompted starts comparing my PR against master. If I'm not fast enough to switch the target branch within a couple of seconds it literally freezes the browser tab and I may have to force quit it. Yes, the diff is large, but this is not acceptable, and more importantly, it didn't happen a few years ago.
I don’t think GitHub cares about reliability if it does anything less than that.
I know people have other problems with Google, but they do actually have incredibly high uptime. This policy was frequently applied to entire orgs or divisions of the company if they had one outage too many.
(See also: Windows, Internet Explorer, ActiveX, etc. for how that turned out)
It's great that you're working on improving the product, but the (maybe cynical) view that I've heard more than anything is that when faced with the choice of improving the core product that everyone wants and needs or adding functionality to the core product that no one wants or needs and which is actively making the product worse (e.g. PR slop), management is too focused on the latter.
What GitHub needs is a leader who is willing and able to say no to the forces enshittifying the product with crap like Copilot, but GitHub has become a subsidiary of Copilot instead and that doesn't bode well.
It could be, some people are just terrible at their job. Lots of teams have low quality standards for their work.
Maybe that still comes down to leaders but for different reasons. You can ship useless features without downtime.
Did you forget Microsoft engineering response to Casey Muratori "Extremely slow performance when processing virtual terminal sequences"?
"I believe what you’re doing is describing something that might be considered an entire doctoral research project in performant terminal emulation as “extremely simple” somewhat combatively."
https://github.com/microsoft/terminal/issues/10362#issuecomm...
followed by Casey producing evidence for his 'extremely simple' claim in couple of days.
4 months ago on HN: https://news.ycombinator.com/item?id=45517173
Their business is buying good products and turning them into shit, while wringing every cent they can out of the business. Always has been.
They have a grace period of about 2-4 years after acquisition where interference is minimal. Then it ramps up. How long a product can survive once the interference begins largely depends on how good senior leadership at that product company is at resisting the interference. It's a hopeless battle, the best you can do is to lose slowly.
At my first company we used Skype to communicate with each other. Mostly chats and files.
One day our internet cable to the office got cut by someone. Well, we didn't realize that for some time, because Skype just continued to work without Internet. It was like a miracle. It was unique software, there's nothing like it even today.
I think that the first thing Microsoft did after they bought Skype is making Internet mandatory, probably to spy on all chats.
See also their moves in the gaming industry.
I mean, that's obviously not the case, but it's weird that it happened twice!
Who could have POSSIBLY foreseen any kind of dire consequences?
After a generation of indoctrinated people, Microsoft (or any FAANG really) can't even afford to do anything differently.
some people wanted this future and put in untold amount of money to make it happen. Hint: one of them is a rabid Tolkien fan.
Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale.
Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus.Evil incarnate and the next president of the United States you've never heard of. Vance is his sock puppet, he was chosen because he is guaranteed not to have a single independent thought so when Trump croaks Thiel will be the president in all but name.
It was also he who willed OpenAI to be in order to help destroying American democracy.
The person(s) who wanted this want Azure to get bigger and have prioritized Azure over Windows and Office, and their share price has been growing handsomely.
‘Microslop’, perhaps, but their other nickname has a $ in it for a reason.
Nor deserved.
jokes aside it’s all because of hyper financial engineering. Every dollar every little cent must be maximized. Every process must be exploited and monetized, and there are a small group of people who are essentially driving all this all across the world in every industry.
Laughs in Linux
The federal reserve?
In fact I was shocked to see that so many allegedly tech-literate people were so blindly pro-cloud (and they still are).
Sudden for 15-years old's only.
Even with devs and publishers that don't die or are killed, they still lay hundreds off when a game is done. Then the studio limps along in pre-production mode on their next game for 4-5 years it seems like...
Maybe the only job stability in the industry is with indies, and... Nintendo?
But they boast the most sold video game in the history of videogames (Tetris a close-ish second), and most downloaded free mobile game, respectively. Each have player bases larger than the population of the country they're from!
Here's to hoping ms is hesitant to gut either!
Just out of curiosity, I guessed Minecraft which tracks, and Subway Surfers respectively, rather than Candy Crush Saga. Is CCS actually the most downloaded free mobile game ever?
Or maybe it's a problem with your English? (Note: I'm being offensive just because you are :-) ).
> when Trump croaks Thiel will be the president in all but name.
This means that he will have the power, but not the title.
Pretty sure it was the biggest at the time at least.
Minecraft has sold 350 million copies. Call of duty: black ops a measly 43 million.
https://en.wikipedia.org/wiki/List_of_best-selling_video_gam...
Given the mobile thing mentioned also, it's most likely to be Activison (which has been acquired by Microsoft).