;)

https://www.dayswithoutgithubincident.com/ (from an above HN item)

I couldn't merge earlier but navigation around my project worked. Maybe if I generate even more PRs some will get through!

I think there should be another page 'GitHub is NOT down'. Needless to say, for all the other times it's Down.

so funny that I was migrating my agent workflow off of github issues today and while I was trying to run a regression test against a run of the old github workflow it gave timeouts related to issues being down, been getting nothing but trouble the last few weeks from having github manage the project/issues

How is Microsoft not able to scale? Is it an architecture limitation? That’s affecting trust big time…

The Internet infrastructure is not ready for all of these "generated" activity.

Eventually it will be!

What's with Github Enterprise? At the workplace it's constantly broken as well.

How is this news? There's as low as a one-in-six chance it's down these days.

It'd be more useful to inform us when it is working at this point

I haven't seen Friendster-level stability like this in a long time.

they are testing the limits of ruby on rails.

"Ultimately, if more companies treated the framework as an extension of the application, it would result in higher resilience and stability. Investment in Rails ensures your foundation will not crumble under the weight of your application. Treating it as an unimportant part of your application is a mistake and many, many leaders make this mistake." https://github.blog/engineering/architecture-optimization/bu...

worked for me pushing some commits about 20 minutes ago.

Honestly, if GH keeps getting worse I may need to migrate away to Forgejo or something. The problem is GHA... Does anyone know of a service that is better and doesn't charge me an arm and a leg for runners (particularly MacOS ones)? I've been wanting to shift away from GHA for ages (because I hate it) but I don't know of any alternatives that are quite like it (or the costs involved).

Wow, GitHub are no longer serious people. Unbelievable.

Selfhosted gitlab alpha versus the cloud Github beta

Soon on your favorite prediction markets.

If this was a business running on its own, without Microsoft’s capital and anti competitive practices, they would have been out of business by now.

re there polymarket bets on if gitub will be up or down today yet?

Probably the solution is not using the centralized Github for all work related to agentic coding but rather a distributed/local github repository from get go. This way only what hits the centralized github and becomes public, is something vetted and signed off by the human in the loop, hopefully reducing the AI slop and increasing the quality of commits.

It's almost as if some people don't understand that it's possible to develop software without using github.

The people who warned about building dependencies on third party services are enjoying some schadenfreude now.

Ironic that a source code control system that was designed to be distributed is now presenting problems due to the failure of centralized services.

Github vs cloudflare

Microslop happens.

Is this trending in relation to Ghostty moving away from github?

oh no, anyway...

I’ve seen this behaviour but IMO it’s a fatal design flaw of Actions. It shouldn’t be possible.

Many of us predicted this.

Microslop, 'xcuse me, Microsoft is killing off GitHub. Not necessarily by design or on purpose, but by slop-neglect effect. AI is eating the corporate brain that Microsoft used to have (well, a long time ago).

I'm moving everything to self hosted GitLab CE now. Will just use GitHub as a backup. This is so ridiculous at this point.

> Co-authored by Co-pilot

Is Git decentralized?

This is genuinely pathetic. I wish I could do my job this poorly and still be employed.

really need competitor for gh rn

Here we go again...

The goals for code-versioning as they existed decades back, might be sliding into irrelevance now. Code is no longer a direct work artifact from humans, as it used to be. Back in the day, people wanted to persist their code and it's changes, because code was hard to write and test. People didn't use a versioning system for their compiled binaries though, because it was a machine output and can be created from source code. But there was no higher level "source" for generating the code itself. Now things have changed. We need to check what exactly is the human artifact that requires treasuring, persisting and versioning.

And where does one store and version this harness?

:) :) Sure buddy - mind showing me yours ?

I, for one, am not paying them enough money to expect any better.

They can't even get two eights, let alone three nines.

Hey there's a nine in there, so it's fine!

At least one 9 … somewhere

ffs can we stop talking about that number and site already.

It treats any service being down as the entire platform being down which is nonsense.

It's just lying with statistics.

GH was going down before AI explosion. The start of the trend is MS buying it, not AI explosion, that is just final nail

It's not new, it's just a DoS, which is a serious crime, just report the attacker to the police if in your country, or block their IP if not. If done accidentally, it's likely not a crime but the police will still scare them to stop doing it.

It's been unacceptable for months, but now it's at the level of "we should actively look for alternatives".

Things are a lot better in Europe. I stopped working hours before this incident started, and I can't really remember any major work-stopping indicents in the past months. I only remember once trying to do hobby stuff in the evening that was impacted recently.

A week? You're going to be happy with more than a day without an incident.

I lost track which Monday morning PST in a row this is.

I would say we are way past unacceptable.

Not a free lunch, data gold mine

In other words, the set of github core services has expanded because you don't use third party tooling for some of those services anymore.

That sounds like their classic EEE

That’s sort of orthogonal. But if GitHub actually invoked an LLM on initial page load, that would be about par for the course, and it would be amusing for GitHub to then complain that they’ve grown so quickly that their systems can’t keep up.

What if I told you most enterprise customers don't even use the cloud offering and aren't impacted by any of this? Companies like Apple use GHES, and honestly thats where most of their revenue comes from, not the free offering.

For all of it's history (up to and including now possibly?) Github was a big Ruby on Rails monolith. [0] Obviously some things run in their own service, but I'm seeing the core github features fall apart which should be the features packed into the big monolith. If load is this much a problem, not being able to only vertically scale the processes that need the extra headroom is a big problem. Scaling horizontally by just throwing more machines at it, or at least cordoning-off some machines as "the ones that people actually pay for" is all I can think of for an application I can only describe as "accidentally working". Urgency is most-definitely high and that pushes decision making towards permanently-temporary patches instead of actual infra/architecture improvements.

[0] https://github.blog/engineering/architecture-optimization/bu...

I suppose the idiocy of their parent company is their job security.

I think this is minimizing the Xbox platform. They are also a massive digital distribution platform where almost every game is a digital download now.

That being said, you are correct. It is absolutely no surprise to me that Actions has the worst uptime.

I don't think it's an inferiority complex, negativity sells more and carefully understanding things doesn't sell as much

How do you throttle bots? Everyone will stop having commit msgs mentioning LLM agents. then what?

I'm curious, do people around you use AI? Because in my own workplace, people use lots of AI, and they ship lots of PRs, which correspond to actual features on the roadmap. I've been doing this a long time, and there is a whole lotta stuff shipping. I'm a manager and in the handful of hours I have I'm shipping the equivalent of what I would have as a full-time eng years ago.

> Is everyone just creating giant piles of throwaway slop?

yes, like the crypto or productivity 'industry', where every project exists to manage your other crypto and productivity projects, the dynamic of 'AI programming' seems to be to make software to manage your other two dozen AI things

mysteriously enough, big open source projects that people actually use in the real world aren't seeing their issues closed at any higher rates, almost as if that requires actual maintenance and programming (which Fred Brooks told us this half a century ago)

The thing about an SLA is that once you’ve broken it you’ve lost the trust. It doesn’t _really_ matter what the cost is for breaking it, nobody chooses their platform based on the refund they’ll get if they’re down. But they absolutely do choose based on reliability and uptime. The enterprise SLA refund credit will show as a (big) metering blip, but the problem is the people who signed the contracts are going to be speaking to Gitlab now

Some people really abuse the f out of the system in a way optimized to take github down. Like they push every minute or for every commit instead of with certain time intervals (e.g. a single push a few times a day for each repo).

I follow some of the accounts that run 24/7 agent sesssion. Their projects are not even that novel for the number of commits that appear on the profile. Many of the commits have the log of beads, claude session etc (no change to the actual code). Some of them are ports of some projects to another language. AI surely will increase the productivity, but the waste and noise that some people are willing to commit ....

It's a nice thought but I think the revealed preference from the history of the internet is that people actually only want centralised services, no matter what they say they want.

People love to clown on the fediverse because of having to choose a server. Which is no different from email. I guess the difference is that their ISP used to give them email.

Not really. It is possible to implement systems that handle a lot more scale than github has. This is proven by systems that exist today.

It might be hard to create such systems using ruby and microslop AI management though

And how many of those actions do uncached downloads instead of building self-contained offline images... Speaking of which, I wonder if GitHub has implemented any HTTP interception for common mirror sites, like used by apt, etc.

Or how many pushes those commits are spread across; oh, neat, big number.

I can guarantee you one thing - github's problem isn't coming from git.

Considering all the ci/cd pipelines, PR & issue discussions, social media tracking, rich data and else that github hosts if their true issue is the actual meat and potatoes of running git I would be gobsmacked.

What are you referring to when you say it's "fundamentally computationally inefficient"? It's pretty efficient because it's content-addressed, plus optimizations to reduce storage and data transfer with packfiles.

Three eights is more than a month of downtime every year. Today is the three eights.

Working in an organization without a mono-repository I've actually found it extremely difficult to keep a tab on PRs and issues across multiple repositories. For a problem that should be resolved by a "For me" page that just lists out all your active incoming and outgoing PRs their multi-page solution involving search filters that often need to be reset feels extremely weak. I've worked on large multi-tenant solutions before and a page where you can "SELECT * FROM everything LIMIT 10" is the absolute last thing you want to give to users.

It is bizarre to me that so much of their tooling defaults to acting across the whole of github data points without guiding the user towards (or even making available as far as I can tell) a way to easily scope requests down outside of a complex search filter.

The GitHub SREs are working for the Copilot company.

Your claim is incorrect. Something purely AI generated may not be covered by copyright in the US. That would make it more valuable to MS as you can reuse it as you like.

However, works with significant human input are covered by copyright, and most code does have such input. Human review, and correction is very common. There is a lot of AI generated code out there, and there are no cases challenging the copyright on it.

You also need to look beyond US law. Software is a global business and most software businesses do not want to write software they can only sell in certain countries.

We aim for 9 5s

> And what if I tell Claude to write the program

Why should this be any different than when telling/paying a human to write the program?

You're free to enter an agreement assigning all rights to the employer or the worker, to license the work ir/revokably and/or non/transferably. There is no need to wait for a court decision to understand what the results will be.

haha thank you.

> Either AI use surged massively in the last quarter

December 2025 is considered by many people to be a major step function in agentic coding (both due to improvements in harnesses and LLMs themselves). I know my coding has forever changed since then.

Before I was basically always hands on the keyboard while working with AI. Now I'm running experiments with multiple agents over the weekend, only periodically checking in if they have any questions or need further instruction.

The last quarter is where I personally first started to see how this was all going to change things (despite having worked on both the research and product side of AI for the last few years).

> I have a feeling they've tuned the models recently to commit more often, which gives the illusion of more work being done.

Agents certainly are committing more often, but I know, at least for these projects, there really is work being done. An example: I had an agent auto-researching a forecast I was working on. This is something I've done manually for over a decade now. The iteration process is tedious and time consuming, and would often take weeks of setting up and ultimately poorly documenting many, many experiments to see what works. Now I can "set it and forget it", and get the same results I would have in hours (with much more surface area covered and much better documentation). Each experiment is a branch (or work-tree) so yes there are a lot of commits happening, but the results are measurably real.

I often think the big divide related to the success with agents is whether or not the quality of ones work can be objectively measured. For those of us doing work that can be measured, the impact of agents is still hard to comprehend.

Many things at once I suspect:

1. Models have got way better, which means you are far more likely to get something working. I know I used to have little 'tool'/'weekend projects' all the time that wouldn't get off the starting blocks before, now it takes a few minutes often to build them, and once I've built them I tend to want to have them saved on github. Quite how useful they turn out to be is another question though...

2. Related, because the models are a lot better I can generate far more code per unit time. On Sonnet last year I'd have to babysit the model and constantly 'steer' it, which meant a lot of the CC time was actually me reviewing it. Now with Opus4.7 it can often just churn away for 10-30minutes and get something reasonable.

3. Most importantly, just the volume of new users to coding agents - loads of new developers shipping far more far frequently.

4. Many users who were not on github, now signing up and pushing code to it. "Vibe coders" basically who don't have SWE experience and their agent tells them git would be a good idea.

Each of these would be a big increase in scale, but combined it is vvv high

I don't think commits per se puts pressure on the infrastructure.

More likely pulls and pushes, and, naturally, the ci minutes they identify as the main issue.

> I think you need to broaden your focus here - I can't really remember any significant downtime before the Microsoft acquisition and the data supports my memories.

This is the opposite of my recollection, actually. I distinctly remember having conversations about Github struggling to scale well before MS was involved, and people claiming that MS had somehow saved Github because it had stabilized and begun adding features again.

> The query performance was fine before they started serving from Azure.

This may be correct though. The Azure migration seems more aligned with the timeline of struggling to scale.

I don't know why this is downvoted. The data backs you up: https://damrnelson.github.io/github-historical-uptime/

I mean, are any of the other forges, which I presume are also seeing logarithmic increase in commits, also failing as hard as Github?

And yet these are still (apparently) implemented as search queries instead of direct database queries.

We can clearly draw a conclusion that their availability is getting worse, but that's not what your original comment claimed.

You said "I can't really remember any significant downtime before the Microsoft acquisition and the data supports my memories", but my memories differ (as do other commenters), and the accuracy of the supporting data seems questionable.

Make it nerdy enough to scare of agentic coders only. Also, blackjack and hookers are said to be helpful in such circumstances.

What do you need users for?

GH is not a social network

> We should make an alternative git site, but how to acquire users?

Buy ad space on Github's outage page?

Forgejo is a thing. But the headlines lately make it sound like it’s not in great shape either.

codeberg is doing fine

> [...]it's just a simple calculation of "is this worth it?" which I suspect for most cases will be 'yes'

Once the landgrab-stage flat-pricing goes away, it will become a case-by-case calculation because unsupervised agents can (and will) run up your billing with zero understanding of the business value of what they're instructed to solve.

Azure’s core hypervisor orchestrator was half-baked at launch and it has never been fixed. This long read blog series explains a lot for me — for example, why the FedRamp certification program was never able to get a straight answer from Azure about how they handled secrets.

https://isolveproblems.substack.com/p/how-microsoft-vaporize...

https://www.kunalganglani.com/blog/microsoft-fedramp-failure...

Re 2, I would generally agree and there is a lot that can be done with caching. However, since writing services in Rust and Golang, there is whole other tier in speed. Architecture matters, code quality also matters, but Golang and Rust help a lot in making very fast services.

> migrate their codebase to a new language[2], continue to drop their inhouse ops for Azure resources and get off MySQL

The recent blog post you're linking to mentioned moving data only for webhooks off MySQL, not all relational data used by the entire site; and moving "performance or scale sensitive code out of Ruby", again not the entire codebase.

Do you have an official source suggesting these migrations are more comprehensive than that?

Why is it easier to submit a bug report if my bug reporting system is run by the same company as your code repository? Why are those things even slightly related?

> same auth system

I'm confused with auth has to do with it?

We've had OAuth 2.0 since 2012

> While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud. This longer-term measure is necessary to achieve the level of resilience, low latency, and flexibility that will be needed in the future.

That paragraph read, to me at least, that the initial targeted changes were just the tip of the iceberg and that much heavier lifting than initially budgeted were now in scope.