GitHub Merge Queue Silently Reverted Code

GitHub Merge Queue Silently Reverted Code(githubstatus.com)

121 points by matthewbauer 24 days ago | 45 comments

cwkcw 24 days ago |

The window to audit is 2026-04-23 16:05 to 20:43 UTC, about 4h 38m, and 'merged incorrectly' is unpacked in exactly no detail on the status page. Practical check is to compare the tree of each merge commit in that window against the tree of the rebased-and-tested commit that the queue ran CI on. GitHub Enterprise Cloud with Data Residency is still affected as of the latest status update, so DR customers shouldn't clear new queue merges yet. Would be kinda useful if GitHub published the specific failure mode, because content-drift and parent-linkage and dropped-changes all need different audit scripts.

matthewbauer 24 days ago |

Might want to check your git logs if you are using the Merge Queue feature. This afternoon, we found that Merge Queue was silently reverting changesets in the merge queue. Acknowledged by GitHub, but could be a very hard problem to debug if you aren't looking for it.

EdwardDiego 24 days ago | |

Same here, ffs GH.

MarkMarine 24 days ago |

4 people spent hours putting our repo back together at my company after this. GH has been unreliable and now they are breaking the core tenant of what I expect from this service.

ragall 24 days ago | |

> core tenant

tenet not tenant

EdwardDiego 24 days ago |

This has caused a morning of fun for our team, how do you break one of the most fundamental bits of your system? Time to look at alternatives...

ramon156 24 days ago | |

Self-hosting has been wonders. I thought I needed the social part of GH (i still kind of do, I like seeing what people are working on or liking) but overall its the same experience.

Randalthorro 19 days ago |

Imagine you pay someone to save your family pictures but sometimes they save only half your data. Instead they try to upsell you on AI filters for your pictures. GitHub, return to your core mission.

acid__ 24 days ago |

Tip: identify PRs with this problem by looking for discrepancies between the metadata (files/lines affected) for PRs via GitHub API and associated commits in the time window.

This was a pain to clean up!

tracker1 23 days ago |

I experienced a weird bug, where the github ci environment was checking out code that was throwing an error on build... the version tag definitely didn't have the code the error was coming from... I messed with the CI workflow yaml and finally cat'd the file in question... the error code was definitely there (2 lines repeated in the file) that was not in the repository.

I nuked the release tag and re-created it, and it started working... was just a really weird bug/experience.

sancho1952007 18 days ago |

Been self-hosting Gitea since ~9 months for my critical projects. Originally I did it because I didn't want AI to train on my private code but seeing what's happening to Github these days, it looks like a bonus that I don't face data integrity issues or uptime issues. I still use Github for my side-projects where I can directly push to main. I just hope these issues get fixed soon. Been using Github for many years now and I can't afford losing my Github history.

NelsonMinar 24 days ago |

How did a bug like this get past their testing?

philpem 24 days ago | |

Given the push to AI/Copilot from MS I'd be impressed if they still did any testing.

tikiman163 19 days ago | |

CoPilot broke github actions in such a way that if too many requests to start a workflow came in at the same time it would reduce the capacity of the build server's queue until you did a full manual reboot of the build server and completely reset the queue.

Some people are saying they aren't doing testing, I know this to be incorrect in one very important way. When it comes to people, the ways in which a co-worker is going to be wrong are predictable. When it comes to AI written code, there is zero predictability as to what could be wrong.

Trying to test AI written code is like writing code with no try catch blocks, and expecting the application to not crash within 15 minutes of giving to the end user. They will try everything you didn't think to explicitly handle, and when it comes to AI there is no try catch block for subtly incorrect or incomplete logic.

estimator7292 24 days ago | |

They're at 88% uptime this year. I don't think they have testing.

tracker1 23 days ago | | |

Microslop®, unmatched quality.

LiquidityC 19 days ago |

Checked all my PRs that merged on the 23:d. Checked out each branch locally, rebased on trunk and ran 'git diff'. One of them had a diff which was the entire change that was marked as merged in the GH web ui.

This happened in complete silence and since the PR in question was just code refactor I would most likely never have noticed.

LiquidityC 19 days ago | |

Considering switching to original method of simple git repo hosting and email patches like the kernel does it. A workflow I favor. Turns out there are no hosting services which provide such solutions. Lots of manual hosting setup and config required... Sad that GitHub sort of cast the mold for how git hosting has been done for the past 2 decades.

sidewndr46 24 days ago |

Github has been doing this for at least 5 years now. Merging a pull request merges whatever the latest commit on that branch is in the specific backend that handles the merge request. It does not merge what was just pushed into that branch.

orpheansodality 24 days ago |

for other folks currently in an incident trying to resolve the chaos this caused, the first commit we've found with issues in our repo is from ~10:30am pacific this morning

classified 24 days ago |

How long will users put up with this before they finally leave? Microslop is inventing new failure modes on a weekly basis.

jonnonz 23 days ago |

Whats going on over there, is clippy in charge of the release button?

mushfiq_rahman 19 days ago |

wow!

minraws 24 days ago |

Wait what? How, what? This doesn't compute