That's almost believable at this point. ;)
Due to GitHub's chronic unreliability, it is guaranteed to continue happening every month.
Looks like avoiding to 'centralize everything to GitHub' has aged very well [1] and at this point you would get better uptime with self-hosting instead of using GitHub.
Just ask many open source organizations like RedoxOS, ReactOS, wireguard, GNOME, KDE, etc.
Edit - Just want to clarify when I say "opened hackernews with a tuna sandwich" I want to clear up that I did indeed full on mash the keyboard letters with my sandwich. It's costing me a fortune in keyboards every day and it's ruining my sandwich most days as well, I think I have an issue.
“Never ever type on keyboard with sandwich not even the most delicious tuna sandwich”
There was a period of time when GitHub basically didn't change, for years. And the platform was relatively stable (although "unicorns" (downtime)) still happened from time to time.
But nowhere near as often as now, but then again, there is a lot of more moving pieces now compared to before.
[1] https://github.com/customer-terms/github-online-services-sla
GitHub is probably as reliable now as it has been for the past 10 years. It's always had downtime.
Wow. This sounds like a broken organization?
You are better off self-hosting at this point, rather than centralizing everything to GitHub [1] as it is just chronically unreliable for years ever since the Microsoft acquisition.
For medium/large companies, I fully agree. For smaller projects, specifically open source projects, I'd look at something like https://codeberg.org/
2) It's not like this push is a big secret: https://www.cnbc.com/2022/10/12/microsoft-github-relying-mor...
Every time some global service goes down, or internal internet/intranet goes down, there is a security breach, or a WFH person has a power outage I'm reminded I'm right.
I'm no luddite, these services make you dependent on them. The worst thing I'm dependent on here is a bad computer. We have backups and keep our files on our network, so it seems fine. We are slowly moving to an online system, and I'm constantly reminded all the problems shifting online.
Meanwhile, if I had a linux server, we would be in control of our own destiny.
Additionally an 80% uptime architecture is really simple to maintain and restore and so on.
Complexity increases exponentially the more 9's you add.
Having our programs offline mean I can run them, even if the internet isnt working.
Instead of getting 0 data from downtime, I can still get the data, run the programs, and give it to the person who needs it.
If we are fully online, if the servers are down, we basically lose the entire time.
Not to mention, I think 'uptime' is a pretty optimistic number, unusable slow service doesnt seem to hit any metrics I'm aware of.
* Where I am
Really, GitHub outages barely hurt at all. It's not like an AWS or Cloudflare outage which is more likely to be a production disaster. Every outage a bunch of people on HN start screaming about owning their own on-prem destiny or wondering why we're still on GitHub. Nothing changes because it's not nearly as bad as those people are making it out to be. Life is all about tradeoffs.
Architecturally this seems rather sub optimal?
EG AWS doesn't roll out changes globally - they start with an internal shard within a region and progressively roll out to more shards and more regions.
Why do GitHub not do the same?
And I guess you're only as strong as your weakest link, which can be not that bad, that is, if it isn't your core tables.
Though take this with a grain of salt, this is mostly hearsay =D
`error: failed to push some refs to ` when using --set-upstream
remote: Resolving deltas: 100% (3/3), completed with 3 local objects.
remote: fatal error in commit_refs
To github.com:acme/foo.git
! [remote rejected] HEAD -> acme/foo (failure)
error: failed to push some refs to 'github.com:acme/foo.git'would sound better without "other" in title
If this is not the first time, hopefully at least it will be the last one :)
As a somewhat large size org, we're now exploring other options for code hosting.
Earlier, I also got GitHub PR comment emails about 6 hours late.
Whatever it is, it’s been happening for more than 6 hours.
Some people have their entire roadmap in GitHub, and every single bug report / feature request, without any offline backup. Don't ask me why, I don't get it. Especially since they have proven for the last few years that they cannot keep the platform up in a stable manner.
Heh, if anything it has gotten worse.
* March - 20 incidents
* April - 12 incidents
* May - 4 incidents (so far)
I don't see how the two situations are comparable
> Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed.
https://about.gitlab.com/blog/2017/02/10/postmortem-of-datab...
There's no evidence that the exact same doesn't happen with GitLab. I've had it (consistently) 500 on me in the past when there's nothing on their status page to indicate any issues.
That's not the point of discussion. I didn't say Gitlab doesn't lie about it or heck, That it doesn't have worse uptime than Github.
My argument is that a company erasing 300GB production database once is not a stain on their competency and that it can not be compared to a company which has very frequent outages which also happens to lie when they have outages.
Perhaps this is a continuing argument for self-hosting, especially if you don't have to expose the instance publicly. But then, if that's an option, you can also self-host GitHub (though I have heard less anecdotes about the stability of that).
I'm confused. You can do zero-downtime backups and replication of databases. I don't know what your company / Gitlab are doing but it seems wrong.