GitHub incident 2022-03-23(githubstatus.com) |
GitHub incident 2022-03-23(githubstatus.com) |
In any case, why not just relocate some vendor engineers on site for a bit? Or, better, why does the vendor not have a small presence in the corner?
Sounds like whatever "the db" is it's probably some (objectively) small but very scary thing that's currently on fire and people are trying to figure out how to put it out without crashing the plane and also making too many waves internally, which is probably even harder. So asking about making vendor noises is (as useful as it may be) probably going down the wrong path - in much the same way this is probably not related to the outages (it may well be, but from the outside it's all coincidence anyway).
Systemctl restart
mysqld
(Or mariadb, if you pronounce "SQL" as "sequel")
https://github.blog/2022-03-23-an-update-on-recent-service-d...
Guide to incidents: Step 1: Stop the bleeding Step 2: Prevent it in the future
Doing Step 1 doesn't make you incompetent.
I also don't appreciate our builds freezing, unable to be cancelled and then eating up hundreds of minutes.
I haven't used GA in a way where it actually costed me anything, but having minutes just tick away while you can't do anything is really stupid if that's the case.
Edit: Another sane solution would probably be to record outage periods and have Billing automatically reconcile for every customer when invoicing. This would require them to admit the outage durations however, so it may be flawed from a human perspective.
At what rate would you do these pings? I don't know how upgrading/downgrading works at GitHub but if they do any sort of refund/credit when you downgrade, it seems like there's some interesting implications for abusing the system (e.g. upgrading/downgrading between pings for "free" service if the time between them is too long) versus performance (e.g. how do you update all users per ping in a timely manner if the time between them is too short?).
Would love to read up more on this approach; seems interesting!
I've created a new discussion in their feedback repo asking for this, three major outages in a week could really do with a post-mortem: https://github.com/github/feedback/discussions/13344
Of course, assuming that a future bug won't affect the timeout-minute itself.
As headcount goes up I think the inability to locally rewrite history into easily reviewable patches would be sorely missed. So it's git for team stuff and fossil for my own.
“Incident” alone makes me think something got hacked or leaked.
It's a nice way of putting it.
I'm trying to run github action for couple of hours now. They don't work at all. But apparently this means they run, but in infinite time, hence == degraded performance, nice.
GrubHub delivers
If I can get all the Github features I had as of ~2020, but on an instance that wont get hit by the public cloud/update bus, I would be exceptionally happy.
The only complaints we have are regarding availability. If we can fix that one problem, this is a perfect product in our view.
Oh dear. Not a good idea to go 'all in' on GitHub.
21 incident outages in just 3 months. At this rate the benefits of running your own gitea or gitlab are starting to become competitive.
I find myself regularly asking this — about every major SaaS used for critical ops stuff like this.
People want to know it isn’t their problem, that makes cloud computing (and things like GitHub) worth their weight in gold. I have real problems to solve I don’t want to deal with a git repo manager on top of that.
This has been my experience as well. I don't know if that means GitHub is being overly transparent about issues or I've just been lucky but I would hate if people punished services for being transparent and informative on their status pages.
These have been minor inconveniences for us - at worst. Most of the time it simply means people jump to something else then come back later in the day.
Failing tests and PR feedback cycles are more of a blocker to our team than these outages.
Most issues have a relatively narrow impact, but the impacted people _still_ benefit from seeing them listed.
When you host things yourself, you still have downtime. And, having worked with Github for over a decade, the actual disruption to my work is from downtime is much less than if I had to host my own.
That being said: I briefly worked for a company that hosted its own source code control system. For us, as a small team, it wasn't worth it. The system was outdated and hosted in an insecure manner. No one ever did any "admin" work except the founder. He ran it because he had irrational fears of switching, not because of any tangible advantages over Github (and competitors.)
Keep in mind that Github (and competitors) are often cheaper than the time needed to invest in hosting your own. (Estimate 10-20 hours a year of invested time. Calculate your hourly rate. Github and competitors are cheaper.) In order to come ahead, you need tangible benefits other than "I think I can have less downtime."
I certainly would not have this problem on self hosted instance, because it would not be behind CF. I'm sure I'd have other problems though. :)
All software is crap. You can be either spending time fixing it yourself, or spending time begging online for fixes/help from some SaaS company/community with resolution time in months, somtimes, all that while you may not be able to use it fully.
Also with SaaS it will be constantly shifting under you. Things will be moved around, restyled, iconized, popupized, etc. This doesn't help productivity either. With self-hosting, you can at least avoid upgrading, if you dislike this kind of thing. Or choose FOSS software that values UX permanency/stability, which seems to be really hard ask from SaaS business.
2019 -> 39 Incidents
2020 -> 67 Incidents
2021 -> 86 Incidents
2022 -> 20 Incidents so far
Edit: Using Linear Regression...Prediction for total end 2022: 111 Incidents.
So perhaps they are not exactly improving, but maybe there is some other way to normalize the data.
[1] https://github.blog/2019-11-06-the-state-of-the-octoverse-20...
How many core incidents? The part that affects whether you can even push to and pull from a repo, and access issues and PRs? Because everything else is nice to have, but you can do work perfectly fine without them if they go down for a few hours.
Just off the top of my head, that's one thing you can do.
Are you kidding? The last 2 incidents were called "degraded performance". Where "degraded" meant I would get nothing but 500 errors accessing GitHub.com either via browser or git itself for the duration of the outage. How is this not lying?
If you're using GitHub in Europe or Asia it's not uncommon for GitHub to be offline for many hours before they acknowledge anying.
But going 'all in' on GitHub just doesn't make any sense anymore.
[0] https://hn.algolia.com/?dateRange=all&page=1&prefix=true&que...
Also, quite a few of the non-profits behind the projects you mentioned have multi-million dollar budgets that they can use to administer their git instance, if needed. I don’t think “if they can do it, you can” is a strong argument for those.
Gitlab just seems better for actually running a software project.
After all, you don't even need Gitea for pure Git hosting. If you have a server with SSH access, just init a bare repo in a directory, push to that, and you're ready to go. No web UI needed.
The reason I'm still using GitHub is not code hosting. It's collaboration.
Used to do that years ago for my personal projects. Honestly does the trick.
If you don't want or need those things, bare git repos are fine and certainly easier to support (not that Gitea's that hard, though a few issues/PRs I've noticed have caused me more than a little concern about the overall quality of the project).
With the advent of the Okta breach I think we will see a reverse in the centralization trend.
Oh stop the drama. Fine. Setup your gitlab.
Git is distributed. GitHub is very much not.
assuming that would be flawless, which it wouldn't
No need, just use Codeberg.org instead. They run Gitea and is a free collaboration platform (+ git hosting) for free projects. FOSS/OSS should really consider alternatives to GitHub and GitLab, especially when there are much more FOSS/OSS friendly platforms around.
Speaking as somebody who's done over a decade of large scale OO applications perl and is actually really good at finding and fixing the leaks, this has often been intellectually aggravating but every time I've set that option instead I rewarded myself with a glass of bourbon for picking the pragmatic choice and then went back to adding (non-leaky) features that were far more useful to the company in question than cleaning up the older code would've been.
Seriously though, IIS 5.0 had no worker recycling. There was no method to fix the issue. Threads would eat up GB's of memory until you killed them.
GitHub Enterprise is confusingly both a "call us for pricing" tier of GitHub the website, and also an on-premise version of GitHub that you can run as an appliance in your own data centre. The first of those is ultimately just GitHub and so has the same outages, the second is running on your own hardware so (shouldn't be) tied to the website's availability.
The parent comment was explicitly about non-closed (e.g. private) orgs.
But for new open source projects of mine the ease of contribution and user expectation of a github repository are a trade-off worth making even so (I also maintain a self hosted master git repo that I consider the source of truth to -me- but these days it syncs from, rather than to, github, just because of the trade-offs involved).
Service Downtime Core Services Only - Cumulative per Month
( Some months with more than one outage)
Jan 2021: 3 hours 53 min
Feb 2021: 1 hour 42 min
Mar 2021: 4 hours 10 min
Apr 2021: 2 hours 20 min
May 2021: 10 hours 34 min
Jun 2021: 0 min
Jul 2021: 0 min
Aug 2021: 4 hours 23 min
Sep 2021: 0 min
Oct 2021: 1 hour 36 min
Nov 2021: 2 hours 50 min
Dec 2021: 0 min
Jan 2022: 26 min
Feb 2022: 13 min
If it was more like 99.80+ I think I would be like "meh", but honestly for the price you pay that's not terrible. Still, for a company at the Microsoft level, it should be 99.80 at least.
Perhaps with decentralization push of web3/QR etc, we'll get over the hump.
These are not easily portable, but honestly is because of this lock-in that I prefer to use separate/independent tools. For my open source project [0], I am putting things on github and it is the link that give to most people, but in reality is just a mirror to the gitlab repository[1], which I use for CI and static page hosting, and the "project management" is done on Taiga [2]
[0]: https://github.com/mushroomlabs/hub20
[1]: https://gitlab.com/mushroomlabs/hub20/hub20
[2]: https://tree.taiga.io/project/lullis-mushroomlabshub20You don't need a 'multi-million dollar budget' to self-host a git repo and may of these open-source projects have been doing so even before GitHub existed for years. Even if they did have such a budget, there isn't an excuse left to self-host and avoid going 'all in' on GitHub.
At the very least I would expect something like what ReactOS is doing by having a self-hosted backup just in case GitHub goes down or vice-versa. [1]
Looks like that is proving to be useful.
I never made that claim. The argument was “if X can do it, so can you”.
I pointed out that _some_of_these_ (Mozilla, likely the most extreme of them, had over $400 million in revenues in 2020), are quite different from the typical ‘you’, invalidating that argument.
As always, invalidating an argument doesn’t mean its conclusion is wrong.
So when are you going to question this user [0] and others here planning to do the same thing for not having a 'multi-million dollar budget' for self-hosting their own services then?
Since clearly according to you they 'can't do it', despite me saying 'if X can do it so can someone else'. Where 'X' can be even a toy project like RedoxOS, or a messenger project like GNU Ring hosted by themselves and accessed via a subdomain.
Seems like they and other lesser known and funded open-source projects are doing just fine like that for years.
I don't even worry about a strong backup strategy (besides just making occasional snapshots of the data volumes) because this was all set up with IaC tools (Terraform, Ansible) and I have copies of all the code in local repositories.
Same thing also happened few months back.
It feels like they do this manually and it's only done when enough people are effected.
Without more detailed numbers, there's literally no conclusion to draw here.
Self-hosting always increases the operational burden of making sure your systems are secure. Maybe you have the engineering resources to spend on patching everything immediately and conducting in-house pen tests, but for most companies it's much, much more secure to let the software's developers host it as well.
In what tech company is hosting or domains the main cost centre? Many companies spend more on a single hour of a dev's time than their entire GH monthly bill.
Moving to gitea solved all of those issues for me (thus far), now I’m looking into adding other stuff like CI through Drone.
Thanks!
I actually wrote a bit about the migration process, as well as the reasons for migrating over to Gitea, Nexus and Drone CI as opposed to using GitLab, GitLab Registry and GitLab CI: https://blog.kronis.dev/articles/goodbye-gitlab-hello-gitea-...
With containers, it's actually a pretty good experience that's not too hard to setup or manage.
(1) An engineer getting paid 120k doesn't "cost" 120k, probably >150k with federal taxes, health insurance, benefits, and so on. Not including the cost to recruit, interview, and train said person.
(2) I don't know of many 1,000 person companies that would trust a new grad software engineer with no experience to manage critical infrastructure.
(3) You need N engineers to manage said service, because what happens when your one engineer gets sick, takes PTO, or quits for some reason? You also need a manager for said engineer(s).
(4) You now need to secure an internal service you never did before, so expect to have to hire external security consultants or re-allocate security engineers, since it's high risk.
(5) Github is FedRAMP compliant, SOC1 and SOC2 compliant and GDPR compliant. If you or your customers need any of those things, expect to hire external auditors on a recurring basis to validate your home-grown solution meets those requirements.
I hate to make these points because I'm a big believer in the scrappy startup mentality, but if you want to do things right, in the context of a large enterprise that is accountable to a lot of people, expect a project like this to cost $1MM per year minimum, and it probably won't reach parity with a cloud offering in terms of reliability, multi-region performance, proper backups, and so on. This is why Github can charge ~$200 per user (Or $200k per year for 1,000 seats) and still come away looking like a bargain.