GitHub was down again(githubstatus.com) |
GitHub was down again(githubstatus.com) |
A UPS and a tethered smartphone would get me three nines uptime-while-anyone-needs-it, which is well in excess of what I need.
I don't regret it.
Not saying it's right, just an observation
Until I decided to no longer do that and my life improved considerably.
I've caught a big cloud provider not reporting a degraded service, I assume they knew but politics and $ come in and it's easier to just gaslight everyone. I get it, but my frustration is worth loosing a trailing 9.
I think there should be some 3rd party continuously testing APIs. Degraded states are downtime!
For example status.digitalocean.com is _not_ real time, it's manually updated.
And it's irritating as fuck.
It's gotten more reliable over time (especially selenium events being dropped on the floor causing tests to stall and fail), but I used to have to babysit it quite a bit and there were quite a number of times where IE instances just would not spool up (with a multiple minute timeout set). Sometimes it was a one-shot thing, other times it went on for hours.
During these incidents the average allocation times listed on their status page would double for Windows VMs (I don't recall the exact numbers but they were on the order of 10 seconds vs 5) but nothing would be red, and most of the time nothing ever did go red.
And that's what you get for using averages for things and divide infinity by n improperly.
The only reason we're using it is because it's free..
At this point, we would probably be better off just bolting some lightweight git solution onto our devops tools (which are 100% custom in-house developed), rather than fighting with some more-durably-hosted offering of GitHub, et. al.
Anyone who posts that "but you cant make it more reliable than microsoft" line is not thinking about the dependencies between systems and the considerable impact incurred on a service just by virtue of it being a publicly-accessible platform without any cost barrier to entry. Sure, bringing it in house might bring additional difficulties, but I think I can eliminate a shitload of existing difficulties if we moved from webhooks across the public internet to a direct method invocation within the same binary image.
Microsoft could be pushing for transparency. Or people are more relaxed about transparency now that GitHub has its exit. How long did GitHub know they were looking to be acquired? Maybe this analysis should look at a longer time interval..
Incident on 2020-07-15 15:41 UTC We are investigating reports of degraded performance. Posted 9 minutes ago. Jul 15, 2020 - 15:41 UTC
If it is not purposefully being swept under the rug, it sure is convenient.
Ideally, I'd like to have two synchronized repositories, for no single point of failure, organizational or otherwise.
https://github.blog/2020-07-08-introducing-the-github-availa...
https://github.blog/2020-05-22-april-service-disruptions-ana...
Even a self-hosted GH Enterprise would suffice for some businesses but this would be overkill for others. I even see the Wireguard author using his own creation (cgit) to self-host on his own git solution for years. [1]
This is problematic since many JS/TS, Go and Rust packages are on GitHub, which many developers rely on. Thus, it would be risky to think about tieing open-source project to (GitHub Actions, Apps, etc).
That is dedication.
FWIW the git backend always seems rock solid in comparison to the front end they have displaying it.
Cloud solutions are great -- however they have a golden rule, don't go down ever. This is seriously damaging to GitHubs reputation.
GitHub has GitHub Enterprise
> Cloud solutions are great -- however they have a golden rule, don't go down ever.
Oh where oh where can I sign up for this mythical unicorn cloud service?
If that is not enough, run your own instance of gitlab.
If that is not enough use Gitlab.
Microsoft is going to attempt to make a profit on Github. That's okay, but based on past experience and current issues, their business model is lock-in not service.
I suspect the same is true for NPM.
Frankly I can't wait. Imagine being able to reference other users across instances with @username:instance or something to that extent, or projects and tickets.
I don't often need a web interface to a git repo. I can pull and do everything locally.
What I do use GitHub for is (1) code review and approval process, (2) CICD / actions, (3) releases to push stuff out.
The branch / tag / file browser is a nice addition, but it's not key. Rendering README.md is almost as important, if not more.
Issues with a green 'X' means they link to a feature on their issue tracker.
And, as far as I know, they are working on integrating CI/CD right now. They already have support for other non-integrated CI/CD platforms: https://docs.gitea.io/en-us/ci-cd/
There are two active contributors for Gogs, while Gitea has 27. Note, the number of contributors can't tell you if one has higher quality or not, I just wanted to point out the difference in development philosophy.
Given that Gitea has significantly more active developers working on it, we can probably assume it can add functionality faster than Gogs though.
Then set up the alternative remote on your repos.
[1]https://www.linux.com/training-tutorials/how-run-your-own-gi...
ssh user@git.example.com
mkdir project-1.git
cd project-1.git
git init —-bare
exit
git remote add alternate user@git.example.com:project-1.git
However git has more: - Bug tickets - PR - Wiki - many projects use their githubpages as primary homepage - newly GitHub actions
And then: Often collaborators are only known and identified by their GitHub handle. Running an own server requires some mechanism to identify them again and creating a way to handle their access credentials (ssh key etc.)
Moving a mildly successful project isn't easy. Good if more people plan for that eventuality, even if they stay on GH for the time being.
git remote set-url --add origin git@somewhere.else:my-project
Building this in git itself is not hard at all, and there's likely a script or plugin for gogs or gitea.
But GitLab does have feature support for extensive importing and mirroring. https://docs.gitlab.com/ee/user/project/import/github.html (Import your project from GitHub to GitLab) has a section on project mirroring.
git add remote NAME URL
git push NAME
git remote add originFoo URL
git push --all originFoo
There are other flags like mirror but never used them.
Severity went something like this (sometimes the numbers flip which always confuses at least 20% of the team about whether things are almost normal or people are hunting each other for sport).
1: data loss
2: some workflows blocked
3: some workflows unavailable w/ workarounds (ie other routes)
4: Everything else except
5: Irritations
Having a UI break but the underlying functionality is still working is not good but people can still do their jobs, if more slowly. It's important to classify these separate from S2 and S4. There is urgency but don't panic. Go eat lunch or have your planning meeting, then go fix it. If data is getting lost ain't nobody doing nothin' until we figure it out, and then some people can go back to work but don't interrupt the people still working on it.
I think the problem is that so many metrically dysfunctional people, to the point of cliché, have rationalized that an S2 means that only 20% of our customers can't do their jobs so we are degraded but still working normally, when really a yellow status should be at S3, while S2 should be at least orange although those affected will be upset that it's not red.
Over time that 20% will shift around to most of your customers. Eventually several times, and then you'll wonder why everyone is talking trash about you on HN. It's not like that many people were affected!
Foobar: for when you are too polite to say FUBAR (Fucked Up Beyond All Recognition).
Named, of course, by the Army when they built the Alaska Highway.
git clone --mirror https://github.com/you/repo
git remote add new https://gitlab.com/you/repo
git push --mirror newI've seen status pages and error counts tied to bonuses, which only caused a giant mess of bad incentive alignment and internal lies, customers are unhappy, developers are unhappy, management are lying to upper management, it's so much easier to focus efforts on real problems and just be honest and improve. Thank goodness I dont work there anymore (cough cough Google)
This could have been caught with a cron job and some curl requests :\
Building infra I have to work around all sorts of 3rd party services going out or having blips throughout the day, docker registries, caches, bgp, etc., it's totally an expected part of infra design but not every team has the time or need to build in the resiliency. I see tons of outages that never get reported or IMO aren't reported adequately enough.
With that said, I'm no angel, I get all my service down notifications through slack, so when slacks down..
Recently, more and more of them take 10-15 minutes until they mention a service outage. I don't work in super HA, I don't want to get an alarm because a single ping failed etc, so I'm lenient and have a few minutes of a delay in alarms. If I'm writing an internal incident report before the official status page is updated, that's bad.
This seems similar: external users noticing the outage and posting on HN before GitHub notices & acknowledges it.