The GitHub Load Balancer

The GitHub Load Balancer(githubengineering.com)

438 points by logicalstack 9 years ago | 125 comments

NicoJuicy 9 years ago |

I notice a lot of negativity arround here. Don't know why that is... But i'll take my 5 cents on it.

NIH - Not invented here and redoing an opensource project.

- Github said they used HAProxy before, i think the use case of github could very well be unique. So they created something that works best for them. They don't have to re-engineer an entire code base. When you work on small projects, you can send a merge request to do changes. I think this is something bigger then just a small bugfix ;). Totally understand them there for creating something new

- They used opensource based on number of open source projects including, haproxy, iptables, FoU and pf_ring. That is what opensource is, use opensource to create what suits you best. Every company has some edge cases. I have no doubt that Github has a lot of them ;)

Now,

Thanks GitHub for sharing, i'll follow up on your posts and hope to learn a couple of new things ;)

otoburb 9 years ago |

Given this is based on HAProxy and seems to improve the director tier of a typical L4/L7 split design, I'm led to believe GLB is an improved TCP-only load balancer.

But they also talk about DNS queries, which are still mainly UDP53, so I'm hoping GLB will have UDP load-balancing capability as gravy on top. I excluded zone transfers, DNSSEC traffic or (growing) IPv6 DNS requests on TCP53 because, at least in carrier networks, we're still seeing a tonne of DNS traffic that still fits within plain old 512-byte UDP packets.

Looking forward to seeing how this develops.

EDIT: Terrible wording on my part to imply that GLB is based off of HAProxy code. I meant to convey that GLB seems to have been designed with deep experience working with HAProxy as evidenced by the quote: "Traditionally we scaled this vertically, running a small set of very large machines running haproxy [...]".

dcgudeman 9 years ago | |

Where did it say that it is based on HAProxy?

bogomipz 9 years ago | | |

If you look under "stay tuned" it says:

"Now that you have a taste of the system that processed and routed the request to this blog post we hope you stay tuned for future posts describing our director design in depth, improving haproxy hot configuration reloads and how we managed to migrate to the new system without anyone noticing."

That leads me to believe it involves HAProxy.

otoburb 9 years ago | |

EDIT: I read the post more carefully - the reference to DNS was merely to highlight that often times a single public [V]IP can get overloaded and common mitigation strategies.

jimjag 9 years ago |

I am increasingly bothered by the "not invented here" syndrome where instead of taking existing projects and enhancing them, in true open source fashion, people instead re-create from scratch.

It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expected user base of 1?

Again, I am not surprised by this. They whole push of Github is not to create a community which works together on a single project in a collaborative, consensus based method, but rather lots of people doing their own thing and only occasionally sharing code. It is no wonder that they follow this meme internally.

alexandros 9 years ago | |

here's an alternative interpretation: Due to their unique requirements, they were forced to investigate alternative approaches to the problem, and the approach they went with, they believe, is useful to a lot more people than just themselves. This is both plausible and matches the announcement, so why take a bad faith approach to someone releasing some code to the open source? Would you prefer if they kept it closed?

And since we're talking about github, haven't they already launched a highly successful pair of projects in atom/electron, in areas where both had competition? why start with negativity before we see what they come out with?

saurik 9 years ago | | |

Huh? Atom/Electron is another great example of GitHub duplicating a ton of existing projects (whether dependencies such as CEF and node-webkit or high-level solutions such as ACE) without seemingly having any interest at all in joining those existing projects. Just because someone is successful at doing this does not make what they are doing any more reasonable: if anything it should just put them in a similar place in your mind to the Microsoft of the 90s which many people here would denigrate. GitHub's model of "open source"--the one which it is, devastatingly, teaching to an entire generation of developers--is only about code being available as opposed to being about community and collaborative design. Asking if one would prefer an alternative where the code is simply kept closed source ignores the premise of the compliant: that the code being an advertised separate project undermines the premise that working with an existing project to solve a problem tons of those users almost certainly also have. :/

olalonde 9 years ago | | |

> Due to their unique requirements [...] This is both plausible

I'm skeptical that Github's load balancing requirements are that different from any of the other large file hosts and SaaS companies. But it's possible and I'm not in a position to tell. That being said, the "NIH syndrome" is a largely overlooked problem in our industry and I think it's reasonable to raise concerns over new projects that may be reinventing the wheel.

ckdarby 9 years ago | |

Have you ever contributed to HaProxy? Have you ever tried committing massive alterations to major open source projects?

It isn't as simple as here's my massive rewrite, click the accept button and everything works out for the open source community.

Let me be the first to say that the level of politics, circle jerking and knowing people is ridiculous.

polpo 9 years ago | | |

Given the good reaction to an out-of-the-blue patch from me on the HAProxy mailing list, I'd imagine that contributing even major changes to HAProxy probably would go rather well. It's one of the best open source development communities I've experienced. Welcoming, but still highly focused on quality contributions. The quality and performance of HAProxy reflects this approach.

zepolen 9 years ago | | |

As someone who's contributed multiple patches to multiple open source projects, this is 100% truth.

I have >3 month old pull requests to add tests which have never been looked at - whereas someone who knows the project maintainer will get a PR looked and merged the next day.

senko 9 years ago | | |

"Contributing to open source is hard, let's write our own and open source it instead."

JungleGymSam 9 years ago | | |

Sounds like you're suggesting you can't use your own modified fork until your PR is accepted.

eikenberry 9 years ago | |

NIH syndrome is overrated. A small application doing specifically what you want is usually more robust, easier to maintain and easier extend as you need than a pre-existing application that includes most of what you need as a subset of its functionality.

jcastro 9 years ago | |

> in true open source fashion, people instead re-create from scratch.

True open source fashion is also the freedom to work on whatever you want.

"Just think about it, if they would have contributed to foo instead of working on bar, then foo would be twice as good!" keeps being thrown around every time someone announces something new around here.

mobiuscog 9 years ago | |

This is just ridiculous.

Open Source is about just that - allowing anyone and everyone to have access to the source and (if free) making their own version.

There never needs to be justification to write something from scratch, even if it's been done a million times before.

You can appreciate open-source, you can wish that proprietary code was open source, but you never have the right to tell people what they should do, nor are you ever likely to be correct as to what they should do - you are not them.

brazzledazzle 9 years ago | |

It may not based on haproxy but they definitely used it prior to switching. If we're being generous in our interpretation that would indicate that they found that it didn't work for them. I'm not saying your overall point isn't valid but I wouldn't be so quick to judge.

gjvc 9 years ago | |

by that rationale, Apache httpd would have stayed Apache httpd, and we wouldn't have gotten nginx.

paulddraper 9 years ago | | |

Shudder.

bogomipz 9 years ago | |

I'm not sure I agree, they mention HAProxy and Foo over UDP so they are leveraging existing open source technologies. Custom additions to suit ones particular use case isn't necessarily the same as NIH syndrome.

jimjag 9 years ago | | |

They mention HAProxy, but it doesn't look like it's based on it.

serge2k 9 years ago | |

> I am increasingly bothered by the "not invented here" syndrome

I'm bothered by the increasing prevalence of "never invent here".

FOSS is great, but if it's not meeting your needs then writing your own is perfectly valid.

Retra 9 years ago | | |

And often one of your needs is "have people on hand who know exactly how it works", and the easiest way to achieve that is to have those people build it from scratch. That's the only way to actually ensure it makes your needs priority #1.

jedberg 9 years ago | |

> It is then justified that their creation is needed because "no one else has these kinds of problems" but then they open source them as if lots of other people could benefit from it. Why open source something if it has an expected user base of 1?

Two reasons.

1) Recruiting. Check out our awesome code! Don't you want to work on this too?

2) Our unique problem today will be the problems of everyone in three years.

This has borne out with the Netflix opes source. At the time it was a problem unique to Netflix -- now a bunch of people are using that software or derivatives.

dismantlethesun 9 years ago | |

For me, NIH is about the idea of already having deep knowledge of the problem domain so a solution is relatively straight forwards. Sure it'll take X time, and there might be some hiccups, but its all about effort and not having to learn anything new.

Joining a pre-existing project is more reasonable when you just can't replicate the basis of the project without learning a lot of new things.

That's the reason why there's a plethora of compile to JS languages, but only a few actual javascript virtual machines.

merb 9 years ago | |

basically a load balancer must often do special stuff and most companies building custom software around haproxy, nginx whatever to support their needs.

im4w1l 9 years ago | |

Why would you start a new company when you can join and enhance an existing one?

Scaevolus 9 years ago |

Related presentations/papers about large scale load balancing:

Facebook: https://www.usenix.org/conference/srecon15europe/program/pre...

Google: http://static.googleusercontent.com/media/research.google.co...

gwright 9 years ago |

While I understand that NIH syndrome is a real thing, it is very dissapointing to read many of the comments here.

I think very few HN readers are really in a position to have an informed opinion regarding Github's decision to build new piece of software rather than using an existing system.

Personally I find this area quite interesting to read about because it is very difficult to build highly available, scalable, and resilient network service endpoints. Plain old TCP/IP isn't really up to the job. Dealing with this without any cooperation from the client side of the connection adds to the difficulty.

I look forward to hearing more about GLB.

Ianvdl 9 years ago |

Given the title and the length of the post I was expecting a lot more detail.

> Over the last year we’ve developed our new load balancer, called GLB (GitHub Load Balancer). Today, and over the next few weeks, we will be sharing the design and releasing its components as open source software.

Is it common practice to do this? Most recent software/framework/service announcements I've read were just a single, longer post with all the details and (where applicable) source code. The only exception I can think of is the Windows Subsystem for Linux (WSL) which was discussed over multiple posts.

wtarreau 9 years ago |

Did people really read the article ? For me it was pretty clear, maybe it involves some regular load-balancing terms that people are not familiar with, because I'm seeing a lot of bullshit written in the comments, but here is what is described there :

- in a traditional L4/L7 load balancing setup (typically what is described in my very old white paper "making applications scalable with load balancing"), the first layer (L3-4 only, stateless or stateful) is often called the "director".

- the second level (L7) necessarily is based on a proxy.

For the director part, LVS used to be used a lot over the last decade, but over the last 3-4 years we're seeing ECMP implemented almost in every router and L3 switch, offering approximately the same benefits without adding machines.

ECMP has some drawbacks (breaks all connections during maintenance due to stateless hashing).

LVS has other drawbacks (requires synchronization, cannot learn previous sessions upon restart, sensitivity to SYN floods).

Basically what they did is something between the two for the director, involving consistent hashing to avoid having to deal with connection synchronization without breaking connections during maintenance periods.

This way they can hack on their L7 layer (HAProxy) without anyone ever noticing because the L4 layer redistributes the traffic targeting stopped nodes, and only these ones.

Thus the new setups is now user->GLB->HAProxy->servers.

And I'm very glad to see that people finally attacked the limitations everyone has been suffering from at the director layer, so good job guys!

gumby 9 years ago |

They talk about running on "bare metal" but when I followed that link it looked like they were simply running under Ubuntu. Is it so much a given that everything is going to be virtualized?

When I think of "bare metal" I think of a single image with disk management, network stack, and what few services they want all running in supervisory mode. Basically the architecture of an embedded system.

wmf 9 years ago | |

Yes, it is assumed that all startups are running in EC2 us-east-1 and "bare metal" is the accepted term for non-virtualized systems.

colemickens 9 years ago | | |

I don't get it. What else is bare metal meant to mean? "bare metal" = "embedded system"? What does "embedded system" mean then? I guess my age / cloud-nativeness is showing?

gumby 9 years ago | | |

Thanks for the clarification. Yet another dilution of a technical term, sigh.

I was quite excited when I read it, and felt quite let down when I followed up.

Veen 9 years ago | |

In the hosting world, bare metal generally means dedicated servers and clusters of dedicated servers. It's just a shorthand for "not cloud".

malodyets 9 years ago | |

I wondered the same thing: "Wow, they have their own git kernel?" But no.

yladiz 9 years ago |

I'm of two minds about this. Part of me agrees with many of the commenters here, in that Not Invented Here syndrome was probably in effect during the development of this. I don't really know Github's specific use case, and I don't know the various open source load balancers outside of Haproxy and Nginx, but I would be surprised if their use case hasn't been seen before and can be handled with the current software (with some modification, pull requests, etc.). On the other hand, I would guess Github would research into all of this, contact knowledgeable people in the business, and explore their options before spending resources on making an entirely new load balancer. Maybe it really is difficult to horizontally scale load balancing, or load balance on "commodity hardware".

That being said, why introduce a new piece of technology without actually releasing it if you're planning to release it, without giving a firm deadline? This isn't a press release, this is a blog post describing the technical details of the load balancer that is apparently already in production and working, so why not release the source when the technology is introduced?

p1mrx 9 years ago |

GitHub only speaks IPv4, so I would be extra-skeptical about using any of their networking code to support a modern service.

NatW 9 years ago |

I'm curious if they looked into pf / CARP as part of their research into allowing horizontal scalability for an ip. See: https://www.openbsd.org/faq/pf/carp.html

logicalstack 9 years ago | |

CARP and similar systems require an active/passive configuration which we did not want since it needs at least twice as many hosts, half of which are not doing any work. We had similar issues with our former Git storage system based on DRDB (http://githubengineering.com/introducing-dgit/).

pfsync, lvs and etc uses multicast to share connection state which we also wanted to avoid.

voltagex_ 9 years ago | | |

Why do you want to avoid multicast?

treve 9 years ago |

I half expect a comment here explaining why Gitlab does it better ;)

sytse 9 years ago | |

:) We're not doing this better.

We're struggling with our load balancers right now. We're using Azure load balancers and then HAproxy. But the Azure ones sometimes don't work. Luckily the new network type on Azure supports floating IPs so we can set something up ourselves https://gitlab.com/gitlab-com/infrastructure/issues/466

jdc0589 9 years ago | | |

do you guys have a blog post or anything talking about why you settled on Azure? I've used both azure and aws a decent amount now, and GitLab falls outside the "right place for the right tool" mentality I've been operating with. Most of our infrastructure is in Azure, but it's also mainly windows (azure has the best windows VM pricing with a moderate was agreement, which isn't surprising). There are some regulatory considerations for us too, buts that's anither conversation.

dogismycopilot 9 years ago | | |

I would love to see the solution. We're also desiring to run HAProxy in Azure with keepalived (even in unicast mode). The black-box "windows based" load balancer that Azure offers is quite limited.

jedberg 9 years ago |

Awesome. The whole time I was reading I was thinking "they need Rendezvous hashing". And then bam, last paragraph mentions that is in fact what they are using.

lifeisstillgood 9 years ago |

I love using GitHub and appreciate the impact it is and has had. But this post is what is wrong with the web today. They have taken a distributed-at-it's-plumbing technology, and centralised it so much that now we need to innovate new load balancing mechanisms.

Years ago I worked at Demon Internet and we tried to give every dial up user a piece of webspace - just a disk always connected. Almost no one ever used them. But it is what the web is for. Storing your Facebook posts and your git pushes and everything else.

No load balancing needed because almost no one reads each repo.

The problem is it is easier to drain each of my different things into globally centralised locations, easier for me to just load it up on GitHub than keep my own repo on my cloud server. Easier to post on Facebook than publish myself.

But it is beginning to creak. GitHub faces scaling challenges, I am frustrated that some people are on whatsapp and some slack and some telegram, and I cannot track who is talking to me.

The web is not meant to be used like this. And it is beginning to show.

contingencies 9 years ago |

I am intrigued by their opening statement of multiple POPs, but the lack of multi-POP discussion further in the system description.

My understanding is that the likes of, for example, Cloudflare or EC2 have a pretty solid system in place for issuing geoDNS responses (historical latency/bandwidth, ASN or geolocation based DNS responses) to direct random internet clients to a nearby POP. Building such a system is not that difficult, I am fairly confident many of us could do so given some time and hardware funding.

Observation #1: No geoDNS strategy.

Observation #2: Limited global POPs.

Given that the inherently distributed nature of git probably makes providing a multi-pop experience easier than for other companies, I wonder why Github's architecture does not appear to have this licked. Is this a case of missing the forest for the trees?

lamontcg 9 years ago |

Why not just use DNS load balancing over VIPs served by HA pairs of load balancers?

Back in the day we did this with Netscalers doing L7 load balancing in clusters, and then Cisco Distributed Directors doing DNS load balancing across those clusters.

It can take days/weeks to bleed off connections from a VIP that is in the DNS load balancing, but since you've got an H/A pair of load balancers on every VIP you can fail over and fail back across each pair to do routine maintenance.

That worked acceptably for a company with a $10B stock valuation at the time.

manigandham 9 years ago | |

Company stock value has nothing to do with their scaling, performance and customized processing requirements.

madmulita 9 years ago |

We are in the process of moving all of our infrastructure to OpenStack, OpenShift, Ansible, DevOps, Microservices, Docker, Agile, SDN and what not.

There are some brainiacs pushing these magic solutions on us and one of the promises is load balancing is not an issue, even better, it's not even being talked about.

Please, please, tell me there's something I'm missing.

squiguy7 9 years ago |

I know they mentioned their SYN flood tool but I recently saw a similar project from a hosting provider and thought it was neat [1]. It seems like everyone wants their own solution to this when it is a very common and non-trivial problem.

[1]: https://github.com/LTD-Beget/syncookied

bogomipz 9 years ago |

Do the Directors use Anycast then? That wasn't clear to me.

jssjr 9 years ago | |

Anycast usually implies traffic will be directed to the nearest node advertising that prefix. The GLB directors leverage ECMP which provides the ability to balance flows across many available paths.

bogomipz 9 years ago | | |

Anycast and ECMP work together in the context of load balancing. ECMP without Anycasted destination IPs would be pointless for horizontally scaling your LB tier.

What Anycast means is just that multiple hosts share the same IP address - as opposed to unicast. When all the nodes sharing the same IP are on the same subnet "nearest" is kind of irrelevant. So the implication is different.

alsadi 9 years ago |

I never like github approach, they alway use larger hammers