One Million Concurrent TCP connections

One Million Concurrent TCP connections(blog.whatsapp.com)

192 points by pvodsevhcm 14 years ago | 65 comments

mrb 14 years ago |

I remember reading around 2002-2004 about a sysadmin managing a very large "supernode" p2p server who was able to fine-tune its Linux kernel, and to recompile an optimized version of the p2p app (to allocate data structures as small as possible for each client) to support up to one million concurrent TCP connections. It wasn't a test system, it was a production server routinely reaching this many connection at its daily peak.

If it was possible in 2002-2004, I am not impressed that it is still possible in 2011.

One of the optimizations was to reduce the per-connection TCP buffers (net.ipv4.tcp_{mem,rmem,wmem}) to only allocate one physical memory page (4kB) per client, so that one million concurrent TCP connections would only need 4GB RAM. His machine had barely more than 4GB RAM (6 or 8? can't remember), which was a lot of RAM at the time.

I cannot find a link to my story though...

gtani 14 years ago | |

(Not your link but) some pointers on tuning erlang/OTP servers:

http://www.erlang-consulting.com/thesis/tcp_optimisation/tcp...

http://www.trapexit.org/Building_a_Non-blocking_TCP_server_u...

http://groups.google.com/group/erlang-programming/browse_thr...

(this erl mailing list thread is pretty typical, if you put up code, describe your app, hardware, network, database/external dependencies, etc, you'll get a ton of good advice about killing off bottlenecks. Another example

http://groups.google.com/group/erlang-programming/browse_frm...

mmaunder 14 years ago | |

Agree with this being underwhelming. Maybe they'll hire someone who understands that Erlang is using kqueue to make this possible.

Running netstat|grep like this on a high concurrency server takes a long time to run. I've never found a faster way to get real-time stats on our busy servers and would be interested if anyone else has.

thomasknowles 14 years ago | | |

Actually I would run ss over the netstat for such high concurrency connection as netstat hits /proc/net/tcp and it's quite slow. However, you need to have the tcp_diag module loaded otherwise it falls back to /proc/net/tcp.

derobert 14 years ago | | |

You might be able to parse /proc/net/tcp yourself faster, especially for just counting by state.

adient 14 years ago | | |

You may have tried but generally add -n to netstat to speed it up by just displaying IPs instead of trying to resolve them. I've never seen a system with so many concurrent connections hands on so there could be some other bottleneck.

tworats 14 years ago |

The WhatsApp guys are very sharp ex-Yahoo guys who've had tremendous experience with scaling systems. Rick Reed is fairly legendary. Yahoo was a long time FreeBSD shop, so it's not surprising they went with that.

I hope they publish how they did it - in fact let me drop them an email and see if I can convince them to do so.

cperciva 14 years ago | |

Yahoo was a long time FreeBSD shop

FWIW, Yahoo still uses FreeBSD extensively.

lenn0x 14 years ago |

It's been done before. Here is an article using Erlang and Linux.

http://www.metabrew.com/article/a-million-user-comet-applica...

Part 3 is my favorite. http://www.metabrew.com/article/a-million-user-comet-applica...

jen_h 14 years ago | |

This article series is kind of a Bible to me. It's not going to solve all of your problems, for sure, depending on yer stack, and setting yourself up for a forkbomb isn't the wisest in all situations, but it's got a lot of good advice & is pretty good about providing you a "Okay, tweak these parameters and then try to break it" baseline.

indygreg2 14 years ago |

Technical details would be interesting. Until then, here's Urban Airship's post from last year on 500k connections on Linux:

http://urbanairship.com/blog/2010/09/29/linux-kernel-tuning-...

jsr 14 years ago |

OK, you hooked me with the title. But "FreeBSD + Erlang" was kind of a dissatisfying reason for how you achieved it. Would love to hear more details! How far we've come since http://www.kegel.com/c10k.html

getsat 14 years ago | |

They could have done it using C, Ruby, Python, or any other language. kqueue is what makes FreeBSD (and OSX) awesome at concurrency.

http://en.wikipedia.org/wiki/Kqueue

silentbicycle 14 years ago | | |

How does kqueue compare to epoll on Linux? I've written C code using kqueue on OpenBSD and OS X, but have only used epoll via libev (and not at especially high load). I thought the big change came from trading level- for edge-triggered nonblocking IO, but maybe the kqueue implementation is superior for sockets somehow?

The main advantage Erlang has over C/Python/Ruby/etc. is that asynchronous IO is the default throughout all its libraries, and it has a novel technique for handling errors. Its asynchronous design is ultimately about fault tolerance, not raw speed. Also, it can automatically and intelligently handle a lot of asynchronous control flow that node.js makes you manage by hand (which is so 70s!).

You can make event-driven asynchronous systems pretty smoothly in languages with first class coroutines/continuations (like Lua and Scheme), but most libraries aren't written with that use case in mind. Erlang's pervasive immutability also makes actual parallelism easier.

With that many connections, another big issue is space usage -- keeping buffers, object overhead, etc. low per connection. Some languages fare far, far better than others here.

asomiv 14 years ago | | |

The kqueue interface is awesome, but unfortunately kqueue is bugged on OS X: http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod#OS_X_AN... (as are poll() and select()!)

Rickasaurus 14 years ago |

This may be a dumb question (I'm not a networks guy) but how do you maintain so many connections with just 65535 ports? Can you have more than one connection per port?

caf 14 years ago | |

The server generally only ever uses one port, no matter how many clients are connected. It is the tuple of (client IP, client port, server IP, server port) that must be unique for each TCP connection - so the limit of 65535 ports is only relevant for how many connections a single client can make to a single server.

nathanappere 14 years ago | | |

I believe this is incorrect. The server usually listen on one port, but everytime it does an accept, a different random port is used, and the client start talking to the server on that new port.

richardw 14 years ago | | |

You can have multiple connections on the same port from the same client. E.g. all connections via HTTP commonly go over the same port and a single browser usually has 2-4 simultaneous connections to the server. They don't use multiple ports to achieve this.

pacala 14 years ago | |

There is one unique connection per (address, port) pair.

huhtenberg 14 years ago | | |

Actually, the quintet of [protocol, src_ip, src_port, dst_ip, dst_port] is what's unique (protocol being TCP or UDP)

Rickasaurus 14 years ago | | |

Ahh I see, you just give one box a ton of IPs. Thanks :)

chubs 14 years ago |

They beat me to it! I've only gotten to 500k on EC2, however i believe there's some trickery in their firewalls / NAT which is holding me back... If anyone's interested in the gory details, see: http://splinter.com.au/tag/comet

forsaken 14 years ago | |

Curious about what you've seen from the EC2 networking gear that's holding you back? Firewall not letting more connections through?

chubs 14 years ago | | |

Well, at the moment i can't pin it down to EC2, but it's the only thing i can imagine it'd be. The network/cpu/memory usage is all healthy, and there's nothing in the kernel log, so that's what i'm guessing is the cause. Although i may be wrong.

huhtenberg 14 years ago |

In absolute numbers - wow, that's impressive.

It was several years ago, but I've done my share of high-concurrency stuff under Linux and the highest I got to was about 200K connections - at which point the single-threaded server bottlenecked at its disk I/O.

The main issue is not the actual connection count, it's what the per-socket OS overhead is (so not exhaust non-swappable kernel memory), how many sockets are concurrently active (have an inbound or outbound data queued) and if the application can handle all the events that epoll/kqueue report. This is not a rocket science by any means, and the kernel is relatively easy to fine-tune even when the actual load is present.

cnlwsu 14 years ago |

I would be curious about the hardware used, it can make it much more or less impressive. I have done a test with 1 million concurrent tcp connections using java (mina) on an Ubuntu system... but it had 64 gb of ram. It kept running for weeks under the load which I felt pretty good about.

getsat 14 years ago |

What is the kernel structure overhead (in bytes) per connection on FreeBSD?

imsy 14 years ago |

Good work. On Imsy (www.imsy.com) we hope to achieve the same with Node.js on EC2. Presently the numbers are smaller (in 100K range).. but looks like it will scale smoothly till 1 million.

henry501 14 years ago | |

It's hard to say that something is going to scale smoothly up an order of magnitude until you're there. 900K requests is a lot of room for things to go wrong.

nivertech 14 years ago | |

Node has memory limit of about 140K connections. I was able to accept and actually do something usable with 1M on EC2 and 3M on physical server (Erlang and Ubuntu/CentOS)

kitsune_ 14 years ago |

How much ram does this single server have, what about its cores?

jen_h 14 years ago |

A. That is AWESOME. Props, guys! B. Just how long did it take for that netstat to return? ;)

spokengent 14 years ago | |

FWIW, netstat isn't fun to use. conntrack is better, or cat /proc/net/ip_conntrack

Luyt 14 years ago | | |

They're on FreeBSD. conntrack is a Linux utility.

KonradKlause 14 years ago | | |

conntrack is not netstat. I requires netfilter's connection tracking feature.

Using conntrack on a 1MC system will waste even more kernel memory!

gabi38 14 years ago |

How would kqueue compare to Windows's IO completion ports in terms of performance?

flzz 14 years ago |

Just because you can doesn't mean you should.

nivertech 14 years ago | |

Not many can, but some fortunate enough to actually need this stuff.