This is our first follow up to last week's web framework benchmarks. Since last week, we have received dozens of comments, thoughts, questions, criticisms, and most importantly pull requests. This post shows data collected from a second run on EC2 and i7 hardware that started on Tuesday of this week. A third round with even more community contribution is already underway.

Thanks especially to those who have contributed! We hope this is useful information.

  > In this week's tests, we have added latency tab (available using the rightmost tab at the top of this panel). On i7, we see that several frameworks are able to provide a response in under 10 milliseconds. Only Cake PHP requires more than 100 milliseconds.

This is still a json API comparison, not a full framework comparison.

In the first test it sees how fast frameworks can serialize json and in the second test it sees how fast frameworks can access the database AND serialize json.

An inefficient json library and a framework is doomed across all tests. There are a lot of things frameworks do beyond serving json. These benchmarks are presented as representative of broad use when really they are only representative of the frameworks if used as a json api.

A raw HTML hello world, and removing the json encoding step in the db test would go a long way towards getting results that can mean what the authors seem to want them to mean.

There's nothing wrong with a json API benchmark, in fact its quite valuable in a lot of cases. But if that's what this is intended to be, the authors should say so.

As an amateur hacker and after 275 days of reading Hacker News I feel I can now navigate in the sea of client side Javascript frameworks. With the introduction of this data on web framework performance, I now have another set of choices that I am completely unqualified to comprehend.

My first impression is that this data shows about 3 levels of web frameworks. At the bottom (slowest) we have Django's and Rails and many other introductory app server frameworks. Lets say after I was able to build an initial product successfully, would I then consider re-building the product in a higher performance framework as Go, Node, etc?

The third level, netty and gemini and servlet etc, I am not familiar with. Googling "netty" I get --"Netty is an asynchronous event-driven network application framework"-- I thought that is what Node is (in js) and Go does with gophers.

What are the use cases for these faster frameworks and do they follow an evolution of performance options that an app might go through?

Are C#/Mono web frameworks an option? Because I would love to see how http://servicestack.net fares:

https://github.com/ServiceStack/ServiceStack/wiki/Real-world...

Once again, the Node examples are wrong.

- Unnecessary parsing.

- Writing strings instead of buffers (they're copied, they aren't sent as-is).

- Using async. It does a lot of really nasty things, most of which break V8 optimization best practices. This is a perf benchmark, not a comparison of how concise your code can be.

- Having the main request handler in a gigantic function that will never be properly optimized by V8.

- Not getting helper functions that are clearly monomorphic warm before accepting requests.

While the code itself is what I would consider fine, when benchmarking against strongly typed compiled languages, performance concerns become important, even if you have to write ugly code.

It might be interesting to track average and peak server side memory usage during these tests. For example, if framework A is only 80% the "speed" of framework B, but uses 1/4 as much memory, for some users this might be a win for A.

A note about Compojure: they're using Korma for database access. Korma is an abstraction layer that's more Clojure-friendly than using SQL directly, but lighter than an ORM. Using clojure.java.jdbc might produce higher performance.

It's a useful data point to know the speed of frameworks, but what matters most to me is speed of development and whether or not I enjoy the process.

Wish there was a way to benchmark those points.

It seems like you're still using Mongo DB as the datastore for some languages and Mysql as the datastore for other languages. This seems like it would bias the results.

I wish Lua was tested in the benchmarks.

In particular, Lua-CJSON [1]

[1] http://www.kyne.com.au/~mark/software/lua-json-performance.h...

(I know, everyone wishes their favorite language was represented and not all can be tested. I'm just posting this in case the OP does a "Round 3")

EDIT: fix typo

This is an awesome project, and thank you for the follow-up.

In particular, this is awesome because it's introduced me to some new frameworks that I hadn't even heard of (ie vert.x) that seem extremely interesting.

Keep it up and many thanks for this contribution!

It is worth pointing out Go 1.0.3 not Go 1.1 beta is being used. I tested the code locally and on my machine 1.1 beta was about 20% "faster" (req/sec).

This is definitely motivation to learn another framework besides Rails and Django.

Compojure's syntax and design seem really attractive... is anyone using that in production? Would it be a reasonable choice for a "serious" web application?

Yesod disappoints! Unfortunate for how much it is hyped. This is definitely good to know, even if it appears that the Warp server was not used.

Very interesting - again.

What I - again - find most interesting, is the large discrepancy in the 20-query test. Common wisdom seems to be that language performance is not important b/c every language hangs in the same IO/DB.

Does anyone know what on earth is Play! doing that makes it is so slow?

I'd like to see Lift in there if you guys do another round.

Gemini conveniently leads the pack all the time. So either OP and his/her colleagues have revolutionized web programming or the benchmark is nothing but a PR piece.

Is anyone else alarmed to see raw PHP spanking Flask, Sinatra, and other Python/Ruby frameworks?

If not, can anyone explain in layman's terms why that should be the case?

This is superb.

1. More sorting options please :) I really want to sort by standard deviation.

2. Also in case of rails and other single threaded frameworks are you spinning up multiple processes?

3. Why is Go not being tested in the Database access tests?

4. Can you test rails with unicorn too?

I apologize if this question has been asked and answered, but why is netty missing from the database tests? No ORM?

Thanks for the tests. Especially the latency is interesting.

I cant't wait for the next round, when the updates of the last two days are included (elli, grizzly, play-java, play-scala, snap etc).

I wonder where the bottleneck for the ruby benchmarks is. Is it in passenger, the framework, or somwhere else? I am sure I have gotten higher requests per second on my workstation when I tuned Ramaze (which should not be faster than using raw rack). My guess is that it is either due to a different method of measuring or the fact that I was using thin rather than passenger.

As a Django guy with hurt feeling right now, I just wanted to say that this is a great project. I look forward to the next versions.

I would love to see ASP .NET and ASP .NET MVC included in the benchmark.

I think I said something similar on the first round, but for the Python tests, I'd be curious to see alternate runtimes and/or concurrency strategies: pypy, gevent, and probably the confluence of the two (though it requires quite a bit more hacking, since gevent doesn't officially support pypy).

Most of these are bound to a language: choose a language first (and my answer to that question is "whichever one you're happiest working in and can get the job done"), then figure out which backend to use. You can fret over the framework more after you've already built something that works.

IMO if you build a product and it is working, there really is not much reason to redo your efforts in a higher performance framework unless your needs require you to do so. If you are at your current hardware's limits and you can't throw more hardware at it (for whatever reason), it may make sense in that case. If you want to reduce the amount of servers you currently have, it also makes sense there. http://blog.iron.io/2013/03/how-we-went-from-30-servers-to-2... This is a great post talking about going from 30 servers using Ruby to 2 servers using Go.

Use case example - Twitter was using Rails on the front-end for a while but it got to the point where using it made little sense. Part of Rails' performance is dependent on caching - which is really hard, if not impossible, to do well when you are dealing with live data.

FYI, Gemini is our framework that we've built and maintain internally. You probably won't find reference to it (except for out blog).

edit This is the link to Twitter's blog about Rails http://engineering.twitter.com/2011/04/twitter-search-is-now...

The third level is java based frameworks that are making heavy use of highly optimized native code paths for string handling.

All metrics are meaningless without context.

I like these comparisons. It's unlikely that anyone is going to pick a framework for their project based on just this alone.

I find its good "food for thought" as it provides me exposure to frameworks that perhaps I can use one day but also to learn more about.

The most interesting aspect is all the code for each framework to do the simple page output, the queries, etc. can be viewed for each framework. I find this an incredible learning tool as it shows the personality of each framework/language.

Netty, gemini and servlet are all java server solutions without a thick framework layer getting in-between. What you can tell from these benchmarks in my opinion is that java (or rather the JVM) has the best "raw" performance. Frameworks introduce a slow-down factor on top of the raw platform, and different frameworks have different slow-down factors.

You have to trade off developer productivity for performance when it comes to choosing frameworks. The slower the framework the more it does for you.

In general, I echo ConstantineXVI's comment. However, I'm also curious to know where the definite line is between the bottom half and the top 5 on the table. At what point does a team go "We're getting hammered and our backend is the problem. We have to switch to X or Y because our API on Z has reached its limit."

This is exactly the sort of question that should be asked in the context of benchmarks like this. There have been a lot of good comments in response, but I just wanted to thank you for asking one of the best questions in this thread.

I think it's a misconception to think of Django and Rails as "introductory frameworks". These are production ready frameworks that can scale to meet a variety of (though not all) business needs.

>what matters most to me is speed of development and whether or not I enjoy the process.

Matters most? There are 100x differences here. Take two companies writing the same application, one in Spring the other in Rails. Both have easy access to knowledgable people, both are industry standard. However the Spring application would be several orders of magnitude more scalable[1]. I'd consider that to be a more important consideration over enjoying the language.

[1] All benchmarks are suspect until proven otherwise. You can write slow software in Java/C/C++, you can write fast software in Python/Ruby etc etc etc

The reason why I don't like to see this same argument somewhere at the top of every single performance related thread is that it is not a law of physics that faster languages/frameworks are necessarily less productive than slower ones. It's not even an empirical truth.

If someone from my team suggested to use Cake PHP I would say no without even looking at it once. Not because I would never trade performance for productivity, but because I simply do not believe even for a second that productivity gains are the reason for that kind of horrific performance.

I 100% agree with this. It has been a huge point of discussion for us here internally. "Developer efficiently with language x." Ultimately, language performance does matter but there are ways to make up for less performant languages. If it takes you 2x as long to get something out the door though, then that is a huge problem.

Just use the language and framework you are most efficient in and enjoy using. Worry about framework performance when you notice server load increasing over time.

> Flask

I don't know about Sinatra or anything else but I can give you the reason why Flask out of the box performs like it does. Unlike PHP Flask is threadsafe and uses locks and thread or greenlet local storage. This level of indirection adds a basic performance penalty when executed on a single threaded server like gunicorn. There are ways to change that if you think that becomes a performance problem by replacing the stack object with one that does not do dispatching and by removing the locks.

This in theory is easy to do, it's just generally not a real issue since most Flask apps will actually wait on database more than they waste CPU time.

Additionally Flask has a huge request handling overhead which will be primarily noticable in dummy benchmarks that don't do much in view functions. If you have an application that actually barely executes anything in the views you can subclass the Flask object to remove some of the functionality you normally always pay for (request processors, session initialization, message flashing etc.)

Lastly Flask has some sanity checks when Python is not run with `-O` that simplify debugging. Depending on which code paths you hit you might be paying for some of that.

In the database test the primary slowness is most likely not Flask but the way the database is configured.

Generally if you ever encounter Flask to be an actual performance problem I am more than happy to point you to ways to get more speed out of it. Generally speaking though there is not much point micro-optimizing until you notice it is a problem.

Why shouldn't that be the case? PHP has been designed from the get go to be super fast and 5.4 introduced a bunch of optimizations that have sped it up even more so. Ruby was designed for programmer happiness, PHP was designed for performance and getting sh*t done.

Not sure about ruby, but django doesn't ship with connection pooling, so that is probably why its database performance is terrible. It could be configured to use pooling.

With such a trivial request handler like the one in benchmark, only a small percentage of the execution is actually taken up in the view, so all of the time you're seeing in the benchmark is pure overhead of the framework. A longer running view across all of these frameworks would cause things to even out a bit. All that being said, however, it's not an unfair comparison. After all, a benchmark of this nature should be a measure of overhead because that's really all there is to measure.