Ask HN: Is there a standard web framework performance benchmark test?

8 points by tcarnell 13 years ago | 2 comments

I want to benchmark my web framework 'Contentigo.com', I want to see how it compares to CodeIgniter, Symfony etc.

Has somebody written a good, clear set of benchmark tests that I can also apply to Contentigo? EG:

- time to serve 100 'realistic' web pages (without any caching)

- time to serve 100 'realistic' web pages with 3 database queries per request (with some defined dataset to retrieve) (without any caching)

- time to serve 100 'realistic' web pages (with caching)

something like this?

rartichoke 13 years ago |

I don't think there's a standard list but make sure to not just run micro benchmarks.

I would probably do:

1. Return a few hundred bytes of json with no I/O.

2. Return tens of thousands of bytes of json with no I/O.

3. Do the same 2 tests as above but read the data from a DB.

4. Do the same 3 tests as above but render templates instead of sending json.

5. Do everything again with it cached.

6. Repeat all steps with 1, 2, 5, 10, 50, 100, 200, 500, and 1000 concurrent connections while paying close attention to the response times at the 99% level in addition to the reqs/second.

Also don't forget to look at the bytes transferred. There's going to be differences in the http headers between frameworks. If you're going to compare them directly to yours try to adjust the headers so it matches.

tcarnell 13 years ago |

Would also be handy to create a simple set of JMeter tests to ensure consistency across frameworks.

I would would be keen to run the test against Contentigo, but not very keen to do this for all the other frameworks! I would hope that if a good set of standard tests could be found, then the various vendors/project could run the tests themselves.