Twitter survives election after Ruby-to-JVM move(theregister.co.uk) |
Twitter survives election after Ruby-to-JVM move(theregister.co.uk) |
It's also interesting to see different companies approach this problem differently - Facebook famously recreated a way to run their PHP source code (by compiling it to C and then running it natively) instead of actually rewriting the source to a different language. I wonder if something similar would have been possible for Twitter, or if they weren't happy with how their existing code was structured in the first place which may have made the rewrite more attractive.
Why do I say that? Because in Twitters case, handling tweets is a "trivial" problem to parallelise, and the potential savings of switching language will be a rounding error in terms of scaling their system compared to getting the architecture right.
(To scale Twitter: Make trees. Split the following list into suitably wide trees, and your problem has now been reduced to an efficient hierarchical data store + efficiently routing messages. Both are well understood, easy to scale "solved" problems)
There are many dynamic user driven sites which have scaled well (far less downtime than Twitter) without switching to static compilation.
Also, Twitter doesn't have Google's infrastructure.
In regards to "static compilation", that's not the important bit, but rather the performance of the virtual machine. The JVM, at its core, is not working with static types. The bytecode itself is free of static types, except for when you want to invoke a method, in which case you need a concrete name for the type for which you invoke that method ... this is because the actual method that gets called is not known, being dispatched based on "this", so you need some kind of lookup strategy and therefore the conventions that Java uses for that lookup are hardcoded in the bytecode. However, invokeDynamic from JDK 7 allows the developer to override that lookup strategy, allowing one complete dynamic freedom at the bytecode level, with good performance characteristics.
The real issue is the JVM versus the reference implementations of Ruby/Python. The JVM is the most advanced mainstream VM (for server-side loads at least).
Unfortunately for Facebook, they didn't have a Charles Oliver Nutter to implement a kickass PHP implementation on top of the JVM - not that it's something feasible, because PHP as a language depends a lot on a multitude of C-based extensions. The more pure a language is (in common usage), the easier it is to port to other platforms. Alternative Python implementations (read Jython, IronPython) have failed because if you want to port Python, you also have to port popular libraries such as NumPy. Which is why the PyPy project is allocating good resources towards that, because otherwise nobody would use it.
what I find even funnier is that people just throw in play and grails without a little of experience in rails. The eco system is entirely different:
- you can use java and therefore java libraries(yes you could do the same in jruby, but nvm) - bundles/gems are an order of magnitude better than classical java dependency hell. - need something in rails? add a gem. need something in grails? search throw the outdated plugins, search for missing documentation(just take a look at stackoverflow). in general -> write it yourself or pay a consultant to do it. - have a question? fight with incomplete documentation.
on top of that grails is just a stack on top of the spring mvc.
what about play? i like play more than I like grails tbh, because it doesn't want to be the rails of java. yes people compare it to one another, but that's just the familiarity effect.
now, where's the computationally intensive stuff? nowhere to be found. it's a web api. where's the computationally intensive stuff in twitter? I don't know, but chances are theres a native extension for that nowadays.
There actually was a time when you simply could not build a scalable system in ruby without too many hoops, but that's no longer the case. Yes the GIL is bad, but keep in mind that things like ruby fibers didn't even exist at the time.
My memory might be playing tricks on me, but I remember play developers talking about rails being a huge influence.
If Play isn't rails for Java, how else one does a Rails for Java? The "familiarity effect" is there because Play is modeled after Rails.
> now, where's the computationally intensive stuff? nowhere to be found. it's a web api. where's the computationally intensive stuff in twitter? I don't know, but chances are theres a native extension for that nowadays.
Search, for one is computationally intensive. I am pretty sure there are more which the outside world doesn't know about.
> There actually was a time when you simply could not build a scalable system in ruby without too many hoops,
What scale are we talking about? At Twitter scale, ruby or anything else has to jump hoops. For example, network load from users coming online and offline on FB chat will break out of box solutions.
This is not unlike those stories where a developer writes a trivial program in a new language that is similar to a trivial program they wrote in another language 1+ years prior and compare the results. "I wrote program foo in 30 lines of code in language X, which is much better than the 120 lines of code it took me in language Y two years ago". It's natural for a developer to write a shorter program two years later since they have 2 more years under their belt. In fact I'd expect the same program to be much better two years later even if written in the same language.
Fine, but what I wish I understood was why RoR is so popular to begin with. The claim is always that Rails is so wonderful that it's worth learning Ruby just to be able to build web apps with Rails. Well, if so, then why isn't there a sequel to RoR: Python on Rails? "The benefit people care most about (Rails, not Ruby) using the language you already know (Python)." Since Python, Perl, and Ruby are so similar except in syntax, and people love the Rails part more than the Ruby part, and so many more people know and love Python, and new Python web app frameworks appear all the time...why isn't one of them Python on Rails after years of Pythonistas being forced to abandon Python and learn Ruby just to be able to use Rails?
Is there some significant difference between Python and Ruby that explains this? What makes Rails so attractive and why isn't the same thing done with Python?
I'm a relative newbie to the Ruby world, but as best I can tell, the Ruby and Rails communities both accepted long ago that they weren't made for Twitter levels of traffic.
Fact is, almost no one has Twitter levels of traffic besides Twitter. That's why Ruby and Rails are still so popular, because for ~99% of performance needs, they're more than capable and also extremely pleasant to work with.
Seems like a no brainer that something that is a lot more strict and static will outperform something that has to deal with many more possibilities at runtime.
Now, now, it's The Register. The tabloid style is part of its identity.
The wise programmer says: "Use the right tool for the job."
Please HN, we're wiser than trending stories would suggest.
(They do use a bit of regular Java, but the majority of the core code is Scala, and it's Scala that should be getting the top line credit here, not Java.)
The truth is, Scala is actually helpful with the architecture they came up with AND its great that it runs on the JVM.
I've always admired rails for its flexibility and its enormous productivity boost, but all my serious applications are coded in Lift. I for one believe in "develop and forget", because I'd like to call myself a business guy than a programmer, though I'm deep into both. I like to spend more time expanding/marketing my business than worry about scaling it. But that's just my perspective.
JVM is terribly under-estimated and I realized this when I got started with Lift+Scala. Scala is a very powerful language and requires a totally different mindset (=functional). And Lift is fairly complex for those wishing to get started with it and has poor documentation, despite being a 5-year old Framework. But once you understand it fully (somehow), there's no looking back. Lift provides so many things out of the box, especially related to security (unlike PLay!), so it's kind of a trade-off you have to choose between. Even if you compare all the benchmarks, most of the JVM-based languages like Scala outperform even something like GO! (Ok, that's not fair, since GO is fairly new)
If you're interested in Scala, Coursera has a course on it by the creator of Scala himself (Martin Odersky).
If you're truly more interested in the business side, why don't you build it as quickly as possible in rails/django and then later if it warrants it you can hire some people to build it in lift/something else?
"Last week, we launched a replacement for our Ruby-on-Rails front-end: a Java server we call Blender. We are pleased to announce that this change has produced a 3x drop in search latencies and will enable us to rapidly iterate on search features in the coming months."
http://engineering.twitter.com/2011/04/twitter-search-is-now...
This articles seems more like link-bait to me.
But in any case, if Twitter's architecture is truly scalable then any intrinsic slowness of the language shouldn't be a big problem, because they can just toss more hardware in to compensate. What is a problem is a buggy VM that leaks memory. To run thousands of instances in a heavily instrumented way, the VM must be stable and predictable.
I think their decision was more influenced by the experience of their team of programmers or by employable talent pool - they went the JVM|Java|Scala route because they had people with experience in high level languages and the JVM. If they happened to start with a team with "C hackers" background they would've gone the Ruby|C way and it would've worked as well.
...I think almost any language and technology can (be made to) scale, even to Twitter scale, at least if it's open-source and you have people with the required technology to hack around the internals and recode performance critical parts in lower level languages (basically C, C++ or Go nowadays)
http://engineering.twitter.com/2011/03/building-faster-ruby-...
While most won't ever encounter the issues Twitter encountered using Ruby and Ruby on Rails these kinds of articles are very damaging for the Ruby language and Rails framework because even though I primarily still use PHP, Ruby & Rails are something I have a vested interest in as well and this will no doubt push potential newcomers away from the language.
Twitter has to deal with very high fanout and low cacheability.
I would have thought that you could just throw cheap hardware at that problem, whereas the database would be a considerably complex scaling issue.
But it is commercially solved. If you turn up with a fistful of money, Oracle, IBM, Sybase and a bunch of other companies would love to handle that for you.
History will remember the entire Ruby industry as a series of compounding failures.
The de facto formalisation and specifications.
The black-box behaviour of core development.
The broken-linked, un-versioned docs.
The rampant cargo-cult mentality.
The arcane exceptions.
The meta-frameworks.
Gem hell.
1.9/2.0
Rails.
If you get to the scale of twitter, and more importantly if you have written a real time messaging server with a web application framework, it doesn't matter which framework or language you started with as you're going to be completely rewriting your entire stack at some stage as different parts fail in order to deal with the load (unless you have an incredibly experienced team who has written a twitter equivalent before and scaled it to 15000 messages a second). There's no difference between ruby or python (or perl, or php) in that regard - they are all interpreted and relatively slow, which starts to matter at this scale. And also even if you had used java or c in the fist place, if you had an architecture not written with massive scale in mind, you probably wouldn't survive that growth without radical changes on every level of your stack.
Re ruby versus python, the popularity of rails is partly historical accident, partly that ruby is a nice language which doesn't get in the way and is ideally suited to this domain, and partly that rails is deals with lots of the basics of web development for you without getting in the way too much when you need to adapt it. None of that means ruby is better than python, but I'd disagree that people put up with ruby in order to use rails - it's a really nice language in its own right, but it is not highly performant (though it is getting better). For most websites of course, many of which can employ caching, that is a non-issue, even at large scale - witness the success of Wordpress in php which would not survive even modest loads without caching.
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-14...
Ruby and Python as languages are not radically different in kind, but their respective developer communities have had different focuses, and as a consequence the library of tools are not identical.
Ruby on Rails became popular because people were dissatisfied with the way that web development was being done, and DHH is very good at marketing/propaganda. Ruby on Rails has had a substantive effect on the way that web development is done, and there have been numerous attempts in other languages (that didn't already have a framework like Django) to recreate the things that Ruby programmers enjoy with Rails (CakePHP and Grails come to mind immediately).
To elaborate a bit, Rails tended to have a lot more generated code (the fabled "magic") that really sped up development of your standard CRUD apps. As far as I can tell, this was fairly novel in web development. Django didn't really have a focus on that, you spent a little (lot?) more time configuring. It's better now though.
Of course, at the complete opposite of the spectrum, you have very minimalist stuff like Flask (Python) or Sinatra (Ruby) which is doesn't include a lot of bells and whistles. You'll have to import your own ORM, templating, etc...
When Rails was introduced, there were no other substantive Ruby web frameworks - Ruby itself was relatively obscure compared to every other web-capable tech. As such, it (Rails) had no competition in the framework space. PHP, Java, Python and other languages all had competing web frameworks to choose from.
I'm not sure that's the case. Perhaps had they started out by trying to scale to their current levels they would never have gotten off the ground. When you start developing a system for a new company your largest obstacle is almost always lack of product/market fit.
Twitter hasn't phased out Rails, it has replaced background services written in Ruby.
> Fine, but what I wish I understood was why RoR is so popular to begin with.
That is hard to explain. You will have to try it out yourself and compare with whatever you are currently using.
> why isn't one of them Python on Rails
As other responses pointed out, Python has Django. And no, Django isn't rails inspired. Django was developed independently. Also, the languages and philosophies differ significantly to have a Python on Rails.
If so, it's impossible for a language designer to do a good job, because it's impossible to improve a programming language.
The "no free lunch" theorem would indicate to me that there is no ultimate language, merely languages that are better for common (to you) use cases.
I think all languages have strengths and weaknesses that matter based on the context.
For example, I hate manually managing memory in C/C++. Java, Ruby, etc automate all of that for you, but it comes at a cost of using a lot more memory. That's not a big concern for many applications, but if you're doing embedded software or real-time systems it can be a deal breaker.
So in this example, is C++ better or worse than Java? The answer is that it depends what problem you're trying to solve, what environment you're running in, and what resources you have available on your development team. Personal preference matters too in terms of programmer satisfaction, but just because you like one language's constructs more than another doesn't mean it's well suited to every problem space.
As to the original article, I think it's great to learn about how different companies change their software stack, but not because "w00t Twitter hates Ruby - Java rocks"; rather it's interesting to see how business context changes over time and the implications that has on the technology. Seeing how other businesses have dealt with these hurdles can help you keep an eye out for them in your own business. I just wish that this was the lens that it was written in rather than "OMG - Ruby = Fail Whale."
If you're a language designer then it's good to hear about the complaints and preferences of programmers so that you can design a better language (and thus a better tool). And like any tool, there will be times when it's better to use one language rather than another.
So I think the (even just sometimes) discussion of "good language" should be limited to the context of purpose and goal of the said language, which is similar to "choosing the right language for the task".
Ideally, every person would have their own programming language (or culture), grown from their own personal experiences and desires. In that case, a qualifier for a good language is one that most easily allows expression of new languages or cultures at the individual level.
For a language designer, this could be an impossible problem due to the unique learning models of each individual. Their best hope may be in attracting those whose mental models have some overlap with their own.
Well, since "improve" is a relative and a matter of taste, it is impossible to improve a language, isn't it? Make it run faster, and it will either take up more memory or be slower to develop with. Make it faster to develop with, and it will run slower or have some other tradeoff.
Hence, "use the right tool for the job". And when your focus shifts from "quickly building a product" or "quickly adding new features" to "rock-solid stable and fast", then you need to switch tools. Simple as that.
No technology stack is perfect, but I've yet to meet a stack that was pure evil. There may be some cargo-cult personalities in the Ruby community, but, if there is, it's only because there is value in it.
An infinitesimal number of sites have to deal with Twitter's scale problems. The rest can work on getting crap done instead of worrying about Maseratti problems.
Cargo-culting is what you accuse others of when they are learning and you do not like them.
The appropriate way to deal with newbies who do not fully understand the consequences of the decisions they have made (perhaps even while they are advocating others join them), is to explain their rhetorical and technological shortcomings in a way that others can learn from. Accusing someone of "cargo-culting" is just unhelpful character assassination.
Also, weak-sauce.
As someone who has seen Ruby cause major problems I'd have to say it's not too far off "pure evil"
This is a bit dramatic. Despite its faults, the language and community around it have been quite a success story. It's had a huge and positive influence on the web development world.
The Ruby community, really really cares about developer tooling and teaching others. Ruby is also one of the nice places where object oriented programming touches metaprogramming.
Don't forget that a lot of sysadmin work is done in Ruby now. Both Puppet and Chef are written in Ruby, and that Github and Heroku both came up out of the Ruby community.
Is this really a problem? Every package/dependency manager seems to blow up on occasion. Gems haven't given me the problems that I've had with Pip/CPAN/Autoconf.
There are things more funny to do than solve bugs introduced by a backported patch.
Almost every one of these "we switched from A to B and got a 3x speed increase" articles conflates a lot of different variables. The first version you build when you have no traffic and product/market fit is the most important thing. Performance is a low priority. Eventually it hits a bottleneck and you begin to look at performance. Perhaps there is another language out there that is faster than the one you're using. At this point nobody says "let's do an exact code translation from A to B". As you rewrite, you keep a constant eye on performance. It often involves ripping out abstractions and moving closer to the metal. The system you end up with usually looks nothing like the one you started out with, nor should it since it is the product of all of your experience scaling up to that point.
The approach I suggested would create massive message spikes, and there's undoubtably optimizations. E.g. 30 million followers for the top users is a lot, but most of them don't tweet all that often. Perhaps it's better to keep the recent tweet of the top 1000 (or 100,000) users in memory in memcached's and "weave" them into the timelines of their followers instead of trying to immediately stored the in timelines for each user.
The point is scaling is pretty much never a language problem once your needs are large enough that they'll exceed a server or even a rack no matter how efficient the language implementation is.
At that point the language choice is an operational cost vs. productivity issue down to whether or not the CPU usage differences are sufficiently high to cost you enough to offset any productivity advantages you might get from the slower language implementation.
That is a valid concern. It's perfectly possible that Twitter is right in switching in their case. E.g. even a 10% reduction in servers could pay for a lot of developer hours when your server park gets large enough.
If you're on the tipping point where switching to a faster language implementation can keep you from having to scale past a single machine, then it's slightly different. But arguably if you're in that position, you should still plan for growth.
So what people used to call "gem hell" was actually "I need to specify all my dependencies and take care of conflicts myself." That's what bundler does. And it uses rubygems to actually retrieve, install and load the gems.
What I found interesting from another twitter blog post linked from the article was that they are actually running on a modified version of Ruby 1.8.7 REE, not Ruby 1.9.x as you might expect, so that must really skew their comparisons![1] Anyway, there are a lot of variables in a huge system like twitter, so boiling it down to just a problem with Ruby performance seems pretty simplistic (as you'd expect from the register), and I seriously doubt they could have avoided this kind of rewrite and refactor of their entire stack in any language when hitting this sort of scale. To me it doesn't say much at all about Ruby as a language or whether it is suitable for websites.
[1] http://engineering.twitter.com/2011/03/building-faster-ruby-...
Depending on the implementation, so is Ruby. The canonical MRI 1.9 compiles to bytecode and interprets it. But unlike python, it doesn't write the bytecode to disk. The loading of bytecodes has been disabled until a bytecode verifier is implemented.
As much as I agree it's delusional to think one language may be adapted for virtually any kind of tasks, it seems way too much consensual to me to tell there is no such thing as bad or deprecated languages.
Also, Nastran got us to the moon (which is basically Fortran)
As a happy user of Ruby, Ruby excels in many of its promises. I wish Ruby-MRI focuses a bit on performance in coming versions. The ruby community needs this guy who developed the V8 javascript engine to do the same performance leap here too.(Maybe Rubinius or JRuby already do) That said, let us not forget that Twitter performance requirement is kind of extreme for most startups, and most JVM based languages and foremost Java lag in ease of use, speed of development etc to be a viable option for a startup. (I exclude here Clojure, Scala, JRuby etc). For most startups speed of development is what matters.
I disagree. There's more than ease of use, speed of development and performance to determine if a language is "better" suited to a problem than others. And they're all trade-offs to some point. So to excel at one means that you fall behind on others. Prolog is still used heavily in some areas since it's a natural fit to logic problems. It's just pure logic theory modeled in a programming language. It's a pityful language to solve most "real world" problems, but it excels at what it was made for. So is it "better" than ruby? I don't think so. Is it possible to write a language that models logic problems so nicely as Prolog and still keep the ease of use of ruby? I very much doubt that.
It sounds like you are spending time/effort worrying about scaling way before needed. If you can solve your problems you have today faster then I would do that, not hinder yourself today for possible problems way down the track.
A certain language might be twice as fast to get work done in, but you can be 10 times as fast if you are enjoying yourself and motivated.
I just wouldn't kid yourself that scaling is the reason for the choice. Just enjoying it is plenty of reason.
I wonder if they would have done anything different, had JRuby been where it is now?
Those 10 tiny tiny Ruby programs are 3-100x slower than the Java programs written for the tasks.
Like Ruby.
> and compiled into bytecode
Like JRuby.
Python and Ruby are comparable languages. They are both high-level dynamic languages, with the same sort of tradeoffs (eg, you can do more with less characters than in most statically typed languages, at the expense of safety and performance). There are however significant differences in terms of philosophy, ecosystem and community between the two.
http://stackoverflow.com/questions/4769004/learning-python-f...
Python's performance is only slightly ahead of Ruby's. Both Python and Ruby are a bad match for CPU intensive tasks(provided we are talking pure python/ruby and not native extensions). Network and IO in general is salvaged by using evented io(or some high level evented framework).
I doubt that. The fact that jruby is close to bug-by-bug compatibility means that for most projects you can go and develop on mri or rubinius for fast development and use jenkins/travis/... to run your testsuite against jruby. You can then do integration tests as well as staging on jruby to catch anything your testsuite didn't catch. I've seen more than one large project take that course of action and do just fine.
Deployment on jruby isn't any harder than deployment on any other ruby. It's still "puma start" or whichever server you chose. You can however package your app in a war use a standard servlet container such as tomcat or jetty to deploy, but that's strictly optional.
The biggest win for me is dependency management. One my maven file is setup right, I only need a JRE on the host.. Dependency management is a bit trickier with Python, perhaps for Ruby as well.
JavaScript
http://shootout.alioth.debian.org/u64/which-programs-are-fas...
Not sure what gave you this impression, as the majority of Java bytecode instructions are typed. For example, the add instruction comes in typed variants: iadd (for ints), ladd (for longs), dadd (for doubles), fadd (floats), etc.
The same is true for most other instructions: the other arithmetic instructions (div, sub, etc.), the comparison instructions (*cmp), pushing constants on the stack, setting and loading local variables, returning values from methods, etc.
http://en.wikipedia.org/wiki/Java_bytecode_instruction_listi...
InvokeDynamic, as you point it out, was added to make implementing dynamic languages on the JVM easier, because the JVM was too statically typed at its core.
Disregarding primitives, the JVM doesn't give a crap about what types you push/pop the stack or what values you return.
invokeDynamic is nothing more than an invokeVirtual or maybe an invokeInterface, with the difference that the actual method lookup logic (specific to the Java language) is overridden by your own logic, otherwise it's subject to the same optimizations that the JVM is able to perform on virtual method calls, like code inlining:
http://cr.openjdk.java.net/~jrose/pres/200910-VMIL.pdf
> ... because the JVM was too statically typed at its core
Nice hand-waving of an argument, by throwing a useless Wikipedia link in there as some kind of appeal to authority.
I can do that too ... the core of the JVM (the HotSpot introduced in 1.4) is actually based on Strongtalk, a Smaltalk implementation that used optional-typing for type-safety, but not for performance:
http://strongtalk.org/ + http://en.wikipedia.org/wiki/HotSpot#History
No need to get agressive over this :) I disagreed with your first comment regarding the dynamic nature of the JVM, and replied trying to explain why.
I posted the wikipedia link not as kind of an "appeal to authority", but to give the readers a full listing of bytecode instructions, so that they can check what I was saying for themselves.
> Disregarding primitives, the JVM doesn't give a crap about what types you push/pop the stack or what values you return.
It depends how you see things: the JVM can't possibly provide instructions for every possible user type, so apart from primitives, the other object types are passed around as pointers or references, but whenever you try to do something other than storing/loading them on the stack, the type checking kicks in, ensuring that the reference being manipulated has the right types.
For instance, the putfield instruction doesn't just take the field name where the top of the stack is going to get stored. It also takes the type of the field as a parameter, to ensure that the types are compatible.
Constrast this to Python's bytecode, where the equivalent STORE_NAME (or the other variants) doesn't ask you to provide type informations.
But then again, we might be splitting hairs here: since this type checking is happening at runtime (when the JVM is running your code), you could indeed question calling it "static typing", which is usually performed at compile time (an is partially performed by the java compiler for example).
Suppose I have 5000 followers. How do you propose to get the timeline? Since you said "single copy of data", I assume you would do something like this:
https://github.com/mitsuhiko/flask/blob/master/examples/mini...
At twitter scale, that query will be a major bottleneck.
Twitter has clearly engineered a working distributed system (or you would have read stories like "5% of Twitter users lost their tweets when our servers caught on fire last night"), so the fact that they wrote it in Scala or Java is largely irrelevant. They need fewer computers than they would have if they wrote it in Ruby again, and that's not something to discount, but I think it's technically possible to have a full Ruby stack do what Twitter does. And, they would have needed even fewer computers if they used C or C++. So it's clear that Java was a social choice (it had the right books, libraries, and stack of resumes) rather than an absolute must.
Besides, Ruby and Java both make the same C library calls to do I/O. So it all ends up being the same for everything except search, trending tweets, and other cpu-intensive analysis.
Unless of course, you work in the marketing department or clean the buildings?
<!-- insert *evil grin* here -->With that in mind, I just couldn't pass up the opportunity to make an amusing comment.
I am not aware of comparable features in Ruby.
Cython is not intrinsic to Python either. I see there is a ruby2c gem, but I have no idea how it compares to Cython.
As for pypy, that's about switching to a different runtime, the same way Ruby people would go to JRuby.
None of this turns Ruby into apples and Python into oranges, I'm afraid.
1. Writing a system the second time is easier since you already know the pitfalls and avoid them.
2. Ruby, Python, Perl, Java... provide thin wrappers for the underlying IO system calls, and hence, Java IO isn't very different from Ruby IO when it comes to performance.
3. There are only a few CPU intensive tasks. It goes without saying that Java will kick Ruby's(pure Ruby; native extensions will be a different story)ass when it comes to CPU intensive tasks.
If you have a link to a Twitter-realistic benchmark I'd be interested see the results.