Comparing Amazon EC2 to Joyent SmartOS(mattconnolly.wordpress.com) |
Comparing Amazon EC2 to Joyent SmartOS(mattconnolly.wordpress.com) |
"Amazon need to lift their game in terms of CPU performance. They offer a great service that obviously extends far beyond a simple CPU benchmark. But when you can get the same work done in Joyent significantly faster for the comparable price, you’ll get far more mileage per instance, which is ultimately going to save the dollars."
Or, "Wow, Amazon sure makes bank on this EC2 stuff, they are selling kilo-core-seconds for a bunch more than Joyent is and they are still the most widely used provider."
There have been a couple of good benchmarks around which instances get you the most 'bang' for the buck, but its pretty clear that Amazon is making a ton of money here.
A good comparison is a rack of 40 SuperMicro 1U dual socket servers with 192G of ram in them. You can create 480 dedicated instances with 16 GB of RAM each, and run the whole thing in a colo facility for $5K/month (power, networking, and floor space). That is 0.2 cents per second, or a bit more than $10/month running them 100% of the time.
Not a bad business to be in.
Other than Amazon, I can only think of Google being in a position in which such an enterprise can take off.
We tried this at Loudcloud, and it turns out you need more than a bunch of smart people, lots of money, and single-minded focus towards building on-demand infrastructure.
I don't care if SmartOS is 'faster' or has dtrace. The fact that it is missing basic things like top (no prstat isn't anything close to top) and 'standard' things like upstart made me want to run for the hills. The Joyent wiki documentation on SmartOS is abysmal at best. Even the pkgsrc repository system feels ancient and pales in comparison to what apt is. I have no desire to spend time re-inventing basic tools that linux has had for ages.
The image version of SmartOS w/ NodeJS isn't up-to-date with the latest Node release and it starts up MongoDB by default. While this may be good for just getting things running quickly, what happens if I don't want Mongo running at all? Generate my own images? No thanks.
I know, lots of hate... but it isn't all bad. Joyent as a cloud platform is great. Basic ab testing against NodeJS had me serving 9900 concurrent connections without any issues on their least expensive vm (which runs on a blazing fast network at 10gb/s). We were on Rackspace before and with a network limited to 20Mbps/s (at the lowest tier) we just couldn't get the throughput we needed.
tl;dr: SmartOS is a train wreck. Don't waste your time. Joyent as a platform is great. Why bother with EC2 when Joyent is faster and less expensive?
More generally, we're definitely committed to making SmartOS the best environment for the modern cloud; I'm sorry that your experience was frustrating, and if you have any additional concrete feedback on what you'd like to see improved, we're all ears. And finally, I know you said that you "don't care" about DTrace, but I just can't help my (biased) self; if you're developing a node.js app, you really should check out some of the things that SmartOS can uniquely do with respect to node:
http://dtrace.org/blogs/dap/2012/04/25/profiling-node-js/
http://dtrace.org/blogs/bmc/2012/05/05/debugging-node-js-mem...
Sure, I didn't spend a lot of time with it, why should I? I saw that it was missing a great deal of basic things out of the box. I decided quickly to not waste my time learning more esoteric stuff that is only useful if I stick with Joyent and SmartOS.
You are an absolute genius for creating dtrace and it is nice to have but certainly isn't a necessity for most applications. Your example is voxer which is a total edge case since they've been pushing the node envelope since day one.
In the end, the reality is that SmartOS doesn't feel modern at all, it feels like Windows98. Actually, it feels like SunOS back when I had pizza boxes under my desk 15 years ago. I could spend time giving you tons of (free) feedback about what's wrong with SmartOS or I could just use Ubuntu and move on.
9900+ concurrent connections on a $22/mo Ubuntu box is 'good enough' for me.
Joyent has to make the experience better by streamlining documentation, not by compromising its core values that make SmartOS great. The quality of the Solaris documentation is great, it's just not organized how Linux people expect it to be and they don't know where to find it.
The bigger issue is Joyent's product lifespans. We've had to migrate our servers twice in that three or so years. Because of that, I moved to a different cloud provider. I don't know if I can expect better.
The only outage I can recall was recently when they began the process of EOLing the current product we use from them.
As for your bash script... I stopped compiling binaries ages ago, why would I want to start again when I can just use Ubuntu and the official NodeJS PPA?
Wait there buddy.... First of all, why would OS choice be of so much significance here, after all this is all executed mainly on the CPU (not much IO). The only difference would be that the random() function might return different things on different systems (libc difference?)... or maybe not (I'm not familiar with ruby's internals).
So I'm not sure what this measures at all...
At least you can run the benchmark shootout (CPU), or something that can test the I/O.
- For the best price, you'd want to use a reserved instance if this is a webserver, and perhaps a spot instance if it's a batch/HPC task. You'd want to use high-CPU instances if you really are all about quickly sorting 5M numbers. If your app is or might be come database-bound, DB perf's super key.
- I'd love to hear more broadly about the practical differences between the clouds--whether anyone seems to be better at managing noisy neighbors than the next guy, network and I/O performance, etc. One CPU-bound task can't really be the end of it.
Small: 6.5 cents/ecu-hr
Small, 1yr: 4.7 cents/ecu-hr
Small, spot: 1 cent/ecu-hr
High-CPU: 3.3 cents/ecu-hr
High-CPU, 1yr: 1.9 cents/ecu-hr
High-CPU, spot: 0.56 cents/ecu-hr
If RAM matters, high-CPU (7G, or 3.5/core) > Small (1.7G) > EC2 Micro and Joyent extra-small (~0.5G). Again, don't rely too much on figures like these; look at all your needs (not just performance, everything) and try to figure out what'll work for you.
cc2.8xlarge: 0.15 cents/ecu-hr
cc2.8xlarge 1yr: 0.058 cents/ecu-hr
Apart from being forced by VC ( because they are heavily invested in Amazon ) i could not understand why startup are choosing AWS at all.
S3 is very reliable, and I'm not aware of anything as reliable (we're talking eleven nines)/simple/cheap/distributed.
EC2 gives you the ability to manage instances in several continents with a dead simple API, which is unheard of anywhere else at this price.
ELB is very reliable, simple to use, and saves you hours of setup.
My point is that if you really wonder why anyone would want to use AWS, you might not be considering AWS, but only EC2, and in a very non-international way.
Cloud computing is a commodity. Many (most?) customers use multiple clouds in parallel. Learning and maintaining the intricacies of a niche OS in this environment is a nonsensical proposition.
And for what? For a few percentage points on a CPU benchmark? Seriously?
To me, the weakest part of AWS is always on the technical support - you have no right to create a ticket to them unless you pay for the support fee (as contrast to Joyent /Rackspace/Linode), even you have discovered a bug in their system!
On the other hand, I also agree with @latchkey that SmartOS /Ubuntu is an important factor when we are choosing the platform - for a lot of people, ease of use and larger community is more important than cutting edge features like dtrace and zfs.
Joyent love micro benchmarks because they allow one customer to grab a lot of CPU in burst, but if you are one of the other customers on the same hardware node, it is better to have a bit slower all the time than high variability like Joyent. Predictability when you run your infrastructure is better.
Also, Joyent will kick you out of their infrastructure if they consider at their sole discretion that they are not making enough money out of you, but this is my personal experience and may not be yours. So basically, they broke the trust I placed in them, I am migrating out at the moment, so take my comment as coming from someone with a bad experience with them.
When you got right down to it, what killed that plan was that you were still using Amazon S3 for storage; the money that we would have saved in CPU/processor time we would be eaten alive with transferring that data back to S3.
It really depends on which cluster you were allocated and what other applications were running at that time. If someone else was running a heavy process at the same time, then that could explain some of the sluggishness..
Even still, the numbers are so drastically different that this question does deserve a second look. Someone from Joyent or EC2 should really put together a solid benchmark to prove once and for that their service is indeed better in a wide range of compute and networking categories.
FWIW, here are my benchmarks, along with pricing for Northern California regions:
Linode 512: 15.2 seconds, 65 cents per day
Storm-on-demand 1 GB: 13.05 seconds, 5 cents per hour
EC2-micro: 168 seconds, 2.5 cents per hour
EC2-small: 63 seconds, 9 cents per hour
EC2-medium (high CPU): 12.13 seconds, 18.6 cents per hour
Linode Cost per day: $0.65 Time per image: 15.2 Images per day: 5684.2105263158 Cost per image: $0.0001143519 Images per dollar : 8744.9392712551
Storm-on-demand Cost per day: $1.2 Time per image: 13.05 Images per day: 6620.6896551724 Cost per image: $0.00018125 Images per dollar : 5517.2413793104
EC2-micro Cost per day: $0.84 Time per image: 168 Images per day: 514.2857142857 Cost per image: 0.0016333333 Images per dollar : 612.2448979592
EC2-small Cost per day: 2.16 Time per image: 63 Images per day: 1371.4285714286 Cost per image: 0.001575 Images per dollar : 634.9206349206
EC2-medium Cost per day: $4.464 Time per image: 12.13 Images per day: 7122.8359439407 Cost per image: 0.0006267167 Images per dollar : 1595.6173709545
The second fault I have is that articles like this, is that they only every look at the OnDemand price, which most people using AWS seriously don't pay. AWS has reserved pricing and spot both of which will dramatically drop the pricing. Maybe not as inexpensively as Joyent, but much much closer.
Does anyone know a good cloud-based Windows host as an alternative to AWS?
You can have either your own domain name or SSL w/ their subdomain.
Who the bleep would settle for that? What kind of "enterprise solution" is this?
It's dead-simple to get started.
It's widely known. I can convince my CTO to use Amazon for cloud-hosting. He's never heard of Rackspace or Joylent - people who are serious about running their infrastructure on the cloud will research and know about them but for quick experimenting and companies just getting their feet wet, Amazon has a big edge here.
The PaaS offer per se is called Hosted Services and it does support SSL on custom domains.[1]
The price for the IaaS offer for a small Windows VM is $0.08/h ($57.60/month).[2] EC2 pricing for a small Windows VM is $0.115/h ($82.80/month).[3]
[1] https://www.windowsazure.com/en-us/develop/net/common-tasks/...
[2] https://www.windowsazure.com/pt-br/pricing/calculator/?scena...
1) Funny that scaling up to one of their largest instance types can improve bang-for-buck, on this metric at least. 2) I wish Amazon made it simpler to sort all this out.
Another downside of cc2.8xl is that the spot market is crazy, with spikes at $6/h or more.
They have a very decent "private cloud" offer. The feedback is good but I am a bare metal + Ganeti[1] to manage my VMs kind of person.
If you like pure instances, I have also instance running on Gandi[2] infrastructure. My blog made several times the front page of HN and without special caching it handled extremely well the load (in fact, you could barely see the increase in CPU usage in the monitoring graph while seeing a huge increase of the outbound bandwidth).
Another good provider is Hetzner[3]. You need to pay attention to the server you select (ECC memory or not) but then they are reliable (I had a server with them for 4 years or so without a single issue).
In Europe, AWS is in Irland, I suppose it was for tax purpose that they selected this location, but from a latency point of view, this is stupid if you want to deliver packets in the South or East of Europe (and even part of Scandinavia because most of the time your route does IE, NL or FR, and then goes North).
Sorry for the long post... but at the end of the day, do your calculations. AWS is great if like Netflix you scale from 20% to 100% during the day[4] (the hit on the cache gives you how they scale) but if you are a normal shop without day/night activity because you are at global level or so well optimized that you just not need to scale (a single dedicated server with SSD is incredibly powerful nowadays) you simply do not need "the cloud". Just test with 1k€, do a bit of maths, then select the best choice.
[1]: http://notes.ceondo.com/ganeti/ [2]: http://www.gandi.net/ [3]: http://www.hetzner.de/ [4]: http://techblog.netflix.com/2012/01/ephemeral-volatile-cachi...
If curious you may have a look at http://www.youtube.com/watch?v=UBHSz3fOP-4&hd=1 where we mix an eucalyptus private cloud on physical server with AWS.
A few days ago there was an article about Linux monoculture. It's even worse than that, it's an Ubuntu monoculture.
If you really believe there is more information in SmartOS' wiki than in man pages & other proper documentation available about Linux, I don't think the problem is with the Linux monoculture, but somewhere else.