Poor Man's Scalability(codypowell.com) |
Poor Man's Scalability(codypowell.com) |
> We were now rendering our site index in 1.5 seconds or less, compared to 3 seconds before.
This is the real WTF. What kind of queries take three seconds to run, that is ridiculous. Even with indexes 1.5 seconds is insane.
Deciding against premature optimisation isn't permission to throw away good, efficient design. It's clear from the rest of the article that these guys really have no idea what they're doing. Quoting the results of all their optimisations:
> In the end, we did not go down. The last round of load tests with httperf showed us handling 200 GETs a second indefinitely, with an average response time of about 1.2 seconds. We served thousands of requests without even a hiccup, and we did it without spending any money.
This is really poor performance, and it shows in the site when you change page or select a different category/filter. I'm guessing that they also chose a nosql solution for the wrong reasons, as it seems everything there could be done much faster/better by using an ACID compliant database.
> I didn't get too upset here; I expected us to do poorly. After all, we'd spent all of our development time in actually building something people want, not on scaling up for thousands of imaginary users.
Perhaps it would be relevant to the conversation if the author could chime in with how many projects or ideas they went through that failed. Perhaps they had to do consulting on the side for income.
Premature optimization is a mistake. It's time wasted on delivering minimal value to you or anyone else.
Claiming basic development practices to be "premature optimization" is a fantastic way to paint yourself into a corner because of stupid decisions in your haste to "get it out the door." Your MVP isn't V if it still takes a second to render your homepage to users, because they'll leave. (Site responsiveness is a huge factor in bounce rate, even for sites that people actually want to look at.)
Wholly agreed and I often say the same thing. But I do not consider basic indexing on a database to be "premature optimization". I do consider it to be a a mistake since you almost certainly find performance issues (unless the table is small enough that indexing is a mistake), then you'll have to track down the cause of the issues (commonly called "a bug") and then will have to add indexes.
Also, we didn't lack all indexes, just a couple of important ones. We've been iterating quickly on the site and we weren't analyzing the performance as our queries were refactored.
Kudos for recommending Yslow (or PageSpeed) - it's indeed brilliant. You didn't mention much about http caching, which I think is probably as important as the other things you mentioned; Not only does it improve performance for the user, but it also reduces load on your server and it enables you to put up an edge side cache for extra performance.
As for the problems mentioned with the load balancer - You could have simply provisioned a new instance and installed your own load balancer. There's dedicated packages like haproxy, but you could also just put up nginx or lighthttpd. This machine can later double as your edge cache (Squid).
Btw. 1.5s to render the front page? That's way beyond acceptable, in my book. But I suppose it depends a bit on what the web site does.
Agreed. This will have an extremely negative impact on your user engagement. Now if this is measuring end user response time rather than server-side page generation it might not be quite so bad, but generally I think most common pages should target < 150ms.
In fact:
* $0.025 per Elastic Load Balancer-hour (or partial hour)
* $0.008 per GB of data processed by an Elastic Load Balancer
Source: http://aws.amazon.com/ec2/pricing/
Obviously these are problems loads of people face / hope to face, so it would have been nice to get some more 'meat' in this post.
Edit: formatting
As a rails site hosted on Heroku, this gave us the ability to scale our site using the gambit of widgets that Heroku offers; but, we didn’t want to spend the money and cheat ourselves from the satisfaction of scaling our site.
Using New Relic (a gem available on Heroku) I identified our first performance bottleneck - database queries. Certain queries were taking over 30 seconds to complete! These database queries all involved a table of BLOBs (on the order of 20MB) that needed to be searched and sorted. I tried adding indexes but that only marginally reduced the query time. The solution we found included moving the BLOBs to their own table while keeping an index to them in a table of their attributes [1]. Doing this, we were able to reduce query times down to less than a 100ms.
The remaining bottlenecks we found using YSlow has helped us reduce the overhead in loading pages substantially.
Even though we were able to weather the storm of visitors to our site, we did leave some tasks for the next one including implementing caching, background processing, and remote file storage. All in time.
Does anyone have other wallet-friendly Rails/Heroku scaling stories share?
[1] This is database 101 stuff - a class I never took.
Yes, yes, a thousand times yes.
The best and most important way to improve database performance is to have a sensible schema. Query planners can do magic with stuff that's sensibly normalised and with things that are sensibly denormalised.
Where they choke is on schemata where everything is just squished into a bunch of tables with fields holding internal datastructures.
We took a slow site running on hardware, put it on EC2 (slower hardware obviously), and with varnish are saving about 60% cost of hosting, and decreased page load time on some of the popular pages by upwards of 50%, all for minimal engineering effort.
Edit: Less obnoxious.
As I was scanning the comments on this post, I was a bit surprised by the all the "This post was too basic. You are clearly an amateur and wasting my time" comments. Well, to those authors, not everyone on HN is an l337 hax0r such as yourselves.
Yes, the author still has some work to do. Maybe if he spent another couple weeks just on this, he could get response times down. Maybe he should use a reverse proxy or maybe he shouldn't. As with most problems of this type, the answer is, it depends.
To a young programmer with a few years experience who may be working on his/her first high traffice website, I thought the blog post was fantastically written. It was clear, concise, explained well the low-hanging fruit of optimization, explained the difference between performance and throughput, and decisions-making/tradeoffs made when preparing for a traffic spike. Well done.
Having said that, nothing I submitted ever made it close to the top of HN. Maybe my understanding of the community is off.
I actually don't know what happened here either. I was at a kid's birthday party, and I look on Twitter to see one of my cofounders had submitted this and that it was shooting up the front page.
Nice job.
And being cached won't help for first-time visitors to the site, of course.
The first is to create a vhost in your webserver that serves an index.htm page on your NON-www url. The contents of that page would simply redirect users to the www.yourdomain url.
If that seems like too much work, an even easier fix (truncated for brevity) is to do something like this:
var url = window.location.href;
if(url.indexOf("http://www") != 0) {
window.location.replace("http://www.yourdomain/");
}
RewriteEngine On
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteRule ^(.*)$ http://www.example.com/$1 [R=301,L]Even better, just use http://wwwizer.com/ (look for the free redirect service banner on top).
A simple solution, which I've used--and I think the OPs situation is similar--is to just have an additional A-record pointed to the ELB, this should work unless the OP is doing something really fancy. No need for route-53, even go-daddy will let you do this.
However, route-53 gives you much better TTL, so if you ever need to re-route..
http://googlewebmastercentral.blogspot.com/2009/02/specify-y...
If you can put all the content inside a <script/> tag at the very end of the HTML, it will at least give the browser a chance to find and download all the page's resources (css, Jquery, logo image, etc etc).
e.g.
<html>
<head><link rel="stylesheet"></head>
<body>
<h1>Home Page</h1>
<p id="content1">cheap content</p>
<p id="content2">place-holder content</p>
<p id="content3">more content</p>
~~flush~~
<script>
$("#content2").html("expensive content");
</script>
</body>
</html>And very often, you can't understand your data model and how it will be requested until you've actually built an app and gathered some data. Users will surprise you, and do things you never expected, and probably render 80% of your app (oftentimes, the 80% you worked hardest on) obsolete immediately.
First, you try and figure out if you can reduce the number of repeated queries. Then you try and figure out if you can get rid of chunks of code that spawn lots of queries altogether, by tweaking the algorithm to do things differently, without needing that data. Then, finally, once you've used every trick to make it all faster, then you apply indices to speed up that handful of remaining queries.
If you apply indices first, then you won't spot the other potential optimisations, and when your indices stop covering up for your poor coding it will be much later, and much harder to fix.
Indices are standard way to achieve basic performance levels of a database. They may have their downsides but "potentially dangerous" is dramatically overstating the case. Furthermore, the dangers of premature optimization are about taking extra time or adding complexity to something that ultimately doesn't matter, not about using very basic features in a sane way.
The correct optimizations to make first are ones that make the biggest impact percentage-wise as well as being the most elegant in the code. Indices typically fit both categories very well. Unless you are doing a lot of stupid things, there's not going to be much lower-hanging fruit, but even if there is, after you apply sane indices that will be when your profiling will start to reveal the real interesting possibilities for optimization. The idea that indices are a good final optimization does not show much interest in real performance.
You want to make the riskiest, most invasive changes first, because those are the ones that the rest of your codebase builds upon. If you've changed your query patterns and the app still isn't fast enough, it's relatively trivial to add indices on top of that. The speed benefits are cumulative, and none of the work you've done examining your data-access patterns has been undone by making that data access faster.
If you add indices first, however, and it still isn't fast enough, you have to examine your query patterns anyway. And this time, the work you did is undone by your further optimizations. The benefits of indices depend a lot on how you access your data: they slow down writes to speed up certain queries. If it turns out that the queries you're speeding up don't occur frequently anymore, your index is counterproductive.
(That said, since adding indices is often a task that takes 5 minutes of your time, you might want to give it a try and see what sort of results you get before investing a lot of effort in other stuff. If it doesn't work, you can back out the change and then start looking at your query patterns.)
Should you spend days analyzing things and creating a million indexes? No. But you should have some idea how the tables will be used and setup a few of the obvious ones, at least. Read any book on DB admin.
Just to reiterate, with a RDBMS with a decent query planner, indexing early is premature optimisation.
Neither strikes me as terribly smart, and the latter strikes me as writing it right the first time, not "avoiding premature optimizations."
If you're building an app that will hopefully be bigger than you expect, build cache into your data layer, it's not complex and will pay off sooner than later.