Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement(fourhourworkweek.com) |
Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement(fourhourworkweek.com) |
To be honest you don't have to use Google Website Optimizer to notice that.
Example: I'm releasing my Rails A/B testing framework this Sunday, with a case study writeup taken from my site. My site currently allows people to defer signup for the trial by means of a guest login. That sort of option is popular here. I'm told it increases usability and user engagement, right?
Without ruining the surprise: that is a testable hypothesis. ;)
I stopped at the first misuse of statistics, but by that point, they had taken out informative links from their web site and turned it into a landing page. Frankly, a 5% change in conversion rate is in the noise compared to the effect of targeted advertising (and I have no faith that these sloppy reporters actually did a properly controlled experiment).
Edit: Downmodded for asking for real math over marketing math?
The old conversion rate was 24.4%. The new one was 29.6%.
(29.6 - 24.4) / 24.4 = .213 = 21.3%.
Is it misleading because their number of conversions is so low? (Not trying to be snarky, I have no idea)
Reporting "a percent of a percent" is downright misleading because it is entirely dependent on the starting baseline and also because conversion rates have high natural variance. Imagine an improvement from 2% conversion (for a truly awful site) to 2.6% (for a site just as awful). That "30% improvement" just isn't. If they were intellectually honest and called it a "0.6% improvement," anyone would be able to see that the claimed improvement is well within stddev.
You appear to know words associated with statistics, but treat them as if they were magic incantations utterly disassociated from actual math.
I'll bite: what is the standard deviation in that example. I'm looking for a two part answer: a) a number, b) what the number is measuring.
Let me be intellectually honest with you: there is no basis for assuming that an improvement from 2.0% conversion to 2.6% conversion is not statistically significant.
You need sample sizes to even attempt to do the math. For example, try doing a chi-squared test on 100,000 people converting at 2% versus 5,000 converting at 2.6%.
I'm thinking you'll find that you reject the "just as awful" hypothesis with over 99% certainty.
First, you're asking the wrong person to supply the missing data. The onus is on the OP to justify their stats, not on the person calling BS on the misuse of statistics.
Given that the original article is missing the critical numbers and presenting the rest in the most self-serving manner possible, the only data anyone has to go by is their own experience with conversion rates. We don't even know what "conversion" means for this website because they don't define it, but that justifies making reasonable assumptions (e.g. users who land on page A end up on page B, for a website in some industry, involving some side-effects to get from A to B that are left undefined by the OP). In my experience, the behavior of users on a web site is quite fickle and dependent on the surrounding advertising campaigns, as well as the weather, the time of day, and the timing of holidays. And maybe a billion other confounding factors. So, is the 5% improvement seen here significant? We don't know, there's no evidence to assume it is, and that's precisely my point.
The number that matters is "how many more users are signing up since we changed"? The answer is NOT 5.2%.
Example: I have a 1% conversion rate on 100,000 visitors. 1,000 customers! Yay! I change something and I get it up to 2% on the next 100,000 visitors. 2,000 customers.
Which is a better description: "I doubled my conversion rate" or "I increased my conversion rate by 1%?" I'd go with the former.
But the other reason why it's a bad idea to quote percentages of percentages is that there is no indication whether the claimed improvements are actually statistically significant. I didn't see any data in the article to indicate they are, and what little I know about conversions from my own observations is that they vary by at least 10% depending on totally random factors.
>Which is a better description: "I doubled my conversion rate" or "I increased my conversion rate by 1%?" I'd go with the former.
I see your point and I'd use it if I needed to flatter myself. But I'm sticking to the latter. And I'd be sure to mention the starting baseline as well.
Help yourself. Just don't use that type of language in any conversation with anyone else on the planet because you'll likely confuse the hell out of them (take note of the "wtf is this guy talking about" downvotes you've received on this thread)
It's not about flattery. It's about metrics that matter and revenue. Doubling your conversion rate pretty much doubles your revenue. That aside, the goal here is communication. I'd wager that if you said that you doubled your conversion rate, people would grok what it meant (1% to 2% or 5% to 10%-- either way, a big win because it'd double revenue and profit down the funnel). If you said you increased your conversion rate by 10%, I'd wager people would assume that meant something like 10% to 11%.
The percentage change in users is the interesting number. If someone said "My conversions went up by 100 per week!" I would say "What percentage increase is that?" If someone said "My conversions went up five percentage points!" I would say "What percentage increase was that?" If they said "My conversions went up 20%," I would know what I wanted to know.
It wouldn't be an increase of 1%. It would be an increase of one percentage point OR an increase of 100%.
This entire thread could be resolved by acknowledging the differences between percent and percentage points.
I think it's just a matter of using confusing labels to refer to the percentages/improvements.
In other words, you don't really care about the second derivative (the rate at which the conversion rate) is improving as much as you care about the initial rate.
I don't rely on opinion-polling to determine who is right, esp. when it comes to questions related to science.
>conversation with anyone else on the planet
On your planet, a move from 0.0001% to 0.001% would be reported as "an improvement of 1000%" No thanks.
> It's about metrics that matter and revenue.
Yes, we established long ago that it's about marketing-speak versus math.
So, since you have so many upmods, perhaps you can tell me what happens in your world when the conversion rate goes from 0% to 1%? :-)
I expect hackers to use metrics that are at least defined over the range they are supposed to be used. Imagine that the conversion rate goes from 0% to 1%. I report 1% improvement, the value I measured. You? (1-0)/0 = what percent?
It's a one percentage point improvement. Wrong terminology is responsible for 95% of this thread.
Look, it's pretty clear that hackers have left the building and let marketroids rush in. Feel free to use your ill-defined metric. If you have the least bit of intellectual curiosity, you'd let everyone know what you'd report when the value goes from 0 to a non-zero value. That is all.