On the pitfalls of A/B testing(stavros.io) |
On the pitfalls of A/B testing(stavros.io) |
Calculate it here: http://elem.com/~btilly/effective-ab-testing/g-test-calculat...
Read more here: http://en.wikipedia.org/wiki/G-test
And plain English here: http://en.wikipedia.org/wiki/Likelihood_ratio_test
When A/B testing, you need to always remember three things:
The smaller your change is, the more data you need to be sure
that the conclusion you have reached is statistically significant.
Simple case: think about trying to decide if as normal distribution has mean 0 or mean 1. If the std dev is 0.001, it won't take you very many samples to be fairly confident to this resolution, but of the deviation is 1000, you'll need a lot of samples.
Similarly if the std deviation is only 1, but you are trying to decide if the mean is 0 or 0.001, Far more samples needed.
The intuition generalizes quite well. In the OP case, typically requires sample size estimates will be proportional to the square of the ratio between the size you want to measure and a deviation estimate.
If X is close to Y, we need a very large sample to achieve statistical significance on the hypothesis that they're not equal, whereas if X and Y are far apart, it is likely that a small sample will already indicate that fact.
The correct way to estimate your sample size is to use the cumulative distribution function of your underlying distribution. See a brief explanation from Wikipedia here: http://en.wikipedia.org/wiki/Sample_size_determination#By_cu...
Now what's the problem with A/B testing? Most of the stuff we test A/B for is incredibly non-normal. Often 99% of visits do not convert. We're looking at extremely skewed data here. Generally the more skewed the distribution, the more samples we need.
For a very basic understanding of why: consider a very simple distribution with 99.99% of the time you get $0 and 0.01% of the time you get $29 - fairly similar to what we A/B test. Do you think a sample of 1000 or 10000 is going to be anywhere near enough here? Of course not.
Our A/B testing data results from a Bernoulli experiment, and thus is binomially distributed. So indeed, if we use tests that assume a normal distribution, if we want to approximate the binomial distribution when p is 0.0001, n needs to be roughly 50k.