Ask HN: What is your one tip about A/B testing you would share?

4 points by raycloyd 11 years ago | 9 comments

Wondering what everyone would tell someone new to a/b testing for website and conversion rate optimization. Could be anything from expectations to approach, etc. Thanks!

Someone1234 11 years ago |

That most A/B tests don't prove what they aim to prove. People just add a new version of the site, then wait until 100 people have seen it, measure a small improvement and migrate to the new version.

However if you then run the test a second time, but make the "old" site the alternative, you might find it too sees a small measure of improvement and thus you could bounce back and forth between A and B until the end of time.

I am not a statistician so I won't try to give you advice on what a statistically significant result is. However many /many/ articles have been written on that topic, and many products have been designed which you can slot your data into to see if the 'B' option is legitimately better.

onion2k 11 years ago |

Learn what "statistical power" (http://en.wikipedia.org/wiki/Statistical_power) means, and then understand that the fact n% of users prefer one option doesn't necessarily make it a good idea. Statistical significance is far more important than the actual result. This is a really good paper about it: http://www.qubitproducts.com/sites/default/files/pdf/most_wi...

dairgram 11 years ago |

Before you get too far, try an A/A test. By this, I mean let the A and B choices be identical. You would certainly expect the outcomes to be equal. Right?

I have seen statistically significant differences in outcomes in A/A testing.

A/B testing has value but being sure to A/A test may temper your expectations and/or point at problems in your setup before you get too far.

domrdy 11 years ago |

http://nginx.org/en/docs/http/ngx_http_upstream_module.html#... - to make clients 'stick' to upstreams, if you're using nginx.

mtmail 11 years ago |

If you radically change a feature then regular users will first start playing with it. Simply because it's new. In that case your test needs be longer or exclude regular users.

seekingcharlie 11 years ago |

Ensure that your testing things that are actually going to impact your conversion rate or goals. You don't have to test everything - just the things that matter.

siddharthdeswal 11 years ago |

Don't A/B test your credibility. Not everything should be (or can be) subjected to hypothesis testing.

hkiely 11 years ago |

Choose and test one independent variable at a time. Then, check your results for statistical significance.

catman01 11 years ago |

Kittens will always improve conversion rates.