How to measure the cost of an A/B test?

4 points by mistercheese 13 years ago | 4 comments

At my past few companies, there has been heavy investment into live A/B tests on the customer population. However, for bigger companies that have more exposure, I have a worry that we may not be able to have the proper ways to measure the real cost of an A/B test that may 1) Have negative PR or branding effects 2) Conflict with the objectives of other features.<p>Sure we can measure customer drop off or engagement, but given the number of tests and projects at any given time, there are too many confounding factors to isolate those measures to a particular test. I'd love to hear from Hackernews suggestions! Thanks!

darkxanthos 13 years ago |

For #1 if customers are assigned to groups that get a certain set of variants for each test and you have enough groups to run the full combination of tests that are active, you should be able to correlate drop off with certain tests.

For #2: That's much harder. How do you handle this for larger products that you don't A/B test? I'd bet this isn't a unique issue.

mistercheese 13 years ago | |

For #1, the issue is actually that even the awareness of Test A might effect Test B. I'm curious if there's anyway to capture that then.

karolisd 13 years ago |

How do you measure PR and branding effects?

If you aren't controlling for variables, it isn't very scientific. It's possible to run multiple tests at the same time and test different things, but you need to be able to isolate what you're measuring and be able to measure something that's statistically significant.

mistercheese 13 years ago | |

Pretty much every test we run has the potential to effect user engagement, so I would suspect unless isolated it's a lost cause. What's more interesting to me is how to measure perhaps how bad branding has lost consumer confidence and the effect on sales thereafter.

It's easy to measure drop off in a funnel, but harder to measure customers who refuse to even enter the funnel due to bad brand. Is there some way of capturing that?