Yuchao's blogspot: A/B test

Saturday, November 12, 2016

A/B test

A/B tests means that you can get data to make decisions rather than relying on intution or highest paying person’s opinion.

It’s not about how to implement an A/B test framework, it’s about how to design a task, choose metrics, and analyze the results.

John Lilly, CEO of Mozilla,

A/B testing is really useful for helping you climb to the peak of your burrent mountain, but isn’t so useful on deciding which mountain you want to be on.

new features
addition
different look

When Amazon first started doing personalized recommendation, they discovered people bought more stuff and actually had a significant increase in revenue.

For every 100 ms latency added to the page, they actually had a 1% decrease in revenue.

limitation

change aversion vs novelty effect
A/B testing can’t really tell you if you’re missing something.
can’t have immediate feedback for long-term, big decision

A/B test in other area:

agriculture
medicine: clinical trial

The key thing is you have a consistent response from your control and experiment group

customer funnel

users are trickling down the funnel:

page visits -> explore the site -> create account -> complete

binomial distribution

binomial distribution: In a sequence of n independent yes/no experiments, each has a success probability of p. Each experiment is also called Bernoulli experiment.

The probability function for k successes is $Pr(k)=\frac{n!}{k!(n-k)!}p^{k}(1-p)^{n-k}$

standard error $SE=\sqrt{\frac{p(1-p)}{n}}$

margin of error m=Z*SE

Z distribution is normal distribution with mean=0, sd=1: $\phi (x)={\frac {e^{-{\frac {1}{ 2}}x^{2}}}{\sqrt {2\pi }}}$

for a confidence interval of 95% in a 2-tailed test, each tail contains 2.5% distribution, corresponding to a z score of 1.96. So 1000 page visit with 100 clicks m=math.sqrt(0.10.9/1000)1.96=0.019, will have a confidence interval of 0.1-0.019~ 0.1+0.019

hypothesis testing

Null hypothesis : no difference
Alternative hypothesis

Yuchao's blogspot