A/B testing: what it takes to trust the result

An A/B test compares two versions of something against a chosen metric to see which does better. Designed properly, it is one of the most effective ways to check a product assumption against data instead of against the loudest opinion in the room.

Four steps that keep a test honest

1. Write the hypothesis first: what you are changing, what you expect to happen, and why. A test without a hypothesis produces a number without a lesson.
2. Calculate the sample size before you start. Statistical significance needs enough users, and stopping early because the first numbers look good is the most common way to fool yourself.
3. Assign users to control and test groups at random, then check that the split you observe matches the split you planned. If the groups differ by device, channel or time, the result measures that difference rather than your change.
4. Look past the headline metric. Conversion can rise while average basket, returns or support contacts move the wrong way. Check the side effects before you declare a winner.

An example from a checkout flow

At Modanisa, A/B tests on the checkout flow lifted conversion by about 18%. The number matters less than the method that produced it: a written hypothesis, enough users, balanced groups, and the side effects checked before anyone called a winner.

What data-driven means

Being data-driven has little to do with testing everything. It asks you to state what you believe, pick the few beliefs worth testing, and accept the answer when it arrives.