Conversion Experiments That Actually Teach You Something

The short version. An experiment is worth running only if it can change a decision. That takes one clear hypothesis, a sample size worked out before launch, a stopping rule you do not break and one metric that pays the bills. Most store tests fail at least one of these.

Most ideas lose. Plan for it.

When I grew sales through paid marketing, continuous testing is what improved conversion. The point of a testing program is not a string of winners. It is a string of decisions made on evidence instead of opinion.

Expect most ideas to fail. Microsoft’s experimentation team reported that only about one third of the ideas it tested improved the metrics they were designed to improve [3]. A team with that much data was wrong two times out of three. A store owner’s instinct is not better.

Conversion also sits inside your customer acquisition cost (CAC). Cost per order from paid traffic is cost per click divided by conversion rate. At an illustrative $1.00 per click, a 2.0% conversion rate means $50.00 per order. Lift conversion to 2.4% and the same clicks cost $41.67 per order. Every real conversion gain is a CAC cut you do not have to win in an ad auction.

One variable at a time, or know why not

A classic example from Kohavi and colleagues: an online retailer’s checkout page went through a redesign with nine differences. The new version lost about 90% of revenue. The critical change was a coupon code field, which made shoppers stop and wonder whether they were paying too much [1].

If you change nine things and revenue drops, you know you lost. You do not know why.

The same paper recommends single-factor tests when you want to learn from incremental changes, and bold, very different designs when you want a big move [1]. Both are valid. Mixing the two goals is not. If a test changes more than one thing, write down before launch that it is a concept test, and that a win will need follow-up tests to explain it.

Size the test before you launch it

Three inputs decide how much traffic you need:

• Baseline conversion rate: what the control converts at today.
• Minimum detectable effect (MDE): the smallest relative lift worth detecting.
• Confidence and power: usually 95% confidence (a 5% chance of a false win) and 80-90% power (the chance of catching a real effect of that size).

Kohavi and colleagues give a quick approximation for 95% confidence and 90% power. At a 5% conversion rate, detecting a 5% relative change needs just under 500,000 users across two variants. Detecting a 20% change needs about 30,400, because the requirement falls with the square of the effect size [1].

Here is what that means for an illustrative store with 4,000 visitors a day:

These rows assume a 5% baseline, 95% confidence and 90% power, using the formula in [1]. If your baseline is 2%, multiply the sample by roughly 2.6.

The lesson for a small store is blunt. You cannot detect button-colour effects. You can detect a new offer, a new landing page concept, a new way of presenting price or a different checkout. Test big things until your traffic lets you test small ones.

Write the stopping rule down before day one

Checking results every morning and stopping at the first significant result is called peeking. It pushes the false-positive rate far above the 5% you think you are running at [4]. There are two honest options:

1. Fixed horizon: decide the sample size, run to it, then read the result once.
2. Sequential testing: use a method built for continuous monitoring, which keeps error rates valid however often you look [4].

Either way, run in whole weeks. Kohavi and colleagues recommend at least a week or two, in multiples of a week, because weekday and weekend visitors behave differently [1]. For a store I would add one more rule: do not let a test straddle a major sale unless the sale is what you are testing.

Check the plumbing first

Before trusting any test, run an A/A test: two identical experiences. It should show no significant difference about 95% of the time, and users should split in the planned proportions [1].

A split that drifts from plan is called a sample ratio mismatch. Researchers who catalogued these failures warn that ignoring one can make a bad change look good [5]. The arithmetic is unforgiving. On 10,000 users, a 52/48 split where you planned 50/50 has less than a one in ten thousand chance of happening by luck. Something in assignment or tracking is broken. Fix it, then rerun.

Also reconcile the test’s order counts with your store’s system of record. If the testing tool says 1,000 orders and your store says 900, find out why before you read any lift.

Pick one metric that pays the bills

Agree on the decision metric before launch. The experimentation literature calls it the Overall Evaluation Criterion [1], [2]. For a store, conversion rate alone is a trap, because a discount or free shipping can buy conversion with margin.

An illustrative free-shipping test:

Conversion is up 15%. Contribution per visitor is down about 8%. The test “won” and the business lost. Make contribution margin per visitor the primary metric, and treat conversion rate as a diagnostic.

What a testing program is for

A single test answers a single question. A program builds a model of your customer: which objections stop them, which proof moves them, how sensitive they are to price and shipping. That model improves the next creative brief, landing page and offer before you test them.

Three habits make that work:

• A log of every test, with hypothesis, MDE, sample, result and decision. Losers included. Losers are half the learning.
• A backlog ranked by expected value: traffic affected, times plausible lift, times margin.
• Reruns of big wins, because some winners are luck. At 95% confidence, about one in 20 tests of a change that does nothing will still show a “win”.

Mistakes I see most often

• Tests too small to detect any change the store can afford to care about.
• Stopping the day the dashboard turns green.
• Calling a nine-change redesign a test of the headline.
• Declaring victory on conversion rate while margin falls.
• No log, so the same idea gets tested again next year.

What I would do this week

1. Write down baseline conversion and daily visitors, and compute the smallest lift you can detect in two weeks.
2. Drop any planned test that cannot reach its sample in four weeks.
3. Make contribution margin per visitor the primary metric.
4. Run an A/A test and check the split and the order counts against your store.
5. Start the experiment log today, including last quarter’s tests.

References

1. R. Kohavi, R. M. Henne, and D. Sommerfield, “Practical guide to controlled experiments on the web: Listen to your customers not to the HiPPO,” in Proc. 13th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD ’07), San Jose, CA, USA, 2007, pp. 959-967.
2. R. Kohavi, D. Tang, and Y. Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge, U.K.: Cambridge University Press, 2020.
3. R. Kohavi, A. Deng, B. Frasca, R. Longbotham, T. Walker, and Y. Xu, “Trustworthy online controlled experiments: Five puzzling outcomes explained,” in Proc. 18th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD ’12), Beijing, China, 2012.
4. R. Johari, P. Koomen, L. Pekelis, and D. Walsh, “Peeking at A/B tests: Why it matters, and what to do about it,” in Proc. 23rd ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD ’17), Halifax, NS, Canada, 2017, pp. 1517-1525.
5. A. Fabijan et al., “Diagnosing sample ratio mismatch in online controlled experiments: A taxonomy and rules of thumb for practitioners,” in Proc. 25th ACM SIGKDD Int. Conf. Knowledge Discovery and Data Mining (KDD ’19), Anchorage, AK, USA, 2019.