A/B Testing

A/B testing means showing two versions of the same page to two groups of visitors at the same time. Half of them see version A and the other half see version B, and since nothing else differs, any gap between the two groups comes from that one change. Then you count which group did the thing you wanted, whether that is buying, booking or sending the form.

The idea is older than the web. A shop has always been able to try one window display for two weeks, then swap it for another and compare the takings. Online you do not wait for that second fortnight, because both versions run together, which means the weather and the season hit them the same way.

The part people get wrong is the number of visitors. A page with two hundred visits a month cannot tell you anything about a new button, because small differences only show up in a lot of traffic. As a rough guide you want a few thousand visits per version, and at least two hundred sales or sign-ups, before you believe the result. In the first two days one version will look far ahead, but that gap is almost always noise and it fades. That is why stopping the test the moment the numbers please you is the most common mistake of all.

Change one thing at a time. If you swap the headline, the photo and the button in the same test and version B wins, you still do not know which of the three did it. So change one element, keep the winner, then start the next test, which looks slower on paper and turns out much faster in practice. Run every test over whole weeks as well, because a Tuesday visitor and a Saturday visitor rarely behave the same way.

A test answers what happened, never why, so you still have to watch real people use the page. When Linkysoft rebuilds a website we usually test the order form first, since that is where the money is lost. Inside a web application the same method works just as well on a sign-up screen or a search filter. Linkysoft also warns clients about a quiet cost, because every week you spend testing is a week you are not fixing something you already know is broken.