An Introduction to A/B Testing

We've recently published a number of articles about data-driven design, or using analytics to inform our UI designs. The short of it is that we can use analytics tools like Google Analytics to conduct user research, to learn about our user demographics and user behavior, and to identify areas of our websites that might be falling short in terms of UX.

When we know where users are having trouble, and we have a little background information as to who is having trouble, we can begin to hypothesize why. We can then employ usability testing to confirm (or disprove!) our hypotheses, and through this we can potentially even see an obvious solution. The result? Happier users, more conversions!

But what happens when the solution doesn't become obvious through usability testing? What happens if there's a solution for one user group, but it doesn't quite work for another user group? You could be fixing the user experience for some users, while breaking it for another demographic.

Let me introduce you to A/B testing.

What is A/B Testing?

A/B testing, in short, is about implementing two variations of a design to see which one is better, where the primary metric to be measured is the number of conversions (this could be sales, signups, subscribers — whatever). There's a methodical technique to A/B testing, and there are also tools to help us carry out these experiments in fair and unobtrusive ways.

We can also use multivariate testing, which is a subset of A/B testing, to carry out experiments when there are multiple variables to consider.

What is Multivariate Testing?

Multivariate testing is more useful for finding out what content users enjoy more, rather than which version of an interface they prefer, but that being said, content is an important aspect of UX too. Just think about it: users don't come to your website to appreciate how intuitive your navigation is. UX is what the users need; content is what the users want (and what they came for).

Consider a heading + CTA that's the main conversion funnel on a home page. Let's say we have various headings and CTA options, and we want to test them all in a variety of different combinations at random. That's where multivariate testing might be used in preference to the standard A/B testing.

Continue reading %An Introduction to A/B Testing%

by Daniel Schwarz via SitePoint