How to Combine ‘Gut Feel’ & Science in Multivariate Testing

As site owners, designers, and developers we often underestimate the value of our intuition.

We prioritize data-driven decisions over instinct when it comes to virtually everything: from optimizing our monetization strategies to styling buttons on our websites. However, we often achieve better results if we learn to complement, rather than substitute our gut feeling with tests and data.

The ability to combine intuition and data-driven thinking is crucial for multivariate testing (MVT). Although site testing is viewed as a “mechanical” process, we often come up with purely intuitive ideas when choosing the features and design elements for further tests.

A typical MVT procedure includes the core stages of the scientific method: question, research, hypothesis, experiment and troubleshooting, data analysis, and implementation. You can increase the value of your ideas by using these stages as a framework for your experiment.

Running Usability Tests to Ask the Right Questions

Start your MVT experiment with the right question. In most cases, asking yourself what exactly you can improve is a good starting point.

As a rule, our intuition makes us visualize dozens of useful tweaks, but we need to concentrate on features which lead to substantial improvements. Running a usability test prior to an MVT experiment is a viable way of reinforcing intuition with concrete data. Depending on your budget you have several options to choose from:

• Hiring experts to perform heuristic evaluation of your website;
• Joining a crowdsourcing testing platform like Peek, FiveSecondTest, or Concept Feedback to have your site reviewed by end-users;
• Getting creative with non-standard UX testing services such as “The User is Drunk” or “The User is My Mom” from Richard Littauer.

Let’s take a look at an example:

When launching native advertising on their websites, publishers often choose full-sized ad units that mimic the structure of their posts. This way the ad gets more prominence.

However, sometimes the ad feels too large, and it is more than just an aesthetic issue. Oversized adverts of this type can erode user experience leading to a decrease of click-through rates (CTR).

Conducting a usability study will most probably confirm our guess — oversized in-feed ads tend to be the first UX-related problem users notice. An obvious solution for this problem would be to test smaller native ad units.

Using Competitor Analysis for Background Research

Conduct background research to ensure a meaningful outcome for your tests. There is a valid reason for you to use competitor analysis at this stage. This type of research often comes down to a brief review of competitor websites with top search rankings, which doesn't take much time.

Studying your competitors will help you better understand what UX solutions work for similar websites. Later on, you will determine if your own ideas work better than these solutions.

If this is the case, you will gain a competitive edge over similar online resources. If not, you will still get to understand what works best for similar websites.

Competitor analysis will add clarity to the example above as well. It's easy to see that the most widespread alternative for an in-feed ad is a compact preview of the same branded post. These previews typically appear alongside recommended content, and are styled similarly, except for minor differences. This is what an average native ad of this type would look like in the “recent articles” section:

In a number of cases, publishers mention the advertiser in an explicit manner:

Constructing a Valid Hypothesis

The hypothesis is the idea you will be testing. In fact, it is likely that your gut feeling will provide you with several worthy ideas.

In this case, you will need to find out which of them have the most practical advantages over your competitors’ features. To select the right items, give proper consideration to these three points:

• Try to predict how your audience will react to your solution. There is plenty of information on the psychology behind different color patterns, shapes, and layouts to help you make precise forecasts.
• Find out if you ideas correspond to the current trends and/or enable you to make use of new technologies.
• Make sure the options you test are suitable for your site in terms of design.

Let’s see, how these points align with the above example. Background color helps the ad stand out, but it may lead some users to mistake the branded post for a banner. To avoid confusion, it will prove worthy to either opt for a less contrasting background or to completely remove it:

Continue reading %How to Combine ‘Gut Feel’ & Science in Multivariate Testing%

by Anton Ruin via SitePoint