Mul­ti­vari­ate testing: the complex process for op­ti­miz­ing the con­ver­sion rate

An optimized con­ver­sion rate is the goal of every developer or web project operator. First-class content and products, as well as con­stant­ly in­creas­ing traffic, are usually promising signs for upcoming stores and companies. Good as these might sound, they are useless if they don’t result in trans­ac­tions, clicks, or ac­qui­si­tions from completed forms. While the number of visitors can be in­flu­enced by quite different measures (social media marketing, SEO, SEA, link building, etc.), the amount of con­ver­sions is mainly due to two factors: how much trust the user has in the store or website, and the web project’s user-friend­li­ness, which is also referred to as usability. In order to con­sis­tent­ly improve the latter, mul­ti­vari­ate testing is becoming in­creas­ing­ly popular among de­vel­op­ers and marketers. The advanced testing method, which is even more complex than the often-performed A/B testing, can detect weak­ness­es among the elements being checked. These can then be rectified to make it easier for the user to access the content. If visitors enjoy their ex­pe­ri­ence they will spend more time in the store or on the website and are more likely to carry out the desired action (con­ver­sion).

The following example il­lus­trates the approach to mul­ti­vari­ate testing: the web page of a certain product should be optimized since the de­scrip­tion and image used were selected as decisive test criteria. The analysis procedure is intended to determine whether and to what extent the two different product de­scrip­tions A and B (as well as the various product images 1 and 2) influence the con­ver­sion rate. For this purpose, the images and de­scrip­tions are combined to form the following com­bi­na­tions:

• De­scrip­tion A, Image 1
• De­scrip­tion A, Image 2
• De­scrip­tion B, Image 1
• De­scrip­tion B, Image 2

This simple example shows that there is a maximum of four different com­bi­na­tions – the number of possible com­bi­na­tions rises with each ad­di­tion­al variable. The four different versions are now presented to potential customers who access the store’s website. The entire traffic is dis­trib­uted evenly among the four variants. The con­ver­sions (completed trans­ac­tions) are analyzed during the review period using an analysis tool such as Webtrends Optimize. This is so that a con­ver­sion rate can be cal­cu­lat­ed for all four variants after the mul­ti­vari­ate test is completed. The bigger the amount of traffic and the longer the ob­ser­va­tion period, the more sig­nif­i­cant the results.

Compared to usability tests (which are un­der­tak­en in the project’s de­vel­op­ment phase), mul­ti­vari­ate tests have the advantage that a much larger number of par­tic­i­pants can be included seeing as these tests are run once the website, app, or store is online. These are known as quan­ti­ta­tive research methods. Although mul­ti­vari­ate testing seems extremely complex, it is actually quite easy when the right tools are used. The various test pages are quickly con­fig­ured and im­ple­ment­ed via JavaScript code snippet in the web project. The results are shown in real-time in clearly arranged tables so that it’s easy to see which com­bi­na­tions promise the greatest success.

Unlike A/B testing, mul­ti­vari­ate testing isn’t limited to two versions; there’s actually no lim­i­ta­tion. This fa­cil­i­tates the ver­i­fi­ca­tion of several different elements and also provides the marketer with com­pre­hen­sive insights into how various com­po­nents interact with each other. Mul­ti­vari­ate tests don’t just reveal which com­bi­na­tions influence the con­ver­sion rate in a positive or negative way when it comes to in­di­vid­ual cases; they also provide a concrete, sta­tis­ti­cal­ly proven picture of which com­po­nents con­tribute to the overall success of a web project and in which ways. The knowledge gained during mul­ti­vari­ate testing can also play a major role in the de­vel­op­ment of future projects.

High traffic is mandatory for a mul­ti­vari­ate test to obtain results that are as accurate as possible since the traffic is split and dis­trib­uted to at least four test examples. An ad­di­tion­al problem with this testing method is when one or more of the tested variables have no influence on the con­ver­sion goal – es­pe­cial­ly when it comes to in­ter­pret­ing the results. Here, mul­ti­vari­ate testing can quickly end up being the wrong choice because it ends up com­pli­cat­ing the eval­u­a­tion process un­nec­es­sar­i­ly. In this case, a simple A/B test might have sufficed.

The best way to test the usability of your web project depends on a number of different factors, with the amount of traffic playing the biggest part. If your web project is only recently up and running and you haven’t had many visitors as of yet, mul­ti­vari­ate testing won’t provide you with reliable results on the success of each variant. In such a case, it is rec­om­mend­ed to test the func­tion­al­i­ty of in­di­vid­ual variables in con­sec­u­tive A/B tests. If on the other hand, your web presence has high traffic, mul­ti­vari­ate tests are the better option since you save time and effort.

A pre-requisite for both methods is that you have to have for­mu­lat­ed clear hy­pothe­ses and result quan­ti­ties for the tested elements. Otherwise, the test results will be difficult to interpret. In contrast to the usability analysis during the de­vel­op­ment process, you must also expect the con­ver­sion rate to de­te­ri­o­rate tem­porar­i­ly during this live test. So when it comes to testing out new ideas or variants, mul­ti­vari­ate tests are neither efficient nor adequate, and A/B testing is often too limited. Pre­lim­i­nary ex­per­i­ments on a smaller scale and with clearly for­mu­lat­ed questions are a much more effective and less risky solution.

If your web project has enough traffic and you decide to perform mul­ti­vari­ate testing, you shouldn’t see this as a carte blanche to create as many variables or versions as possible. For an optimal result, it is advisable to proceed as strate­gi­cal­ly as possible and only have the main opponents compete against each other by pre-selecting which ones these are. In order to play it safe, you can even check the test result with a sub­se­quent A/B test. However, you should be aware that using one of these testing methods doesn’t guarantee an increase in the con­ver­sion rate, but merely gives you ideas for what could increase it.