Conversion Rate Optimization 101 – Part II: Multivariate Testing

In Internet marketing, a/b or split testing is a process of measuring the performance of a few different versions of a web page to determine which one has the highest conversion rate. Split testing is very useful for determining which page layout performs the best, but it is very limited in its ability to find a version of the page that fully maximizes a conversion rate potential.

Multivariate testing, on the other hand, can be viewed as a process of a deep page testing where you are measuring the impact of simultaneous changes of several page components on the overall page conversion rate. The number of possible page combinations that can be created by changing multiple page components can quickly escalate into the thousands and sometimes even into the millions. Testing such a large number of page combinations is a science that requires automation and effective multivariate testing technology.

In the not so distant past, multivariate testing solutions were available exclusively to those who had deep pockets, technical resources, and high volume of web traffic. In last year or two the situation has dramatically changed making it possible for companies of any size to use multivariate testing.

Multivariate testing technology is still evolving and it is critical to recognize that not all multivariate testing methods are the same. On a high level one can identify three main product categories:

Although the multivariate testing methodology is a very critical element of any multivariate testing solution, it is not the only criteria that you should use in making a tool selection. The table below summarizes other product features that can impact effectiveness of your test in significant way:

The following are the practical steps that you should follow during the experiment setup.

Step 1. Determine If You Need a New Page Design

Sometimes your existing landing page is so poorly designed that ‘perfuming the pig’ will not be the best use of your time and resources. We suggest that you take a critical look at a page you want to test, and conduct a simple ’smoke test’. All you have to do is to verify that all three critical page elements exist and that they are all placed above the fold (see below).

If you page is not satisfying these basic requirements we would recommend that you create a new layout.

Step 2: Determine Duration of Your Test

Different multivariate testing methods will perform at different speeds, but you will still have to create an optimization experiment that can be completed within a reasonable period of time.

Just to give you a feel for test duration, let’s use the two calculators from above and apply it to a few typical real world situations

As numbers indicate, the bigger the number of test combinations the bigger the difference in test duration between two multivariate testing methodologies.

Step 3: Design Sections and Variation

Often website owners are paralyzed by a luck of ideas about what elements of the web page they should be testing and how to test them. Let us share with you few resources that will help you jumpstart your experimental design.

However, many marketers try to improve results by changing page elements like font colors and sizes, button shapes, images, incentives, and so on, when the first step should really be focusing on strengthening their value propositions.

Step 4: Define Conversion Goals

Defining your conversion goals is usually a simple task of identifying the most desirable action that should be taken by the test page visitor.

Some tools will allow selection of multiple actions on the same test page. This can be very convenient for landing pages that offer multiple calls for action leading to different pages in the sales funnel.

Others will force you into defining conversion goal as tracking of visitors who reach a pre-defined conversion page after visiting the test page. This is quite straightforward but can limit your ability to measure multi-call conversion actions.

Step 5: Optimization Enable Your Test Page

Enabling your test to participate in the multivariate test is a technical step that can be more or less complicated.

For example, if you decide to use Google’s Website Optimizer you will need to insert few different types of scripts into your test page and tracking scripts into conversion page, as shown below:

The scripts above are test and section specific so each time you decide to make a change in your test, or to run a new test you have to update scripts into your test and conversion pages. This can be a complicated error prone process that fully depends on the availability or IT resources.

Hiconversion’s transparent enabling method simplifies enabling process by requiring you to insert script into a generic page locations only one time in the life of the web page.

After that you can make changes in your test or run a whole different test without any need to touch your web page source code.

The third method of optimization enabling is called reverse proxy method. Here you have to deploy a proxy server in front of your existing web server. Although this approach can provide the highest optimization speed, many companies are very reluctant to make changes in the website hosting setup.

If you are using Google’s Website Optimizer or Hiconversion Pro, all you have to do is to start your test by pressing a ‘play button‘.

Know when to stop your test

One of the pitfalls of multivariate testing is that marketers get excited about results too early. Often they immediately stop the test and apply unreliable winning page combination.

If you desire to learn more about how to calculate statistical confidence of your results, please visit an excellent writup produced by Marketo.com.