Adaptive MVT | The Secret Sauce in Hiconversion’s CRO Product @CRO Community

Adaptive multivariate testing

The adaptive multivariate testing method breaks the performance barriers making conversion rate optimization
posible for companies of any size.

Unleash the Power of Multivariate Testing

As online marketing professional you understand the importance of the conversion rates of your web pages. You know that if your page conversion rates are low your marketing ROI is going to be poor.

Many elements of your web page can impact web visitor behavior: main headline, graphics, product description, call for action, design of buttons, labels,… Testing multiple variations of different elements at the same time is called multivariate testing. Those who decide to run multivariate testing experiments will quickly discover that permutations of a few page elements with a few variations each quickly creates a large number of combinations.

The existing methods of testing all page combinations (so called full factorial testing) or testing of a subset of page combinations in concert with mathematical modeling (so called fractional factorial testing) are web traffic ‘hogs’. To complete your test within a reasonable time you need to have use hundreds of thousands of web visitors. Very few web pages on the internet can provide such traffic within reasonable period of time.

Full factorial method

This method is in essence a brute force approach where every page combination is tested until full statistical reliability is achieved. Google’s Website Optimizer uses this methodology. To get a feel about the web traffic you need to perform a multivariate test please check Google calculator.

Fractional factorial method

To increase speed, the fractional factorial method only tests a small number of all possible page combinations. Then mathematical modeling is used to predict a winning page combination. The most commonly used mathematical modeling is Taguchi. That is why fractional factorial method is often called Taguchi method.

This is providing a significant improvement in comparison with the full factorial method but not significant enough to satisfy the traffic limitations of normal web sites. In addition, if prediction did not work, another batch of page combinations need to be tested.

A fatal flaw of traditional methods is that they are batch processing methods. They are non-intelligent testing methods that keep showing certain page combinations until statistical reliability is achieved. In real life it is often the case that certain page combinations are so bad that further testing is just a waste of web traffic. The traditional multivariate testing tools do not track or act on such information.

Adaptive methodology

Adaptive multivariate testing is designed to respond to web visitor behavior in real time. It can be described as a closed loop system whose optimization process is initialized by a random page combination. During the test the system is collecting measurements and building a statistical model of the test space. At each iteration an algorithm is continually learning and predicting a winning page combination, while converging toward a statistically reliable winning result.

This method is minimizing the amount of web traffic that is going toward lossy page combinations and maximizing the traffic allocated to winners. This is dramatically increasing test speed while at the same time lifting conversion rates even during the test itself. Unlike traditional multivariate testing where users are experiencing a temporary dip in conversion rate during the test, our method is lifting conversion rates and our customers are making money even as they test.

How adaptive algorithm works?

Technically speaking, the adaptive multivariate testing methodology is in essence a multi-dimensional gradient search algorithm.

To better understand its logic lets first imagine that a surface below represents conversion rates of a multivariate test with two sections that have five and six variations respectively.

Then a multivariate testing problem can be visualized as a search in total darkness for a highest point on that surface.

Our algorithm starts the search by randomly picking a point on the surface (Page Variation 1) and determining its position in reference to Control (old page). If gradient (an angle between x axis and the line that connects two page variations) is negative that means that the Variation 1 was not a good choice.

The algorithm will then pick the next page variation (Variation 2). In this example Variation 2 also has a negative gradient. In the background the algorithm starts to correlate winning and loosing page variations. As result it will conclude that the entire line of page variations is less likely to be a winner (line of red dots) and it will move to another side of surface and test Variation 3. Since Variation 3 was a good choice, it will continue to explore that area of the surface but this time it will use Variation 3 as referent point.

Thanks to the speed of the gradient based search and self learning nature of our algorithm it is not then a surprise that our method is producing great results with dramatically less web visitors and that it starts lifting page conversion rates even during the test itself.

Frequently Asked Questions

1. When my test results are statistically significant?

Answer: Between 25 to 50 conversions are required for a combination to be somewhat confident (80% confidence) in a given page’s reported conversion rate. To be 95% confident you need to get 100+ conversions.

Note: As a subscriber to our service you do not need to know anything about the statistical significance of your test results. Our algorithm will converge toward the best page combination and then continue to show it to your visitors.

2. Is it possible that you will discard a winning page combination too soon?

Answer: The probability of our algorithm discarding a winning page combination too quickly is very low. Our decision is not based on a single round of measurements. Instead we continue to map the search space and if our statistical model shows a probability of maximum being in the area of winning combination, the algorithm will test again regardless of the fact that this page combination was already tested before.

3. Should there be a discrepancy between conversion rates achieved during the test and on going use of a winning page combination?

Answer: In general, if your page is getting the same traffic your test results should be within the margin of error. However, some anomalies are possible if your web page is experiencing day-parting or seasonal demographic changes.

Note: Letting our system continually manage your conversion rate will prevent surprises. In case of unexpected change in conversion rate our system will act as auto pilot and automatically stabilize page performance.

blog comments powered by Disqus