Background: My name is Matthew Roche.  I am Co-President and CEO 
of Offermatica, a sophisticated ASP Landing Page Optimization 
service for A/B Split, Multivariate and Taguchi testing. 
 

You have a great idea for a different page layout, some new copy 
or image, or a promotion that will give your site the kick in 
the pants it needs to break through stagnant conversion.  You 
want to make the change, but you have been through this drill 
before.  Some efforts make things better, some of them don't, 
and for the rest, it is unclear, so you assume they must have 
had some positive impact.
 
This time through, you are going to test the idea against what 
you already have so that you know that you won't lose ground.  
You have decided to enter the ranks of scientific marketers  
 
In this column, we will clarify the muddle of techniques that 
can help you more reliably increase the conversion of visitors 
to customers on line, including A/B testing, split run testing, 
multivariate testing, and even something called Taguchi testing 
that sounds suspiciously similar to a "virtual pet" popular in 
the 90s.
 

Scientific Optimization
 
At its heart, scientific optimization uses the computer to 
increase the odds that the page or session that your customer 
sees is the best possible.  I find it useful to break the 
approaches into three levels of sophistication:
 
Level 1:  A/B Split Testing - Simple test of one element of 
          a page against another to see which is more effective.
Level 2:  Multi Variable Testing or Multivariate Testing - 
          Testing more than one element at a time to test new 
          page treatments or offers
Level 3:  Experimental Design - Using advanced statistics to 
          determine the "best" layout or configuration of 
          elements using the smallest possible number of 
          visitors.
 

A/B Split Testing
 
Testing two alternatives at the same time is the easiest but 
often least valuable way to improve conversion through testing.
 
Most companies begin scientific web site optimization using AB 
Split or Split Run Testing.  AB split testing allows you to 
randomly divide your visitors into two groups and show each 
group a different version of a page to determine which version 
leads to higher conversion, average order value, application 
completion, or other target.  These visitors are then tracked 
and a report is generated that describes the impact of the 
A or B page version on this outcome.  
 
A common use of the AB Split Test is to evaluate the impact of 
a new page layout on the likelihood of generating a sale.  Two 
versions of the page are created.  Often companies test a new 
page design versus the existing design.  Traffic is randomly 
split between the two pages using custom programming, by 
splitting the application servers or using ASP tools like 
Offermatica.  The visitors are identified as belonging to the 
A or B test group and are watched through their visit, and 
occasionally across multiple visits to see if they are more 
likely to purchase based on which page they saw.
 
AB Split Testing is a simple approach to letting your customers 
"vote" on changes through their behavior. It also suffers from 
significant limitations.  First, many changes have either a 
negative or not measurable effect.  Basically not every element 
on a page influences a purchase, making it necessary to run 
several tests in a row to find the element that matters. 
 
To add insult to injury, it can often take 10,000 visits and 
50-200 orders over a minimum of two weeks to achieve confidence 
in an outcome, so running enough tests one after the other can 
take a long time.  If a significant event happens during that 
period, like Valentines day or tax time, it can compromise the 
result.
 
The second issue is that an AB Split Test of a new page 
treatment against the existing may confirm a positive impact, 
but it cannot tell you what elements of the new design actually 
made the contribution.  Imaging a situation where a new copy 
treatment on a home page was very effective, but the new 
navigation was actually worse.  The overall test of the page 
may show a slight improvement, but hides the fact that you 
could have done even better with just the copy and not the 
new navigation. 
 
 
Multiple Variable Testing or Multivariate Testing
 
Multiple Variable Testing isolates the elements on a page and 
helps you find out what elements matter and which combination 
is the strongest.
 
One way to improve your chances of finding a winner is to test 3 
or 4 or more versions instead of just testing old versus new and 
pick the best performer from among a the larger group.  This is 
called A..n testing and improves your chances of finding a 
winning version, but also increases your content development 
burden.  A better way is to test elements on the page in 
different combinations of "recipes."  This approach is called 
Multiple Variable Testing or Multivariate Testing.
 
Multi Variable Testing allows you to test the elements on a 
page that you believe impact sales.  When planned and executed 
carefully, Multiple Variable Testing virtually guarantees a 
positive change over your existing page and offers insights into 
how to market to your customers and prospects elsewhere on your 
site.
 
A Multi Variable Test on a product landing page might test the 
product image, the headline and the product description copy.  
The goal is to create the most compelling page possible so that 
visitors to this page, often paid for through search or banner 
advertising, convert to customers at the highest possible rate. 
Two or more alternatives of the picture, description and 
headline are created and a page is composed for every 
combination of these elements in each of their versions.  
If there are 3 elements with two alternatives, this requires 
8 combinations or "recipes."  
 
By splitting the traffic randomly and showing each visitor only 
one version, we can determine the optimal recipe.  The advantage 
of Multi Variable Testing over AB Split Testing is that you can 
nearly always find a recipe that outperforms existing.  The 
problem with Multi Variable Testing is that if you have more 
than three elements or more than two alternatives, the number of 
combinations becomes so large that it takes too many visitors to 
run a conclusive test.  
 
 
Advanced Testing and Automated Optimization using the Taguchi 
Method
 
The Taguchi Method is the most powerful and most likely testing 
method to create a significant improvement without creating an 
overwhelming amount of incremental work. 
 
If you have four elements in a multivariate test including the 
product picture, headline, copy, navigation and a promotion, and 
you have four alternatives for each, you need to run 64 recipes. 
It still takes 40-200 conversions for each recipe to achieve a 
conclusive test and the volume of traffic required is too great 
for most applications.  Because of this limitation, experimental 
design methods have been created to test a small, indicative 
subset of recipes and estimate the theoretical best recipe even 
if it was explicitly tested.  This approach is called fractional 
factorial testing and can be done using a number of methods 
including the Taguchi Method.
 
The Taguchi Method was developed 50 years ago and has been used 
with great success to optimize automobile and other product 
manufacturing.  More recently, The Taguchi Method was applied 
to direct mail and web applications.  The Taguchi Method takes 
a number of elements on a page with one or more alternatives for 
each element and dictates exact combinations that will allow you 
to estimate the positive or negative effect of each 
element/alternative.  
 
There are three extremely exciting aspects to this approach.  
First, by creating a "best page" using the best performing 
alternatives for each element, significant improvement can be 
achieved.  Second, the length of the test cycle and the number 
of visitors required is surprisingly small.  And finally, since 
the "recipes" are created using modular element/alternatives, 
using a solution like Offermatica, Taguchi tests can be designed 
and executed in a surprisingly small amount of time. 
 
Taguchi tests have been run on email, PPC ads and Landing Pages 
with great success.  Where an AB Split Test might create a 5-10% 
improvement, a Taguchi test cycle will regularly return 25-45% 
improvement and has been known to improve results by 100% or 
more.  A test cycle includes two weeks of testing a large number 
of elements in just two alternatives to identify which elements 
increase the likelihood of converting a visitor to a customer, 
a second test where the high-impact elements are tested with a 
greater number of alternatives, and a final test of the "best 
recipe" against the original page.  The test cycle takes from a 
couple of days to a month depending on traffic and variance and 
can be designed and run without significant quantitative 
marketing or statistics experience.
 
After using Taguchi Testing on a single page to optimize the 
conversion for general traffic, things get a little more cutting 
edge and complicated.  Early attempts have shown promise in 
applying the Taguchi approach to multiple pages, the entire 
session and even multiple sessions.  Also tests can be targeted 
to optimize the page or pages to a subset of visitors.  
 
Scientific website optimization using AB Split Testing, 
Multivariate Testing and Taguchi is not a substitute for great 
marketing ideas, or a strong sense for what will work.  The raw 
materials for great test results are always great ideas.  But 
finally, it is possible to quickly and easily quantify these 
great ideas and see the result on the bottom line. 
 
Testing is not a replacement for marketing, and no technology 
will design your pages for you.  However, these alternatives can 
provide a reasonable map of how to use technology to make your 
brilliant ideas hit home.

 
For more information about AB Split Testing, Multi Variable 
Testing or Taguchi Testing please contact: 
mailto:mjroche@offermatica.com 
or visit: http://www.offermatica.com