Multivariate and A/B testing – the power of competition

In joined up web marketing teams there are systematic approaches to designing multivariate and A/B tests.  Specialist groups interpret analytics, identify key landing pages and the route taken to the ultimate goal – and then look for the worst performing steps – those that leak visitors away from the site and out of that sacred conversion funnel.  With plenty of resource this is very feasible, and results should come in thick and fast.

However, I would estimate that the vast majority of web teams do not have the resources to manage ongoing testing alongside all the usability, redesign and fixing of their site.  The clever e-business knows that their tech/analytics talent is valuable to them, but few pour resources into an area that is hard to calculate a clear ROI for.  So, when opportunities for the web/tech/ecommerce teams come along that put them in the spotlight and show their worth, they need to be taken.  So how is web-team-public-relations related to multivariate testing?

Done properly, multivariate and A/B testing will significantly increase conversion rates – ideally conversion in a process that involves revenue.  If you can show that your team have increased the ‘add to cart’ or the ‘checkout’ conversion by 10% – that makes a good start for justification for more resource.  But wait – we don’t have that resource yet.  So this is where we get to do some PR and get test results at the same time.

Set up your test framework in whatever system you like – Omniture Test & Target is good if you have the cash but Google’s Website Optimizer comes for free.

Then after you have identified an area to focus your tests on, announce to the rest of the company that you are running a competition to see who can come up with the best content/design for that area and that you will be testing the 6 best entries on the live website.  Getting people from other departments is great – you get a fresh insight that is more layman, and probably closer to that of your customer/visitor than your web team.  Hopefully they come up with good ideas, some of which you take forward into the test.

The nice thing about this from a PR point of view is that it has several clear stages with a specific overall purpose.  The purpose is to increase revenue, something that the whole company should easily buy into, the thing being tinkered with (the website) is easy to understand and relate to, and the process fits a competition structure well with opportunity for regular communications with the rest of the company on progress.  The end result should be a formula for a better conversion rate (and thus revenue) that has had lots of internal publicity – great well done to the people that came up with the winning test version and well done web team for coming up with this idea.

Exec: “You want more resource to do more of the same next year?  Sure!”

Google’s website optimizer and ajax

A couple of weeks ago Google launched their Website Optimizer product out of beta – it is now a fully fledged standalone product (previously you had to use it via an Adwords account). I was playing with it today because I wanted to make sure that we could test with dynamic content.

A typical A/B or multivariate test might take a page portion and then serve up several static variations. Sometimes static variations aren’t good enough though. Most eCommerce websites are database driven and use templates for product pages that are populated with information specific to that product. The template knows what product info to load in because the page might be accessed via a URL with identifiers in the query string: e.g. http://somesite.com/product.html?productid=1234

This product page knows that it has to load up the details for product 1234.

When you want to start doing more complex A/B tests, where the data for your variations also comes from the database, you have a slight problem in that the alternative content for the test is managed in the website optimizer interface – how do you get dynamic content out of your database for your test variations?

To get around this, you can use Ajax to grab the dynamic content relevant to that particular product page, and use the Website Optimizer to simply modify parameters in the Ajax call. This might be implemented by creating four server-side functions that are accessed by Ajax, each returning a variation on the original test content.

In Website Optimizer, when you declare which part of the page you are testing, rather than wrapping the content section, wrap the piece of Javascript that sets which function the Ajax request will call (or Javascript that sets an Ajax parameter):

<script>utmx_section(“AjaxSection”)</script>
<script>aj_fn = “variation1″;</script>
</noscript>

Then, when you proceed through the experiment designer to add new variations, just add:

<script>aj_fn = “variation2″;</script>

Where “variation2″ is the name of the function the Ajax will call to return the “variation2″ content.

Alternatively, as mentioned previously, instead of creating a function per content variation just alter an Ajax parameter so that the function returns different content.

This combination of Website Optimizer and Ajax makes for an extremely powerful technique. It’s pretty easy to implement too.