Multivariate and A/B testing – the power of competition

In joined up web marketing teams there are systematic approaches to designing multivariate and A/B tests.  Specialist groups interpret analytics, identify key landing pages and the route taken to the ultimate goal – and then look for the worst performing steps – those that leak visitors away from the site and out of that sacred conversion funnel.  With plenty of resource this is very feasible, and results should come in thick and fast.

However, I would estimate that the vast majority of web teams do not have the resources to manage ongoing testing alongside all the usability, redesign and fixing of their site.  The clever e-business knows that their tech/analytics talent is valuable to them, but few pour resources into an area that is hard to calculate a clear ROI for.  So, when opportunities for the web/tech/ecommerce teams come along that put them in the spotlight and show their worth, they need to be taken.  So how is web-team-public-relations related to multivariate testing?

Done properly, multivariate and A/B testing will significantly increase conversion rates – ideally conversion in a process that involves revenue.  If you can show that your team have increased the ‘add to cart’ or the ‘checkout’ conversion by 10% – that makes a good start for justification for more resource.  But wait – we don’t have that resource yet.  So this is where we get to do some PR and get test results at the same time.

Set up your test framework in whatever system you like – Omniture Test & Target is good if you have the cash but Google’s Website Optimizer comes for free.

Then after you have identified an area to focus your tests on, announce to the rest of the company that you are running a competition to see who can come up with the best content/design for that area and that you will be testing the 6 best entries on the live website.  Getting people from other departments is great – you get a fresh insight that is more layman, and probably closer to that of your customer/visitor than your web team.  Hopefully they come up with good ideas, some of which you take forward into the test.

The nice thing about this from a PR point of view is that it has several clear stages with a specific overall purpose.  The purpose is to increase revenue, something that the whole company should easily buy into, the thing being tinkered with (the website) is easy to understand and relate to, and the process fits a competition structure well with opportunity for regular communications with the rest of the company on progress.  The end result should be a formula for a better conversion rate (and thus revenue) that has had lots of internal publicity – great well done to the people that came up with the winning test version and well done web team for coming up with this idea.

Exec: “You want more resource to do more of the same next year?  Sure!”

4 Responses to Multivariate and A/B testing – the power of competition

  1. Great post! But here is the catch: what do you do *after* the first or second round? The idea of continuing to run internal contests to generate ideas will exhaust itself soon enough.

    In our experience, the only way to get quantum leap improvements on the conversion rate (and *all* our clients at have seen conversion rate lift ranging from 10% to 270%!) is to commit to *continuous improvement* through ongoing experimentation… but the biggest challenge is coming up with new hypothesis to test, leveraging results on to the next experiment and the next and the one after that.

    This is why you need to hire experts in Conversion Optimization: they do this relentlessly. It’s not about the testing technology (as you point out, Google Website Optimizer is free and easy) – it’s all about *what to test*

  2. Ed says:

    > what do you do *after* the first or second round…
    Good point. If you have a big site, then the opportunities are everywhere. Especially if you have never tested before. The idea really is to work on building up an expert group of testers. Getting a wider group of people to help come up with things is just one idea to help the web group drum up ideas.

    > hire experts in Conversion Optimization…
    One thing I’d like to say is that website testing will be mainstream (if it isn’t already) and a fundamental skill that should be developed within every web/ecommerce group. As with everything a business will have to go through the process of proving why it is worth doing – but after that it depends on the amount of resource available as to what you do. A company with enough resource may go straight out to an expert third-party – certainly one good way of getting a quick return. More technically savvy businesses with existing in-house resource could easily build up that expertise themselves through tactical hiring or a good learning and development programme. Why pay someone else to do it if you can build that resource yourselves? That is a question for the business.

  3. Chris Goward says:


    > Why pay someone else to do it if you can build that resource yourselves? That is a question for the business.

    You’re correct there, Ed. Some businesses like to do everything in-house and others identify their core competency in another area and are happy to outsource to companies like WiderFunnel that have focused expertise in conversion rate improvement.

    Some good posts on this blog. Keep it up.

  4. Somehow i missed the point. Probably lost in translation 🙂 Anyway … nice blog to visit.

    cheers, Photolithography!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: