There’s a lot of love about online marketing from a strategic standpoint. Possibly the biggest differentiator between online marketing and other marketing methods is the degree to which it can be tracked. After launching a website, social media presence, or campaign, the next logical step is to understand and optimise the performance you’re receiving. Introducing A/B or multivariate testing.
While marketers often rely on industry best practices, past performance, and trends to optimise performance, one method of optimization seems to often get swept under the rug: A/B and/or multivariate testing. A/B and multi-what? Let’s start with what Wikipedia says and bring it down a notch:
A/B Testing
A/B testing, split testing, or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates.
In layman’s terms, two (or several) different versions of an ad, email, or web page are tested and measured to see which one performs best. Pretty straightforward. The two promotional banners below could be tested to see which encouraged more sales.
Multivariate Testing
This method is different than multivariate testing which applies statistical modelling which allows a tester to try multiple variables within the samples distributed.
Again, in (hopefully more) simple English, multivariate testing allows you to identify several items on a page, for example, that for each, you wish to test different one or several different options. An example would be an email where you want to test three different headlines, two alternate images, as well as three variations of the call-to-action to see which combination is the most successful.
Sounds great. Nah, I’m not interested
The goal of these A/B or multivariate testing methods is to serve the best possible option to the end-user and in doing so to optimise performance. So, why doesn’t everyone run tests like this?
Here are three common reasons that A/B Multivariate testing isn’t done:
“It costs too much money”
Unless budgets are incredibly constrained, this shouldn’t be an issue. Testing doesn’t have to be expensive, especially with the tools available today, including free one’s like Google’s Website Optimizer (note that Website Optimizer is for just that-websites, not email, ads or other media).
And, if you’re working with an agency, they likely have tools that they’ve already licensed. The cost is then limited to concepting the different versions and putting the test in place.
“We tried it once and didn’t see much of a difference between versions”
You definitely won’t hit a home run every time. Just because testing on one occasion revealed that between the options tested there wasn’t much of a difference, that doesn’t mean that nothing should be tested.
“We know better”
Clients often simply think that they understand their offering, brand and customer so well that A/B or multivariate testing is just a waste of time and money. While I can understand (and respect) a client’s knowledge and intimacy with their brand, what sometimes comes to light is that what performs best can defy common logic or perceived wisdom. Sometimes the consumer just sees it differently from their side of the equation.
What you don’t know can cost you money
To put it as bluntly as possible, if you don’t test something you simply won’t know if it could have performed better.
Yes, you can run tests and find that other options offer marginable, offer only negligible gains, or offer even worse results. But, you may also find out that there’s an option that performs significantly better, sometimes by a long shot, pushing the impact of your marketing dollars far further.
If your budget allows for it, I’d recommend testing when it makes sense.
Some final advice
First of all, you certainly don’t need to test everything. That tagline that the new guy brought up in the marketing meeting that clearly sucked will still suck no matter how many ways you test it.
A great way to get started is to test something that will effect the return on your marketing investment. Possibly an email that is going out that has a call-to-action to redeem an offer or sign up for a seminar, or a landing page on your website where you drive paid marketing traffic and hope for a conversion.
To find a great candidate for A/B or multivariate testing, identify something that has created some uncertainty internally. Are you struggling with two subject lines that both state the same thing, just with a different tone? Are you considering changing the placement or styling of a form?
Wondering if you might capture more leads if less information were required of the visitor? Each of these situations would create a great testing opportunity.
Good luck!