Should You Try Testing Multiple Variants of Your Landing Pages?


Have you ever heard the saying that “two heads are better than one?” Makes sense, right?

But what about three heads? Are three heads better than two? And what about four heads? Five heads…when does it stop? Isn’t there such a thing as too many cooks in the kitchen?

Right about now, you might be wondering, Jake, What on earth does all this have to do with improving my online marketing?

As luck would have it, these two little phrases can actually teach us quite a bit about landing page testing.

More heads is always better, until you… overflow the kitchen.

The Secret: Test Multiple Variants

When you think of landing page testing, you might immediately think of A/B tests.

Now, that’s not bad at all. Most marketers will really benefit from A/B testing—you can think of it like our “two heads is better than one” phrase. Each variant is like a head and—if you only have one—you’ll never know as much as you could if you had two.

A/B testing is a great way to improve the performance of your landing pages. You come up with a new page design, test it against your original and see if your results improve.

But, getting back to our “two heads are better than one” analogy, wouldn’t you get even better results from testing 2 new pages against your original?

This kind of testing, multiple variant testing (not to be confused with multivariate testing), is effectively A/B/C/D/etc testing. If you have enough traffic, multiple variant testing can deliver quick, effective results.

In fact, Optimizely did a study this year on what defines the best testing companies in the world. Here are the top 4 predictors of testing success:

  1. Testing the elements that bring in the greatest amount of revenue
  2. Testing each and every change
  3. Testing with real problems in mind (and testing for their solutions)
  4. Simultaneously testing multiple variants

Notice that last one? Turns out the best landing page testing doesn’t come from A/B testing—it comes from multiple variant testing.

So, two heads are better than just one, but three, four, five, etc. heads are even better.

Just to put this idea into more concrete terms, Optimizely reported that only 14% of A/B tests significantly improve conversion rates. However, tests with 4 variants improved conversion rates 27% of the time.

That means that testing 4 variants makes you 90% more likely to significantly improve your conversion rate than by just running an A/B test. But guess what? According to Optimizely, 65% of landing page tests are A/B tests!

Testing Multiple Variants Works Better

There are two major reasons why multiple variant testing outperforms A/B tests: 1) it’s quicker and 2) it lets you use the same testing conditions for more variants.

Multiple Variant Testing is Quicker

Now, with enough time, you could test all of the elements you want with a ton of A/B tests—or you can test several variants at once with one multiple variant test.

Which one sounds like it will take more time?

Think of it like unloading your groceries from your car. You can take just one bag at a time…or you can take as many as you can carry and make a lot fewer trips.

With an A/B test, you can only really learn one thing from each test. Your variant will either be better, worse or just the same as your original. Once you know that, you need to run more tests before you can learn anything else.

Now, if you have a smart A/B testing strategy, you’ll learn more about your audience from each test and be well-equipped to run better tests in the future, but it doesn’t change the fact that you only learn one thing from every test. On the other hand, multiple variant testing allows you to test several hypotheses at once.

See where this is going? With multiple variant testing, you can learn multiple things at the same time!

So, if you want to test your hero shot, you aren’t limited to comparing a photo of a crotchety old man to one of a smiling woman. You can also test a disgruntled grandma…or even see if an inquisitive man brings in the best results.

You can even do multiple combinations—like changing your CTA or putting in a new headline with any of the options. These simultaneous tests will let you optimize your site or page much faster.

The end result? You’ll be getting more from your website even sooner.

Multiple Variant Testing Also Happens to be More Reliable

Is the world any different today than it was a month ago? I can think of a few big things that happened in the past few weeks…

Whether its politics or seasonality, changes in the world change the traffic you’re driving to your A/B tests, which can make it hard to really compare the results of A/B tests.

As a result, it can sometimes be hard to know if one of your variants really succeeded or failed due to factors outside of your knowledge and even your control. Running more successive tests won’t make things better—in fact, it might make your results more murky.

But, with multiple variant tests, everything is tested under the same conditions. You’ll be able to really compare your results and get reliable, valid conclusions from your tests.

What Does Testing Multiple Variants Really Look Like?

We recently had the following experience with one of our clients that I think will help you see how testing multiple variants can really improve your CRO results.

Our client was trying to get people to visit their “Find Your Local Chapter” page, so we figured we would add a simple “Find Your Local Chapter Link” to their footer. That way, as many people as possible would see it.

Simple, right?

What we put together looked like this:

We were about to just run the test when it occurred to us that we had enough traffic to test another variant or two. The client wanted traffic drawn to this page, so we figured that we might as well make our link more obvious.

So, we thought, let’s try making the link stand out a bit more:

This seemed like a logical idea, but at Disruptive, we always prefer testing to gut instinct, so we thought, “You know, there’s enough traffic here to test 3 variants, let’s try something really bold!”

The only potential problem was, the client loved their site’s design—it was gorgeous, seamlessly designed and modern. It took a bit of persuasion, but we eventually sold them on testing a page element that interrupted that seamless flow.

So, we ended up with a third variant:

Though it was quite different from anything the client had previously tried, they agreed to run with it and let us test out the idea.

A few weeks later, the winner was clear:

Simply adding a “Find Your Local Chapter” link increased their visits to that page by more than 60% for every variant.

Awesome, right?

But here’s the thing—if we had only run an A/B test with the grey link, we only would have found a way to increase their traffic by 63%.

However, with the extra variants, we were able to find an option that increased their traffic to that page by 83%—even though it was a design our client had thought would “interrupt” their site experience.

Now, we might have been able to reach the same conclusion if we had just run a bunch of A/B tests, but with multiple variant testing, we found these results much quicker and much more reliably than we would have with A/B testing.

So…Should You Be Testing Multiple Variants?

This is where the “too many cooks” saying comes into play. If you have a kitchen that only fits two cooks, then two is really what you should stick to. But, if your kitchen can fit seven cooks, by all means, have seven!

Makes sense, right? Many hands make light work…now apply this to your landing page testing.

The size of your kitchen is determined by how much traffic it gets and—if you think of each variant as a cook—you can probably see where this analogy is headed.

Multiple variant testing is awesome, but if you’ve only got enough traffic to run an A/B test, adding more variants will actually slow your process way down. After all, each variant will need a certain amount of traffic to yield statistically significant results.

So, if you don’t have a lot of traffic, you might have to stick with A/B testing for now. But, if you’ve got the traffic, go for it!

If you’re wondering how long it might take to run a multiple variant test on your site, check out this free calculator from VWO for sample sizes and test durations. If your business is ready for the time frames this calculator suggests, start testing multiple variants!


So, when it comes to landing page testing, don’t be afraid to move beyond simple A/B testing. Testing multiple variants will enable you to improve your conversion rate far faster and more reliably than any A/B testing strategy.

You’ve heard my two cents, now it’s your turn.

Have you tried testing multiple variables at once? How did it go? Did you find any of the data in this article surprising?

About the author:

Jacob is a passionate entrepreneur on a mission to help businesses achieve online marketing success. As the Founder & CEO of Disruptive Advertising, Jacob has created an award-winning, world-class organization that has helped hundreds of businesses grow using pay-per-click advertising and conversion rate optimization.


Leave a Reply

Your email address will not be published. Required fields are marked *