The Ultimate Guide to A/B Split Testing with WordPress

The Ultimate Guide to A/B Split Testing with WordPress

Does the thought of your website keep you up at night? Do all the “what ifs” have you worrying about the design, the functionality, the calls to action, the wording and, ultimately, your conversions??

You could have your designer/developer make changes based on what you think might help with improving the number of site visitors, the time they spend on site, as well as their conversion rate. But what if you’re wrong and you spend all that time redesigning your site only to find that they actually liked the way it looked before?

You know that your website serves many purposes. One of those is to showcase your business and the solution or service it provides to customers. Another is to give your customers an easy way to reach you in the form of a phone call, email, form submission, or online purchase.

If you’re worried that your website may be standing in the way of successfully making this connection with your audience, then it’s time you took a look into the underlying cause and devise a solution.

A/B testing is a method of comparing two versions of a webpage or app against each other to determine which one performs better.
A/B testing is a method of comparing two versions of a webpage or app against each other to determine which one performs better.

What Is A/B Testing?

Supposing your website is causing you stress, you could leave it be and try to compensate from its lack of performance by investing in other ways to target new and existing customers. That wouldn’t make much sense though. Your website is the online face of your business. You need it to perform just as well as anyone else who works for your company.

You can’t assume that because you love your website and you think it looks great that your online visitors feel the same way. More importantly, a nice-looking website is only part of what’s needed here. It needs to be easy to navigate, have a logical flow, and clearly explain what you do and what you offer. No matter how excited you are about your website, if it’s not performing well or you want it to do even more, it’s time to make a change.

According to Merriam-Webster’s dictionary, the Scientific Method is defined as:

… Principles and procedures for the systematic pursuit of knowledge involving the recognition and formulation of a problem, the collection of data through observation and experiment, and the formulation and testing of hypotheses.

A/B testing is exactly that. It’s a digital version of the Scientific Method whereby a website’s design, layout, copy, and/or functionality can be tested without permanently altering the makeup of the site. By establishing very specific variations on pages of your website, you can study the resulting differences in behavior of your visitors and consequently make well-informed decisions regarding what improvements are needed.

There are a number of approaches to consider when conducting A/B testing:

  • Split Testing: This is the typical method when someone refers to “A/B testing.” A singular element on a page is chosen to be altered and then two versions of that page are created with an A variation and a B variation. Those two-page versions are then shared with different visitors in order to test the varying responses between the two.
  • Multivariate Testing: For those who have successfully conducted A/B tests in the past and want to look into whether more complicated pairings of elements on a page may affect the user experience, that’s when multivariate testing comes into play. Rather than focus on changing one element, a variety of elements and combinations of those elements are reworked simultaneously.
  • Experimental Testing: In this method, all elements on a page are under consideration for testing, so this is really only for users who have extensive experience conducting multivariate tests and understand what to do with the results.

Whether this is your first time with A/B testing or if you already have experience in this, split testing is the optimal choice. It ensures a level of simplicity and focus that allows you to work with a control (your original design) and a variation (your new design). By testing two very clear differences among your audience, you can identify changes in behavior (if any) as it relates to that one element.

It’s important to remember that while you may have a good understanding of your audience as a customer, you may not totally understand what they want or need from the online experience. Rather than try and make that decision on your own, A/B testing can help you gain those insights. With carefully measured steps and results, you don’t have to worry about whether you’ve made the right call or not—the data will let you know what to do.

When Should You Conduct A/B Testing?

Now that you know what A/B testing is, it’s important to consider the “when” of the thing.

Most people probably assume that A/B testing is meant solely for websites that are underperforming. While A/B testing may be one solution for that, it won’t always be the case. If oversized images are bogging down your site’s loading time or if you have broken forms and incorrect contact information keeping visitors from reaching you, there is no amount of A/B testing that is going to fix those problems. Remember that you need to employ the use of the right tools for the right problems.

When it comes to A/B testing, there are a number of scenarios where you may need it:

  • Site re-launches: Rather than wait to see how a new site design (or messaging) performs with your audience in the long-run, start with split testing as soon as possible. There was a reason for your website’s re-launch, so you should do what you can this time around to better shape all the pieces of it based on what your visitors want.
  • Marketing campaigns: Marketing campaigns can be a highly effective way of expanding your business’s reach. However, these can often take a lot of time and money to strategize and create. While in the design and copy phases of your campaign, create A/B tests for your landing pages. In the off chance one of your ideas doesn’t go over well with your audience, you can potentially save this marketing venture from being a total loss.
  • Decreased conversions: If you’re noticing a change in the amount of website conversions, now is the time to take action. If the problem  isn’t due to a back-end issue, A/B testing may help you sort out where things are going wrong on the site.
  • Increased conversions: While increased conversions are definitely a cause for celebration, this is also a great time to work on keeping that momentum going. Conduct some A/B tests (one at a time) to figure out what is working and what your visitors would like to see more of across your website.
  • Product launches: When releasing a new product or service, there is never a guarantee it’s going to perform as well as your current offerings. Much like the site re-launch, you don’t want to have a cross-your-fingers-and-see approach. You want to set yourself up for success and test out a number of options during the launch so you can act immediately if something about the page or information isn’t working.

Ultimately, the choice of which elements to test will fall to you. You’re looking to make changes to your website for a reason, so you must have a gut feeling that something isn’t working the way you need it to. Based on where the problem lies—a high bounce rate on a specific page, a lack of conversions on a certain form, confused messages from visitors who can’t find something—you can then make a determination of where the test should occur.

Once you’ve got the page figured out, you’ll need to determine which element you want to rework. It could be something like one of the following:

  • Headlines
  • Sub-headlines
  • Paragraph text
  • Layout
  • Design
  • Copy
  • Contact form
  • Call to action text
  • Call to action button
  • Images
  • Page length
  • Social proof
  • Awards and badges

And within those elements, it could be any number of issues causing disinterested visitors:

  • Distracting color choice
  • Too many steps required
  • Confusing button placement
  • Negative messaging
  • Too many ads

You may want to step things up a notch with more advanced test

Overall, you really won’t know what could affect your user’s experience until you get started with the process.

Obama 2012 Campaign banner
The Obama 2012 Campaign ran over 500 experiments over 20 months to drive online fundraising.

How Do You Conduct an A/B Test?

An A/B test runs just like the Scientific Method: research, hypothesize, test, monitor. So if you haven’t taken high school chemistry (or it’s been a while) here is a recap of all the steps you’ll need in your testing procedure:

#1: Research

The first place to start is by asking yourself questions: Why do you want to A/B test? Have you identified a problem page or process on your website? Why do you think visitors aren’t responding well to that page or process? How do you think it should be updated?

While it’s important to ask these questions and to know why you want to do this, you also need to conduct some research to find out a bit more about what’s going on.

  • Start by digging into your analytics and see if you can identify any troublesome exit pages where activity and interest seems to stop.
  • Analyze the popular pathways visitors take on your website. If they’re not doing what you want them to, then you need to make changes to better guide them.
  • Use heat maps to see where people’s eyes are drawn to on your website. It may not be where you think.
  • Conduct interviews with your sales or customer service team members to see if they have any insights on what your audience is thinking.
  • Conduct online surveys with your audience and encourage them to provide feedback. That will be the most straight-forward approach to finding out what needs to be changed.

#2: Establish the Goal

Once you’ve identified a problem area on your website (or an area that you think could perform even better than it already does), you need a goal. When your team updates this page, what is the end result you want to see? More conversions? Longer time on the page? More visits to a different page that shoots off from this one? Without a clear and achievable goal, you won’t be able to verify whether the test worked or not.

#3: Hypothesize

You’ve identified your goal, so now you need a hypothesis.

Which elements on this page could stand to be updated? What sort of change could you make to each of those elements? What result would you expect to see with each of these variations? Why? Jot down your expectations for analysis later.

#4: Choose One Variable

You’ve compiled a list of elements on your web page that could potentially affect audience reception or behavior, so it’s now up to you to choose one element to focus on for this test. Perhaps it’s the color of the call-to-action button that just doesn’t stand out enough. Perhaps the form is in a weird spot that people don’t think to scroll to. Perhaps the headline at the top of the page is turning people off. Use what you know to identify the variable and determine how you will alter it.

#5: Develop a Timeline

A/B tests cannot run indefinitely. The point of these is to test a variation, monitor the reaction, and take action.

Depending on the amount of traffic to your website, you may want to allow the test to run for longer so you have enough activity to measure. Regardless of how long you feel your test should run, start with a test duration calculator to get started. This will help you determine where that sweet spot will be. Ending your test too early may lead to a lack of valuable stats. On the other hand, ending a test too late could be detrimental for a number of reasons. For one, the search engines may look unfavorably upon that (see more on that below). For another, if the variable was not a good pick, it may be the reason there are no results to work with and should be ended sooner rather than later.

When conducting a test, it’s important to develop a hypothesis so you have a clear vision of what you think will happen. It’s equally important to closely monitor the real results against the hypothesis in case there are severe deviations that require you to take action.

#6: Set Up

The next step is to create the variation on your page and track it.

For A/B split testing, you’ll need to create a duplication of your web page within WordPress and then apply the desired change to the new one. Your testing software or tool will then push part of your web traffic to the variation on a separate URL. The purpose of this test is to find out what these changes mean to visitors. In order to effectively test the reception of your page’s change, the variation should only be sent to new visitors. You don’t know if the alteration to your site will be a welcome addition, so it’s safer to test this on new visitors who have no preconceived notions about your brand.

When adding new pages to your website or making any changes that affect the user experience, it’s important to consult with Google. A/B testing can introduce some potential issues for you with the search engines, so keep the following in mind before proceeding with the setup and launch of your first test:

  • Cloaking: Google is very sensitive about websites attempting to deceive the search engines for better search results. Cloaking is one such example whereby a split test is created to show one version of a website or web page to the search engines and a different one to users (which is a big no-no).
  • Indexing: Any time you set up a duplicate web page, it needs to be set up for proper indexing (or none at all) by the search engines. Since the purpose of A/B testing is not to create a permanent duplication of your site’s content, your variable page should not be marked with the noindex tag. Instead, you should use the rel=”canonical” link attribute to properly establish its purpose.
  • Redirecting: Another way you can indicate to Google that this is only a temporary change is by setting up the proper redirect. A 301 redirect indicates a permanent redirect to the new, alternate URL. A 302 redirect indicates a temporary redirect to the new, alternate URL and, in so doing, lets search engines know they should continue to direct traffic to the original URL.
  • Conclusion: Your test needs to come to an end and, according to Google, within a reasonable timeframe. If the search engines feel your test is running unnecessarily long, they may interpret that as a form of deception and penalize your site. So make sure to closely monitor your testing tool to ensure you’ve collected as much data as you need and then close out the split test completely.

There are many tools you can use to help you set up and track the performance of your A/B test (see below). These tools may not help you account for Google’s tips though, so make sure these are built into your setup process so as not to run counter to what the search engines want.

#7: Run the Test

Using your A/B split testing tool, kick off your test. As with any true scientific test, consistency is key:

  • Your control and variation need to be tested simultaneously. If they are not tested under the same conditions (especially the time period), the results may not portray an accurate representation of how your change was received. The only difference between your control page and the altered one should be the actual variation.
  • For any update you make to the inherent design of your website–the color of a call-to-action (CTA) button, the layout or length or a form, etc.–you should apply this variation throughout all pages of your website where the element occurs. Even though you may be testing a specific page, you still want visitors to have a consistent experience across your website.

#8: Monitor

Once you’ve kicked off your test, make sure to regularly monitor the results. A timeline is established, but that doesn’t mean you need to wait until the very end of it to see how the page variations are performing (especially if something isn’t working or stops working in the middle of the test).

Monitoring isn’t just about logging into whatever tracking tool you’ve signed up with. If you have a customer service or sales team in place who gave you feedback while doing your initial research, they should be able to tell you if they’ve noticed any changes since the testing took place as well.

#9: Take Action

Once you have solid data to work with, take time to analyze it. What have you learned from the A/B test? Which variation worked best? Can you tell why? Is this change something you can apply elsewhere on your website? Is there a different element on this page that may still be causing issues? Was the difference in performance between the variations worth making any changes at all?

The results of your A/B test may be surprising. No matter how extreme or unexpected they may be, the results of the test–if conducted properly–don’t lie. So if you’re feeling wary about implementing a change that doesn’t sit well with you personally but that was well-received by your audience, don’t let your own aesthetic preferences get in the way. Remember that if the A/B test result goes live and does not work in the long-run, you can always change it back or set up a new test to find a more successful variation.

Once you’re comfortable with the process of A/B split testing and have a clearly defined approach, plan on making this a regular part of your website maintenance. With each test you’ll learn more about your audience and be able to more easily make decisions related to website and business improvements.

With a clearly defined approach and carefully measured results in hand, you should be able to more easily make decisions related to website improvements.

WWF Switzerland used A/B to increase their donations by 360%.
WWF Switzerland used A/B to increase their donations by 360%.

What Should an A/B Test Look Like?

There are no hard and fast rules about which elements you need to update for greater traffic and better conversion rates. That determination will be yours to make based on the current state of your site, the feedback and insights you have into its performance, and your ultimate goal in A/B testing.

Real-Life Examples

While the choice to test a specific variable will depend on what’s happening with your own website, that doesn’t mean you can’t look to others for inspiration on what has worked for them. Here are some well-documented cases of brands and people that have used A/B testing to success on their website:

  • Gyminee: The first and simplest of the tests conducted on this website were to the home page. The original version of the home page included 25 above-the-fold calls-to-action. In the B version of the website, they cut that number to 5 CTAs. They ran the test twice and the simplified B version of the home page resulted in 19-20% more conversions.
  • Adore Me: This clothing retailer uses A/B testing to ensure that the product images they use on their website perfectly are framed. So when it comes time to shoot the photos of the lingerie and the models that wear them, they make sure to include variations in the model, the color of lingerie, the placement of the model’s hands, etc. This way, they have a variety of images to test out for every single product on the website.
  • Soocial: Soocial wanted to experiment with conversion rates when changing up the language of their home page’s CTA button. They conducted a number of tests to find out what would perform the best. The design and words they originally had on the site proved to be fairly effective, but when the words “It’s free!” were placed beside the button, the conversions increased by 28%.
  • ComScore: The number of demo requests on ComScore’s software product page were underperforming. They believed part of the problem was the lack of emphasis on the social proof (the testimonials). So they tested a number of layouts and designs against the original. All of the variations outperformed the control, but there was one clear winner they identified when all the testing was done.
  • ABC Family: This case study is the perfect example of why user behavior statistics can help create more effective tests if you have an idea of what drives your audience in the first place. When the original page design showing a promotional image for one of their shows didn’t perform as well as they thought it would, they turned to a text-heavy menu of options instead. Because they knew site visitors were more apt to do a search for the show rather than click on that image, the results of this A/B test were a huge success and led to a 600% increase in engagement.

A/B Test Ideas

Once you’ve got the motivation to conduct your own A/B tests and have found some inspiration, it’s time to start focusing on the specifics that can be done for your website.

Here are some test ideas to get you started:

  • Add a video testimonial
  • Include logos from former clients or current partners
  • Use a different promotional offer
  • Use a different promotional photo
  • Update the call-to-action wording
  • Update the call-to-action button color or size
  • Change the text layout from multiple columns to single columns
  • Alter the headline above a form
  • Shorten the length or appearance of a form
  • Reduce the number of steps to checkout
  • Revise the navigation structure
  • Change the font size
  • Use strategic bolding or italics in the copy
  • Rewrite copy to include less paragraphs and more bulleted or numbered lists
  • Revise the content that appears above-the-fold
  • Change pricing or pricing structures
  • Change ad content or design

For your first A/B test, start out small and simple. Make sure your research and planning is guiding you toward the right hypotheses and testing variables before committing to more drastic or extensive updates. If you find that the suggestions or case studies above aren’t applicable to your current situation or you simply want more help, feel free to check out Hubspot or Wired for more information

Many of the A/B testing tools we’re about to mention below (like Virtual Website Optimizer and Optimizely) also serve as strong resources on the subject of A/B testing. So regardless of whether you use them for testing or not, the information they produce around the topic is still worth checking out if you’re serious about this testing business.

What Are Some Tools for A/B Testing?

You don’t have to set up your A/B tests on your own. There are a number of services you can sign up with (most for a fee) that will help you create, conduct, and manage your A/B tests.

  • Optimizely

    When looking up information on A/B testing, chances are good that you’re going to see Optimizely in the list of top solutions or mentioned as a reference for more information. With customers like The New York Times, Disney, and Priceline under their belt (and case studies to back it up), Optimizely is the A/B testing tool to trust. And it also helps that their Starter plan is free.

  • Nelio

    Nelio has two especially strong traits to pay attention to. First, they specifically target WordPress websites, and second, they have custom-tailored solutions for different industry and website types. When software providers are able to speak to the specifics of your business and thoroughly understand the platform on which you work, there is usually a level of expertise there worth paying for.

  • Maxymiser

    Oracle’s Maxymiser testing tool is another one that drills down into industry specialty. They’ve also included mobile app A/B testing which you don’t see in many other tools, so this is another differentiator worth considering if you have a large amount of traffic visiting your website from mobile devices. There isn’t any pricing listed on their website which means it’s probably more expensive than the competition, so be prepared for that before reaching out for more info.

  • Visual Website Optimizer

    VWO is another one of the leading A/B testing tools currently available. They’ve worked with some of the top brands around the globe and have the results to prove it (like the 62% increase in inquiries for Hyundai). If your website has a lot of web traffic and you can afford the higher price tag on these services, this is the tool for you.

    Interested in Visual Website Optimizer?

  • Adobe Target

    If you’re already a fan of Adobe’s tools, then this may be the next logical step (and even if you’re not familiar with their other tools, this still might be the right solution). Their home page alone offers many tools to help you make that decision: an interactive demo, a self assessment, and an ROI calculator are all right there to help you get started.

  • Unbounce

    Unbounce’s client list includes the likes of Hootsuite, Thomson Reuters, and Zoho and they’ve got the testimonials and statistics to prove that the solution they’ve provided to those companies has worked. With a specialization in the creation of landing pages that convert, Unbounce is a good tool to consider for your website and newsletter integration needs. They even have optimized landing page templates to get you started.

  • Usability Hub

    If what you’re looking for is less of an all-encompassing A/B testing tool and more of a means to find out where to start, Usability Hub offers a number of tests to help you identify problem areas on your website. Start with the Five Second Test and found out what first impression your website is making with visitors. You can move on from there to try out their other tests and discover what more can be done to your web design and layout to improve functionality, performance, and user experience.

    Interested in Usability Hub?

Note: If you’re willing to pay extra for a testing service – which may indeed be the right solution if you plan on doing regular tests – make sure to check in the WordPress Repository for their associated plugins before you sign up.

WordPress Plugins

For those of you who don’t want to make that level of commitment just yet, but still want to be able to run tests without having to update the coding of your website or set everything up from scratch each time, look into the following plugins to get started:

  • WordPress Calls to Action

    This tool is an all-encompassing call-to-action solution for WordPress. If you need help building CTAs on your website, this will guide you through the steps of creating and placing them on your site. In addition, there is an A/B testing feature that displays the details and results of your test in a very clear and easy to understand format.

    Interested in WordPress Calls to Action?

  • WordPress Landing Pages

    This plugin is very similar to (and can be managed in conjunction with) the WordPress Calls to Action plugin mentioned above. Creating landing pages for your website can be a tedious task, so having a tool like this is invaluable. It will guide you through setting up a well-planned out landing page and provide you with testing opportunities within the same interface so you don’t need to recreate the page with the alteration from-scratch.

    Interested in WordPress Landing Pages?

Wrapping Up

A/B testing is often focused on improving the user experience on your website. While that is definitely true, the underlying fact is that you need to do what you can to increase your chances to recover missed opportunities. By identifying and fixing small elements that may be distracting your visitors from taking the desired action, you are actively moving your website and business into a better place.

Your website is an important piece of your business’s identity and is often the place people go to learn more about you before seeking out your services or products. It would be a shame to invest all that time and energy into building this piece of your brand for it to not work properly. Through A/B testing, you can make sure your website—whatever state it’s currently in—is well-received and functioning at the highest level possible.

Image credits: Optimizely.

Now over to you. Have you ever set up an A/B test? How did it go? Feel free to share your thoughts and experiences below.

Brenda Barron

Brenda Barron Brenda is a freelance writer from Southern California. She specializes in WordPress, tech, and business and founded WP Theme Roundups. When not writing about all things, she's spending time with her family.