CRO glossary: split testing

A split test distributes website traffic between two different versions of a webpage—the original or baseline (version A) and a variation (version B)—which differ from each other in terms of design, content structure, page elements, etc. Observing how traffic groups respond to the version they're exposed to helps marketing and optimization teams determine which version offers the greatest conversion rate and opportunities for business growth.

Split testing vs. A/B testing

The term ‘split testing’ is often used interchangeably with A/B testing. The difference is simply one of emphasis:

• A/B refers to the two web pages or website variations that are competing against each other

• Split refers to the fact that the traffic is equally split between the existing variations 

Like A/B testing, split testing can evaluate small changes to a single website element (such as a different image, header, call to action, button color, signup form, etc.) or be run between two completely different styles of design and web content.

All available users will be split into groups (without their knowledge) and half of them will see the original version (the control) while the other half will see a new version (the variation). Split tests are typically conducted on landing pages or product pages (if you’re an ecommerce company), though you can split test any page on your website. Once the test has reached a statistically significant sample size, the design and optimization team will investigate differences in behavior and declare a winner (or an inconclusive test result if no measurable differences are obtained).

Why is split testing important?

Like A/B testing, split or multivariate testing ensures that decisions aren’t made by gut feel or guesswork. 

Without split testing, companies often make changes based on so-called ‘best practices’ or based on the Highest Paid Person’s Opinion (HiPPO). But best practices can kill conversions because, by definition, they’re based on what worked in the past for others (as in they still can't guarantee that something that worked elsewhere will work for your business). And of course, the highest-paid person’s opinion can be just as flawed as anyone else’s. 

Even the most experienced marketers, designers, and copywriters can be wrong when trying to figure out what users will respond to. Split testing lets the users decide, and can prevent a conversion optimization team from going down a dead end.

Split testing is NOT about discovering new ideas

It’s easy to mistake split testing for CRO (conversion rate optimization)—imagining that CRO is just an ongoing series of split tests and A/B tests, helping you stumble across new ideas—but that’s not how it works.

Before you split test anything, you must come up with evidence-based hypotheses about how to improve your user experience and, in the end, boost conversions. Split testing is about exploring designs and solutions based on what you’ve learned studying your users and markets, and collecting answers to questions like: 

• Which website elements do users currently interact with?  What is the click-through-rate for action buttons?

• Which elements do they ignore?

• Which Unique Selling Propositions do customers respond to?

• What are they looking to accomplish and how can you help them get there?

6 steps to form split test hypotheses

1. Conduct informal research: take a look at what customers say about your brand and your products. Explore customer reviews, speak with your product designers, sales, and support staff. Look at metrics for your marketing campaigns. Find the common themes in these different sources of feedback.

2. See where users leave your website: use general analytics tools (such as Google Analytics) to see where users drop off, where they get stuck, and what small changes might remove barriers and increase conversions.

3. Find out which page elements users interact with: heatmaps can show you where large groups of users click, scroll, and hover their mouse pointers.

4. Gather customer feedback: use customer polls, on-page surveys and open-ended feedback to get direct feedback from your website visitors and customers.

5. Study individual session recordings: observe individual users as they navigate your website, and watch those recordings keeping in mind the feedback users have left. For example, if customers tell you in surveys they don’t understand your signup page, watch several sessions of users interacting with the page to get a better sense of what that behavior looks like on the page. 

6. Conduct usability testing: usability testing tools allow you to observe real people using your website so you can create an easy, frictionless user experience.