CRO glossary: A/B testing

In an A/B test, half of the users landing on the website will see the original or 'control' version A, and the other half will see a 'variation' B that features a change or group of changes such as a different header, images, call to action, page structure, etc.

This methodology is used to test measurable elements of a website that can affect a visitor’s decision to convert—including page structure, registration and sign-up forms, calls to action. 

After testing enough visitors to have a statistically significant sample size, you’ll know which version converts more users. You will then use the winner of the A/B test as the control in future versions of the website, and any new variations will have to perform even better to become the new control.

Why is A/B testing important?

A/B testing removes gut-feel decision-making and guesswork from CRO: A/B testing lets the users decide, and they vote with their clicks and conversions.

Without the help of A/B testing tools, optimization teams are more likely to make changes based on so-called ‘best practices’ or intuition, or based on the Highest Paid Person’s Opinion (HiPPO).

What’s wrong with that?

• Best practices can lead you in the wrong direction

• Even the best designers, copywriters, and marketers can be biased and wrong when trying to figure out what makes users convert.

• The highest-paid person can be just as blind as everyone else when it comes to figuring out what the market wants. Even if your executives built the business from the ground up, customers can (and will) have different ideas about what's valuable and what isn't. 

A/B testing is NOT about discovering new ideas

Many people think that A/B testing and CRO are the same thing—that CRO is just a series of A/B tests that allow optimizing teams to stumble across new ideas—but that’s missing the point.

A/B testing is the last step in the CRO process, and it’s never about coming up with new ideas. It’s about testing ideas that came about by studying your market and your users. In fact, before you A/B test anything, you need to come up with some evidence-based hypotheses about how to improve user experience and boost conversions.

In other words, before you even start thinking about A/B testing, you need to answer questions like:

• Where are customers dropping off?

• Which website elements do they interact with?

• Which ones do they ignore?

• What are they looking to accomplish and where does your website fall short?

6 steps to come up with A/B testing ideas

1. Do some informal research: look at product reviews and Customer Support feedback to see what customers are saying about your products and brand. Talk to your product designers, sales, and support staff. Ask yourself: what does all this input have in common?

2. Identify where people leave the website: you can use traditional analytics tools (such as Google Analytics) to see where visitors leave your website and complement them with Hotjar’s conversion funnels tool to study this data as well.

3. Determine which page elements visitors engage with: heat maps on your pages can show where users click, scroll, and hover their mouse pointers in aggregate. Look for trends in the way people interact with your most important pages so you can zero in on elements that need preserving vs. changing. Here's how to monitor A/B tests with heat maps.

4. Collect customer feedback: use on-page surveys and feedback widgets to get open-ended feedback about what customers think about your site.

5. Study session recordings: observe individual users (anonymized) as they make their way through your website, so you can see the experience from their perspective and observe what they do right before leaving your site.

6. Run usability testing: usability testing tools give you insight into how real people use your website and allow you to get their direct, spoken feedback about the issues they encounter and the solutions they'd like to see.

It's only after having collected some (or all) of this data that you and your optimization teams will be ready to formulate a hypothesis and start thinking about a testable B variation for your website that addresses the issues and offers potential solutions.