How to find bugs and issues after changing your website

You can’t know in advance the impact of your changes, so it’s crucial to have a plan of action to catch urgent issues (so you can deploy solutions in record time) and discover elements you can fix or re-optimize as soon as possible.

This piece covers a 3-step process to help you stay on top of your website in the first minutes/hours/days after launching changes:

• Step 1: watch Recordings to find and prioritize what needs fixing

• Step 2: review Heatmaps to identify issues on individual pages

• Step 3: investigate WHY something doesn’t work with feedback Surveys

Note: this framework requires you to use specialist software like Hotjar (which you can use for free) or any equivalent set of behavior analytics tools.

Step 1: watch recordings to find and prioritize what needs fixing

The fastest way to figure out if there are bugs on your website is to watch people use it. This is especially helpful if you don’t have large volumes of traffic or have no resources to do in-person usability testing after a new page launch or a complete redesign.

Session recordings (also known as session replays or visitor recordings) are renderings of the actions your visitors took as they browsed through your site—we’re talking mouse movement, clicks, taps, and scrolling across multiple pages on desktop or mobile devices:

Recordings give everybody involved in developing and/or fixing your site a chance to see, and therefore understand, what your visitors are going through. Without recordings, you can’t be aware of all the bugs and glitches that come up, because people don't always report them: they just get frustrated and leave.

How to do it: in Hotjar, set up recordings as soon as possible after launching your new site or releasing page changes. Hotjar will start recording samples of visitor sessions when new people land on your site, so you are able to watch videos fairly quickly.

From the sidebar menu, select Recordings and set them up as you need (here are step-by-step instructions for starting Recordings if you haven’t done it before).

Watch a selection of user journeys across the site

You've launched your new website and started recordings—now, you're ready to click 'Play'. To catch bugs or pages that misbehave post-launch, sort them by length and/or number of pages viewed and watch a mix:

• Shorter recordings might be caused by visitors who encounter an issue and leave immediately after

• Longer recordings help you paint a more accurate picture of the whole journey and potential stumbling blocks that happen between pages

USING FILTERS TO NARROW DOWN RECORDINGS

Depending on how much traffic you get to your website, you might find yourself with tens or even hundreds of recordings in the hour(s) after launch. Based on experience, it takes just about 5-10 recordings for you to get into a good watching flow. To get started:

• Get everyone on your team to watch 20-30 recordings in one go and report back, or

• Watch 30-50 recordings if you are on your own

• Skip pauses and/or watch at increased speed to save some time (see the two buttons below, highlighted in red)

5 common website bugs you can spot with recordings

As you go through recordings, look out for the following bugs:

• Compatibility bugs, where pages load incorrectly on different devices/browsers

• Functionality bugs, where the website doesn’t do something it should (example: users try to login but cannot, or clickable buttons or elements don’t work)

• Layout and design bugs, where elements of a page don’t render correctly

• Unusual mouse activity, such as wild scrolling or repeated clicking, which can indicate visitors are not finding what they need

• Navigation issues, for example 404 pages (to find them, filter by the relevant URL, such https://www.yoursite.com/404. If you find any, following the journey backward will help you see how people got there)

Pro tip: log each bug by tagging recordings; you can later filter by individual tags and re-watch relevant files only. Pick a naming convention ahead of time: some of the ones we’re using at Hotjar include Broken link to contact us, Error, Filters used, Unsuccessful, To investigate, Didn’t seem to find pricing, Verification issue—maybe these can spark ideas for your team, too.

Share findings with your product/dev team

If you and your product/dev team are using a project management system like Jira or Trello, create and add recording links to individual tickets or requests for fixes: this gives whoever is in charge of dealing with the problem the opportunity to see it happen before going in and working on a Javascript or HTML fix.

Step 2: use heatmaps to find issues on individual pages

Recordings let you dive deep into individual journeys across multiple pages, but you’ll also want to understand how your visitors as a whole experience each page—enter heatmaps, which aggregate data from multiple visitors and turn it into colorful visualizations of what people are clicking on, scrolling past, and/or ignoring.

EXAMPLE OF A SCROLL HEATMAP (LEFT) AND A CLICK MAP (RIGHT)

All heatmaps use a color scale to show elements that get the least (blue) to the most (red) interaction—and they are useful when you release substantial changes to your page layout, such as by moving elements around or restructuring page hierarchy. Comparing a heatmap of the page pre- and post-change helps you spot differences in behavior, for example in the way people interact with the main call-to-action.

AN EXAMPLE OF PRE- AND POST-CHANGE HEATMAPS FROM BANNERSNACK THAT IMPROVED CLICK-THROUGH-RATE ON THE HOMEPAGE CTA

In Hotjar, follow a two-step approach:

1) Before you release your changes, set up a heatmap on your most relevant/important pages:

2) After you release your changes, set up a new heatmap on the same page(s). Now that you have heatmaps for both versions of the page, you are able to compare the before-and-after and spot any changes in behavior.

Note: it only takes one person to visit the page for a heatmap to start working, but data based on just one visitor will not be very useful—so remember to give the tool some time to collect a good amount of data (100-200 visits are a good starting point).

Use click heatmaps to see what gets the most interaction

Click and tap heatmaps show where visitors click their cursor on desktop devices and tap on mobile and tablet screens:

The red elements on a click map are the ones that get the most interaction(s). If the reddest elements on your heatmap are not links, you might be creating confusion in visitors who expect to be taken elsewhere when clicking or tapping on them.

Red elements can also help you find out if your visitors’ attention is where you want it to be. If people are ignoring or missing something you’d rather they focused on, you can investigate further and eventually change (or test changing) page elements accordingly.

Editor’s note: back in 2018, we redesigned the Hotjar homepage and included a video where there hadn’t been one before. We also placed heatmaps on the newly redesigned page:

Several desktop and mobile clickmaps proved that the video wasn’t receiving any interaction (see the lack of orange-red color on the ‘play’ button?), wasting valuable page space. We eventually changed the design again—and because of this heatmap, we also removed the video.

Use scroll heatmaps to see how much of the page visitors are viewing

Scroll heatmaps show you how far down the page your visitors scroll:

As with click heatmaps, red indicates the most viewed page portion and blue the least; views are also displayed as a percentage of page visits, showing how many visitors reached a particular point on the page.

If your heatmaps reveal that the majority of visitors don’t scroll far enough to see important information, you can consider moving elements around to make them more visible—and obviously, you can compare your before-and-after change heatmaps to see if the situation used to be better.

If the change in color is sharp rather than gradual, there might be an issue with page design/layout that makes people exit en masse because they think they reached the end of the page and there is no content below.

Check for differences across devices

Session recording software will make you aware of individual glitches or problems on specific devices, but heatmaps help you get a more comprehensive understanding of their impact and show you obvious post-launch inconsistencies that need fixing. For example, after your redesign valuable content that is high on a desktop page could be far lower on a phone, and a scroll map from a visitor using a mobile app would help you understand just how many people actually see it (or don’t).

Some bugs also may only be present with specific browsers, and if you’re getting a substantial amount of traffic from these browsers then such bugs can be damaging. Something as simple as clicking around your website using the different browsers (e.g. Google Chrome, Safari, Firefox, or Internet Explorer for the really old school) can give you an idea of whether or not your website performs well on each browser.

Share heatmaps and findings with your team and/or clients

Because of their visual impact, heatmaps are usually very powerful in helping your product or web development teams and clients quickly understand how the changes you made are performing. You can download heatmaps and/or generate unique URLs to share with them as a link—it’s hard to argue with a heatmap when it shows that your new page design performs better than the old one 😉

FROM OUR CUSTOMERS: THE TEAM @ SKYSCANNER REVIEWING PAGE HEATMAPS

Step 3: investigate WHY something doesn’t work with on-site feedback

In addition to uncovering bugs and broken elements that need fixing in the first few minutes/hours post-launch, you might find that something doesn’t work as intended but you don’t know why. For example, individual recordings might show that people reach a certain web page then do not progress further; a page-specific heatmap might then confirm that these are not isolated incidents, because the ‘next page’ button (or equivalent call-to-action) does not get many clicks.

To know why things are happening, you have to take one extra step. In addition to watching your visitors’ actions, you have to ask for their feedback on the page itself using on-site surveys like this one:

In Hotjar, create an on-site survey and place it on any page(s) you want to investigate; customize the color and appearance to match your brand style:

Hotjar will start showing the survey to visitors as soon as you launch it, but the speed at which you get answers will depend on people’s willingness to submit them. This will give users an opportunity to describe issues in real-time as they happen. If you haven’t received many answers after a few days and you are sure the page has been visited plenty of times, consider changing the position or color of the poll to make it stand out more, and/or re-formulating the question you are asking.

(NOTE: Surveys work better if you have around 1000+ users a day. When you have lower traffic volumes, you will likely benefit from ‘guerilla research’ instead—where you just show the website to a few people and ask them questions directly.)

Ask visitors why they are leaving

If you’ve seen potential customers drop-off at a specific page in your ecommerce flow, set up a survey on that page to investigate what is going on. Asking questions close to the ‘moment of truth’ (i.e. the decision point on whether to continue or abandon), you will understand the context of their decision to leave the page.

You can trigger the survey to appear on exit, and use very direct questions:

• Why are you leaving this page?

• What is unclear about this step/page?

• Before you go: what’s missing from this page?

• Not for you? Tell us why:

Or you can try a less direct approach:

• What would you expect to do after this page?

• Is there something you think should be added to this page?

• If there was ONE thing you could add to this page, what would it be?

When you set up a Survey, you can decide to trigger it at specific points—for example, on exit when people are about to leave a page, or after a specified time delay:

Ask visitors for guidance

If you have spotted a recurring problem or behavior (for example, something is stopping people from continuing down a registration path) but are not sure what the best solution to it is, ask your visitors for guidance. A simple open-ended question allows potential customers to express their concerns, doubt, or objections:

• What would you expect to happen when you click [insert description of clicked element]?

• What would you expect to happen after this page?

• How could this page/section/form/[etc.] be improved?

• What would make you more likely to [insert desired action]?

You don’t need to stick with open-ended questions. You can use a YES/NO approach to give people options and pair them with a follow-up question:

• Is there anything on this page that doesn't work the way you expected it to? Yes/No (If people answer ‘yes’, follow up with: Please specify what)

• Would you prefer to see [X] on this page? Yes/No (if no: why?)

Your 3-step checklist is ready to use: