How an A/B test reveals the revenue you’re missing
Someone recently asked me to explain a bit more about A/B testing. How do we set one up? Which module do we use? What do you need to get started? Happily, we’re running A/B tests for our clients more and more often. They’re a great way to put your optimizations head to head and find out how much revenue you’re missing!
A quick recap
In an A/B test, you compare two variations of something, such as a design change or a feature, to see which one is more efficient or converts better. Users are split into two groups, A and B, and each group sees its own version. This is handled with cookies.
Benefits
- Measure, don’t guess
- Conversion-focused
- Optimize your store
- Data-driven
Collecting data
Sound conclusions take time. After just 3 user visits, we can’t say what works better or worse. You’ll often hear the term significant data in this context. Put simply, it means we keep collecting data until we can draw real conclusions that aren’t down to chance.
Hypothesis
An A/B test isn’t the only way to optimize. Optimization starts with reviewing your own data and applying any relevant “best practices.” Only then is it time to set up an A/B test. We always begin by asking what you’d like to test. What are you curious about, and what question should the test answer? That question becomes your assumption, or hypothesis. Here’s an example: “All key buttons and price summaries are consistent throughout the site, except in the cart and checkout. If users can see the price summary sooner in the cart, they’ll move in the right direction more quickly. Recognition reduces hesitation, so users will convert faster.”
What do you want to find out?
The example above focuses on conversion: how does the change affect it? Of course, there’s more you can measure in an A/B test, such as click-through rate, downloads, or page views. Always keep in mind that the learnings from an A/B test are there to help you communicate better with your end users. That’s why we want clear metrics. You also decide whether the change applies only to mobile or to every device (mobile, tablet, and desktop), which we configure as part of the test. And it bears repeating: you need significant data before you roll out the change.
The module
Many small optimizations can be handled perfectly well with a tool such as Google Optimize. Modules are also available, from Amasty for example. We always use our own A/B testing module, developed by Cream, which lets us set up and run multiple A/B tests. Within each test, you can also choose what percentage of visitors will see a variation. The module adds a grid to your Magento installation, where we enter the different tests and connect them to Google Analytics for tracking and measurement. Our module records the results of every test and compiles them into a report that shows whether an outcome is significant. That way, you can clearly see which optimizations were the right ones.
No-brainers?
There’s often debate about whether you should test everything. Do you really need to? Aren’t there any no-brainers? Reading patterns are generally well known, and there’s plenty of UX feedback showing, for example, that users expect a button when an action is involved. In my opinion, there are indeed no-brainers you can implement right away, although of course everything depends on context. The classic example is that green buttons convert better than red ones. But if your entire site is green, does that still hold? What works well for one store won’t necessarily work for another. It all depends on context. Small changes that have already proven to be “low-hanging fruit” for other stores are still worth a quick test. For major functional changes, test thoroughly and for longer.
Recommendation
When a test is complete, we use its data to make a recommendation as your e-commerce partner, backed by a report with all the results.
Interested?
Want to know how much revenue you’re missing? Interested in running a test of your own? Feel free to get in touch.
