A Guide To Start A/B Testing In 7 Days

In the past few weeks, I went through a training program with an A/B testing consultancy that’s a 3-star partner with Optimizely. If you’ve been around this space, you know Optimizely is one of the leading split testing tools that’s been around since 2009. Prior to this program, I had thought A/B testing was for larger companies with plenty of resources to devote – but it’s totally not the case.

My initial interest in A/B testing came from doing consulting work for other stores with Facebook Ads or PPC. While we could change the copy, photos, display, and targeting options – they were secondary metrics if the down funnel experiences were not optimized. In essence, there’s no point spending more money on traffic if your site is still leaking conversions.

Going through this training, I got a peek inside on how to set up the proper infrastructure so that you can make data-driven experiments to improve the conversions of your store. Some of their clients include Uber, Facebook, Altassian, and plenty of large software companies. There are plenty of learnings that can be applied to an e-commerce store, so here they are.

3 Reasons Why You’re Not Currently A/B Testing

1. Resource Constraints: Testing, researching, and analyzing your data is hard to handle on your own, especially when you are running other aspects of the business.
2. Infrastructure & Tools: Experiment programs require proper setup and monitoring of data to keep iterating on them and improving.
3. Expertise & Culture: Using an A/B testing tool for the first time is like getting airplane keys with no piloting experience – you don’t know where to start.

Understanding A/B Testing At A High Level

A big misconception about A/B testing is that it’s just about changing button colors and hopefully seeing your business double. That couldn’t be further from the truth. It’s much more of an iterative process where you find wins that compound over time that bring long-term results.  There is also no silver bullet that will magically solve all your problems too.

Fundamentally speaking for an e-commerce store, you’re looking to provide a great product and great user experience (UX) when shopping. Often times starting out, you are just trying to find product/market fit where people will actually buy your stuff. As passes on and the product is validated, improving the user experience bring greater returns because it scales at a certain volume.

A/B testing is about experimentation and implementing what works, while learning from losers to create better ideas down the line. There is no loss to having a test variation go down because those are signals you can analyze and apply to the next series of tests. You can only improve your performance with data – not hunches or gut feelings.

Find Your One Key Metric

Part of what makes A/B testing so confusing is the mountain of ideas you can test. Before you get into it, you have to figure out what metrics matter the most to your business that move the needle. For e-commerce stores, it’s obviously checkout conversions. There are limited tweaks you can do for the checkout process when it comes to a platform like Shopify so you’ll have to move slightly up the funnel of folks adding products to the cart.

Once you know your “one key metric” you can base your up funnel or down funnel goals based on this. It’s important to kno because one number does not tell the whole story. For example, if you are seeing a 10% traffic drop for a new test, but seeing 20% increase in conversions at checkout – this test is still a winner even though on the surface it doesn’t look good. You may just be getting more quality traffic and can look into scaling that channel up.

Without having a baseline metric that you are measuring, it’s easy to get lost in the data and lose focus. To give you an idea of what other companies are using as their main metric:

• Uber/Lyft: Weekly Rides
• AirBnB: Nights booked
• Snapchat: Daily Active Users

For example: Is your shipping policy displayed on your homepage? If not, it could be causing people to search the site and dropping off instead of adding what they want to the cart. To verify this, you’ll need to look at behavior flow combined with heat-maps to see if the data supports that hypothesis I just brought up. In the end, you want to have the right data points to make the best decisions. Perhaps this results in 5% more people adding products to their cart. Now you know this is a winner and you could also iterate on this idea – what if we try free shipping? 2-day shipping? How does this affect the add to cart rate?

Setting Up Your A/B Testing Infrastructure

This part usually scares most people away as the investment in tools seems daunting. But you really only need three things to get started to play on the level with some of the biggest companies in the world. Two out of three of them are essentially free and you can install in less than an hour on your site:

1. Google Analytics (Free): Most folks are using Google Analytics which is the industry standard
2. Hotjar (Freemium): You need to see what people are clicking because analytics does not tell you that.
3. Optimiziely Test Results: If you’re just starting, you won’t have any but find out more here if you need help.

That’s all you really need to get started as an e-commerce store. In some ways, it’s less complicated than software companies that have to acquire users to use an application. You just want people to buy something from your store which simplifies goals across the board if you think about it.

You Don’t Need To Test Everything At Once

There’s no need to test everything under the sun at once. A high test rate would be 4-6 ideas per month even for a large company like Uber. Tests require time to get significance, and chances are pretty high that we are not Amazon and can’t get meaningful test results in one day. I’ve seen tests default to running for 2-4 weeks at a time before a decision is made. So there’s no need to put pressure on yourself to change everything at once.

Remember – it’s a long term game and you need to keep at it for months and years to really reap the benefits of A/B testing over the long term and really eclipse your competition. I’m not saying that you can’t find quick wins and must struggle, but like the plane that goes off course by one degree, over 15 hours the gap has become halfway around the world. That’s the mindset you need to approach with A/B testing. Its not a quick win and must be bolted into your business culture, process, and mindset.

Need Help Getting Setup?

If you need some help getting your A/B testing infrastructure and systems setup, happy to walk you through the steps if you can contact us here.