Overview
This article will walk you through creating an A/B test for your order protection program. The goal is to better understand how customers respond to your pricing and displays throughout the website. Once a campaign is completed, the winning settings can be applied across your program.
Get Started
To begin creating your A/B testing campaign, go to: Analytics > A/B Testing
When you first land on the A/B Testing page, we provide an overview of your current and past campaigns. From here there are a few options:
Active - Any campaigns with the "Active" campaign represent an campaign currently in progress. To review the current results click "View results" to be taken to the analytics page. Alternatively, if you'd like to cancel the campaign early click "Mark as completed".
Paused - Any campaigns with the "Paused" campaign represent a draft or a campaign that has not started yet due to setting changes. Once you're ready to get started, click "Edit" and update the settings to be "Active".
Note: Only 1 campaign can be active at a time.
Completed - Any campaigns which have completed their time limit or have been marked as completed manually will show here. These campaigns cannot be started again.
Create Test
To get started and create a new Test click the button on the top right. You'll now land into the Campaign settings page. Your existing program values can be found on the right side under "Control program values".
General Settings
Test Name - internal name for your campaign
Test Duration - how long the A/B testing campaign should run
Test Groups
This section indicates the different protection values to test in your campaign against your existing account settings (control %). Let's say for example, your default account setting is 3% for Domestic protection, you may wany to set the Test Group A to 3.5% to see how customers respond to a higher % cost.
Flat % or Dynamic - this controls if the % fee is based on the total cart as a single value, or is it a dynamic % based on the cart total. By going dynamic, you can test different values.
Display Opt-out popup - this option will display the opt-out modal if the customer initially adds the protection to their cart and later removes it. The goal of the modal is to encourage the customer to maintain the protection in their order. This modal design can be customized under: Display > Cart Widget.
Give bonus store credit for opt-in - this option will give the customer store credit on their Shopify store account if they decide to add protection to their order. This is a fixed value and is awarded to the customer after the order has been fulfilled.
Give bonus loyalty points for opt-in - we've partnered with Rivo to give your customers the opportunity to earn loyalty points in your program for adding protection to their order. This is option can be used in conjunction with store credit, or stand alone. The Rivo integration must be connected to use this feature (general settings > integrations).
Group display percentage
The final step in your campaign is to specify what % of customers should see your control group versus your test groups. By default we recommend starting with a 60/40 approach, however if you want to heavily push your test groups you can assign a larger number such as: 30% control / 70% group A.
Once you're ready to launch your new campaign, select "Active" from the Status dropdown and click Save at the top.
A/B Test Results
Once a campaign has been running for a few days into your campaign duration, we recommend to review the results page to see how customers are interacting with your various test groups.
The goal of the results page is to give a complete overview of how well the Groups are performing against your control. When first entering the page there are various charts at the top:
Test Duration - how long the campaign is set to last.
Total Views - how many unique customer views saw one of your groups.
CVR Winner - this value provides the conversion rate of customers who placed an order after viewing the a/b test group widget in their cart. This doesn’t mean they purchased protection, but rather if an actual order event occurred. We will record the “attach rate” on the next performance box.
Attach Winner - this measures which Group / Control led more customers to add the order protection to their cart.
Revenue Winner - this measures which Group / Control contributed more revenue to your company.
At the bottom of the reporting page is an complete overview of each Group / Control and it's underlying values and $ generated. This allows you to see the raw data and winners within each underlying category.
Campaign Completion
Once the campaign has finished it's scheduled duration or if you decide to stop it early you can now apply the winning settings to your account.
At the top right of the results page, click "Apply Results" > Select the Group and you're done.