Build your first Shopify A/B test

If you're interested in A/B testing on your Shopify site, there are two steps you will need to take.

1. Forward event data to Statsig so you can analyze the impact of new features on your PDP to Add to Cart and Purchase funnel
2. Integrate the Statsig javascript SDK to bucket users into you different test variants

To get data into Statsig, the easiest way is to use an existing "Data Connector" integration. If you use Segment or RudderStack, for instance, you can set up the Statsig integration to get your events flowing in. See the integrations docs for a complete list of available data connectors.

Next, you'll need to hook your site up to the statsig-js sdk. In the remainder of this guide, we'll set up an A/B test to optimize the landing page hero banner of our Shopify store. Our hypothesis is that optimizing the hero banner will engage new visitors and returning buyers in a way that leads them to buy more from the store. We'll measure this using metrics that include 'add to cart' and 'purchase event' from buyers. Let's dive in!

Step 1: Set up your Shopify Store with the test variants

Here's a detailed guide to creating your Shopify store, adding the products you want to sell, and picking a theme. To follow along with this guide, you'll want to use a Dawn-based theme for your store.

Once your store is up and running, log into your Shopify store as an admin. You might already have done this and here's just a quick recap so we're all on the same page:

• Log into https://partners.shopify.com/
• Click on Stores in the left navigation menu
• Click on Log in link next to the store where you want to run the A/B test

Here's my store on the Shopify Partners portal. It's called Fall Colors, where I sell photographs that I've taken in the fall season when everything's so colorful and beautiful. My store happens to still be in development stage.

• On the admin landing page, click on Customize Theme tab and click the Customize theme button shown below

• Click on the Customize button

• From the left hand navigation menu, click on Image banner

• Under the First Image, click Select Image button

• Select the hero banner image that you want to use as control from the library or upload a new image

• Repeat the previous two steps for the Second Image as well
• At this point, you should see both images in your hero banner on the store

Step 2: Create an experiment on Statsig

To create an experiment on Statsig,

• Log into https://console.statsig.com/
• Click on Experiments from the left hand navigation menu
• Click on Create, then enter the name and description of the experiment
• In the experiment Setup tab, enter the names of the variants that you want to test in the Groups panel; in this case I'm just calling them Control and Test.
• Staying in the Groups panel, click on sample_parameter and enter a parameter name of your choice; in this case, I'm calling the parameter image_selector. Enter the parameter values that you want to use; in this case, I simply want to toggle between the two images that we've already uploaded to the store in the previous step

Step 3: Customize your theme

After you log into your store in step 1, click on Themes under the Online Store in the left hand navigation panel. Then,

• Click on the Actions drop down, then click on Edit Code
• Click on theme.liquid file under the Layout section
• Load the SDK by copying the code the below in the head section; use the experiment config to show the right image to your visitors

<script src="https://cdn.jsdelivr.net/npm/statsig-js/build/statsig-prod-web-sdk.min.js"></script>

• Initialize the SDK and get the experiment config by copying the following in the body section

<script>
    // hide the page until we fetch the configuration from statsig
    document.getElementsByTagName("BODY")[0].style.display = "none";
    
    //initialize the Statsig SDK using your Statsig client SDK key 
    window.statsig.initialize("<CLIENT_SDK_KEY>").then(() => {
        // Assuming you're on the Dawn theme and you've uploaded two images for the hero image
        const heroImages = document.querySelectorAll(".banner > .banner__media");
        if (heroImages.length == 2) {
            heroImages[0].classList.remove("banner__media-half");
            heroImages[1].classList.remove("banner__media-half");
            
            //get experiment config, control (default) is 0
            const switchImage = statsig.getExperiment("shopify_hero_image_abtest").get("image_selector", false)

            // swap the images if the user is assigned to treatment  
            if (switchImage){
              const primaryHTML = heroImages[0].innerHTML;
              const primaryOuter = heroImages[0].outerHTML;
              
              heroImages[0].innerHTML = heroImages[1].innerHTML;
              heroImages[0].outerHTML = heroImages[1].outerHTML;
              
              heroImages[1].innerHTML = primaryHTML;
              heroImages[1].outerHTML = primaryOuter;
            }
        }
        // show the page
        document.getElementsByTagName("BODY")[0].style.display = "block";                                         
    });
</script>

• Log the key events that you want to track as metrics (along with any metadata) by copying the code below into the pages where customers take key actions

statsig.logEvent('purchase_event', 'SKU_12345', {'price': '9.99', 'item_name': 'diet_coke_48_pack'});

• Click Save after you make these changes

Step 4: Verify your experiment

Make sure that your store visitors are being assigned to control and test groups using the Diagnostics tab in your experiment.

As visitors hit your store's landing page, you should start to see ongoing experiment checks in the Checks panel as shown below.

As your visitors engage with products on your store, you'll also start to see the hourly event count for the events you're logging from your store in the Event Count by Group panel.

Most importantly, you can see a live event stream in the Exposure Stream panel in case you're trying to validate if your store application is running the experiment checks and sending events to Statsig as expected.

Step 5: Review experiment results

As the experiment runs, you'll see cumulative exposures in the Results tab.

The Metric Lifts panel will show you which of your metrics are seeing a statistically significant change based on your experiment.

In this case, the new test image is performing worse than the control on nearly all the metrics that I care about. For example, product views are down by a little more than 0.3%, purchases are down by about 0.4%, and daily active users (DAU) are down about 0.2%. We have statistically significant data to make a decision on this experiment: we don't want to launch the new test hero banner image.