Skip to main content

Email AB Testing with Customer.io

Email campaigns are a critical tool for any Marketing team. Finding the best performing Email template is a perfect use-case for an A/B test. Statsig allows you to run simple but powerful A/B tests on different parts of your email content. Since Statsig can integrate seamlessly with product analytics, you can run email experiments and understand deeper business level impact on product metrics easily.

info

This guide assumes you have an existing Statsig account. Please go here to create a new free account if you don't already have one: https://statsig.com/signup

Step 1: Create an experiment

Start by creating a new Experiment on Statsig console. Put in a name and leave the rest of the fields empty/default. For the purposes of this walkthrough, that should do.

image

Step 2: Start the experiment

Since you can't start an experiment without a parameter, let's go ahead and add a dummy parameter.

image

image

Save the experiment setup and Start it. We're all set with the experiment set up.

image

While you're at it, copy the Experiment Name. We'll use this in a bit.

image

Step 3: Set up Exposure Webhooks

In your Customer.io campaign, create a Random Cohort Branch which a flow similar to the following:

image

In the Webhook actions, put in a post request similar to the following, with your api key and experiment name filled in:

image

Pass in any other custom IDs and user attributes inside the post body.

For each webhook, make sure to expose the correct group that you'd like to attribute your branch to. In the above webhook, we are exposing the "Control" group.

image

In Statsig, you'll now have exposures for each of your experiment groups.

Holdouts

To use Statsig Holdouts with Customer.io, it's recommended to identify users that are part of a holdout via customer.io's identify function: https://customer.io/docs/sdk/ios/identify/

Where you call Customer.io's identify method, you could check a Statsig holdout gate, and add an attribute to the user to mark that user as being in a holdout.

In your campaign, you'll be able to create a True/False branch to check whether a user is in the holdout.

image