A/B Test Email Campaigns

How to run controlled A/B tests on email campaigns with Statsig, including audience targeting, exposure logging, and metric attribution best practices.

A/B testing an email campaign and measuring experiment results on downstream product metrics is a common use case for Statsig customers. Email marketing tools often have native A/B testing capabilities but are limited to measuring email open rates or link click rates. These patterns work across most email marketing tools, including Braze, Customer.io, Sendgrid, and Salesforce Marketing Cloud.

Two common patterns used on Statsig are:

1. Do assignments in the marketing automation tool, do analysis in Statsig (requires Statsig WHN)

Use case: The marketing automation tool only measures email open and click-through rates, but you need to measure impact on product usage.

Solution: Use the email automation tool to bucket users into Control or Test groups. Write these assignments to a table in your warehouse. Statsig Warehouse Native uses this table as the assignment list for the experiment and analyzes experiment impact using the full catalog of business and product metrics configured in Statsig.

Go to this guide for more best practices with email experimentation using Warehouse Native.

2. Do assignments and analysis using Statsig

Use case: You want email campaigns to be coordinated with in-app messaging or promotions.

Solution: Call the Statsig SDK to assign each user into Control or Test groups and use that assignment in the email marketing system. For example, Salesforce Marketing Cloud lets you create a custom extension for this. You can perform a bulk evaluation for a list of email addresses selected for a campaign using a Python script and the Statsig Python SDK, or call the SDK one at a time. This works with both Statsig Warehouse Native and Statsig Cloud.

When using a Python script to call the SDK, call either flush() or shutdown() before exiting the script so Statsig logs the exposures. Verify that exposures flushed on the Diagnostics page of your experiment.

Was this helpful?