Running Email Experiments

Run email experiments in Statsig Warehouse Native by mapping send and open events from your warehouse to assignment sources and metric definitions.

Email marketing experimentation is a common use case that Statsig Warehouse Native handles by analyzing the downstream outcomes of an email experiment.

This guide covers the key considerations for configuring this type of experiment.

Data access offered by your marketing platform

Does the platform offer the ability to ETL email engagement events (sends, deliveries, email opens) to your data warehouse either directly, or through a CDP or other pipeline?

Data access may be critical to the viability and quality of your experiment, depending on your specific use case. In particular, email open events are useful for controlling the population of users Statsig counts for analysis.

For example, if you send 20,000 emails and only 12,000 are delivered and opened, counting only the 12,000 email-open users in the test population (rather than the entire recipient list) avoids diluting the population and achieves more statistical power.

Use email open events to manually create an assignment source for an Analyze Only experiment, or to filter your analysis population for an Analyze Only experiment.

Configuring scorecard metrics

What are your key down-funnel (for example, on-site or in-app) success metrics for the experiment, and can you unify email recipients with their subsequent down-funnel interactions?

Email open events are useful for filtering the test population or for monitoring purposes, but aren't typically a primary metric unless the experiment is testing different subject lines.

Consider which down-funnel metrics to include as primary metrics for deciding on your experiment. Attributing down-funnel interactions that occur before user login can be complex, but ID resolution in Statsig solves this.

Down-funnel metrics that occur after login are easy to attribute to the experiment, because you'll have the user ID and email address to map directly to the identifiers on your assignment source.

Designing test groups and running the test

What options does your email platform offer for defining test and control groups? Does the platform require you to upload a list of users for each group, or does it natively support A/B cohort definitions?

If your platform offers native A/B split capabilities, configure your test as an Analyze Only experiment in Statsig. Select Analyze and leave the Experiment Completed checkbox unchecked. Statsig then continues computing results from the first exposure time until you decide to conclude the experiment.

If you need to manually create the test groups, set up an Assign and Analyze experiment and use Statsig's SDKs to generate your test and control lists. Loop over all intended recipients in code and call getExperiment. This captures each test group assignment and automatically writes the assignment events to your data warehouse through the Statsig managed exposure pipeline. Select Start before running your assignments script, then keep the test in a running state until you decide to conclude the experiment.

Was this helpful?