Running an A/A Test using Sidecar

This guide explains how to use Statsig’s Sidecar to run an A/A test on your product.

Why run an A/A test?

The most common reason to run an A/A test is to validate a new experimentation engine you're integrating with, such as Statsig. For new users, running an A/A test first provides a low-stakes environment to confirm that your metrics are set up correctly and that exposures are flowing through as expected before you start your first real A/B test.

How to run an A/A test

Step 1: Create a new Experiment in Sidecar

Navigate to the page on your website where you want to run an A/A Test. Open the Statsig Sidecar extension and select New Experiment. Fill in the title of your A/A test.

Then, determine the URI filter (for example, All Pages, contains). After you configure the URI, set up the variants.

For the variant 'Control', choose an action. In this example, the action changes the content of an element: specifically the title 'Getting Started is Simple'.

Repeat the step above for the variant 'Test', using the exact same action. Your setup should match the following.

Select Publish to push the experiment to Statsig as a draft.

Step 2: Configure experiment scorecard in Statsig console

After Sidecar pushes the experiment to end users, edit the scorecard for your experiment in the console. Navigate to the console, select the Experiments tab, and open the experiment you just created.

In the Setup tab, fill out the scorecard with the experiment hypothesis and any primary metrics you want to monitor. Statsig shows results for all your metrics, but these key metrics represent your hypothesis. Establishing a hypothesis upfront ensures the experiment improves your understanding of users, rather than providing data points to support a predetermined decision.

In the Allocation and Targeting section, for an A/A test, allocate 100% of users to the experiment while targeting everyone.

Save and push your experiment. Your test is now ready to start measuring metrics associated with the A/A test.

Step 3: Review A/A test results

Within 24 hours of starting your experiment, cumulative exposures appear in the Pulse Results tab of your experiment.

This shows a breakdown of your logged exposures and their distribution. If something looks off, check the Diagnostics tab for more granular, day-by-day exposure breakdowns at both the Checks and User level.

In the Scorecard panel, you can see the full picture of how all your tagged metrics perform.

What to expect:

• Exposures: Confirm that exposures are flowing through as expected from your product. If exposures are missing, use the Diagnostics tab and the Exposure Stream to debug.
• Pulse results: About 5% of your metrics in Pulse should show a statistically significant change, because Statsig’s stats engine uses a 95% confidence interval.

Run your A/A test long enough to reach most of your weekly active users, or at least one week.