Running an A/A Test

An A/A test splits traffic into two groups that receive the same experience, so you can confirm that exposures flow into Statsig and that your metrics compute correctly before you run a real A/B test. Because neither group changes the product, the test is a low-stakes way to check your setup. Run an A/A test before your first A/B test, or after you change metric definitions. This tutorial creates and implements an A/A test as a feature gate in Statsig. Statsig also runs simulated A/A tests for you every day without any code on your side; use those results when you want to validate the stats engine itself rather than your instrumentation.

You need a Statsig account and the Statsig client SDK integrated into an existing application.

Step 1: Create a feature gate in the console

The easiest way to run an A/A test in Statsig is with a feature gate. You can also use an experiment, but this tutorial uses a feature gate.

Log into the Statsig console and select Feature Gates in the left-hand navigation panel.

Click Create and enter a name and optional description for your feature gate. This example calls the feature gate aatest_example. Click Create.

On the Setup tab, define the rules for this feature gate. Click + Add New Rule. You can run an A/A test on a specific user group or platform, but this tutorial divides all traffic 50/50 and delivers the same experience to each group.

Under Criteria, select Everyone (you may need to scroll up), name your rule, and set the Pass Percentage to 50%. Click Add Rule, then click Save Changes in the upper right-hand corner.

Your feature gate setup now looks like this:

Verify that the gate works by entering some sample user IDs in the console. Roughly 50% of the IDs should pass and 50% should fail.

Step 2: Check the feature gate in your code

Copy the code snippet from the upper right corner of your feature gate page (under the < > symbol) and add it to your application at the point where you want to call the A/A check.

statsig.checkGate("aatest_example") 

When a user renders this page in their client application, a live log stream appears in the Statsig console on the Diagnostics tab of your feature gate.

Step 3: Review A/A test results

Within 24 hours of the gate going live, cumulative exposures appear on the Pulse Results tab of your feature gate.

The Pulse Results tab shows your logged exposures and their distribution. If the distribution looks incorrect, check the Diagnostics tab for day-by-day exposure breakdowns at both the check and user level.

The Metric Lifts panel shows how all your tagged metrics perform.

Check two things in the results:

• Exposures: Confirm that exposures flow through from your product as expected. If you don't see exposures, use the Diagnostics tab and the Exposure Stream to debug.
• Pulse results: Roughly 5% of your metrics in Pulse should show a statistically significant change, because Statsig's stats engine uses a 95% confidence interval.

Run your A/A test for at least a week, and long enough to reach most of your weekly active users.

Simulated A/A tests Statsig runs for you

Statsig runs simulated A/A tests every day in the background for every company on the platform. Like the test in this tutorial, a simulated A/A test has the structure of an A/B test, but both groups receive the same experience.

A/A tests can be online or offline. An online A/A test, like the one in this tutorial, runs on real users: your app checks a gate or experiment through the Statsig SDK, Statsig logs the assignment, and users see no difference in experience. Because there is no real effect, you expect to see only statistical noise. With 95% confidence intervals, only about 1 in 20 metrics shows a stat-sig difference between control and test.

Offline A/A tests

An offline A/A test runs on one unit type per request and works like this:

1. Query a representative sample of your data.
2. Randomly assign subjects to test or control.
3. Compute the relevant metrics for test and control and run them through the stats engine.
4. Measure the percentage of false positives. With a typical p-value cutoff of 0.05, expect a false positive rate of about 5%.

Statsig runs 100 tests per request. Download the running history of your simulated A/A test performance from the Tools menu in the Statsig console.

Simulated A/A test file columns