Running an A/A Test

Learn how to run A/A tests to validate your experimentation setup and ensure proper metrics configuration.

An A/A test splits traffic into two groups that receive the same experience, so you can confirm that exposures flow into Statsig and that your metrics compute correctly before you run a real A/B test. Because neither group changes the product, the test is a low-stakes way to check your setup. Run an A/A test before your first A/B test, or after you change metric definitions. This tutorial creates and implements an A/A test as a feature gate in Statsig. Statsig also runs simulated A/A tests for you every day without any code on your side; use those results when you want to validate the stats engine itself rather than your instrumentation.

You need a Statsig account and the Statsig client SDK integrated into an existing application.

Step 1: Create a feature gate in the console

The easiest way to run an A/A test in Statsig is with a feature gate. You can also use an experiment, but this tutorial uses a feature gate.

Log into the Statsig console and select Feature Management > Feature Gates in the left-hand navigation panel.

Click Create and enter a name and optional description for your feature gate. This example calls the feature gate aatest_example. Click Create.

Create New Feature Gate dialog with the name aatest_example

On the Setup tab, define the rules for this feature gate. Click Add New Rule. You can run an A/A test on a specific user group or platform, but this tutorial divides all traffic 50/50 and delivers the same experience to each group.

The new rule opens in the rules list. Name the rule, keep Criteria set to Everyone, and set Pass to 50%. Then save your changes at the bottom of the page, or submit them for review if your project requires reviews.

Gate rule named A/A test with Everyone criteria and a 50% pass split

Verify that the gate works by entering some sample user IDs in the console. Roughly 50% of the IDs should pass and 50% should fail.

Gate check showing a sample user ID passing

Gate check showing a sample user ID failing

Step 2: Check the feature gate in your code

Copy the code snippet from the upper right corner of your feature gate page (under the < > symbol) and add it to your application at the point where you want to call the A/A check.

jsx
statsig.checkGate("aatest_example") 

When a user renders this page in their client application, a live log stream appears in the Statsig console on the Diagnostics tab of your feature gate.

Diagnostics log stream showing live gate checks

Step 3: Review A/A test results

Within 24 hours of the gate going live, cumulative exposures appear on the Metric Impact tab of your feature gate.

Exposures start on the third day with a launch burst, then climb steadily. After a month the gate has logged about 65,000 exposures, split almost exactly 50/50 between Pass and Fail.
Cumulative exposures by gate result
DateFailPass
Mar 10, 202600
Mar 11, 202600
Mar 12, 20267,9007,850
Mar 13, 20268,7238,668
Mar 14, 20269,2279,212
Mar 15, 20269,7409,740
Mar 16, 202610,77410,942
Mar 17, 202611,77112,006
Mar 18, 202612,70513,026
Mar 19, 202613,56713,947
Mar 20, 202614,41114,766
Mar 21, 202614,92215,265
Mar 22, 202615,50015,777
Mar 23, 202616,63716,925
Mar 24, 202617,71017,899
Mar 25, 202618,68518,880
Mar 26, 202619,55919,827
Mar 27, 202620,46120,641
Mar 28, 202621,07121,158
Mar 29, 202621,66121,643
Mar 30, 202622,72922,803
Mar 31, 202623,63723,760
Apr 1, 202624,57124,612
Apr 2, 202625,50525,590
Apr 3, 202626,42926,377
Apr 4, 202627,03026,898
Apr 5, 202627,50927,382
Apr 6, 202628,51928,484
Apr 7, 202629,43229,390
Apr 8, 202630,30530,204
Apr 9, 202631,12031,082
Apr 10, 202631,99531,905
Apr 11, 202632,50032,450

The Metric Impact tab shows your logged exposures and their distribution. In a healthy A/A test, Pass and Fail each hold close to 50% of exposures. If the distribution looks incorrect, check the Diagnostics tab for day-by-day exposure breakdowns at both the check and user level.

The Metric Lifts panel shows how all your tagged metrics perform.

Pulse results with metric lifts for an A/A test

Check two things in the results:

  • Exposures: Confirm that exposures flow through from your product as expected. If you don't see exposures, use the Diagnostics tab and the Exposure Stream to debug.
  • Metric results: Roughly 5% of your metrics should show a statistically significant change, because Statsig's stats engine uses a 95% confidence interval.

Run your A/A test for at least a week, and long enough to reach most of your weekly active users.

Simulated A/A tests Statsig runs for you

Statsig runs simulated A/A tests every day in the background for every company on the platform. Like the test in this tutorial, a simulated A/A test has the structure of an A/B test, but both groups receive the same experience.

A/A tests can be online or offline. An online A/A test, like the one in this tutorial, runs on real users: your app checks a gate or experiment through the Statsig SDK, Statsig logs the assignment, and users see no difference in experience. Because there is no real effect, you expect to see only statistical noise. With 95% confidence intervals, only about 1 in 20 metrics shows a stat-sig difference between control and test.

Offline A/A tests

An offline A/A test runs on one unit type per request and works like this:

  1. Query a representative sample of your data.
  2. Randomly assign subjects to test or control.
  3. Compute the relevant metrics for test and control and run them through the stats engine.
  4. Measure the percentage of false positives. With a typical p-value cutoff of 0.05, expect a false positive rate of about 5%.

Statsig runs 100 tests per request. To export the running history of your simulated A/A test performance, open the Help menu (?) in the Statsig console navigation and select Automated A/A Testing. When the export finishes, download it from Settings > Reports, where it appears with the export type A/A Test Summary.

Simulated A/A test file columns

Was this helpful?

For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.

On this page