Skip to main content

Your First A/B Test

In this guide, you will create and implement your first A/B/n test. While you can use Statsig's Feature Gates to roll out new features, Statsig's Experiments enable you to run all kinds of A/B tests, from simple bivariant (A vs. B) experiments to multi-variant experiments ( A vs. B/C/D/n) and mutually exclusive experiments.

This guide is for a user-level experiment

Most experiments randomize the users that are exposed to different variants. We call these user-level experiments. To randomize based on devices, say when you don't yet have a userID (e.g. for unregistered users), consider a Device-level Experiment.

Do I need a large sample size?

This is a common question that comes up in the context of A/B tests. Most assume (incorrectly) that they can't get statistically significant results with small sample sizes. Here's a good article that clears this up:

You don't need large sample sizes to run A/B Tests

Step 1 - Create a new experiment in the Statsig console

Log into the Statsig console at and navigate to Experiments in the left-hand navigation panel.

Click on the Create button and enter the name and description for your experiment. Click Create.


In the Setup tab, define the test groups for this experiment. You can define what percentage of users should receive the group's configured values. Click on the pencil icon next to the sample_parameter and enter the name and type of your experiment parameter, and click Confirm. Enter the values that the parameter will take for the control and test groups.


In the Allocation panel, enter the percentage of users that you want to allocate to this experiment.


Enter the key metrics you want to track in this experiment. While Statsig will show you experiment results for all your metrics, these key metrics represent your hypothesis for the experiment. Establishing a hypothesis upfront ensures that the experiment serves to improve your understanding of users rather than simply serving data points to bolster the case for shipping or not shipping your experiment.
Finally, don't forget to click on the Save button at the top right hand side of the page to complete your experiment setup.


Step 2: Install and Initialize the Statsig SDK

In this guide, we use the Statsig Javascript client SDK. See the Statsig client or server SDKs docs for your desired programming language and technology.

You can install the statsig SDK via npm, yarn or jsdelivr:

<script src=""></script>

via jsdelivr:

Statsig is available from jsdelivr, an open source CDN. We use this installation method for using Statsig on! Go ahead, inspect the page source.

To access the current primary JavaScript bundle, use:

To access specific files/versions:{version}/{file}

After you install the SDK, you must initialize the SDK using an active Client API Key from the API Keys tab under Settings in the Statsig console.

In addition to the Client API key, you must also pass in a Statsig User ID to ensure users can be assigned and tracked for different variants of the experiment.

// Include this if using statsig via npm
// import statsig from 'statsig-js';

await statsig.initialize(
{ userID: 'some_user_id' },
{ environment: { tier: 'staging' } }, // optional, pass options here if needed

Step 3: Check the experiment in your application and log an event

Get your experiment's configuration parameters to enable the user to see the experience you want to create with each variant. In this case, we fetch the value of the shape parameter that we had created earlier.

const expConfig = statsig.getExperiment("onboarding_banner");

const showBanner = expConfig.get("showbanner", false);

Now when a user renders this page in their client application, you will automatically start to see a live log stream in the Statsig console when you navigate to your experiment and select the Diagnostics tab.


At this stage, you may also want to track downstream events to measure how your users are responding to different variants of the experiment. To do this, call the LogEvent API with the key measure that you may want to aggregate as well as other metadata that you may use as additional dimensions to breakdown the aggregate results.

statsig.logEvent('add_to_cart', 'groceries', {'price': '9.99', 'SKU_ID': 'SKU-123'});

As the experiment progresses, you will see how many experiment checks occur and how many events are logged on an hourly basis in the Diagnostics tab.

Step 4: Review experiment results

Within 24 hours of starting your experiment, you'll see the cumulative exposures in the Results tab.


You will also start to see the key results for each test group compared to the control in the Results tab.


In the Metric Lifts panel, you can see the full picture of how all your tagged metrics have shifted in the experiment.


Once you start to see statistically significant results, either green (positive) or red (negative), you're getting ready to make a decision about whether to ship the test variant.

Alternatively, you can choose to ramp up the experiment to a larger user base by increasing the allocation in the Setup tab. We recommend scaling up allocation slowly in the initial stages, and then by a larger margin in later stages, say 0% -> 2% -> 10% -> 50% -> 100%. This helps you get more data and build more confidence before you decide to ship the experiment.