Create an Experiment

Creating an experiment in Statsig sets up a controlled A/B/n test: you split users into groups, show each group a different variant, and measure which variant moves your key metrics. You build and configure experiments in the Statsig console, where you name the experiment, set up a scorecard, and define allocation and targeting. To start, open the Statsig console, go to Experiments, and click Create.

User-level experiments

To create a user-level experiment, follow these steps:

1. Log into the Statsig console at https://console.statsig.com/.
2. Navigate to Experiments in the left-hand navigation panel.
3. Click on the Create button.
4. Enter a name for your experiment.
5. Add the hypothesis you're trying to prove.
6. Select at least one Target Application. The Create button stays disabled until you select a Target Application.
7. By default, your experiment runs in its own Layer. A Layer allows you to manage multiple experiments and feature flags together. To add this experiment to an existing Layer, select Advanced, then click Add Layer. To create a new Layer, select Create New Layer.
8. Click Create.

The creation modal includes several additional controls:

• Template: Pre-fill the experiment's configuration from a saved template. Statsig creates templates from the Save as Template action in the experiment overflow (...) menu.
• Team: Assign the experiment to a team.
• Tags: Add tags to organize and filter your experiments.
• Experiment Type: Under Advanced, select the experiment type. The default is Standard A/B/n. Statsig also supports other types, such as Switchback Tests and A/A tests.

Configure your scorecard

When running an experiment, it’s common to test a specific hypothesis using a set of key metrics. The Scorecard feature makes this easy by letting you enter your hypothesis and select both primary and secondary metrics.

• Primary Metrics are those you expect the experiment to directly impact.
• Secondary Metrics are important to monitor to ensure there are no unintended side effects, but they aren’t the primary focus of your experiment.

The Hypothesis field is a rich-text editor. Use its toolbar to apply headings, bold, italic, underline, strikethrough, links, text color, highlighting, bulleted and numbered lists, blockquotes, images, and code blocks. The field also provides a built-in template that you can fill in:

We believe that [change or feature] for [user segment] will [desired outcome] because [reason or insight]. We will measure success using [primary metric], and monitor [guardrail metrics] to ensure no negative impact.

Configure allocation and targeting

The Allocation and Targeting section is where most of your experiment configuration happens.

Allocation

For Allocation, enter the percentage of users you want to assign to this experiment. You can allocate up to 100% of eligible users. It’s good practice to start with a smaller percentage, verify the experiment’s stability, and then increase the allocation.

You can increase the allocation of your experiment anywhere from 0% to 100% at any time after experiment start. However, you can't decrease allocation without resetting your experiment, because decreasing allocation causes biases in group allocation and pollutes your metric results.

Targeting

To configure Targeting criteria, click to edit the Targeting section. You can either set new targeting criteria or use an existing Feature Gate. Targeting limits the experiment to only the users who meet the defined conditions.

• For basic targeting criteria, creating it through Inline Targeting works well. (Select "Criteria: Everyone" to get started.)
• For more advanced targeting (for example, progressive rollouts) or if you want to maintain targeting criteria when you launch your experiment, reference an existing Feature Gate.

By default, no targeting criteria apply, so your experiment includes all allocated users within the defined Layer or exposed user base.

Configure your groups and parameters

When you configure Groups and Parameters, define your parameters first. These are the variables that control the behavior of the different experiment variants.

• Enter the values the experiment parameter takes for each variant. For more about the difference between Groups and Parameters, go to Groups vs. Parameters.

You can add additional groups by clicking the "+" next to the existing groups. The user allocation automatically adjusts as you add more groups.

You can also name, describe, and add variant images for each group under the Groups section. Only the parameters and values affect what users see; the experiment code doesn't use group names and descriptions.

Device-level and custom ID experiments

By default, experiments randomize users based on User ID. If you need to use a different ID type (for example, device-level), follow steps 1–4 from the "User-level Experiments" section, then:

1. Click the ID Type dropdown menu and choose the ID type you need.
2. Click Create.

Afterward, continue with the same steps described above to finish configuring the experiment.

ID mapping capabilities

When you run experiments, you may want to start with one ID type (like stableID for device-level targeting) but analyze results using events from another ID type (like userID for logged-in user metrics).

Warehouse Native: Supports ID mapping between different identifier types (for example, stableID to userID) through Entity Property Source configuration.

Cloud: Doesn't support mapping between different ID types. Experiments started with stableID only analyze events with stableID, and experiments started with userID only analyze events with userID.

For advanced ID mapping requirements, consider using Statsig Warehouse Native.

Isolated experiments

If you want to create an experiment that excludes users exposed to other experiments, follow steps 1–4 from the "User-level Experiments" section. Then:

1. Select Advanced options.
2. Select an existing Layer or create a new one.
3. Click Create.

Now, complete the rest of the experiment setup as described above.

Reusing experiment salts

Statsig lets you copy and set the salts used for deterministic hashing. This capability is useful for advanced use cases, such as a series of related experiments that need to reuse the control and test buckets. Use this feature carefully; only Project Administrators can access it. It's in the overflow (...) menu in Experiments.

Experiment actions menu

The overflow (...) menu on an experiment's page provides these actions:

• Add Decision Framework: Attach a decision framework that defines success criteria and ship-versus-iterate logic for the experiment.
• Clone: Create a copy of the experiment.
• Rename: Change the experiment's name.
• Save as Template: Save the experiment's configuration as a reusable template, which then appears as a Template option when you create a new experiment.
• Share Link: Copy a link to the experiment.
• Modify Salt: Copy and set the salt used for deterministic hashing. Only Project Administrators can access it. Refer to Reusing experiment salts.
• Export Report: Export a report of the experiment's results. This action is available after the experiment starts.
• Archive: Archive the experiment.
• Delete: Delete the experiment.

Significance level adjustments

By default, Experiment Results display with 95% confidence intervals and without Bonferroni correction. You can customize this during experiment setup or later when viewing results in Experiment Results.

• Bonferroni Correction: Apply this to reduce the risk of false positives in experiments with multiple test groups. Statsig divides the significance level (α) by the number of test variants.
• Default Confidence Interval: Choose a lower confidence interval (for example, 80%) if you prefer faster results with higher tolerance for false positives, or use 95% for greater certainty.

Target duration

Target durations longer than 90 days: By default, Statsig computes Experiment Results for the first 90 days, though the experiment itself can run longer. Before you set a duration beyond 90 days, consider whether results past that period are still relevant, and whether earlier data might already provide the insights you need.

After you set the target, you can track progress against the target duration/exposures in the experiment header. You also receive notifications through email and Slack (if integrated) when you reach the target.

Hypothesis Advisor

Writing good experiment hypotheses is key to a strong experimentation culture. Statsig gives instant feedback on experiment hypotheses, flagging what’s missing. Admins can set custom requirements, which Statsig uses to guide experimenters toward stronger, more complete hypotheses.

This Statsig AI feature is off by default. Enable it from Settings > Experiment > Project > Statsig AI. This settings page is also where you configure any custom requirements you want Hypothesis Advisor to enforce (for example, "Strongly recommend that a validation plan be mentioned").