Create an Experiment

Create an experiment in the Statsig console when you want to compare two or more variants of a feature against your key metrics. If you only need to roll out a single change safely, without measuring variant lift, create a feature gate instead. To create an experiment, you need a name, a hypothesis, and at least one target application. Add at least one primary metric on the Setup tab before you start the experiment.

Creating the experiment registers it. You then configure it on the Setup tab and start it. For the full order of steps, from sizing the experiment through reading its results, go to Experiment process overview. For a continuous tutorial that also covers integrating the Statsig SDK, go to Run your first experiment.

Size the experiment before you create it. A power analysis tells you how much traffic and how many days it needs, which is the target duration you set during setup.

User-level experiments

To create a user-level experiment:

1. In the Statsig console, go to Experimentation > Experiments.
2. Click Create. The Create New Experiment dialog opens.
3. Enter an Experiment Name. Statsig generates the Experiment ID from the name.
4. Click Add Hypothesis and describe the change and the result you expect.
5. In the Targeting section, click Target Applications and select at least one application. The Create button stays disabled until you select one.
6. Optional: In the Targeting section, click Layer to add the experiment to an existing layer, or choose Create new Layer at the bottom of the list. If you leave Layer empty, the experiment runs in its own layer.
7. Optional: In the Details section, set any of the following, then click Create:
   • Experiment Type: The default is Standard A/B/n. The other options are Switchback Test, Visual Editor, and Grid Search. Refer to Switchback tests.
   • Team: Assign the experiment to an internal team.
   • Tags: Add tags to organize and filter your experiments.

To pre-fill the dialog from a saved configuration, click Template at the top of the dialog and choose a template. Refer to Templates.

After you click Create, Statsig opens the experiment's Setup tab.

Device-level and custom ID experiments

The first control in the Targeting section sets the unit that Statsig randomizes. It shows your project's default ID type. To use a different ID type, such as Stable ID for device-level experiments or a custom ID, follow the user-level experiment steps with one change before you click Create:

1. In the Targeting section, click the ID type button, then choose the ID type you need. The list includes User ID, Stable ID, and any custom ID types defined in your project.
2. Click Create.

You can also change it after creation with the edit control next to ID Type in the experiment's Details panel.

Then finish configuring the experiment as you would a user-level experiment.

ID mapping capabilities

You may want to start an experiment with one ID type (such as stableID for device-level targeting) but analyze results using events from another ID type (such as userID for logged-in user metrics).

Warehouse Native: Supports ID mapping between different identifier types (for example, stableID to userID) through Entity Property Source configuration.

Cloud: Doesn't support mapping between ID types. Experiments started with stableID analyze only events with stableID, and experiments started with userID analyze only events with userID.

For advanced ID mapping requirements, consider Statsig Warehouse Native.

Isolated experiments

To create an experiment that excludes users exposed to other experiments, follow the user-level experiment steps with one change before you click Create:

1. In the Targeting section, click Layer, then select an existing layer or choose Create new Layer. The list shows each layer's ID type and how much of it is already used, for example (20% used).
2. Click Create.

Then finish configuring the experiment.

Configure the experiment

A newly created experiment isn't running yet. Configure the following on the Setup tab, then start the experiment:

1. Configure the scorecard: record the hypothesis and choose the primary and secondary metrics that measure it.
2. Set allocation and targeting: choose how many eligible users enter the experiment and which users qualify.
3. Define groups and parameters: name the variants and set the values your code reads.
4. Review the analysis settings: set a target duration and choose the significance level Statsig applies to results.

When the setup is complete, implement the experiment in your application, check it with overrides and the quality score, then start it. Experiment process overview lists every step in order.

Reusing experiment salts

Statsig SDKs use deterministic hashing to bucket users. Statsig buckets the same user identically for the same experiment, regardless of where evaluation occurs. Every experiment has its own bucketing salt so that each experiment's user assignment is random.

You can copy and set the bucketing salts that Statsig uses for deterministic hashing. This capability is useful for advanced use cases, such as a series of related experiments that need to reuse the control and test buckets. Only Project Administrators can copy bucketing salts. To view or set an experiment's salt:

1. Open the experiment and click the ⋯ menu next to Start.
2. Choose Modify Salt. The Experiment Salt dialog shows the current Salt.
3. Enter the new salt and save it. If your project requires reviews, the button reads Submit for Review and the change applies after approval.