Create an Experiment

An experiment is a randomized controlled trial: Statsig splits traffic into a control group and one or more test groups, serves each group a different variant, and reports how your metrics moved for each one. Assignment is random but deterministic, so a user sees the same variant every time they're evaluated. Use an experiment when you need evidence that a change caused a metric to move, not just a safe way to ship it. If you want to roll out a single change safely, without measuring variant lift, create a feature gate instead.

For the full order of steps, from sizing the experiment through reading its results, go to Experiment process overview. For a continuous tutorial that also covers integrating the Statsig SDK, go to Run your first experiment.

Size the experiment before you create it. A power analysis tells you how much traffic and how many days it needs, which is the target duration you set during setup.

User-level experiments

To create a user-level experiment:

1. In the Statsig console, go to Experimentation > Experiments.
2. Click Create. The Create New Experiment dialog opens.
3. Enter an Experiment Name. Statsig generates the Experiment ID from the name.
4. Click Add Hypothesis and describe the change and the result you expect.
5. In the Targeting section, click Target Applications and select at least one application. The Create button stays disabled until you select one.
6. Optional: In the Targeting section, click Layer to add the experiment to an existing layer, or choose Create new Layer at the bottom of the list. If you leave Layer empty, the experiment runs in its own layer.
7. Optional: In the Details section, set any of the following, then click Create:
   • Experiment Type: The default is Standard A/B/n. The other options are Switchback Test and Visual Editor. Refer to Switchback tests.
   • Team: Assign the experiment to an internal team.
   • Tags: Add tags to organize and filter your experiments.

To pre-fill the dialog from a saved configuration, click Template at the top of the dialog and choose a template. Refer to Templates.

Device-level and custom ID experiments

The first control in the Targeting section sets the unit that Statsig randomizes. It shows your project's default ID type. To use a different ID type, such as Stable ID for device-level experiments or a custom ID, use the user-level experiment steps with the following change:

1. In the Targeting section, click the ID type button, then choose the ID type you need. The list includes User ID, Stable ID, and any custom ID types defined in your project.
2. Click Create.

You can change the ID type after the experiment is created by clicking the edit icon next to ID Type in the experiment's Details panel.

Then, finish configuring the experiment as you would a user-level experiment.

ID mapping capabilities

You may want to start an experiment with one ID type (such as stableID for device-level targeting) but analyze results using events from another ID type (such as userID for logged-in user metrics).

Cloud: Doesn't support mapping between ID types. Experiments started with stableID analyze only events with stableID, and experiments started with userID analyze only events with userID. Warehouse Native: Supports ID mapping between different identifier types (for example, stableID to userID) through Entity Property Source configuration. For advanced ID mapping requirements, consider Statsig Warehouse Native.

Isolated experiments

To create an experiment that excludes users exposed to other experiments, use the user-level experiment steps with the following change:

1. In the Targeting section, click Layer, then select an existing layer or choose Create new Layer. The list shows each layer's ID type and how much of it is already used, for example (20% used).
2. Click Create.

Then finish configuring the experiment.

Configure the experiment

A newly created experiment isn't running yet. Configure the following on the Setup tab, then start the experiment:

1. Configure the scorecard: Record the hypothesis and choose the primary and secondary metrics that measure it.
2. Set allocation and targeting: Choose how many eligible users enter the experiment and which users qualify.
3. Define groups and parameters: Name the variants and set the values your code reads.
4. Review the analysis settings: Set a target duration and choose the significance level Statsig applies to results.

If you use Statsig exposure forwarding, review the group setup guidance and warehouse forwarding requirements. Each group's traffic share affects forwarding eligibility separately from the experiment's overall allocation.

When the setup is complete, implement the experiment in your application, check it with overrides and the quality score, then start it. Experiment process overview lists every step in order.

Reusing experiment salts

Statsig SDKs use deterministic hashing to bucket users. Statsig buckets the same user identically for the same experiment, regardless of where evaluation occurs. Every experiment has its own bucketing salt so that each experiment's user assignment is random.

You can copy and set the bucketing salts that Statsig uses for deterministic hashing. This capability is useful for advanced use cases, such as a series of related experiments that need to reuse the control and test buckets. Only Project Administrators can copy bucketing salts. To view or set an experiment's salt:

1. Open the experiment and click the ⋯ menu next to Start.
2. Choose Modify Salt. The Experiment Salt dialog shows the current Salt.
3. Enter the new salt and save it. If your project requires reviews, the button reads Submit for Review and the change applies after approval.