To create a user-level experiment,
Log into the Statsig console at https://console.statsig.com/
Navigate to Experiments in the left-hand navigation panel
Click on the Create button
Enter the name and description for your experiment as shown in the figure below
In the Allocation panel, enter the percentage of users that you want to allocate to this experiment. By default, each experiment runs in its own layer and you can enter a value up to 100%
In the Variations panel, enter the variant names and click on Add Another Variant to enter more variants
You'll find that the split percentages between these variants automatically change to evenly distribute users between these variants.
Continuing in the Variations pabel, click on the pencil icon next the sample_parameter and enter the name of your experiment parameter and the type, and click Confirm
Enter the values that the experiment parameter will take for each variant
Enter the key metrics you want to track in this experiment
Click on the Save button at the top right hand side of the page to complete your experiment setup.
Device-level and Custom ID Experiments
The default randomization unit for experiments is User ID. To create an experiment with a different unit ID type, follow steps 1 - 4 above. Then,
Click the ID Type drop down menu and make a selection.
Now follow the remaining steps as described in the previous section to complete your experiment setup.
By default, each experiment runs in its own layer. When you want to create an experiment that excludes any users exposed to other experiments, follow steps 1 -4 in 1 -4 n User-level Experiments above. Then,
- Click the checkbox for Add Layer
- Select an existing layer or create a new layer.
- Click Create
Now follow the remaining steps as described in the top section on this page to complete your experiment setup.
Significance Level Adjustments
By default, Pulse results are displayed with 95% confidence intervals and without Bonferroni correction. This default can be changed during the experiment creation and can also be adjusted in the settings when viewing the results.
- Bonferroni Correction: Select this option to automatically apply the correction in experiments with more than one test group. This reduces the probability of Type I errors (false positives) by adjusting the significance level alpha, which will be divided by the number of test variants in the experiment.
- Default Confidence Interval: The selection will be used by default every time Pulse results are shown for this experiment. Choose lower confidence intervals (e.g.: 80%) when there's higher tolerance for false positives and fast iteration with directional results is preferred over longer/larger experiments with increased certainty.
By default, this is hidden under the advanced configuration of an experiment (at the bottom of the experiment setup page). A targeting gate will restrict the users elegible for the experiment to those who pass the list of conditions definied in the linked Feature Gate. This continues to apply after making a decision on an experiment in the Statsig UI.