Skip to main content

Create an experiment

User-level Experiments

To create a user-level experiment,

  1. Log into the Statsig console at https://console.statsig.com/

  2. Navigate to Experiments in the left-hand navigation panel

  3. Click on the Create button

  4. Enter the name and description for your experiment as shown in the figure below

  5. Click Create

    image

  6. In the Allocation panel, enter the percentage of users that you want to allocate to this experiment. By default, each experiment runs in its own layer and you can enter a value up to 100%

    allocation image

  7. In the Variations panel, enter the variant names and click on Add Another Variant to enter more variants

    image

    You'll find that the split percentages between these variants automatically change to evenly distribute users between these variants.

  8. Continuing in the Variations pabel, click on the pencil icon next the sample_parameter and enter the name of your experiment parameter and the type, and click Confirm

    image

  9. Enter the values that the experiment parameter will take for each variant

    image

  10. Enter the key metrics you want to track in this experiment

    image

  11. Click on the Save button at the top right hand side of the page to complete your experiment setup.

Device-level and Custom ID Experiments

The default randomization unit for experiments is User ID. To create an experiment with a different unit ID type, follow steps 1 - 4 above. Then,

  1. Click the ID Type drop down menu and make a selection.

  2. Click Create

    customID_experiment

Now follow the remaining steps as described in the previous section to complete your experiment setup.

Isolated Experiments

By default, each experiment runs in its own layer. When you want to create an experiment that excludes any users exposed to other experiments, follow steps 1 -4 in 1 -4 n User-level Experiments above. Then,

  1. Click the checkbox for Add Layer
  2. Select an existing layer or create a new layer.
  3. Click Create

image

Now follow the remaining steps as described in the top section on this page to complete your experiment setup.

Significance Level Adjustments

image

By default, Pulse results are displayed with 95% confidence intervals and without Bonferroni correction. This default can be changed during the experiment creation and can also be adjusted in the settings when viewing the results.

  • Bonferroni Correction: Select this option to automatically apply the correction in experiments with more than one test group. This reduces the probability of Type I errors (false positives) by adjusting the significance level alpha, which will be divided by the number of test variants in the experiment.
  • Default Confidence Interval: The selection will be used by default every time Pulse results are shown for this experiment. Choose lower confidence intervals (e.g.: 80%) when there's higher tolerance for false positives and fast iteration with directional results is preferred over longer/larger experiments with increased certainty.

Targeting Gates

By default, this is hidden under the advanced configuration of an experiment (at the bottom of the experiment setup page). A targeting gate will restrict the users elegible for the experiment to those who pass the list of conditions definied in the linked Feature Gate. This continues to apply after making a decision on an experiment in the Statsig UI.