Make a Decision

Making a decision for an experiment ships the winning group to all your users. After you make the decision, the variant that users see depends on whether you're using a targeting gate for your experiment. Experiment results remain accessible after you make a decision, but they stop updating. The last day of metrics is the day you make a decision on the experiment.

Experiments with no targeting gate

When you ship a group in an experiment with no targeting gate, the parameter values from the shipped group become the default values for all your users going forward.

If the experiment uses parameters from a layer, the layer's parameters take on the shipped group's parameter values as their defaults. These are the values that all your users see going forward.

For example, suppose you have a Demo Layer configured with a parameter, a_param. Its default value is set to layer_default as shown below.

To run an experiment in Demo Layer, create Demo Experiment as shown below.

You set up Demo Experiment with two groups: Control and Test, intending to experiment with new values for the layer-level parameter, a_param as shown below.

Now if you decide to ship the Control group for the Demo Experiment, a_param takes the value set for the Control group as its default: experiment_one_control

On the other hand, if you decide to ship the Test group, a_param takes the value set for the Test group as its default: experiment_two_test

Experiment with a targeting gate or targeting rules

When you ship a group in an experiment configured with a targeting gate or targeting rules, you can decide whether to continue targeting after shipping.

• If you discontinue targeting, the parameter values from the shipped group become the default values for all your users going forward. If the experiment uses parameters from a layer, the layer's parameters take on the shipped group's parameter values as their defaults.
• If you decide to continue targeting with a targeting gate, Statsig adds an override to the experiment layer so that:
  • all users who pass the targeting gate see the shipped group's parameter values
  • all users who fail the targeting gate see the default value (layer-level parameter defaults or the defaults you set for the parameter in your code)
• If you decide to continue targeting with targeting rules, the experiment is only shipped to users who pass the inline targeting rules as set in the experiment setup.

For example, suppose Demo Experiment is in Demo Layer, which has a parameter called targeted_layer_param with a default value of targeted_layer_default_value.

When you ship Demo Experiment and discontinue targeting, targeted_layer_param takes on the value from the Control group, targeted_layer_control, as its default.

If you continue targeting, targeted_layer_param acquires an override so that:

• all users who pass the targeting gate see the overridden value of the parameter
• all users who fail the targeting gate see the default value of the parameter

In this case, the default value of targeted_layer_param in Demo Layer doesn't change. Users who pass the targeting gate aren't eligible for future experiments run in this layer. Shipping experiments with targeting on is not recommended, especially when the experiment is in a layer.

Rolling out an experiment group

Rolling out an experiment group is an option when you have decided the winning variant but want to avoid a sudden, large shift of traffic into the winning variant group. Use automated rollouts to schedule gradual rollout phases, which increase your shipped group size to the target percentage by reallocating users from all other groups proportionally.

Setting up rollouts

To set up rollouts, open the make decision form and select the winning group. From here you can use automated rollouts or ship with rollout.

The ship with rollout option immediately updates the shipped group size. Manual rollouts clear any automated rollout phases.

Alternatively, you can set up automated rollouts, which opens the following dialog to populate with rollout phases:

Configure each phase of your Scheduled Rollout. To add phases to your rollout, select Add Phase and configure as many phases as needed.

Each scheduled rollout phase includes:

• Rollout date
• Rollout time*
• Pass percentage

After saving, you can preview the rollout and commit the schedule:

Resizing logic

During each phase, the rollout group is resized to the desired percentage, and all other groups are scaled proportionally in the following way:

You have n groups with sizes:

G₁, G₂, ..., Gₙ

Their total sum is:

100 = G₁ + G₂ + ... + Gₙ.

Now, suppose you set one group (say the k-th group) to a new size Gₖ′. Let

Delta = Gₖ′ − Gₖ.

Because the grand total must remain 100, you need to adjust the remaining groups proportionally. Let

T = 100 - Gₖ

Then, for each group i ≠ k,

Gᵢ′ = Gᵢ − (Delta × Gᵢ / T).

In other words, each group other than k is decreased (or increased, if Delta < 0) by its fraction of T.

Notes

1. Experiment results are frozen to a snapshot of when the rollout decision is made.
2. Rolling out a group to 100% doesn't fully ship the experiment; configurations such as experiment/layer allocation, targeting, and overrides don't change. To fully ship the experiment, go through the usual flow without specifying any rollouts.
3. Groups can be rolled out and back, but the rollout percentage can't be lower than the group's original size.

Shipping with a holdback

Shipping with holdback lets you release an experiment variant (the “shipped” group) to most of your users while keeping a percentage in the control group for ongoing comparisons.

1. Make a decision and select a group to ship: From the Make Decision dropdown, choose which variant you want to ship.

2. Turn on the Ship with holdback option and specify the percentage of users you want to keep in the control (holdback) group.

3. Allocation of users: The control (holdback) group is set to your specified percentage. The remaining users are assigned to the shipped experience.

4. Splitting the shipped group: The shipped group is divided into two segments: Test and Launched.

   • The size of Test segment has the same percentage allocation as the holdback group for an equal sized comparison of 50:50.
   • The Launched segment no longer appears in the pulse results, but users in this segment continue to receive the shipped experience.

5. Continue monitoring the pulse results that evaluate the Test segment vs. Control (holdback).

6. When you decide to end the holdback, you can make a decision to ship the variant to everyone, which includes users in holdback.

The new pulse results (Test segment vs. Holdback) start when you ship with holdback. Your original experiment's results from the point of the holdback decision are retained and remain accessible.

From the experiment history, you can see the new log of the experiment decision and a View Results Snapshot button where you can view the read-only snapshot of original results.

Shipping with holdback maintains a dedicated, stable control group alongside a representative test segment of the shipped experience, making it easy to measure ongoing performance and user behavior after launch.