Make a Decision

Making a decision for an experiment enables you to ‘ship’ the winning group to all your users. After you make the decision, the variant that your users see depends on whether you’re using a targeting gate for your experiment. The results for your experiment will still be accessible after you make a decision, but they will stop updating. The last day of metrics will be the day you “make a decision” on the experiment.

​

Experiments With No Targeting Gate

When you ship a group in an experiment with no targeting gate, the parameter values from the shipped group will become the default values for all your users going forward. If the experiment happens to use parameters from a layer, the layer’s parameters will now take on the shipped group’s parameter values as their defaults. These are the values that all your users will see going forward. For example, suppose you have a Demo Layer that’s configured with a parameter, a_param. It’s default value is set to layer_default as shown below. Say you decide to create an experiment, Demo Experiment in Demo Layer as shown below. You set up Demo Experiment with two groups: Control and Test, intending to experiment with new values for the layer-level parameter, a_param as shown below. Now if you decide to ship the Control group for the Demo Experiment, a_param will take the value set for the Control group as its default: experiment_one_control On the other hand, if you decide to ship the Test group, a_param will take the value set for the Test group as its default: experiment_two_test

​

Experiment With a Targeting Gate / Targeting Rules

When you decide to ship a group in an experiment configured with a targeting gate or targeting rules, you can decide whether to continue targeting after shipping.

• If you decide to discontinue targeting, the parameter values from the shipped group will become the default values for all your users going forward. If the experiment happens to use parameters from a layer, the layer’s parameters will now take on the shipped group’s parameter values as their defaults. These are the values that your users will see going forward.
• If you decide to continue targeting with a targeting gate, this will add an override to the experiment layer so that:
  • all users who pass the targeting gate will see shipped group’s parameter values
  • all users who fail the targeting gate will see the default value (layer-level parameter defaults or the defaults you set for the parameter in your code)
• If you decide to continue targeting with targeting rules, the experiment will only be shipped to users who pass the inline targeting rules as set in the experiment setup.

For example, suppose Demo Experiment in a Demo Layer that has a parameter called targeted_layer_param, whose default value is set to targeted_layer_default_value. When you decide to ship Demo Experiment, if you discontinue targeting, targeted_layer_param will now take on the value from the Control group, targeted_layer_control, as its default. On the other hand, if you decide to continue targeting, targeted_layer_param will now acquire an override so that:

• all users who pass the targeting gate will see the overridden value of the parameter
• all users who fail the targeting gate will see the default value of the parameter

In this case, the default value of targeted_layer_param in Demo Layer will not change. Also, any users who pass the targeting gate will not be eligible for future experiments run in this layer. For this reason, we do not encourage shipping experiments with targeting on, especially when the experiment is in a layer.

​

Rolling Out an Experiment Group

Rolling out an experiment group is an option available when you have decided the winning variant, but want to avoid a sudden, large shift of traffic into the winning variant group. You can use automated rollouts to schedule gradual rollout phases, which will increase your shipped group size to the desired percentage by reallocating users from all other groups proportionally.

​

Setting up Rollouts

To set up rollouts, open the make decision form and select the winning group. Here you will be able to either use automated rollouts, or ship with rollout. The ship with rollout option allows you to immediately update the shipped group size. Manual rollouts will clear any automated rollout phases. Alternatively, you can set up automated rollouts, which will open the following dialog that you can populate with the rollout phases: From here, you can configure each phase of your Scheduled Rollout. To add phases to your rollout, click Add Phase and configure as many phases as you want. Each scheduled rollout phase includes-

• Rollout date
• Rollout time*
• Pass percentage

Upon saving, you will be able to see a preview of the rollout and commit the schedule:

​

Resizing Logic

During each phase, the rollout group is resized to the desired percentage, and all other groups are scaled proportionally in the following way: You have n groups with sizes: G₁, G₂, …, Gₙ Their total sum is: 100 = G₁ + G₂ + … + Gₙ. Now, suppose you set one group (say the k-th group) to a new size Gₖ′. Let Delta = Gₖ′ − Gₖ. Because the grand total must remain 100, you need to adjust the remaining groups proportionally. Let T = 100 - Gₖ Then, for each group i ≠ k, Gᵢ′ = Gᵢ − (Delta × Gᵢ / T). In other words, each group other than k is decreased (or increased, if Delta < 0) by its fraction of T.

​

A few notes

1. Experiment results will be frozen to a snapshot of when the rollout decision is made.
2. Rolling out a group to 100% does not fully ship the experiment, meaning configurations such as experiment/layer allocation, targeting, overrides, and so on will not change. To fully ship the experiment go through the usual flow without specifying any rollouts.
3. Groups can be rolled out and back, but the rollout % cannot be lower than the group’s original size.

​

Shipping with a Holdback

Shipping with holdback lets you release an experiment variant (the “shipped” group) to most of your users while keeping a percentage in the control group for ongoing comparisons. Here’s how it works:

1. Make a decision and select a group to ship: From the Make Decision dropdown, choose which variant you want to ship.
2. Turn on the Ship with holdback option and specify the percentage of users you want to keep in the control (holdback) group.

3. Allocation of users: The control (holdback) group is set to your specified percentage. The remaining users are assigned to the shipped experience.
4. Splitting the shipped group: The shipped group is divided into two segments: Test and Launched.
   • The size of Test segment will have the same percentage allocation as the holdback group for an equal sized comparison of 50:50.
   • The Launched segment will no longer appear in the pulse results, but users in this segment will continue to receive the shipped experience.

5. Continue monitoring the pulse results that evaluates the Test segment vs. Control (holdback)
6. Once you decide to end the holdback, you can make a decision to ship the variant to everyone which will include users in holdback.

Note that the new pulse results (Test segment vs. Holdback) will start when you ship with holdback, but your original experiment’s results from the point of holdback decision will be retained and remain available to you. From the history of experiment, you will see the new log of experiment decision and a ‘View Results Snapshot’ button where you can view the read-only snapshot of original results. By using shipping with holdback, you maintain a dedicated, stable control group alongside a representative test segment of the shipped experience, making it easy to measure ongoing performance and user behavior post-launch.