Conclude Experiment & Defer Decision

How Conclude Experiment and Defer Decision works

The "Conclude Experiment and Defer Decision" feature stops any further user allocation and data collection on an ongoing experiment. Use this feature when you want to decouple running an experiment from making a ship decision.

Benefits

• Focused decision making: Decoupling the experiment from the decision gives you time to evaluate results, weigh options, and review findings with stakeholders across teams.
• Flexibility in outcomes: Decide whether to ship the control, implement the test, or abandon/restart the experiment based on comprehensive insights.

Enabling this option

You must first enable the "Conclude Experiment and Defer Decision" option in Project Settings for it to appear in the Make Decision modal. To enable this option, go to Settings > Experimentation in the Product Configuration section and toggle on the Enable decision type - Conclude Experiment and Defer Decision setting.

Key details

When you conclude an experiment:

• After you conclude the experiment, no new users are enrolled.
• All previously exposed users start receiving the default experience.
• Statsig stops further data collection on the experiment and concludes results.
• After concluding the experiment, you can still decide to ship control vs test variant, reset, or abandon the experiment.

Reloads after experiment conclusion

You may notice slight changes in exposure and metric data if you trigger another reload after concluding an experiment. This doesn't mean new users are still being exposed. Depending on the timestamp of when the experiment was concluded versus when the latest exposure data landed in your warehouse in the previous reload, additional exposures from that time window may appear in the refreshed data.

Summary

The "Conclude Experiment and Defer Decision" feature lets you make data-driven decisions without the risk of diluting results by continuously adding new participants or new data from existing participants. This structured approach ensures that all stakeholders are aligned before you make a decision.