What is a holdout?
A holdout is a group of users that are held out of one or more experiments to measure the combined impact of that set of experiments. You can think of it as a "global control group" - while each experiment you run will compare the control and test groups for that feature, a holdout will compare the "global control" (users who never got ANY experiment) with users who may have been in one or more experiments.
Want to measure the combined impact of a number of different experiments easily? Perhaps you want to measure the combined impact of your team's work on your company's KPIs over the course of a quarter, or even a year. But with new features shipping every week, maintaining a holdout over the course of a long period can seem difficult and add cognitive load to setting up experiments.
With Statsig, creating and managing a holdout is nearly automatic. With just a few clicks, you can create a new holdout, update existing experiments to respect this holdout, and ensure all new feautures hold out the same set of users! Let's take a look.
How do I create a Holdout with Statsig?
Statsig has a separate tool just for this called "Holdouts" (what a name!). Start by navigating there.
Next, click "Create"
Fill out the resulting form.
- If you want all future gates to respect this holdout, make sure you mark it as a "global" holdout.
- Select a Holdout size in terms of a percentage of all users. Depending on how many users you have, how long you are keeping this holdout, and how many experiments you are running, you may choose to make this a bit bigger, but generally its quite small (1-5%).
- If there are any existing features that are already gated, you can select those gates at the bottom to make sure they respect the holdout moving forward.
Click create, and you are DONE! Now, when you look at a Feature Gate that has a holdout applied to it, the holdout will show at the top like this
How can I analyze my holdout group?
In order to actually analyze a holdout, you would need to collect data over a long period of time. Don't worry - we have some demo companies set up which have been using Statsig for some time. Let's take a look at one of their holdouts.
Step 1. Log in to the demo account Go to statsig.com/demo and complete the captcha
Step 2. Switch to the "Swagger Exclusive Retail" demo company
Step 3. Navigate to Pulse Analyzing a holdout is no different that analyzing an experiment - we always start from the Pulse page!
Step 4. Select the "2021_h1_product_holdout" Holdout
Step 5. Interpret the results! Pulse will pull all the same metrics comparisons as your experiments, only for the holdout group. If you need help interpreting Pulse results, check out our Getting Started with Experiments guide