Skip to main content

Creating your First Holdout

What is a holdout?​

A holdout is a group of users that are held out of one or more experiments to measure the combined impact of that set of experiments. You can think of it as a "global control group" - while each experiment you run will compare the control and test groups for that feature, a holdout will compare the "global control" (users who never got ANY experiment) with users who may have been in one or more experiments.

Want to measure the combined impact of a number of different experiments easily? Perhaps you want to measure the combined impact of your team's work on your company's KPIs over the course of a quarter, or even a year. But with new features shipping every week, maintaining a holdout over the course of a long period can seem difficult and add cognitive load to setting up experiments.

With Statsig, creating and managing a holdout is nearly automatic. With just a few clicks, you can create a new holdout, update existing experiments to respect this holdout, and ensure all new features hold out the same set of users! Let's take a look.

How do I create a Holdout with Statsig?​

Statsig has a separate tool just for this called "Holdouts" (what a name!). Start by navigating there.

Screen Shot 2022-06-26 at 11 00 07 AM

Next, click "Create New", and fill in the name/description, as well as the ID type for the holdout

Screen Shot 2022-06-26 at 11 01 52 AM

Fill out the resulting form and click create. Then, you will be landed on the setup tab to complete your holdout configuration:

  • If you want all future gates to respect this holdout, make sure you mark it as a "global" holdout.
  • Select a Holdout size in terms of a percentage of all users. Depending on how many users you have, how long you are keeping this holdout, and how many experiments you are running, you may choose to make this a bit bigger, but generally it's quite small (1-5%).
  • If there are any existing features that are already gated, you can select those gates at the bottom to make sure they respect the holdout moving forward.

Click Save Changes to finalize the setup, but you can always come back and add new gates/experiments to the holdout later if you need to. When you looking at a Feature Gate or Experiment that has a holdout applied to it, the holdout will show at the top like this:

Screen Shot 2022-06-26 at 11 09 08 AM

How can I analyze my holdout group?​

In order to actually analyze a holdout, you would need to collect data over a long period of time. Don't worry - we have some demo companies set up which have been using Statsig for a while. Let's take a look at one of their holdouts.

Step 1. Log in to the demo account Go to statsig.com/demo and complete the captcha

Screen Shot 2022-06-26 at 11 13 18 AM

This will land you on the "Demo: Swagger Exclusive Retail" company.

Step 2. Navigate to Holdouts and select the "Frontend Team Holdout"

Screen Shot 2022-06-26 at 11 20 47 AM

Step 3. You should land on the "Results" tab. Now you can analyze the impact of the set of features that respect this holdout together vs the holdout group. In this case, the changes the front end team has been making have moved a number of metrics in the conversion funnel!

Screen Shot 2022-06-26 at 11 19 37 AM