Variance Reduction

Variance reduction shrinks the noise in an experiment metric so the same effect reaches statistical significance with fewer users or in less time. Statsig applies CUPED to every experiment automatically, applies winsorization to count and sum metrics, and offers stratified sampling when you set up an experiment. Use this page to find which technique applies to your metric; the CUPED, winsorization, and stratified sampling pages cover the derivations and setup.

Variance measures the noise in a metric or in experiment results. Higher variance produces wider confidence intervals and requires a larger sample to observe a statistically significant result for the same effect size. Reducing variance shortens experiment run times.

CUPED

CUPED (Controlled-experiment Using Pre-Existing Data) uses each user's data from before the experiment to reduce variance and increase confidence in experiment metrics. Statsig's implementation follows Deng, Xu, Kohavi, and Walker (2013). Statsig applies CUPED automatically to experiments and to the topline results on key metrics in Pulse, and it reduces variance for most metrics where Statsig can apply it. For the derivation, refer to CUPED; for background, refer to the CUPED launch post.

At Statsig, the pre-experiment data covers the 7 days before each user's exposure, rather than a fixed window before the experiment starts for all users. This per-user window helps reduce bias in experiments where the groups differed by chance before exposure.

Statsig Cloud uses stratification alongside CUPED to account for users without pre-experiment data. Statsig groups users into strata based on the available pre-experiment information, estimates treatment and control effects within each stratum, aggregates them into an overall result, and then applies the standard difference-in-means and variance estimation. This approach retains users with missing pre-experiment data while still reducing variance where it can.

Winsorization

Winsorization reduces noise by limiting the influence of outliers. It measures the percentile $P_x$ of a metric and sets all values above $P_x$ to $P_x$, which reduces the influence of extreme outliers from causes such as logging errors or bad actors. For the default percentile and variants, refer to winsorization.

Stratified sampling

Stratified sampling balances the test and control groups on a metric or attribute before the experiment starts, which reduces the variance of the measured delta for low-volume or high-variance populations.

Metric selection

The metrics you choose also affect the sensitivity of your analysis. Winsorization and CUPED, combined with techniques such as threshold-based flags, let you trade exact numbers for more statistical power. For more, refer to the blog post on understanding and reducing variance.