On this page

Variance Reduction

Overview of variance reduction techniques in Statsig Warehouse Native, including CUPED, stratified sampling, and regression adjustment for sensitivity.

Variance reduction

Variance measures the dispersion (or "noise") in a metric or experiment results. Higher variance produces larger confidence intervals, requiring more sample to detect a statistically significant result for the same effect size.Lower variance reduces required sample size, which leads to shorter experiment run times. Statsig uses a form of CUPED based on a 2013 Microsoft paper (Deng, Xu, Kohavi, & Walker). Statsig automatically applies CUPED to experiments and runs it for the topline results on key metrics in Pulse, producing significant variance reduction for most metrics.Go to the CUPED launch post for more details.

CUPED - Controlled-experiment Using Pre-Existing Data

CUPED (Controlled-experiment Using Pre-Existing Data) uses user information from before an experiment to reduce variance and increase confidence in experimental metrics. In Statsig, the pre-experiment data window is the 7 days before each user's exposure, rather than a fixed window before the experiment starts. This approach helps debias experiments where groups were randomly different before you apply any treatment.

The Cloud product uses stratification alongside CUPED to account for users who may not have pre-experiment data. Statsig groups users into strata based on available pre-experimentation information. Statsig estimates treatment and control effects within each stratum, then aggregates them to produce an overall result. Statsig then applies standard difference-in-means and variance estimation. This approach retains users with missing pre-data while still benefiting from variance reduction where applicable.

Winsorization

Another technique for reducing noise is Winsorization, which manages the influence of outliers. Winsorization measures the percentile Px of a metric and sets all values over Px to Px. Winsorization reduces the influence of extreme outliers caused by factors such as logging errors or bad actors.

Metric selection

The metrics you use can dramatically influence the sensitivity of your analysis. CUPED and Winsorization, along with techniques like creating threshold-based flags, let you trade exact numbers for significantly more statistical power. Go to the Statsig blog post on variance reduction for more information.

Was this helpful?