Skip to main content

CURE

CURE (variance Control Using Regression Estimates) is Statsig's extension of CUPED, a technique which uses pre-experiment data for each experimental unit to control some of the data observed in an online experiment. CUPED can reduce variance moderately to significantly, and can correct for some amount of pre-existing differences between experiment groups at the time of the experiment.

CUPED is a powerful technique that provides additional statistical power for "free". However, it has one major limitation: it relies on using pre-experiment data for a given metric to reduce the same metric's variance. This means that:

  • new user experiments cannot leverage the technique
  • metrics which are not autocorrelated do not see significant variance reduction

How CURE Improves CUPED​

CURE solves the covariate problem by allowing users to specify additional covariates, which are used in a lasso regression. The predictions from this regression are used to generate an unbiased set of estimators with reduced variance.

This can reduce experiment runtimes far more than CUPED in isolation, and when combined with Statsig's entity property sources allows end users to trivially plug in existing feature stores to this variance reduction technique.

How it's different​

This technique is similar to standard regression estimate used across other experimentation tools, but stands out in several ways:

  • CURE is applied on all major experimental surfaces that Statsig offers; instead of showing mismatching results between scorecards and explore queries due to incomplete variance reduction implementations, Statsig offers CURE across experiment scorecards, drill-down explore queries, filtered queries, and even power analysis
  • Multivariate regressions allow you to experiment with reduced variance on new users or users without pre-experiment data
  • CURE manages feature selection, preventing observed overfitting issues seen in similar multivariate approaches, and making the regression adjustments significantly more transparent and less of a vendor black-box

Approach

The original version of CUPED specified by Microsoft (Deng, et. al.) is equivalent to running an OLS regression with one independent variables. However, this adjustment does not need to be a single-variable regression (or a regression at all). A regression has the benefit of guaranteeing identical or lowered variance, but so long as an adjustment is unbiased, any adjustment is mathematically valid.

This by calculating the covariance matrix between covariates and experiment outcomes in SQL, you can cheaply generate the required inputs to generate multi-variate regressions and use these to adjust post-exposure values in an unbiased manner -- so long as this estimator is unbiased (critically, meaning the regression is demeaned and only uses pre-experiment data). Similarly to the original paper's proof, this yields an estimator that has variance less than or equal to the existing estimator. That means that:

  • if covariates do not exist, nothing happens and the adjusted estimator is identical to the unadjusted
  • in the worst case where covariates are uncorrelated, the adjusted estimator is identical to the unadjusted
  • if no additional covariates are applied, CURE is mathematically equivalent to CUPED
  • if additional covariates are applied that increase the R2 or portion of explained variance, CURE reduces variance more than CUPED

Data Sources

When CUPED is enabled, pre-experiment metric data will still be an input to the CURE regression.

The extension CURE offers is allowing you to pull data from:

  • assignment records; if using e2e experiments with Statsig, you can log user attributes using the SDK and they will be exported for your use in CURE, if desired
  • entity property sources: you can use any timestamp-enabled entity property source to provide covariate data. Note that at launch, non-timestamped entity property sources are disabled due to the potential for data leakage into CURE

The latter option is a convenience feature; if desired, you could manage joining user features to your assignment data. In practice, this unlocks scalable use of the platform, unlocking use cases like providing your own table of regression estimates as covariates, e.g. Doordash's CUPAC approach.

Configuration

CURE can be managed in two places:

  • Project Settings under Experimentation. The default CURE Covariates setting allows you to specify covariates you want to use across any CUPED-enabled metric in your project
  • Advanced Settings in the Experiment Settings page. This allows you to specify covariates specific to one experiment

Datasets

Statsig will surface information on CURE in experiment diagnostics, giving details on coefficients used and relevant covariate for any given metric. You can also see the adjustments used under the Regression Adjustment job, and check the coefficients table to run your own analysis on the regression model.

Notes

Similar to CUPED, CURE can modify the point estimates of groups; though the total value across all experimental groups will sum to the same value as in the unadjusted dataset, is it expected that there may be drift in value between the groups (particularly if there's some pre-experiment differences and the correlation is high). Generally this can be seen as the algorithm adjusting for pre-existing deltas in experiment groups, but please reach out in slack if you are concerned about a concerning change.

References