On this page

CURE

Introduction to Statsig CURE (Causal Uplift Recommendations) for Warehouse Native, used to identify which users are most likely to respond to a treatment.

CURE (variance Control Using Regression Estimates) is Statsig's extended implementation of CUPED, a technique that uses pre-experiment data for each experimental unit to control variance observed in an online experiment. CUPED can reduce variance moderately to significantly and can correct for some pre-existing differences between experiment groups.

CUPED provides additional statistical power without requiring additional users or experiment runtime. However, CUPED has one major limitation: it relies on pre-experiment data for a given metric to reduce that metric's variance. This means that:

  • New user experiments can't leverage the technique
  • Metrics which aren't autocorrelated don't see significant variance reduction

What is CUPED

Refer to the CUPED documentation or the CUPED blog post.

How CURE improves on CUPED

CURE solves the no-covariate problem by allowing practitioners to specify additional covariates, which are used in the CUPED regression. The predictions from this regression are used to generate an estimator set that is unbiased, but has lower variance than the original estimator.

CURE can reduce experiment runtimes more than CUPED alone. When combined with Statsig's Entity Properties, CURE allows practitioners to connect existing feature stores to this variance reduction technique.

Like CUPED, CURE can modify the point estimates of groups, though the total value across all experimental groups sums to the same value as in the unadjusted dataset.

Drift in value between groups is expected, particularly when there are pre-experiment differences and the correlation is high. This is the algorithm adjusting for pre-existing deltas in experiment groups. Contact the team in Slack if you have concerns.

How CURE differs from standard regression

This technique is similar to standard regression estimates used in other experimentation tools, but differs in several ways:

  • Statsig applies CURE across all major experimental surfaces, including experiment scorecards, drill-down explore queries, filtered queries, and power analysis. This ensures results match between scorecards and explore queries, which can diverge in tools with incomplete CUPED implementations.
  • Multivariate regressions enable variance reduction for new users or users without pre-experiment data.
  • CURE manages feature selection, preventing overfitting issues seen in similar multivariate approaches, and making regression adjustments more transparent.

How CURE calculates variance reduction

The original CUPED implementation by Microsoft (Deng, et. al.) is equivalent to an OLS regression with one independent variable. However, the adjustment doesn't need to be a single-variable regression. A regression guarantees identical or lower variance, but any unbiased adjustment is mathematically valid.

By calculating the covariance matrix between covariates and experiment outcomes in SQL, you can generate the inputs for multivariate regressions and use these to adjust post-exposure values in an unbiased manner, meaning the regression is demeaned and uses only pre-experiment data. This yields an estimator with variance less than or equal to the original estimator. Specifically:

  • if covariates don't exist, nothing happens and the adjusted estimator is identical to the unadjusted
  • in the worst case where covariates are uncorrelated, the adjusted estimator is identical to the unadjusted
  • if no additional covariates are applied, CURE is mathematically equivalent to CUPED
  • if additional covariates are applied that increase the $R^2$ (i.e. portion of explained variance), CURE reduces variance more than CUPED

CURE for ratio metrics

Combining Statsig's CUPED implementation for ratio metrics with CURE covariates enables ratio metrics to use both pre-experimental data and user-provided covariates. With an adjusted formula to calculate the covariance of a ratio random variable to a simple variable, Statsig establishes a multi-covariate regression similar to CURE for simple metrics. The variance and mean adjustments follow the ratio CUPED approach.

  • As an artifact of the regression, Statsig obtains the shrinkage $R^2$, which it uses to scale the post-adjustment variance.
  • Statsig constructs the mean adjustment as the inner product of the coefficient vector $\theta$ and the covariate group means $C$. By calculating the difference in adjustment between the test and control group, only the test group mean is adjusted while holding the control group mean the same.

Data sources

When CUPED is enabled, pre-experiment metric data is still an input to the CURE regression.

CURE allows you to pull data from:

  • Assignment records: if using end-to-end experiments with Statsig, log user attributes using the SDK and they are exported for use in CURE.
  • Entity property sources: use any timestamp-enabled entity property source to provide covariate data. Non-timestamped entity property sources are disabled to prevent data leakage into CURE.
Entity property sources are a convenience feature; you could alternatively manage joining user features to your assignment data. In practice, this enables use cases such as providing your own table of regression estimates as covariates, similar to DoorDash's CUPAC approach.

Configuration

You can manage CURE in two places:

  • Project Settings under Experimentation: the default CURE Covariates setting lets you specify covariates to use across any CUPED-enabled metric in your project.
  • Advanced Settings in the Experiment Settings page: specify covariates for a single experiment.

Datasets

Statsig surfaces CURE information in experiment diagnostics, providing details on coefficients and relevant covariates for any given metric. View the adjustments used under the Regression Adjustment job, and check the coefficients table to run your own analysis on the regression model.

Outputs

View the DAG history to see the exact coefficients used and the table where they are stored. Statsig also renders an in-console explanation of which variables contributed to variance reduction and by how much, which helps users decide which features to use.

Known limitations

Like CUPED, CURE can modify the point estimates of groups. The total value across all experimental groups sums to the same value as in the unadjusted dataset, but drift in value between groups is expected, particularly when there are pre-experiment differences and the correlation is high. This is the algorithm adjusting for pre-existing deltas in experiment groups. Contact the team in Slack if you observe a concerning change.

Feature selection

Statsig uses Lasso regression to select important features, reducing the computational cost of adjusting the unit metric level and controlling for multicollinearity. View the contribution of each feature on the variance reduction card. The estimated proportion of variance reduction is:

$$ \text{Variance Reduction}{i} = \frac{{Coefficient{i}}^2Var(feature_{i})}{\sum {Coefficient_{i}}^2Var(feature_{i})} $$

by that,

∑Variance Reductioni=100%\sum \text{Variance Reduction}_{i} = 100\%

References

Was this helpful?