Measuring multiple rollout stages

Continuous analysis

For gate rules that roll out with a pass percentage ≤ 50% and without any rollback, Statsig consolidates data collected at earlier and later stages of the rollout into one analysis. This produces a more complete analysis.

Valid continuous analysis rollouts include:

10% → 20%

10% → 20% → 50%

Invalid continuous analysis rollouts include:

10% → 5% (rollback)

10% → 70% (exceeds 50%)

When a rollout is no longer valid for continuous analysis, Statsig analyzes the new rollout step separately from previous steps.

Compatibility with other gate features

Balanced gates

In cases where pass percentage ≤ 50%, Statsig uses the same hashing as the pass/fail and takes an equal percentage of the other group.

When pass percentage > 50%, Statsig uses an orthogonal random hashing to sample the larger group at a rate of $\frac{1-p}{p}$ where p is the fraction of users in the larger group. In the case of a gradual rollout, this prevents bias based on enrollment time period.

CUPED/CURE

CUPED/CURE is available only when there have been no rollbacks and when the rollout percentage hasn't exceeded 50% within a gate rule. Analysis is no longer continuous after a rollback or after 50% is exceeded, because earlier treatments bias the pre-exposure data of a given unit.