Fieller Intervals

How Statsig uses Fieller intervals to construct confidence intervals for ratio metrics in experiment analysis, with formulas and interpretation notes.

Fieller intervals give an exact confidence interval for the relative change between test and control groups, accounting for the uncertainty in the control mean that forms the denominator. Statsig recommends Fieller intervals over the delta method, which approximates the variance of the ratio and builds a confidence interval from that approximation. Fieller intervals stay reliable when sample sizes are small or denominators are noisy; the two methods agree in most other cases.

How Statsig calculates a Fieller interval

Check that the Fieller interval is well defined

Before applying Fieller's theorem, Statsig checks that the denominator of the relative lift metric, $\overline{X_C}$ , is significantly distinct from 0.

Statsig calculates the parameter $g$ :

g = \frac{Z_{\alpha/2}^2 \cdot \mathrm{var}(X_C)}{(n_C-1) \cdot \overline{X_C}^2}

Where:

$Z_{\alpha/2}$ is the critical value for the confidence level you select.
$\mathrm{var}(X_C)$ is the variance of the control group metric values.
$n_C$ is the number of units in the control group.
$\overline{X_C}$ is the mean of the control group metric values.

When $g$ is less than 1, the control mean is significantly different from 0, and Statsig uses the Fieller interval.

Apply the Fieller interval formula

Because the control and test group results are independent of each other, Statsig drops the covariance terms in Fieller's theorem.

CI(\% \Delta \overline{X} ) = \frac{1}{1-g} \left( \frac{\overline{X_T}}{\overline{X_C}} \pm \frac{Z_{\alpha/2}}{\overline{X_C}} \sqrt{ \frac{\overline{X_T}^2}{\overline{X_C}^2} \cdot \frac{\mathrm{var}(X_C)}{n_C-1} + (1-g)\frac{\mathrm{var}(X_T)}{n_T-1} } \right) - 1

Where:

$\overline{X_T}$ is the mean of the test group metric values.
$\mathrm{var}(X_T)$ is the variance of the test group metric values.
$n_T$ is the number of units in the test group.

Handle a control mean that isn't distinct from zero

In rare cases (less than 5% of observed metric comparisons on Statsig), $g$ is greater than or equal to 1, which means the control group's mean isn't statistically distinguishable from 0.

When $\overline{X_C}$ isn't statistically different from zero, the denominator of the relative lift calculation is unstable, so the confidence interval for the percent difference between test and control is unbounded.

When the confidence interval is unbounded, Statsig surfaces the relative lift observed during the experiment:

\% \Delta \overline{X} = \frac{\overline{X_T}-\overline{X_C}}{\overline{X_C}}

Enable Fieller intervals in Statsig

Configure the relative confidence interval methodology in your Experimentation Settings at the organization level. Changing this setting affects only experiments created after the change.

Experimentation settings configuration interface

In the experiment scorecard, Fieller intervals appear as shown:

Experiment scorecard with Fieller intervals

Was this helpful?