Metric Deltas

Learn how Statsig computes metric deltas to compare absolute and relative differences between experiment groups.

A metric delta is the difference in mean metric value per user between two experiment groups, by default test minus control. Pulse shows an absolute delta (the difference in means) and a relative delta (the absolute delta as a percentage of the control mean). Read the absolute delta to size an impact in metric units; read the relative delta to compare impact across metrics with different baselines.

Computing metric deltas

The metric delta represents the impact measured in experiment results. To account for the different number of users (or units) in each group, Statsig compares the mean metric value per user, not the total.

Statsig defines all deltas as the difference between a treatment group and a presumably unchanged control group, but you can compare any two groups. If you reverse the order of comparison in Pulse to control versus treatment, Statsig reverses all deltas and inverts the direction of change.

The absolute delta is the difference between the two means:

\Delta \overline{X}=\overline{X}_t-\overline{X}_c

The relative delta expresses the impact relative to the baseline value of the metric. For example, an absolute delta of +1 clicks per user is a 100% increase against a baseline of 1 and a 1% increase against a baseline of 100. Statsig computes the relative delta with the control group mean as the baseline:

\Delta \overline{X} \%=\frac{\overline{X}_t-\overline{X}_c}{\overline{X}_c} \times 100 \%

Computing means

The method for calculating metric means depends on the metric type.

Event count and sum metrics

These metrics represent totals: the number of times an event occurs, the sum of time spent, the total purchase amount, and similar values. The mean is the average user-level total during the analysis period.

The mean value of the metric $X$ for a group is:

\overline{X}=\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} X_{i, d}

where:

$N$ is the number of users in the group.
$n_i$ is the number of days during the analysis period that user $i$ was in the experiment.
$X_{i,d}$ is the metric value for user $i$ on day $d$ .

In the group mean, Statsig includes only user metrics recorded after it exposes a user to the experiment.

User accounting and event user metrics

event_user metrics set to Daily Participation Rate capture the number of distinct users who have the event each day. In Pulse results, Statsig normalizes these values by the number of days the user is in the experiment, which gives the probability that a user is daily active for that event: the daily participation rate. The group mean is:

\overline{X}=\frac{1}{N} \sum_{i=0}^N \frac{1}{n_i} \sum_{d=0}^{n_i} X_{i, d}

where:

$X_{i,d}$ takes the value 0 or 1 depending on whether user $i$ has the event on a given day $d$ .

Statsig computes the user accounting metrics in the following table in the same way. The new user accounting metrics use a different group mean.

Metric	Type
`DAU`	User accounting
`WAU`	User accounting
`MAU_28day`	User accounting
`L7`	User accounting
`L14`	User accounting
`L28`	User accounting
`new_DAU`	New user accounting
`new_WAU`	New user accounting
`new_MAU_28day`	New user accounting

For new user accounting metrics, Statsig counts users that are new xAU (new daily, weekly, or monthly active users) at some point during the analysis window. The group mean is:

\overline{X}=\frac{1}{N} \sum_{i=0}^N \max \left(X_i\right)

where $\max(X_i)$ is the maximum value of the new xAU metric for user $i$ .

event_dau metrics are in legacy support only, and Statsig no longer creates them for new events. Existing event_dau metrics remain available for your new experiments, and Statsig continues to compute them daily. For new events, create an event_user metric to measure daily active users.

Custom ratios, means, retention, and stickiness metrics

These are metrics such as click-through rate, average purchase value, and sessions per user. Statsig computes them by dividing a numerator value, $X$ , by a denominator value, $Y$ . The mean value of a ratio metric $R$ for an experiment group is:

\overline{R}=\frac{\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} X_{i, d}}{\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} Y_{i, d}}=\frac{\overline{X}}{\overline{Y}}

where $N$ is the number of users in the experiment group that participate in the metric, that is, users with a non-zero denominator value. $X_{i,d}$ and $Y_{i,d}$ are the $X$ and $Y$ values for user $i$ on day $d$ .

Different approaches exist for ratio metrics in experiments. Statsig chose this implementation because:

$R$ is the ratio of two means of independent observations: a set of user-level $X$ values and a set of user-level $Y$ values. You can therefore use the central limit theorem to obtain the summary statistics of $X$ and $Y$ separately.
Statsig computes the group means in the same way as the topline metric value, which makes the means easier to interpret and relate to the topline metric.

Event user one-time event

For custom event_user metrics with One-Time Event selected, Statsig computes how many users have the event at any time after the user enters the experiment. Statsig doesn't normalize this result by the number of days a user is in the experiment. The group mean is:

\overline{X}=\frac{1}{N} \sum_{i=0}^N X_{i}

where:

$N$ is the number of users in the group.
$X_{i}$ takes the value 0 or 1 depending on whether user $i$ has the event at any point after entering the experiment.

Was this helpful?