Skip to main content

Pulse Metrics

Statsig computes experiment results, also known as Statsig's Pulse results, depending on the metric type. Most metrics are aggregated across all users in the group. Some ratio type metrics are only aggregated across partipating users (users that have non-null value for that metric).

Pulse Statistics by Metric Type

Metric TypeTotal CalculationMeanUnits
event_countSum of events (99.9% winsorization)Average events per user (99.9% winsorization)All users
event_dauSum of event DAU (distinct user-day pairs)Average event_dau value per user per day. Note that we call this "Event Participation Rate" as this can be interpreted as the probability a user is DAU for that event.All users
ratioOverall ratio: sum(numerator values)/sum(denominator values)Overall ratioParticipating users
sumTotal sum of values (99.9% winsorization)Average value per user (99.9% winsorization)All users
meanOverall mean valueOverall mean valueParticipating users
user: dau, wau, mau_28dayNot shownAverage metric value per user per day. The probability that a user is xAUAll users
user: new_dau, new_wau, new_mau_28dayCount of distinct users that are new xAU at some point in the experimentFraction of users that are new xAUAll users
user: retention metricsOverall average retention rateOverall average retention rateParticipating users
user: L7, L14, L28Not shownAverage L-ness value per user per dayAll users

Event Count and Event DAU in Pulse

From Metrics 101,

  • event_count measures the volume of the activity based on count of events triggered
  • event_dau measures unique daily users who triggered a given event

For example, the table below shows the event_count and event_dau metrics for two event types,Page Views and Add to Cart, for three users over three days. image

Over the duration of an experiment, Pulse results measure the change in:

  • the mean event_count, or the average event count per user
  • the mean event_dau, or average active days per user; we call this the Daily Event Participation Rate

For example, the table below shows the Total event_count, Total Units, and Mean event_count over the same three days as above, now in the context of an experiment.

image

Similarly, the table below shows the Total event_dau, Total Units, and Mean event_dau over the same three days of the experiment. Alice was 'active' on three days for the Page View event and on one day for the Add to Cart event. Therefore, average eventdau for Alice is 3/3 for the _Page View event and 1/3 for the Add to Cart event. In other words, Alice's daily participation rate is 1.00 for the Page View event and 0.33 for the Add to Cart event so far in the experiment. Statsig aggregates this average event_dau for each user in the experiment, with each user weighted equally.

image

To measure the change in engagement for a call to action link or button, use event_count to measure the change in average clicks per user, and use event_dau to measure the change in users’ daily participation rate for the click.

Event Count and Event DAU in Custom Metrics

When creating a custom ratio metric, use event_count to include all events (counting all events triggered by the same user). Use event_dau to count unique active users on a given day (all events triggered by the same user are counted as one).

Winsorization

To reduce the impact of outliers, Statsig caps event_count and sum metric types at the 99.9th percentile. This mitigates the risk of bots and extreme values significantly swaying experiment results.

Frequently Asked Questions

1. Is it possible for a ratio metric to move in the opposite direction than both the numerator and denominator metrics?

Yes, it is possible for the ratio to rise while both the numerator and denominator metrics decline. For example, this happens when the denominator is falls more than the numerator. As a best practice, Statsig recommends tracking the numerator and denominator as independent metrics when monitoring ratio metric. Ratio metrics are often subject to statistical noise and can be tricky to use for obtaining a statistically significant result.

2. For ratio metrics, how does Statsig determine participating users?

Ratio metrics are computed only for users that have a non-zero value in the denominator, i.e. the user must have triggered the denominator event on a given day to be included in the daily ratio. Users that don't trigger the denominator event during an experiment are not included in the test vs. control comparison of a ratio metric.