Skip to main content

How to Read Pulse

How to read Pulse​

Exposures​

Exposures Chart

At the top of Pulse is the Exposures Chart. Exposures are the unique experimental units enrolled in the experiment. This is typically the number of unique users, and for device-level experimentation, this is the number of devices. The timeline shows you when the experiment was started, and how many exposures were enrolled on any given day. You can see the rate at which users were added into each group of the experiment, how many total users were exposed, and confirm the target ratio matches what you configured in experiment setup.

Metric Lifts​

Screen Shot 2022-12-29 at 1 47 32 PM

The Metric Lifts section, a.k.a. Pulse calculates the difference between the comparable randomization groups (eg. test and control) across your company's suite of metrics, and applies a statistical test to the results. You can read more about Statsig's stats engine here.

For every metric, we will show you:

  • The calculated relative difference (Delta %)
  • The confidence interval
  • Whether the result is statistically significant
    • Positive lifts are green
    • Negative lifts are red
    • Non-significant results are grey

The formula for calculating lift is:

Delta(%) = (Test - Control) / Control

Confidence intervals are reported at the selected significance level (95% by default). In a typical two-sided Z-test, we show the confidence interval as +/- 1.96 * standard error.

99.9% winsorization is automatically applied to event_count, event_count_custom, and sum metrics. This caps extreme outlier values to reduce their impact on experiment results. For metrics added to the Scorecard or Monitoring Metrics sections of your experiment or gate, you can also apply other optional statistical treatments, such as CUPED (pre-experiment bias reduction) and sequential testing adapted confidence intervals. Read more here.

Dimensions​

There are two ways in which we can breakdown a given Pulse metric - one is by user dimension, the other is by value dimension.

User Dimensions​

User dimensions refer to user level attributes that are either part of the user object you log, or additional metadata that Statsig extracts. Examples of these attributes are operating system, country, and region.

You can create custom queries to create queries that filter on or group by available user dimensions. For example, you could "See results for users in the US", or "See results for users using iOS, grouped by their country".

Value dimensions​

Value Dimensions refer to the set of distinct values logged alongside a given metric. If you want to see Pulse results for a metric further broken down by categories that are specific to that metric, specify the dimension you want to break down by in the value or metadata attributes when you log the event. For example, when you log a click event on your web or mobile application, you may also log the target category using the value attribute as shown below. Pulse will automatically generate results for each category in addition to the top level metric.

To see the Pulse result breakdowns for all categories within a metric, click on the (+) sign next to the metric. Read more about configuring metric dimensions here.

image image