Skip to main content

How to Read Pulse

How to read Pulse​

Exposures​

Exposures Chart

At the top of Pulse is the Exposures Chart. Exposures are the unique experimental units enrolled in the experiment. This is typically the number of unique users, and for device-level experimentation, this is the number of devices. The timeline shows you when the experiment was started, and how many exposures were enrolled on any given day. You can see the rate at which users were added into each group of the experiment, how many total users were exposed, and confirm the target ratio matches what you configured in experiment setup.

Metric Lifts​

Immediately Post-experiment Start​

For up to the first 24 hours after starting your experiment (before our daily Pulse run), the Metric Lifts section, a.k.a. Pulse, is calculated in near real-time (this only applies to Statsig Cloud, for WHN projects you will need to reload pulse on demand or set up a daily schedule). This more real-time Pulse is designed to enable you to confirm that exposures and metrics are being calculated as expected and debug your experiment or gate setup if needed.

Please note that you should not make any experiment decisions based on more real-time Pulse data in this first 24 hour window after experiment start. Experiments should only be called once the experiment has hit target duration, as set by your primary metric(s) hitting experimental power. Read more about target duration here.

Screen Shot 2023-06-19 at 3 29 13 PM

Given data during this early post-experiment start window is designed for diagnostic, not decision-making purposes, you will notice a few key differences between this Pulse view and the Pulse results that will start showing after daily runs have initiated:

  • Metric lifts do not have confidence intervals
  • No time-series view of metric trends
  • No projected topline impact analysis
  • No option to apply more advanced statistical tactics, such as CUPED or Sequential Testing
Screen Shot 2023-06-19 at 3 32 13 PM

All of these are available in daily Pulse results, which will start showing in the next daily Pulse run.

Post-first Day Metric Lifts​

Screen Shot 2023-09-20 at 11 35 05 AM

The Pulse daily run calculates the difference between the comparable randomization groups (eg. test and control) across your company's suite of metrics, and applies a statistical test to the results. You can read more about Statsig's stats engine here.

For every metric, we will show you:

  • The calculated relative difference (Delta %)
  • The confidence interval
  • Whether the result is statistically significant
    • Positive lifts are green
    • Negative lifts are red
    • Non-significant results are grey

The formula for calculating lift is:

Delta(%) = (Test - Control) / Control

Confidence intervals are reported at the selected significance level (95% by default). In a typical two-sided Z-test, we show the confidence interval as +/- 1.96 * standard error.

99.9% winsorization is automatically applied to event_count, event_count_custom, and sum metrics. This caps extreme outlier values to reduce their impact on experiment results. For metrics added to the Scorecard or Monitoring Metrics sections of your experiment or gate, you can also apply other optional statistical treatments, such as CUPED (pre-experiment bias reduction) and sequential testing adapted confidence intervals. Read more here.

Pulse Views​

There are a few different views to see your Pulse metric lifts, namely:

  • Cumulative results (default view)
  • Daily results
  • Days since exposure

Cumulative results includes a detailed view on hover, where you can additionally view the raw statistics used in the Pulse metric lift calculations, as well as topline impact.

Screen Shot 2023-09-20 at 11 35 16 AM Screen Shot 2023-09-20 at 11 35 35 AM

Dimensions​

There are two ways in which we can breakdown a given Pulse metric - one is by a User Dimension, the other is by an Event Dimension.

User Dimensions​

User Dimensions refer to user level attributes that are either part of the user object you log, or additional metadata that Statsig extracts. Examples of these user attributes could be operating system, country, and region.

You can create custom queries to create queries that filter on or group by available user dimensions. For example, you could "See results for users in the US", or "See results for users using iOS, grouped by their country".

Event dimensions​

Events Dimensions refer to the value or metadata logged as part of a custom event that is used to define the metric. If you want to see Pulse results for a metric broken down by categories that are specific to that metric, specify the dimension you want to break down by in the value or metadata attributes when you log the source event. For example, when you log a "click" event on your web or mobile application, you may also log the target category using the value attribute as shown below. Pulse will automatically generate results for each category in addition to the top level metric.

To see the Pulse result breakdowns for all categories within a metric, click on the (+) sign next to the metric.

!image

image image