Unit Count (One-Time Event) Metrics

Use cases

This is an extremely common metric type, used to measure participation rates among users in the experiment.

Calculation

At the unit level, unit count metrics create a 1/0 flag for if they participated.

At the group level, Statsig calculates the mean as the SUM of the unit-level flags, divided by the count of UNIQUE UNITS exposed to the experiment.

The SQL for this calculation:

-- Unit Level
SELECT distinct
  source_data.unit_id,
  exposure_data.group_id,
  1 as value
FROM source_data
JOIN exposure_data
ON
  -- Only include users who saw the experiment
  source_data.unit_id = exposure_data.unit_id
  -- Only include data from after the user saw the experiment
  -- In this case exposure_data is already deduped to the "first exposure"
  AND source_data.timestamp >= exposure_data.timestamp
;

-- Experiment
SELECT
  group_id,
  COUNT(distinct unit_id) total_units
FROM exposure_data
GROUP BY group_id;

-- Group Level
SELECT
  group_id,
  SUM(value)/SUM(total_units) as mean
FROM unit_data
JOIN group_data
USING (group_id)
GROUP BY group_id;

Methodology notes

Statsig calculates daily and days-since-exposure views for user-day metrics per day. If a user is active on all 14 days of an experiment, they contribute 1 to the overall cumulative numerator, but 1 to each day of the daily view. The daily view doesn't deduplicate users. This yields a more intuitive interpretation and produces less sparse timeseries data on long experiments. However, this can cause mix-shift effects where the daily trend moves in the opposite direction of the cumulative timeseries, because daily returning users have already been counted and don't add metric value to the cumulative view.

Unit count metrics use the SQL SUM aggregation. See the Options section below for many advanced options.

Options

• Metric Breakdowns: configure Metadata Columns to group results by for dimensional views in pulse results.
• Multi Source: build metrics from multiple metric sources, equivalent to a UNION in SQL. Useful when the same measure exists in sharded or disparate tables.
• Rollup Mode: controls the specific way that Unit Count metrics are aggregated.