Count Metrics

Use cases

Count metrics are commonly used to measure how often events occurred. Common examples are:

• Counting click frequency by counting event logs filtered to the click event
• Making a threshold metric of if a user read more than 2 articles

Calculation

At the unit level, count metrics run a COUNT(1) across their metric source.

At the group level, Statsig calculates the mean as the SUM of the unit-level count, divided by the count of UNIQUE UNITS exposed to the experiment.

The SQL looks like the following:

-- Unit Level
SELECT
  source_data.unit_id,
  exposure_data.group_id,
  COUNT(1) as value
FROM source_data
JOIN exposure_data
ON
  -- Only include users who saw the experiment
  source_data.unit_id = exposure_data.unit_id
  -- Only include data from after the user saw the experiment
  -- In this case exposure_data is already deduped to the "first exposure"
  AND source_data.timestamp >= exposure_data.timestamp
GROUP BY
  source_data.unit_id,
  exposure_data.group_id;

-- Experiment
SELECT
  group_id,
  COUNT(distinct unit_id) total_units
FROM exposure_data
GROUP BY group_id;

-- Group Level
SELECT
  group_id,
  SUM(value)/SUM(total_units) as mean
FROM unit_data
JOIN group_data
USING (group_id)
GROUP BY group_id;

Methodology notes

Count metrics use the SQL COUNT aggregation. Many advanced options are available.

Options

• Metric Breakdowns
  • You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results
• Multi Source
  • Counts can be built with inputs from multiple metric sources; think of this as a UNION in SQL. This can be useful if you have the same measure in sharded or disparate tables
• Winsorization
  • Specify a lower and/or upper percentile bound to winsorize at. Statsig clamps all values below the lower threshold, or above the upper threshold, to that threshold to reduce the outsized impact of outliers on your analysis.
• Capping
  • Specify an exact cap value (per unit type) to limit a unit's daily value for the count. For example, you might want daily purchases to be capped at a high value like 100 on an e-commerce site to ignore reseller behavior
• CUPED
  • Specify if you want to calculate CUPED, and the lookback window for CUPED's pre-experiment data inputs
• Thresholding
  • Turn this metric into a 1/0 unit count metric counting if the unit's total count equals to or surpasses (>=) a given threshold
• Cohort Windows
  • You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment
    • Select Only include units with a completed window to remove units from pulse analysis for this metric until the cohort window has completed.
• Baked Metrics
  • Baked Metrics let you specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig delays loading the data until the window has elapsed, and only calculates pulse results for that metric after a unit's metric has matured.