Percentile Metrics

Use cases

The mean of a metric can be acceptable while the tail end (for example, the 99th percentile) is not. Consider a website with an average TTL of 300ms but a p99 TTL of 1 minute. A small portion of the population has an unusable experience that doesn't appear in the mean, making it difficult to measure whether the p99 value changed with traditional A/B/n metrics.

Percentile metrics are common as guardrail metrics for performance regression, and for measuring improvements from investments in performance and infrastructure.

Calculation

For percentile metrics, there is no unit-level calculation; Statsig runs the analysis at the group level.

The SQL looks like the following:

-- Group Level
SELECT
  exposures_data.group_id,
  PERCENTILE(user_data.value, percentile_level) as value,
  COUNT(distinct user_data.user_id) as population
FROM user_data
JOIN exposure_data
ON
  -- Only include users who saw the experiment
  source_data.unit_id = exposure_data.unit_id
  -- Only include data from after the user saw the experiment
  -- In this case exposure_data is already deduped to the "first exposure"
  AND source_data.timestamp >= exposure_data.timestamp
WHERE value IS NOT NULL
GROUP BY group_id;

Methodology notes

Options

• Metric Breakdowns
  • You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results