First or Latest Value Metrics

Use cases

First Value

First value metrics track the initial experience a user has with your product. This could be first purchase value, initial time-to-load, or any other initial result you are expecting to influence with your change.

Latest Value

Latest value metrics track how an experiment is impacting the state of your population. For example, you might want to measure if the test group has a higher net account balance, or if the current loyalty rewards balance is higher for users in one arm of an experiment.

Calculation

At the unit level, first value metrics calculate each day's first non-null value within cohort bounds. Similarly, latest value metrics calculate each day's latest non-null value within any cohort bounds.

For pulse, Statsig carries the first value forward from the first date with data, and determines the latest value by taking the latest value from the latest day available.

At the group level, Statsig calculates the mean as the SUM of the unit-level values, divided by the count of UNIQUE UNITS exposed to the experiment.

The SQL looks like the following:

-- Unit Level
SELECT
  source_data.unit_id,
  exposure_data.group_id,
  LATEST_VALUE(source_data.value_field) as value
  -- or FIRST_VALUE
FROM source_data
JOIN exposure_data
ON
  -- Only include users who saw the experiment
  source_data.unit_id = exposure_data.unit_id
  -- Only include data from after the user saw the experiment
  -- In this case exposure_data is already deduped to the "first exposure"
  AND source_data.timestamp >= exposure_data.timestamp
WHERE value_field IS NOT NULL
GROUP BY
  source_data.unit_id,
  exposure_data.group_id;

-- Experiment
SELECT
  group_id,
  COUNT(distinct unit_id) total_units
FROM exposure_data
GROUP BY group_id;

-- Group Level
SELECT
  group_id,
  SUM(value)/SUM(total_units) as mean
FROM unit_data
JOIN group_data
USING (group_id)
GROUP BY group_id;

Methodology notes

Statsig treats users without a value as 0s. If an existing value is present, the user retains that value unless a 0 or a new value is provided later.

Options

• Metric Breakdowns
  • You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results
• Cohort Windows
  • You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment
    • Select Only include units with a completed window to remove units from pulse analysis for this metric until the cohort window has completed
• CUPED
  • Specify if you want to calculate CUPED, and the lookback window for CUPED's pre-experiment data inputs

Special Case: Surrogate Metrics

You can use latest value metrics to implement surrogate metrics. Surrogate metrics (aka proxy metrics or predictive metrics) are a prediction of some long term metric that's impractical to measure over the duration of an experiment, and have some inherent prediction error associated with the model used to derive the metric values.

In advanced settings for latest value metrics, you can indicate a metric as a surrogate metric with a mean squared error (MSE). Prediction accuracy is then accounted for in variance calculation, which adjusts p-values and confidence intervals accordingly.

Consider the variable X to be the true north metric which is being predicted by the surrogate metric S. The surrogate metric S is assumed to be an unbiased estimator with an error term $\epsilon$.