SRM Checks
Sample ratio mismatch (SRM) detection in Statsig Warehouse Native flags skewed traffic splits in experiment scorecards so you can investigate and fix logging.
How SRM checks work
Sample ratio mismatch (SRM) occurs when the observed allocation of unique users between test groups differs from the expected allocation or "split" of the test. Go to the Statsig blog post on SRM for background.SRM is a signal that there could be unknown bias in the test. Unless you can clearly diagnose the reason for the imbalance, there's no reliable way to know how much this bias affects your results.
Statsig runs SRM checks on all experiments and feature gates as part of Health Checks (described in Health Checks). Statsig uses a Chi-squared test to identify whether the split of users between groups indicates a sample ratio mismatch.
Statsig automatically analyzes data by common dimensions logged by the Statsig SDK to identify potential drivers of SRM. These include sdk_type, sdk_version, reason, is_bot, browser_name, browser_version, os, os_version, and region.
Was this helpful?