Skip to main content

SRM Checks

SRM - Sample Ratio Mismatch

Sample ratio mismatch (SRM for short) is when the observed allocation of users between test groups differs from the expected allocation or "split" of the test. We have a brief rundown on this topic here on our blog.

This is a signal that there could be some unknown bias in the test. This is a major problem because unless you can clearly diagnose the reason for the imbalance, there's not an easy way to know how much this bias impacts your results.

SRM Checks

Statsig runs SRM checks on all experiments and feature gates as part of our Health Checks (described here). We use a Chi-squared test to identify if the split of users between groups is indicative of a Sample Ratio Mismatch.


We automatically analyze data by common dimensions logged by the Statsig SDK to identify potential drivers of SRM. These include sdk_type, sdk_version, reason, is_bot, browser_name, browser_version, os, os_version and region to identify potential causes. image