Skip to main content

SRM Checks

SRM - Sample Ratio Mismatch​

Sample ratio mismatch (SRM for short) is when the observed allocation of users between test groups differs from the expected allocation or "split" of the test. We have a brief rundown on this topic here on our blog.

This is a signal that there could be some unknown bias in the test. This is a major problem because unless you can clearly diagnose the reason for the imbalance, there's not an easy way to know how much this bias impacts your results.

SRM Checks​

Statsig runs SRM checks on all experiments and feature gates as part of our Health Checks (described here). We use a Chi-squared test to identify if the split of users between groups is indicative of a Sample Ratio Mismatch.

image

We automatically analyze data by common dimensions logged by the Statsig SDK to identify potential drivers of SRM. These include sdk_type, sdk_version, reason, is_bot, browser_name, browser_version, os, os_version and region to identify potential causes. image