Pulse is inspired by Deltoid, Facebook's internal A/B testing tool. You can learn more about the general ideas here: Itamar Rosenn (Facebook) - A/B Tests and Deltoid
Statsig's Pulse, which measures the impact of features rolled out behind Feature Gates, allows you to bring powerful experimentation to your application as well! If you want to get a feel for what Pulse can do, the best way is through the demo accounts, accessible via www.statsig.com/demo
Here is an example of pulse showing the impact of an experiment on an ecommerce website. Here, you can see the new feature improved the usage of the shopping cart, at the expense of daily active users (DAU) and product details views. These sorts of tradeoffs might make sense (say, adding an item to the cart takes you to the cart, instead of to browse additional product details), and you will have to decide if these tradeoffs are worth it before shipping the feature more broadly.
- Mean: The average daily value per user (Inactive users are counted as 0)
- Std: Standard deviation of this metric
- Abs Delta: Absolute difference of the Mean between groups (Pass Mean - Fail Mean)
- Delta %: Relative difference of the Mean. Calculated as the Pass mean divided by the Fail mean, minus 100%
- P-Value: The p-value is the (two-sided) probability of achieving the observed difference (or more extreme) assuming that there is no difference between the experimental groups. In classical statistics, a low enough p-value means that the assumption of no difference is incorrect