WHN Statistics Overview

Statsig's statistics engine is designed to deliver high-quality analysis of experiment results. The Stats Engine has four core values: Transparency, Trust, Flexibility, and Power.

The pages below cover the Pulse Statistics Engine in detail, including how Statsig calculates results.

Transparency

Statsig documents all of its methodology, and the analysis is fully visible in Warehouse Native.

Statsig provides the tools, data, and queries you need to reproduce the results you see in the console, with no hidden logic. Most enterprise data science teams validate results as part of their evaluation process.

Transparency includes:

• Console-based access to the SQL Statsig runs. You have access to the tables and queries on your warehouse, and Statsig surfaces SQL snippets in the console whenever it runs a query so you can verify what is being calculated.
• Visibility into costs. Most customers can run pulse analyses for very low cost, and Statsig surfaces run time and resource utilization to help you manage your warehouse bill for experimentation.
• Support: Statsig offers support with access to the data science team for open discussion of methodology, approaches, and collaborative development of new features.

Trust

Experiments drive important business decisions, and it's critical that you can trust the analysis and statistics being run. Statsig has a rigorous evaluation process for its methodologies, including peer review, simulations, and publishing the thought process behind statistical designs.

You can trust Statsig's results to be accurate and reliable as they help guide your decisions.

Trust includes:

• Detailed blog posts on the rationale behind decisions in new features
• Documentation of methodologies in blogs, with references to prior art
• A full suite of diagnostic health checks on experiment results to warn you when statistical assumptions or data quality have been compromised

Flexibility

Experimentation isn't a one-size-fits-all tool. Depending on your industry, philosophy, or the setup of a specific experiment, Statsig lets you configure your analysis to suit your needs, offering:

• Standard T-Tests
• Sequential Testing
• Bayesian Tests
• Switchback Tests
• Multi-armed bandits

Statsig also offers many options to control for multiple comparisons, outliers, and regression adjustment.

Power

Effective experimentation relies on having trustworthy results quickly. Statsig has invested heavily in accuracy and power, so your results are faster and more reliable.

Examples of features focused on the power of Statsig's stats engine:

• CUPED: reduces experiment run times and accounts for pre-experiment bias
• Stratified Sampling: makes experiment results more accurate and consistent