Skip to main content

Scenarios

Statsig sees two broad scenarios for experimentation.

1. Experiment to grow faster​

Experiments can help climb a hill to a local optimum based on your current business strategy and product portfolio. For example,

  • Experiments can optimize for the ideal user experience for a given functionality
  • Experiments can help iterate on the functionality, algorithms, and infrastructure that matter the most to your users and your business
  • Experiments can identify proposals with the highest return for effort required

Identifying the metrics that both reflect your strategic direction and are sensitive to the changes you make ensures that you don’t waste resources. Identifying guardrail metrics that you want to hold regardless of the changes you make compels explicit tradeoffs and prevents you from regressing on the fundamental needs of your business.

2. Experiment to discover faster​

Experiments can help develop a portfolio of ideas that may point to a larger hill or opportunity. Navigating these bigger jumps may require:

  • Experiments that run for a longer duration to mitigate any novelty effects and to ensure that you have given the new product version enough time to build adoption
  • Experiments that ramp slowly and progressively to more users to limit risk and to build more statistical power before launch
  • Many different experiments that test several related hypotheses that form a new business strategy