Types of Experiments
Statsig offers many forms of Experiment Analysis
Analysis only
A/B/n
Analysis-only A/B/n tests run analysis on top of assignment and metric data from your warehouse. In these experiments, Statsig operates as a statistics engine to help make analyses more reproducible.
Assign and analyze
A/B/n
The most common Assign-and-Analyze experiments are A/B/n tests that integrate warehouse data with Statsig's live SDK. In these experiments, Statsig operates as an assignment tool, a real-time diagnostics tool, and as a statistics engine. Assignment related features supported include:
Stratified Sampling
Dividing your population into homogeneous groups (based on a metric or classification).
Configurable Allocation Duration
You can enroll users for a subset of an experiment's duration. If you are experimenting on a one-time experience (such as signup flows), no additional configuration is needed. If the experience isn't one-time (enrolled users need to keep being assigned), configure your SDKs to use Persistent Assignment (you provide a store to save user enrollment states; the SDK manages this state). Go to Persistent Assignment for Client and Server SDKs.
Switchback experiments
Statsig offers Switchback tests for experimenting in the presence of meaningful network effects or in ecosystems where changing an experience for one group affects other groups (for example, a ride-service app changing prices for some users changes driver demand for all users).You can configure Switchback tests in Statsig's console. They use time periods (and optional buckets such as city or country) to alternate experiment conditions and run a bootstrapping analysis to estimate test statistics. Advanced options include burn-in and burn-out periods, allocation windows, and configurable window lengths.
Geo testing
Statsig offers Geotesting to support marketing and product causal inference techniques where you can't run a traditional A/B test. Geotesting treats geographic units (such as postal codes and DMAs) as the unit of analysis, enabling new experimental methodologies like rigorous testing of paid marketing or search on platforms like Google and Facebook Ads.Geotesting uses Synthetic Control methodologies built on GeoLift, an open-source package from Meta. All existing metrics and metric sources are available, with the addition of geographic labels.
MABs
- Autotune is Statsig's multi-armed bandit solution. It balances explore and exploit to deliver the optimal global treatment to users, making it useful for evaluating many options and dynamically adjusting traffic to avoid over-delivering underperforming variants.
- Statsig also offers a Contextual Multi-Armed Bandit, which extends the multi-armed bandit by personalizing the experience served to users based on "context" (user or event attributes provided to the Statsig SDK). This balances explore and exploit by optimizing for potential upside in its predictions.
getExperiment call. For the CMAB approach, provide relevant attributes to the user object. Contact the Statsig support team, your sales contact, or the Slack community for assistance with Contextual Multi-Armed Bandits.Was this helpful?