Experiments Overview

Learn the fundamentals of experimentation with Statsig, including key concepts, randomization units, and statistical significance.

Statsig experimentation runs randomized controlled trials (A/B or A/B/n tests) that measure how product changes affect your key metrics. Control variables, randomization units, and statistical significance determine whether an observed result reflects the change or random variation. Use experiments to validate product changes, discover new opportunities, and confirm causal impact before you ship.

What are experiments

Experiments enable you to run randomized controlled trials (A/B or A/B/n tests) to measure the impact of product changes on key metrics.

Statsig’s experimentation platform helps you create, manage, and analyze experiments, ensuring you ship features that deliver value to your users and business.

Experiments are ideal when you want to:

Test multiple variants (A/B or A/B/n) of a product feature.
Run mutually exclusive experiments in parallel.
Measure the direct impact of changes on product and business metrics.

Why you want to experiment

Experiments are the most scientifically reliable way to establish causality between the changes you make to your product and their effect on customer behavior. By running experiments, you can:

Validate Hypotheses: Ship features only after experiments prove they improve the customer experience or drive key business metrics.
Measure Success: Measure feature performance post-launch and detect any unexpected side effects.
Drive Innovation: Provide teams with real-time feedback on product performance, enabling faster iteration and better, data-driven decisions.

Historical metrics may show correlation, but experiments allow you to establish causal relationships. Experiments reduce the influence of uncontrolled external factors. This ensures that observed effects are due to the testable changes.

Key concepts

Make sure you're familiar with the following concepts. These concepts are fundamental to every experiment you run on Statsig.

Control variables

A control variable is the variable in an experiment that you manipulate to observe its effect on key metrics. In a simple A/B test, the control variable usually has two values (A and B). More complex experiments may have additional values (for example: A, B, C, D). This is also known as multivariate experiments.

Variants

A variant is a specific version of the product or feature you're testing. For example, in an A/B test:

A (Control): Represents the current state of the product or feature. It's the control variant because it represents the existing, current state of your environment.
B (Treatment): Represents the modified state you want to evaluate. This is the single change you're experimenting with in your environment.

Statsig randomly assigns each variant to users, allowing you to compare the performance of each variant with an equal representation of users.

Randomization unit

The randomization unit is the entity (such as a user, device, or session) that Statsig randomly assigns to either the control or treatment groups in an experiment. Choosing the right randomization unit ensures consistency in user experience and reliable experiment results.

This choice is critical to ensure that experiment results reflect real-world user behavior and that unintentional crossovers between groups don't skew the data.

Statistical significance

Statistical significance determines whether the observed changes in metrics are likely due to the product change or random variation. Two commonly used methods are:

p-value: The p-value measures the probability of observing the results by chance if the variant had no effect. A p-value below 0.05 typically indicates statistical significance.
Confidence Interval: A confidence interval defines the range in which the true effect of a variant lies, with a given level of confidence (for example, 95%). If the confidence interval doesn't overlap zero, the effect is statistically significant.

For more information on designing, monitoring, and analyzing experiments, go to Product Experimentation Best Practices.

Common scenarios for experimentation

Optimize product growth

Use experiments to refine and optimize user experiences. This helps you move towards the highest level in your product strategy. Common goals include:

Optimizing a specific user journey (for example, improving onboarding).
Iterating on features to identify high-return opportunities.
Aligning experiments with business-critical metrics and guardrails to prevent negative side effects on fundamental business needs.

Explore new opportunities

Use experiments to discover entirely new directions for your product strategy. These experiments help you develop new ideas, validate strategies, and uncover long-term opportunities. For example:

Run experiments over longer durations to account for novelty effects and adoption time.
Slowly ramp up experiments to minimize risk and build statistical power.
Test multiple related hypotheses to explore a broader business strategy.

Choosing the right randomization unit

A randomization unit is the variable that determines how Statsig distributes users across your groups. (For example, users assigned to either the Control or Variant groups.) When you set a variable as the randomization unit, any value associated with that variable always receives the same experience. The randomization unit is also the reference unit for your metrics. For example, if you set the variable userID (User ID) as the randomization unit, Statsig deterministically buckets each user and userID serves as the basis of measurement. For example, your analysis might look at Revenue per userID. The following are common units of randomization and when to use them.

User identifiers

The most commonly used randomization unit is the User ID. Your application generates this identifier when a user registers or signs in to your product. Selecting User IDs as the randomization unit ensures a consistent user experience across sessions and devices because Statsig always assigns the user the same variant regardless of where or when they access your product.

Advantages

Persistent across sessions and devices: Statsig assigns the same variant to a user wherever and whenever they sign in.
Independent of client-side cookies: User IDs don't rely on cookies, which users can clear.

Drawbacks

Unavailable before sign-in: You can't randomize logged-out or anonymous traffic, so User IDs don't fit flows like landing pages or account creation, where the user has no ID at the time of assignment.
Misses pre-login behavior: Because assignment starts at sign-in, User IDs can't capture the full journey that spans logged-out and logged-in activity.

For more details on using User IDs with Statsig, go to User Identifiers.

Device identifiers

Using Device IDs or Anonymous User IDs is common to track users who haven't registered or signed in to your product. These identifiers track users based on their device and are ideal when experimenting with unregistered or guest users.

For example, you can use Device IDs to experiment on landing page optimizations aimed at improving user registration rates. Statsig can generate a Stable ID, which is a consistent identifier for a single device. It's a Device ID that is unique to Statsig. For example, if a guest interacts with your product through an iOS device, Statsig creates a Stable ID for that device. You can then track the actions of that device through the Stable ID. For more details about Statsig-specific Device IDs, go to Statsig Guide for Device-Level Experiments.

Advantages

Reaches anonymous users: Device IDs let you randomize logged-out and first-time visitors who don't yet have a User ID.
Consistent per-device experience: Statsig serves the same variant on a given device regardless of whether the user signs in.
Connects logged-out and logged-in activity: Because assignment stays with the device, you can measure behavior before and after sign-in for a fuller view of the journey.

Drawbacks

Device-specific: If the same user accesses your app from multiple devices, they may have different experiences.
Shared devices: If multiple users share a device, the experiment may mistakenly treat their behavior as belonging to one individual.
Not suited to long-term studies: If a user clears their cookies or switches devices, Statsig generates a new Stable ID that isn't linked to the previous one.

Session identifiers

In certain cases, you may use Session IDs as the unit of randomization, particularly when testing behavior during a single session. (For example, optimizing a checkout flow). Session-based randomization assumes each session is independent of all other sessions. This assumption may not hold if users return in multiple sessions. If a user returns over multiple sessions, they may experience different variants.

For example: you may use Session IDs when experimenting with conversion funnels for guest checkouts, which users typically complete within a single session.

Advantages

Isolates single-session behavior: Session IDs scope the experiment to one visit, so you can measure the effect of a change on a self-contained flow without cross-session carryover.
No persistent identity required: Like Device IDs, Session IDs let you randomize anonymous and guest traffic without a User ID.
Faster to reach statistical power: Each session is its own unit, so exposures accumulate quickly, which can shorten the time needed to detect an effect.

Drawbacks

Users may remember their experience from one session to another, undermining the assumption of session independence.
If users return in future sessions, Statsig may place them in different variants, leading to inconsistent user experiences.

Experiment detail tabs

When you open an experiment, its detail page organizes configuration and analysis into the following tabs:

Setup: Configure the experiment's scorecard, allocation, targeting, groups, and parameters. Go to Create an Experiment for more details.
Diagnostics: View a live log stream of checks and events from your application to confirm your integration works as expected. Go to Monitor an Experiment for more details.
Results: Read exposures and Scorecard metric lifts. Go to How to Read Experiment Results for more details.
Explore: Build custom queries to break down results by user or event dimensions.
Summary: Review a high-level summary of the experiment.

Tutorials

Was this helpful?