How Evaluation Works

Why evaluation matters

The essential function of the Statsig SDKs is reliable, consistent, high-performance allocation of users to the correct bucket in your experiment or feature gate. Understanding how evaluation works can help you answer questions like:

• Why do I have to pass every user attribute every time?
• Why do I have to wait for initialization to complete?
• When does Statsig decide each user's bucket?

How evaluation works

Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always returns the same result, even across different platforms (client or server). The process is:

1. Salt Creation: Each experiment or feature gate rule generates a unique salt.
2. Hashing: The user identifier (e.g., userId, organizationId) is passed through a SHA256 hashing function, combined with the salt, which produces a large integer.
3. Bucket Assignment: The large integer is then subjected to a modulus operation with 10000 (or 1000 for layers), assigning the user to a bucket.
4. Bucket Determination: The result defines the specific bucket out of 10000 (or 1000 for layers) where the user is placed.

For advanced use cases (for example, a series of related experiments that must reuse control and test buckets), Statsig exposes the ability to copy and set the salts used for deterministic hashing. Use this with care. This option is only available to Project Administrators and appears in the Overflow (...) menu in Experiments.

Evaluation order

When evaluating gates, experiments, and layers, the SDK iterates through a list of rules generated by the server. Rules are evaluated sequentially and the first matching rule determines the result. Overrides always take precedence because they appear first in the rule list.

Experiments

When an experiment is evaluated (you call getExperiment), it follows this evaluation order:

1. ID overrides: Specific user/unit IDs mapped to a group
2. Conditional overrides: Segment or gate-based overrides, evaluated in order
3. Layer holdouts: If the experiment is in a layer, layer-level holdout gates are checked
4. Holdout gates: Experiment-level holdout gates; users in a holdout receive default values
5. Experiment exclusion: Mutual exclusion segments that prevent users from being in multiple experiments
6. Start status: If the experiment is not started, users receive default values (with optional non-production environment exceptions)
7. Layer allocation: For experiments in a layer, the user's bucket (based on the layer's universe salt) must fall within the experiment's allocated segments. This is checked before targeting.
8. Targeting gate: Users who fail the targeting gate receive default values. This is checked after layer allocation.
9. Group assignment: The user's bucket (based on the experiment salt) determines which group they fall into. Groups are cumulative ranges across 1000 buckets.

Layers

When a layer is evaluated (you call getLayer), it follows this evaluation order:

1. Override rules: ID overrides from all experiments in the layer
2. Layer holdout gates: Holdout gates attached to the layer
3. Experiment allocation: Each experiment in the layer has a configDelegate rule. The user's bucket determines which experiment they are delegated to.
4. Delegated experiment evaluation: Once delegated, the experiment's own evaluation runs (start status, targeting gate, group bucketing as described above)

If no experiment allocation rule matches, the user receives the layer's default values.

Holdouts

Holdout gates evaluate in this order:

1. Experiment exclusion: Exclusion segments (if applicable)
2. ID overrides: Specific user/unit IDs
3. Population targeting gate: If the holdout has a targeting gate, users who fail it are not held out
4. Holdout percentage: The pass percentage on the holdout rule determines the holdout rate

Gates

When a feature gate is evaluated (you call checkGate), it follows this evaluation order:

1. ID overrides: Specific user/unit IDs mapped to pass/fail
2. Conditional overrides: Segment or gate-based overrides
3. Holdout rules: If the gate has holdouts attached
4. Rules: The gate's targeting rules, evaluated in the order they appear in the console. Each rule has its own conditions and pass percentage.

When evaluation happens

On server SDKs, evaluation happens when the gate or experiment is checked. Server SDKs hold the entire ruleset of your project in memory (a JSON representation of each gate or experiment). On client SDKs, Statsig evaluates all gates and experiments server-side when you call initialize.

This logic applies to both SDK types. In both cases, the user's assignment bucket isn't sent to Statsig until you call getExperiment or checkGate.

What this means

• Performant evaluation: No evaluations require a network request. Checks take under 1 ms after initialization.
• SDKs don't remember user attributes or previous evaluations: Pass all required user attributes consistently on every call. Statsig returns the same value for the same inputs.

A common assumption is that Statsig maintains a list of all IDs and their assigned groups. Statsig's data pipelines track users exposed to each variant to compute experiment results, but Statsig doesn't cache previous evaluations or maintain distributed evaluation state across client and server SDKs.

• Server SDKs can handle multiple users: Because server SDKs hold the ruleset in memory, they can evaluate any user without a network request. Pass a user object into getExperiment on server SDKs; on client SDKs, pass the user object into initialize.
• Each user receives the same bucket: ID-based hashing guarantees consistency. If you make a change in the console that could affect user bucketing on an experiment, Statsig provides a warning.

Evaluation of null or empty unit IDs

Statsig doesn't apply any filtering or business logic before assigning an individual user ID. Statsig buckets a null or empty unit ID based on the salt.