# API Keys Source: https://docs.statsig.com/access-management/api-keys Reference for Statsig API keys, including client keys, server secret keys, and console API keys, with guidance on rotation, scopes, and security. ## Overview There are three main types of API keys: 1. **[Client API Key](#client-api-keys)**: Intended for getting configuration and logging events on the client side. 2. **[Server Secret Key](#server-secret-keys)**: Intended for getting configuration and logging events on the server side. 3. **[Console API Key](#console-api-keys)**: The most powerful key, intended for server side use for full CRUD operations on your Statsig project. You can generate API keys using the Statsig Console UI by going to **Project Settings** > [Environments & Keys](https://console.statsig.com/api_keys). ### Client API Keys Client API keys (Statsig Client SDK keys) are required to initialize all Statsig client SDKs. They are intended to be used in a client environment, such as a mobile app or a web app, where the key itself cannot be secret. Client API keys have access to the following: * /initialize endpoint, which returns all evaluated gates/configs/experiments/layers for a given user, with the names hashed. * /log\_event endpoint, which logs events to Statsig. Because they cannot access the actual names, and because the gates/configs/experiments/layers are all pre-evaluated, the project definition is not accessible via a Client API Key. This means that any conditions you create in the console, say, for a gate to pass for certain users or email addresses, will not be accessible via a Client API Key. ### Server Secret Keys Server Secret keys (Statsig Server SDK keys) are required to initialize all Statsig server SDKs. They are intended to be used on webservers or in server scripts, and have access to the following: * /download\_config\_specs endpoint, which downloads the configuration for a project * /log\_event endpoint, which logs events to Statsig. ### Console API Keys Console API keys (Statsig CAPI keys) should be treated as the most powerful keys in your project. Not only can they read all project configuration, they can also create, update, and delete entities in your project. They have access to the entire suite of [console api](/console-api/all-endpoints-generated) endpoints. You can also create **Personal Console API Keys** that are scoped to your role in Statsig - if your role has limited access, the same access conditions will be applied to the personal key. The intended use case is for multi-user projects where each user can have their own keys for clean access and audit logs. The ability to generate personal keys can be controlled in the [organization info settings tab](https://console.statsig.com/settings?tab=organization). ## Additional Considerations You can use [target apps](/sdks/target-apps) in combination with server secret keys and client keys to control which gates/configs/experiments/layers are accessible via each key. You can also create [per-environment API keys](/guides/using-environments#per-environment-api-keys) to control which rules are sent to the SDK based on the environment. Both client and server secret keys can also be used to access individual entities via the http API (/check\_gate, /get\_config, /get\_layer) ### Client Keys with Server Permissions Client keys can also get access to the download\_config\_specs endpoint via a scope you can add to the key. This is intended only for client local evaluation SDKs, like the [js-on-device-eval sdk](/client/js-on-device-eval-client), which we generally only encourage using in specific situations. Creating a key like this requires additional consideration. Your entire project scope is exposed to clients that can access this key: the names and configurations of all experiments and feature flags accessible by your client key are exposed. This risk may be acceptable (in fact, many other experimentation platforms operate in this manner by default), but you may choose to put additional processes in place to avoid the inclusion of certain information - like plaintext email addresses used in targeting, or confidential config names. We encourage you to: 1. Consider if on-device evaluation is the right choice for your use case. While on-device has [some benefits](/client/js-on-device-eval-client#pros), we generally view precomputed SDKs as the default solution. Discuss this with us in your [slack channel](https://statsig.com/slack). 2. Educate Statsig users on your team of the types of information that should be used in configs based on your info security policy. 3. Use [target apps](/sdks/target-apps) to filter your configs to only the ones needed for the client side, both for privacy and performance. # Discussions in Statsig Source: https://docs.statsig.com/access-management/discussions Use Statsig Discussions to collaborate on feature rollouts and experiments in context, keeping observations and decisions visible to your team. ## Discussions Feature rollout and experimentation are collaborative exercises where teams work together. Often this collaboration requires people taking screenshots, looking at metrics and DMing team mates to ask them questions. This information is siloed and lost to others. In-context Discussions allow teams to discuss aspects of the feature rollout or experiment while making this context available to other team members. Common reasons include discussing a surprising metric lift, asking questions on validation performed before ramping up a feature rollout or even keeping a running log of observations/escalations that can be referenced when the experiment is complete. Discussions interface in Statsig console # Initial Setup Guide of your Workspace Source: https://docs.statsig.com/access-management/guide Initial setup guide for configuring your Statsig workspace, organization, projects, and team access permissions for new Enterprise deployments. Organizations and their related features are for Enterprise contracts only. Please reach out to our support team, your sales contact, or our [Slack channel](https://statsig.com/slack) if you need to enable Enterprise features as you use Statsig. ## Overview This initial setup guide aims to provide a **step-by-step guide** for a set of essential configurations that you would need to get started with Statsig on an Enterprise plan. This guide is primarily written for **organization admins** who need to set up the initial environment and put the guardrails in place for the broader teams to start using Statsig. It also includes some **best practices** to increase your operational efficiency and set your team up for success in the long run. ### Statsig Workspace Structure Statsig structure diagram In Statsig, we have three constructs to help you organize your workspace and scale it out more broadly within your organization. **Organization** is an enterprise-level environment that allows companies to create project(s) and bring members to work inside the project. **Project** is a workspace within the organization where the configs (e.g., feature gates, experiments, layers, etc.), metrics, and SDK keys you and your team created lives. **Team** is a group of members at the project level that can help your organization manage the permissions and ownership of resources. #### Our Recommendation In Statsig, each project within the organization is isolated from one another. This means that **none** of the resources or data is shared among different projects, even if they are part of the same organization. The Statsig team believe that the **sensible default structure** for most customers is having an organization with a **single project** where multiple teams contribute and collaborate inside. We believe that different teams and functions across your organization can stay well-organized within a single project by leveraging features such as **[teams](/access-management/teams), [roles](/access-management/projects#roles), [tags](/access-management/tags),** and **[templates](/experiments/templates/templates)**. This also removes the risk of costly migrations if some projects ever need to be consolidated in the future for cross-functional collaborations. Statsig structure diagram anti pattern Note that Statsig has rich support for **environments** within our resources. We consider it an *anti-pattern* to create multiple projects just to manage lower environments separately from production. ### Setup Guide #### Step 1: Setting up the Organization When you sign an enterprise contract with Statsig, we will provision your existing Statsig account into an enterprise account. Once provisioned, you will see that the [tab for Organization](https://console.statsig.com/settings?tab=organization) is now added to your account along with additional security and governance settings. #### Step 2: Setting up SSO We recommend configuring [**SSO**](/access-management/sso/overview) (Single Sign-On) for your Statsig organization to simplify the user experience while improving your security. Statsig supports any Identity Provider that implements the **OIDC protocol** for SSO, such as [Okta](/access-management/sso/okta_sso), [Microsoft Entra ID](/access-management/sso/azuread), [Google](/access-management/sso/google), and more. Additionally, consider integrating [**SCIM**](/access-management/scim/overview) (System for Cross-domain Identity Management) if you are using Okta as your identity provider. New users who are provisioned via SSO will be assigned the *Member* Role unless you're using SCIM. #### Step 3: Creating your Project Organization project administration interface A project in Statsig serves as a workspace that contains everything you and your team will create. This includes configs (e.g., feature gates, experiments, dynamic configs, layers), metrics, integrations, and more. When creating a new project, you can set its type to *Open* which would allow anyone in the organization to join freely, or to *Closed* which would allow people to join only by invitation or request. #### Step 4: Setting up Roles Project roles interface Each person is assigned a role that defines their level of access within the project and organization. We recommend reviewing the permissions set for each predefined roles so you can [assign the appropriate roles](/access-management/organizations#organization-members) or create a custom role that aligns with your organization's needs. Common examples of custom roles include a *Metrics Admin* (responsible for managing metrics) and a *Data Warehouse Admin* (in Warehouse Native projects, responsible for managing warehouse connections). #### Step 5: Inviting your team to the Organization Project invite interface If you’ve successfully configured SSO, your teams can now join the Statsig organization by signing in from the console and going through the request flow within your identity provider. If you haven’t set up SSO yet, you can invite individual members to your organization and/or project in the console. #### Step 6: Creating Teams within the Project [Teams](/access-management/teams) can be created within the project to help with organization and ownership. Once a team is configured and users are added to team, any configs created will be associated with the team that the user belongs to. And these configs will automatically inherit all the default settings and templates from the team when they are created. Team filter in experiments Team becomes extremely useful as multiple teams begin to use Statsig, as it helps with organization and governance, as well as onboarding of new users as it provides a set of defaults to work from. As a best practice, you can require all resource creations to be attached to a team and have team admins (e.g., tech leads, engineering managers, etc.) add people to their corresponding teams, enabling them to start creating configs and metrics. ### Best practices #### Setting up Review Policies Review setting interface Within Statsig, you can enable [reviews](/guides/setting-up-reviews) for any changes to config and metric, requiring a second pair of eyes to review the change before they get deployed to production. Note that you can **customize** your review policies to align with your organization’s needs. For example, you can set up a team of reviewers by giving their roles a permission to approve the reviews. You can also give certain roles (e.g., oncalls) the ability to self-approve or require reviews for production environment only to allow for agility when needed. Each **team** also has team-level **review settings** that can require reviews for configs and metrics owned by specific team and allow only members and/or admins (with roles that permit approving reviews) of that specific team to be able to approve the reviews. #### Setting up your API Keys SDK environment interface In Statsig, you have **Client API Keys** to initialize all Statsig [client SDKs](/client/introduction) and **Server Secret Keys** to initialize all Statsig [server SDKs](/server/introduction). We believe there is a **"Crawl, Walk, Run"** phase when it comes to configuring your API keys: * **Crawl:** Begin by creating API keys that are scoped to specific environment within Statsig for [environment-based evaluation](/guides/using-environments#1-environment-specific-sdk-keys). * **Walk:** Next, create different API keys for each of the frontend clients (e.g., iOS, Android, and Web). * **Run:** At scale, consider having different API keys at the service level where each backend service and its environments have their own keys. This setup integrates seamlessly with [Target Apps](/sdks/target-apps) that can unlock additional performance and security. #### Configuring Target Apps for API keys Target apps interface [Target app](/sdks/target-apps) is an attribute you can associate to your SDK keys and configs. Through target apps, you can precisely set the scope of configs that will be accessed from each SDK key (which can also reduce the size of your payloads). We highly recommend setting up target apps as you scale the usage of Statsig for [additional performance and security](/sdks/target-apps#motivation). #### Setting up Notifications Statsig has an integration with [Slack](/integrations/slack) that you can set up, so you can receive real-time notifications about the activities, status changes, and other useful alerts directly in your Slack workspace. Additionally, you can configure to receive notifications in your [email](https://www.statsig.com/updates/update/email-notifs) as well for any alerts, reviews, reports, and more. # Workspace Management Overview Source: https://docs.statsig.com/access-management/introduction Overview of Statsig access management features for organizations, projects, teams, and SSO so you can scale adoption across your company securely. Statsig provides a few different solutions for access management as you scale out adoption in your team, organization, or company. We have simple settings like automatically adding new users with the same email domain to your project, but we also offer SSO to simplify the process for inviting your team to your Projects. To get started, learn more about Statsig's [organization](/access-management/organizations) and [project](/access-management/projects) abstractions. *** ## SSO vs SCIM In many enterprise environments, [SSO](/access-management/sso/overview) and [SCIM](/access-management/scim/overview) are used together to enhance both security and usability. * SSO handles the authentication aspect, allowing users to log in once to access multiple applications. * SCIM ensures that user accounts and permissions are consistently managed across those applications. ### Example Scenario: * Onboarding a New Employee: * SCIM: Automatically provisions the employee’s user accounts in all necessary applications based on their role. * SSO: Allows the employee to access all these applications with a single set of login credentials. * Offboarding an Employee: * SCIM: Automatically deactivates or deletes user accounts, removing access. * SSO: No longer authenticates the user since their credentials are disabled. ### Summary: * SSO focuses on simplifying the user login experience by centralizing authentication across multiple applications. * SCIM focuses on automating the management of user identities and attributes across different systems. By using both SSO and SCIM, organizations can achieve a more secure, efficient, and user-friendly approach to identity and access management. # Experiment Policy Source: https://docs.statsig.com/access-management/org-admin/experiment_policy Configure organization-level experiment policies in Statsig to require approvals, reviewers, and quality gates before launching A/B tests across projects. Organization level Experiment Policies are an Enterprise only feature. ## Experiment Policy grants organization admins the ability to: * Set defaults for new experiments: While Statsig aims to provide an intuitive platform for all users, we understand the unique requirements of every organization. Now, data scientists and experts can decide on experiment best practices for their business, such as default experiment duration and use of sequential testing. * Enforce these defaults: This ensures consistency across all experiments, and that best practices are enforced for experimenters. * Tailor Statsig for the business: Safeguard against unintended misuse by limiting the set of experimentation practices to ones effective to the organization. ## Where can these be configured? Organization Experiment Policies can be managed by visiting Product Configuration → Experimentation → [Organization tab](https://console.statsig.com/settings/products/experimentation). Only organization admins have the ability to modify these settings. Experiment policies # Feature Gates Policy Source: https://docs.statsig.com/access-management/org-admin/gates_policy Configure organization-level feature gate policies in Statsig to require approvals, reviewers, and review checks before publishing changes to gates. Organization level Feature Gate Policies are an Enterprise only feature. ## Overview Feature Gates Policy grants organization admins the ability to: **Configure the Custom Fields leveraged in Feature Gate targeting:** With this setting, admins can pre-define allowed Custom Fields by ID type, add a description to provide more context on the Custom Field to end users, and pre-define the allowed values. Defining allowed values is optional. Organization gate policy page showing allowed custom fields per ID type **Configure allowed Targeting Criteria for Feature Gates:** With this setting, admins can select which targeting criteria will surface as options during gate creation for end users. These allowed targeting criteria are defined at the ID type as well, enabling you to, for example, create one set of targeting criteria for logged-in (UserID) vs. logged-out (stableID) feature rollouts. Allowed targeting criteria configuration for feature gates by ID type **Require Templates for Gate or Dynamic Config Creation:** Enables admins to enforce that all gate or dynamic config creations are from a template. Restrictions on who can create/ edit templates (as well as which templates are allowed per-team) can be managed under **People** -> **Teams** and **Roles** Requirement toggle enforcing templates for new gates or configs ## Where can these be configured? Organization Gate Policy can be managed in settings by visiting Product Configuration → Feature Management → [Organization tab](https://console.statsig.com/settings/feature_management?tab=org). Only organization admins have the ability to modify these settings. Product Configuration settings highlighting Feature Management organization policies # Organization Policies Source: https://docs.statsig.com/access-management/org-admin/organization_policies Manage organization-wide policies in Statsig, including review requirements, approval workflows, and quality gates that apply across all projects. Organization-level Experiment and Gate Policies are an Enterprise only feature. ## Overview You can configure Organization-level [Experimentation](https://console.statsig.com/settings/products/experimentation?tab=org) and [Feature Management](https://console.statsig.com/settings/feature_management?tab=org) policies via your **[Settings](https://console.statsig.com/settings)** page. These policies will apply to all experiments and gates created within any Project in your Organization. Read more about: * [Experiment Policy](/access-management/org-admin/experiment_policy) * [Feature Gate Policy](/access-management/org-admin/gates_policy) # Organization Settings & Administration Source: https://docs.statsig.com/access-management/organizations Configure Statsig organizations for Enterprise customers, including creating organizations, assigning admins, and managing settings across multiple projects. Organizations and their related features are for Enterprise contracts only. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you need to enable Enterprise features as you use Statsig. ## Creating an Organization Statsig creates an organization for companies with an Enterprise contract. When you move a project to the organization, * All existing **members** will receive an invitation to join the organization * All existing project **administrators** will remain the same * The project will be added as **Open** i.e. any registered Statsig user with the same email domain as the project owner will have access to the project; you can change this setting in the **Project Settings** tab If you don't move a project to the organization, you may lose access to it as it will be archived. ## Managing Organizations Creating an organization will automatically make you the organization's **Owner** and **Admin**. There can only be one organization Owner but you can have as many Admins as needed. You can view your Organization page by navigating to [Account Settings](https://console.statsig.com/account_settings) in the settings tab and clicking on **Manage** next to your organization. Account Settings organization management interface #### Organization Information Your [**Info** setting's **Organization** tab](https://console.statsig.com/settings?tab=organization) includes the organization's name, SSO configuration, and other settings on access management and security settings. As the organization's **Admin**, you can enable or disable SSO for all projects in the organization from this one place. Organization information interface #### Organization Members All members in the organization will be able to view all current members in the organization from [Members Settings](https://console.statsig.com/settings/members?tab=organization). As the organization's **Owner** or **Admin**, you can also manage its membership by: * Promoting members to the organization's **Admin** * Removing members from the organization * Managing invitations including sending, viewing, and canceling invitations as well as tracking all pending invitations Organization member management interface To change the member's roles within the organization or remove member from the organization, check the member's name from the checkbox in the list and click one of the icons at the top right of the list. ## Managing Projects Members of an organization will be able to view all public projects in the organization. As the organization's **Owner** or **Admin**, you can view every project in the organization as well as **archive** the project if it's no longer necessary. Organization project administration interface # Project Access Management Source: https://docs.statsig.com/access-management/projects Manage project-level access in Statsig by assigning roles to invited users, configuring permissions, and controlling who can edit gates and experiments. This guide applies only to our on-demand customers. If you are an organization who has set up SSO, this guide will not apply to you. Please see our [SSO Guides](/access-management/sso/overview) for more information about how to manage access permissions through SSO. In the [Basic Settings page](https://console.statsig.com/settings) you are able to configure who has access to your project. Each person invited to a project is assigned a Role that specifies their level of access. ## Roles The different Project Roles available are: | Role | Description | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Read-Only | Users with this role only have access to read data on the Project. This includes reading gate and dynamic config configurations, experiment data, and metrics. Any actions users with these roles take that attempt to edit the configuration for the Project will fail. | | Member | Users with this role are able read data for a Project and are able to edit configurations within the Project. This includes being able to create and modify: Feature Gates, Dynamic Configs, Holdouts, Experiments+, etc. | | Admin | Users with this role have the same access as `Member` but are additionally able to modify Project Access settings. This includes inviting new users to the Project and changing Roles for existing users of the Project. | | Owner | Only the user that has created a project is given the Role of `Owner`. The Owner of a Project has the same access as `Admin` but is additionally able to delete the Project and configure automatic invitations for the Project. If you need to change the Owner of a project, the current owner can change it by going to Settings -> Project Members & Invites -> select the person and edit role to be Owner. | ## Custom Roles Custom roles are for Enterprise contracts only. Please reach out to our support team, your sales contact, or via our [Slack channel](https://statsig.com/community) if you need to enable Enterprise features as you use Statsig. Enterprise customers can customize roles used to assign permissions in Statsig. You can create new roles beyond Admin, Member and Read-Only and choose what permissions these roles have. Common use cases include creating a Metrics Admin role or a Warehouse Admin role (for Statsig Warehouse Native). Custom roles configuration interface ## Automatic Project Invitations To simplify sending invitations for a Project, you can allow users creating a new Statsig account to automatically join your project if their work email domain matches the `Owner's`. For example, a Project Owner with an `@statsig.com` email can enable all new users signing up with an `@statsig.com` email to automatically join their project. To enable this feature: 1. Go to your [Project Basic Settings page](https://console.statsig.com/settings). 2. Click on the `Edit Project Settings` button. 3. Toggle the checkbox labeled `Anyone who signs up with the same email domain can join` and choose the role that users will be assigned to. Project settings automatic invitation configuration # SCIM Concepts Source: https://docs.statsig.com/access-management/scim/concepts Core SCIM concepts in Statsig, including users, groups, role mappings, and how provisioning sync works between your identity provider and Statsig.
Our SCIM implementation represents both Statsig users at the Organization level and Project level with their associated roles. There are two major resources in our SCIM implementation: Users and Groups. ## Users Users in SCIM correspond to Statsig users at the Organization level. Users that are part of the Statsig Organization and have the organization email domain will be recognized as SCIM users. Each user has the following attributes: * First name * Last name * Email address (main identifier, must be unique within the IdP) * Statsig ID (used by the IDP to identify the user in the Statsig SCIM API) ## Groups Groups in SCIM represent Statsig Projects with specific roles. For example, a Project named "Project A" with a role of "Admin" would be represented as a group in SCIM. The group name for this example project would be `Statsig-ProjectA-Admin`. ### Team x Role Groups Team x Role Groups are a special type of group that represent Statsig Teams with specific roles. For example, a Team named "Team A" with a role of "Admin" would be represented as a group in SCIM. The group name for this example team would be `Statsig-ProjectName-TeamA-Admin`. Important notes about groups: * We do not allow project level deletion via SCIM, we do support team level deletion. * We currently do not support updating group names via SCIM. ### General Mapping Between SCIM and Statsig Mapping diagram showing Okta group synced to Statsig project and team roles via SCIM # Okta SCIM Org Roles Source: https://docs.statsig.com/access-management/scim/okta_scim_org_roles Map Okta groups to Statsig organization roles via SCIM provisioning so user permissions stay in sync between your identity provider and Statsig. ## Update Okta User Org Role For every user, Statsig surfaces a SCIM field named `statsigOrgRole`. Through this field, you can manage organization user roles. Currently, Okta can only push role updates to Statsig. We currently support the following org roles: `Member` `Admin` `Owner` ### Step 1. Create the Custom Attribute in Okta Navigate to `Directory > Profile Editor` and select the User (default) Okta profile. This represents all of the Okta users' attributes. Scroll down and press `Add Attribute` and fill out the new attribute to have the variable name `statsigOrgRole`. Okta profile editor showing custom statsigOrgRole attribute being added ### Step 2. Create the Custom Attribute in the Statsig SCIM Integration Now Navigate to the `Statsig SCIM Integration's User Profile` in the `Profile Editor`. Add an new attribute that matches the following format: * Variable name: `statsigOrgRole` * External namespace: `urn:ietf:params:scim:schemas:core:2.0:User` * Attribute type: either `Personal` or `Group`, depending if using groups for app assignment Statsig SCIM integration user profile with statsigOrgRole attribute definition ### Step 3. Create a Mapping from Statsig to Okta for the Custom Attribute On the same Statsig SCIM profile editor, navigate to the `Mappings` button. Scroll down to the new attribute `statsigOrgRole` and map `user.statsigOrgRole` to the Okta attribute `statsigOrgRole`. Okta mapping editor linking statsigOrgRole between Statsig and Okta profiles ### Step 4. Create a mapping from Okta to Statsig for the Custom Attribute Now navigate to the Okta User to Statsig SCIM user mapping. Okta to Statsig user mapping screen for statsigOrgRole attribute Scroll down to the `statsigOrgRole` attribute and map `user.statsigOrgRole` to the Okta attribute `statsigOrgRole`. statsigOrgRole attribute mapping row pointing from user field to Okta attribute Now all users will be synced with their organization role. On the Statsig SCIM integration you can modify a user's role directly as well. ### Step 5. Modify Integration Mappings Navigate to the Statsig SCIM integration provisioning section. Under the "To App" tab, scroll down to the `statsigOrgRole` attribute. Provisioning To App tab showing statsigOrgRole attribute settings Set the attribute value to `Map from Okta Profile` and `statsigOrgRole`. Set apply on `Create and update`. Set value dialog choosing Map from Okta Profile for statsigOrgRole Navigate to the "To Okta" tab and scroll down to the `statsigOrgRole` attribute. Provisioning To Okta tab listing statsigOrgRole attribute Set the attribute value to `Map from Statsig Profile` and `statsigOrgRole`. Set apply on `Create`. Set value dialog mapping statsigOrgRole from Statsig profile back to Okta # Okta SCIM Setup Source: https://docs.statsig.com/access-management/scim/okta_scim_setup Step-by-step guide to set up Okta SCIM provisioning with Statsig, including the integration app, attribute mappings, and group assignment configuration. This guide outlines the process for setting up SCIM (System for Cross-domain Identity Management) integration between Statsig and Okta. This integration allows for automated user provisioning and management. ## Prerequisites * An Okta account with admin access * A SCIM Key from the [Statsig Console](/access-management/scim/overview#how-to-obtain-scim-auth-key) (requires Statsig Org Admin rights) ### Integration Notes * User email management is not enabled on SCIM yet. * When a user is removed from Statsig, they will be automatically unassigned in Okta. Conversely, if a user is unassigned or deactivated in Okta, they will be removed from the Statsig Organization. * Creation of Statsig Projects and Roles is not supported via SCIM. ## Step 1: Create a New App Integration in Okta * Log in to your Okta admin console * Navigate to Applications > Applications > Create App Integration * Select "SWA - Secure Web Authentication" Okta Create App Integration dialog selecting Secure Web Authentication ## Step 2: Configure App Settings * Set the App name to "Statsig SCIM" * Enter a placeholder URL for the App Login Page (this is a required field but not used for SCIM). Ex: `https://console.statsig.com/` Okta app settings form with Statsig SCIM name and placeholder login URL ## Step 3: Enable SCIM Provisioning * After creating the integration, go to the "General" tab * Click on "Edit" in the "Provisioning" section * Enable "SCIM Provisioning" Okta application general tab highlighting provisioning section and SCIM toggle ## Step 4: Configure SCIM Settings :::info `Import Groups` requires an Okta flag `SELECTIVE_APP_IMPORT_PLATFORM`. If this flag is enabled for your organization, please select this option. If it is not, leave it unchecked. ::: * Navigate to the `Provisioning` tab * Set the SCIM connector base URL to: [https://statsigapi.net/scim](https://statsigapi.net/scim) * Set "Unique identifier field for users" to `userName` * Enable * `Import New Users and Profile Update` * `Push New Users` * `Push Profile Updates` * `Push Groups` * `Import Groups` (Only if your organization has the `SELECTIVE_APP_IMPORT_PLATFORM` flag enabled, see note above) * Set the authentication mode to "HTTP Header" * For the authorization header, use the SCIM Bearer token generated in Statsig by your Org Admin. See [How to Obtain SCIM Auth Key](/access-management/scim/overview#how-to-obtain-scim-auth-key) for more details. Okta provisioning tab showing SCIM base URL and push settings ## Step 5: Configure Okta to Statsig Settings * Enable "Create Users" * Enable "Update User Attributes" * Enable "Deactivate Users" Provisioning To App settings enabling create, update, and deactivate actions ## Step 6: Import Existing Statsig Users and Groups * In Okta, go to the Statsig app's "Import" tab * Click "Import Now" to fetch existing Statsig users and groups * Process the imported users as needed Okta Import tab with Import Now button for Statsig users and groups # Okta SCIM Team Management Source: https://docs.statsig.com/access-management/scim/okta_scim_team_management Manage Statsig teams from Okta using SCIM provisioning so team membership stays in sync with Okta groups and reflects organizational changes automatically. ## Import Existing Statsig Teams Teams are specific Statsig groups that exist within projects. They are not shared across projects. They also have two possible roles: `Admin` and `Member`. ## Setup * Ensure you have some teams created in Statsig. * Navigate to the Statsig Project and create a team if you have none. Statsig project team creation dialog showing fields for team name and members ## Import Existing Groups * In Okta, go to the Statsig app's "Import" tab * Click "Import Now" to fetch existing Statsig users, groups, and teams * After importing, you should see the group in your Okta groups following the format `Statsig-ProjectName-TeamName-RoleName`. Okta Import tab displaying Statsig team groups discovered from SCIM ## Create a Mapping Group * To push a group of Okta users to a Statsig team, you must first create a mapping group. * This group will contain the Okta members which will be part of the Team x Role Group. Okta group creation screen for mapping Statsig team members ## Push Teams to Statsig 1. In Okta, go to the Statsig Integration's "Push Teams" tab and select "Push Groups". Then pick the mapping group created in the previous step. Okta Push Groups dialog selecting mapping group to link with Statsig team 2. Change the Match result & push action to Link group. Then select the Statsig team you want to push to. It should follow the format `Statsig-ProjectName-TeamName-RoleName`. Match result settings linking Okta group to Statsig-Project-Team role name 3. When you are finished setting up the push, click "Save". ## Verify the Push * Navigate to the Statsig Project and verify that the team has the new members. * This could take some time for Okta to push the members to the team. Statsig team detail page showing synced Okta members ## Delete the Team * On Okta, navigate to push groups and find the pushed group that maps to the Statsig team. * Click the push status section and select "Unlink Group". Okta push groups status view with unlink group option * When prompted to delete the group or leave it in the app, select "Delete". * After Okta pushes the deletion, the team should be deleted from Statsig. This may take a few minutes to complete. Confirmation dialog asking to delete pushed group from app ## Optional: Create a Team Via SCIM * Find an existing Okta group with the members for a new Statsig Team. * Navigate to Push Groups and Select "Push Groups". * Select the group you want to push to Statsig. * Change the Match result & push action to Create Group. * The naming for this group must match the format `Statsig-ProjectName-TeamName-Member`. Okta push groups workflow set to create new Statsig team group * When you are finished setting up the push, click "Save". * Afterwards, you should see the new team in the Statsig Project you specified. Statsig console showing newly created team populated from SCIM push # Okta SCIM Troubleshooting Source: https://docs.statsig.com/access-management/scim/okta_scim_troubleshooting Troubleshoot common Okta SCIM provisioning issues with Statsig, including failed assignments, attribute mismatches, and role mapping errors. ## Common SCIM Errors | Error | Solution | | ------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | User userID is not allowed to be created in this organization | Ensure that the users you are assigning to Statsig have the correct email domain. We require users to have the org's email domain. Contact Statsig if you need to add additional domains. | | Owner cannot be modified | The owner of the organization can only be replaced by another user. | | Cannot set multiple owners | We do not allow multiple owners for an organization. Ensure that you are not trying to set multiple owners for the organization. | | Project creation is not supported via SCIM | Project creation is not supported via SCIM. Please create projects in the Statsig Console. | | Project deletion is not supported via SCIM | Project deletion is not supported via SCIM. Please delete projects in the Statsig Console. | ## Common Issues * I'm not seeing any Statsig groups in Okta * Your organization may have the Okta flag `SELECTIVE_APP_IMPORT_PLATFORM` enabled. If this is the case, you will need to select `Import Groups` in the SCIM integration settings. See the [setup page](/access-management/scim/okta_scim_setup) for more info. * I can't select `Import Groups` in the SCIM integration settings * Your organization may have the Okta flag `SELECTIVE_APP_IMPORT_PLATFORM` disabled. If this is the case, you cannot select `Import Groups` in the SCIM integration settings. However, when importing you will see the groups in the Okta SCIM integration. * I'm trying to push groups but not seeing users populate those groups * Ensure that the users you are pushing to Statsig are assigned to the integration. * If the users are assigned, verify that the groups you are pushing to Statsig are the Statsig Groups generated on import. * I don't want to use Statsig generated groups, can I use my own? * You can use your own groups in the push groups section. However, you will need to push these groups to a specific Statsig Project x Role Group. There is no way to push your own groups without pushing to a Statsig Project x Role Group. # Okta SCIM User and Project/Role Management Source: https://docs.statsig.com/access-management/scim/okta_scim_user_management Manage Statsig users from Okta using SCIM provisioning so accounts are created, updated, and deactivated automatically when Okta state changes. ## Import Existing Statsig Users and Groups Users not assigned to the integration cannot be pushed into groups. * In Okta, go to the Statsig app's "Import" tab * Click "Import Now" to fetch existing Statsig users and groups * Process the imported users as needed Okta Import tab listing Statsig users ready to be brought into Okta ## Manage User Assignments * Use the "Assignments" tab in Okta to add or remove users from Statsig * Adding a user assignment in Okta will create the user in Statsig, while removing the assignment will deactivate the user's Statsig account Okta Assignments tab showing Statsig app user assignment controls ## Push Groups to Statsig 1. In Okta, go to the Statsig Integration's "Push Groups" tab Push Groups tab in Okta Statsig integration 2. Click the settings button and disable "Rename Groups" Push group settings dialog with Rename Groups toggle 3. Click "Push Groups" and select the method for finding groups in Okta. Okta modal prompting to find groups by name for push 4. Type in and select the Okta group that will push to a Statsig Project x Role Group. * You can find Groups in left nav of Okta: `Directory > Groups`. In there, you will see the groups created from Okta and groups created by Statsig. * The required groups are groups you created from Okta. You can filter by choosing `Group source type` and set to `Okta groups`. If you don't have any, go ahead and create it with members as well. Directory listing of Okta groups filtered to Okta source 5. Now let's link/assign Okta group you created from Okta to the Statsig groups with role you want. * Change `Match Result & Push Action` to `Link Group` Push group configuration selecting Link group action 6. Select the Statsig Project x Role Group that the Okta group will push to. * We display the Statsig Project x Role Group with the format `Statsig--` on Okta. * By default Okta only allows you to map 1 Okta Group to 1 Statsig Group. Statsig project role group dropdown showing Statsig-Project-Role format 7. Then link the Okta group to a Statsig Project x Role Group. On save the group should push to Statsig. All future group changes on Okta will be pushed to Statsig. Summary screen confirming Okta group linked to Statsig project role # SCIM User Provisioning Source: https://docs.statsig.com/access-management/scim/overview Overview of SCIM user and group provisioning in Statsig, supported identity providers, and how automated sync works for Enterprise customers. ## Introduction SCIM (System for Cross-domain Identity Management) is a standardized protocol that simplifies the automation of user provisioning and management across multiple platforms. By integrating SCIM with your preferred Identity Provider (IdP), such as Okta, you can securely and efficiently manage user creation, updates, and de-provisioning within Statsig. We currently only offer an Okta SCIM integration. ## How To Obtain SCIM Auth Key You must be a Statsig Organization Admin in order to enable SCIM. The SCIM key includes the `scim` prefix. Statsig organization settings showing SCIM key management panel 1. Navigate to Organization Access Management: Go to [Settings > Organization > Organization Info > Access Management](https://console.statsig.com/settings?tab=organization). 2. Generate a Key: If SCIM is not yet enabled, generate a new authentication key. 3. Deactivate the Key: To disable the key, select Deactivate. 4. Regenerate the Key: If you suspect the key has been compromised, you can regenerate a new one to replace it. ## Current SCIM Offering ### Okta * **Push Users.** Assigning users to the Statsig application in Okta automatically adds the users as members of your organization in Statsig. Unassigning users deactivates them and wipes all roles/permissions. * **Import Users.** Users in Statsig can be imported into Okta and either matched to existing Okta users, or created as new Okta users. * **Import Groups.** Import project/team roles as groups in Okta. Note that imported groups cannot be modified in Okta. * **Push Groups.** Groups can be pushed to Statsig to update user project/team roles. [Okta Setup Guide](/access-management/scim/okta_scim_setup) # SCIM API Overview Source: https://docs.statsig.com/access-management/scim/scim-endpoints Reference for the SCIM 2.0 endpoints supported by Statsig, including users, groups, schemas, and resource types for identity provider integrations. The System for Cross-domain Identity Management (SCIM) specification is designed to make managing user identities in cloud-based applications and services easier. Statsig's SCIM API allows you to automate user provisioning and deprovisioning between your identity provider and Statsig. This API reference provides details for all available SCIM endpoints and operations. # Single Sign-On With Entra ID/Azure AD/Office 365 Source: https://docs.statsig.com/access-management/sso/azuread Configure Single Sign-On for Statsig with Azure AD (Microsoft Entra ID) using OIDC, including app registration, claims, and group-based role assignments. Microsoft Entra ID, formerly known as Azure AD, is a supported IdP for SSO into Statsig. ## Requirements * You will need to be the `Admin` of the Statsig Project you intend to add SSO with Azure AD to. * You will need to be the Administrator of the Azure AD tenant you want to link. ## Supported Features Service Provider(SP)-Initiated Authentication for Single Sign-On (SSO) using OIDC can be enabled on Statsig to connect your Azure AD account to your Statsig Projects. ## Configuration ### Adding the Statsig OIDC Application in Azure AD 1. Navigate to App Registrations in the Azure portal. 2. Click on the "New Registration" button to register the Statsig App 3. Provide a name (eg. Statsig Console) to the app and finish creation. For the Redirect URI use [https://console.statsig.com/sso/oidc](https://console.statsig.com/sso/oidc) 4. Select Certificates and secrets for this app from the left nav and create a new client secret. Save the value; this can't be retrieved except immediately after creation. Azure AD app registration interface You will need to enter three items in the Statsig SSO configuration - 1. Find the Azure AD OIDC URL for this app under Overview -> Endpoints -> OpenID Connect metadata document. Truncate the string to end at "/v2.0". 2. Get the client ID for this app from Overview -> Application (client) ID. 3. Get the client secret value for when you saved this after creating a new client secret. Once these steps have been completed, the prep on Azure AD is complete. Following this, you will need to follow the steps [here to enable configuration of SSO on your Statsig Project](/access-management/sso/overview#configuration). # SSO with Google as your IdP Source: https://docs.statsig.com/access-management/sso/google Configure Single Sign-On for Statsig with Google Workspace using OIDC, including app registration, claims, and group-based role assignments. ## Requirements * You will need to be the `Admin` of the Statsig Project you intend to add SSO with Google Apps to. * You will need to be the Administrator of the Google Apps account you want to link. ## Supported Features Service Provider(SP)-Initiated Authentication for Single Sign-On (SSO) using OIDC can be enabled on Statsig to connect your Google Apps account to your Statsig Projects. ## Configuration 1. Follow the instructions listed [here](https://developers.google.com/identity/openid-connect/openid-connect#getcredentials) to obtain the client ID and client secret from your Google account. At [Google Credentials](https://console.cloud.google.com/apis/credentials), you can create a new "OAuth 2.0 Client ID" or select an existing one to find the client ID and client secret (under "Additional information"). 2. Add Statsig's redirect URI ([more detail here](/access-management/sso/overview#configuration)) under "Authorized redirect URIs" on the chosen OIDC Client: `https://console.statsig.com/sso/oidc`. 3. Follow the steps listed [here to enable configuration of SSO on your Statsig Project](/access-management/sso/overview#configuration). The OIDC Domain / Issuer field should be set to `https://accounts.google.com`, and the client ID / secret come from step 1. # Single Sign-On With Okta Source: https://docs.statsig.com/access-management/sso/okta_sso Configure Single Sign-On for Statsig with Okta using OIDC, including app integration, claim mappings, and role assignment for invited users. ## Requirements * You will need to be the `Admin` of the Statsig Organization you intend to add SSO with Okta to. * You will need to be the Administrator of the Okta account you want to link. ## Supported Features Statsig supports the OIDC protocol for SSO with the following flows: * Service Provider(SP)-Initiated Authentication for Single Sign-On (SSO). This flow is initialized when logging in on the Statsig website. * Identity Provider(IDP)-Initiated Authentication for SSO. This flow is initialized when launching the Statsig App from Okta. * Just-In-Time (JIT) provisioning for SSO. Upon successful login for the first time, Statsig automatically provisions an account for the user. ## Configuration ### Adding the Statsig OIDC Application in Okta 1. Navigate to your Okta portal. 2. On your Okta portal, click on `Applications` on the left-hand-column, and click into `Applications` in the dropdown. Okta portal navigation highlighting Applications menu 3. On the Applications page, click on the `Browse App Catalog` button. Okta Applications page with Browse App Catalog button 4. On the App Catalog page, use the searchbox to search for Statsig and click on the Statsig OIDC Application. 5. In the Statsig Application, click on the `Add` button. Statsig app listing within Okta catalog showing Add button 6. After creating the Statsig OIDC Application in Okta, navigate to the `Sign On` tab in the Application, note the `Client ID` and `Client Secret` fields that will be needed to enable Single Sign-On with OIDC on the Statsig Project. Also note that when adding the Statsig OIDC Application in Okta, the sign-in and sign-out redirect URIs are automatically configured. Okta Sign On tab showing Client ID and Client Secret values Once these steps have been completed, the Statsig OIDC Application in Okta has been successfully configured. Next, you will need to follow the steps [here to enable configuration of SSO on your Statsig Organization](/access-management/sso/overview#in-statsig-console). ## SP-Initiated SSO 1. Navigate to [https://console.statsig.com/sso](https://console.statsig.com/sso) 2. Enter your email address and click on "Authenticate" 3. You will be redirected to authenticate with Okta. If prompted, enter your Okta credentials. 4. Upon successful authentication, you will be redirected and logged in to Statsig. ### Proof Key for Code Exchange (PKCE) Statsig does not currently support the PKCE Flow, so you will need to turn off the feature in Okta when you enable SSO with Statsig. # Single Sign-On With OIDC Source: https://docs.statsig.com/access-management/sso/overview Overview of Single Sign-On with OIDC in Statsig, supported identity providers, and how to enable SSO for Enterprise customers and large organizations. SSO is an Enterprise feature. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you need to enable Enterprise features as you try out Statsig. *This documentation assumes that you already have an OIDC Provider up and running.* Single Sign-On (SSO) with OIDC can be configured for your Statsig Organization to continue using your company's identity store with Statsig and simplify the process for inviting your team to your Projects. New users will be automatically provisioned, once authenticated by your Identity Provider. Organizations are an Enterprise Tier feature. If your SSO requires multi-factor authentication (MFA), it is automatically required when your users sign into Statsig with SSO enabled. ## Supported Providers We support any Identity Provider (IdP) that implements the OIDC protocol for SSO. We have custom documentation for some of the following OIDC providers: * [Okta](/access-management/sso/okta_sso) * [Microsoft Entra ID (AzureAD)](/access-management/sso/azuread) * [Google](/access-management/sso/google) * Ping Identity * Be sure to include `openid` and `email` in the scopes * OneLogin ## Configuration ### In your Identity Provider You will need to specify the following for your Statsig App: * Sign-in redirect URI: [https://console.statsig.com/sso/oidc](https://console.statsig.com/sso/oidc) (and [https://latest.console.statsig.com/sso/oidc](https://latest.console.statsig.com/sso/oidc) if possible) * Sign-out redirect URI: [https://console.statsig.com](https://console.statsig.com) * Sign-in URI: [https://console.statsig.com/sso](https://console.statsig.com/sso) To enable SSO in Statsig, you will need to collect the following from your OIDC Provider: * OIDC Domain * Client ID * Client Secret ### In Statsig Console Once you have obtained all of the information mentioned above: 1. Navigate to your Organization's [`Info Settings` page](https://console.statsig.com/settings?tab=organization) and click the `Enable` button for Single Sign-on. An `Owner`/`Admin` role in your Statsig organization is required to configure SSO on Statsig SSO enable button in organization settings 2. Provide the information acquired from your OIDC Provider into the fields in the dialog and click `Enable`. SSO configuration dialog with OIDC provider fields 3. After clicking `Enable`, an SSO link will be shown that can be sent to your team to allow them to login to Statsig through your OIDC Provider. SSO link generated for team login By default, users who are provisioned via SSO will be assigned the "Member" role in the organization. If the organization has only one open project, users that sign in through an SSO link will automatically join any Projects that have SSO enabled with the same OIDC Provider. If there are multiple projects, users will be added to the organization but will need to request to join open projects or be invited to closed projects. Enabling `Strict SSO` will require that all members of a Project besides the `Owner` must log in to the Statsig Console through SSO with the configured provider to access the Project. ## Break Glass Scenarios If you have configured SSO to be required, but corrupt your SSO config this will block people from logging in. In case of emergency, the user with the Owner role in the organization can use the break glass URL to sign in with a password (bypassing SSO). The break glass URL is [https://console.statsig.com/login?method=password-only](https://console.statsig.com/login?method=password-only) # Tags in Statsig Source: https://docs.statsig.com/access-management/tags Use tags in Statsig to organize gates, experiments, and metrics for easy filtering by team, objective, or initiative across your project. ## Tags for organization Tags let you apply light-weight organization to your Statsig config (e.g. gates, experiments and metrics) to allow easy filtering by team (or organization objectives). E.g. You can tag growth related experiments and metrics with a tag called "Growth", easily filtering down to just these when needed. This is particularly useful with metrics - since you can now just add the Growth tag to your scorecard metrics, and pull in all the key Growth metrics instead of individually picking them. The Core tag is meant to be used for company critical metrics. These are defaulted into experiment scorecards so experimenters are looking at impact on these metrics as part of their experiments. Tags organization interface # Teams Source: https://docs.statsig.com/access-management/teams Configure Statsig Teams to add an organizational and permissions layer on top of a project, enabling team-scoped settings, reviewers, and ownership. Teams are an Enterprise-only feature. If you are on the Developer or Pro tiers, this guide will not apply to you. To upgrade to Enterprise, feel free to reach out to our team [here](https://www.statsig.com/contact/demo). ## Overview For larger organizations, the Teams feature enables an organizational and settings/ permissions layer on top of a Project. Teams are configured at the Project (not Organization) level, and are default-editable by all Project Admins. Once teams are configured and a user is assigned to a team, any config (gates/ experiments/ metrics, etc.) they create will be associated with the team they belong to, and will inherit the settings of that team. Users who are members of multiple teams will have the choice of which team to associate their config with at creation time. ## Creating Teams To create a team, navigate to **Settings** -> **People** -> **Teams**. Create a new team via the **+Create** button, where you'll be asked to name the team and add members. You can add/ remove members from a team at any time, not just at initial team creation. Each team has a **Members** page and a **Settings** page. Within **Members** you can see all members of the team, including the team members project role and team role (which can be member or admin). Team members can be promoted or removed. Statsig Teams page showing member list with project and team roles ## Configuring Team Settings At the Project-level, you can require all config creations are associated with a team via the "Require teams" setting under **Settings** -> **Product Configuration** -> **General**. Note that this will block anyone who isn't yet assigned to a team from creating a config, so should only be enabled after all members of the project have been added to (at least) one team. Project configuration toggle requiring configs to be associated with teams Within each team, there are a number of settings you can configure: **Default Monitoring Metrics/ Scorecard Metrics:** This setting enables pre-configuration of a set of metrics to add to every new gate/ experiment/ holdout at the team level. These might be a mix of top-line company metrics every team must monitor (e.g. revenue/ app performance), as well as a set of team-specific KPIs all rollouts and experiments should be tracking. Statsig Team settings showing default monitoring metrics selection **Require Reviews:** If reviews are not already required at the Project-level, this setting enables you to require reviews at the individual team level. Note that this setting won't appear if you're already requiring reviews at the Project level (controlled via **Settings** -> **Product Configuration** -> **Reviews**). Team require reviews option within settings **Default Allowed Reviewers:** This setting enables more granular control of *who* is allowed to review and approve changes to a team's configs. There are three options here- "Anyone in the Project" (least restrictive), "Team Members Only" (keep reviews within the team), and "Team or Project Admins Only" (most restrictive). Note that team-based review configurations layer on top of [role-based review settings](/guides/setting-up-reviews#enforcing-team-reviews). For example, if your role has permission to approve reviews and your team has review settings set to “Team members only”, then an approver would need to both be in a role with review approval permission AND be on the team to approve a review pending for that team’s config. Default allowed reviewers dropdown specifying who can approve changes **Create/ Edit Configs and Metrics:** This setting dictates which members of a team are allowed to edit or create configs tagged with the team. There are two options here- "Anyone in the Project" (no restrictions, anyone can edit the team's configs), or "Team Members Only". Create and edit configs permissions for team members only or entire project **Default Target Applications:** This setting will auto-apply any assigned Target Applications to all configs created that are associated with this team. Note that this only impacts which Target Applications are added to the config by default at creation time, but can be edited/ overridden as needed. Team default target applications selector **Default Holdout:** It's a common use-case for teams to want to measure cumulative impact of their new features/ experiments over the course of a Quarter/ Half, etc. To make this easier and more automatic, you can associate a default Holdout to a team. This will cause all subsequent configs associated with the team will be auto-added to this default Holdout. Team default holdout configuration UI ## How Teams are Used Throughout the Console Once a user is associated with a team, every config they create will now be default-associated with their team. For users on multiple teams, they will be able to choose which team to associate their config with at creation time. This will apply the team’s relevant settings to that config. Config creation screen with team selection dropdown Every config will have a field in the header for “Team”. This field is separate from “Owner"- whereby "Owner" is a single individual, "Team" is a group of individuals and will not automatically update if, for example, the Owner moves to a different team within the organization. Teams must be manually changed (subject to the review requirements) at the config level. Config header showing Team field separate from Owner Finally, with the addition of teams, every user can now filter Gate/ Experiment/ Metric lists and the Home Feed by team. The Home Feed will default to a user's team(s), ensuring the most relevant content is surfaced. Console list filters for gates experiments and metrics by team # Node AI SDK Source: https://docs.statsig.com/ai-evals/node Statsig's Node SDK for AI Application Configuration & Telemetry Node AI SDK on Github, NPM Package The Statsig AI Node SDK is currently in beta. We are no longer accepting new beta customers at this time. ## Overview The Statsig Node AI SDK lets you manage your prompts, online and offline evals, and debug your LLM applications in production. It depends upon the [Statsig Node Server SDK](/server-core/node-core), but provides convenient hooks for AI-specific functionality. ```js npm theme={null} npm install @statsig/statsig-ai ``` ```js pnpm theme={null} pnpm add @statsig/statsig-ai ``` ```js yarn theme={null} yarn add @statsig/statsig-ai ``` If you have unique setup needs like a frozen lockfile, take a look at the [Node Server SDK docs](/server-core/node-core#installation) - the AI SDK will install Node Server if you don't already have it. If you already have a Statsig instance, you can pass it into the SDK. Otherwise, we'll create an instance for you internally. Initialize the AI SDK with a Server Secret Key from the Statsig console. Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. ```js theme={null} import { StatsigAI } from '@statsig/statsig-ai'; const statsigAI = new StatsigAI({'YOUR_SERVER_SECRET_KEY'}); await statsigAI.initialize(); ``` Optionally, you can configure [StatsigOptions](/server-core/node-core#statsig-options) for your Statsig instance: ```js theme={null} import { StatsigAI, StatsigAIOptions } from '@statsig/statsig-ai'; import { StatsigOptions } from '@statsig/statsig-server-core-node'; // if you want to configure any statsig options, this is optional: const statsigOptions: StatsigOptions = { environment: 'production', }; const statsigAI = new StatsigAI({'YOUR_SERVER_SECRET_KEY', statsigOptions}); await statsigAI.initialize(); // if you would like to use any statsig methods, you can access the statsig instance from the statsigAI instance: const gate = statsigAI.getStatsig().checkGate(statsigUser, 'my_gate'); ``` After installation, initialize the SDK with a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. If you initialize this way, the AI SDK won’t handle initialization, flushing, or shutdown. ```js theme={null} import { Statsig } from '@statsig/statsig-server-core-node'; import { StatsigAI } from '@statsig/statsig-ai'; const statsig = new Statsig('YOUR_SERVER_SECRET_KEY'); await statsig.initialize(); const statsigAI = new StatsigAI({statsig}); await statsigAI.initialize(); ``` Optionally, you can configure [StatsigOptions](/server-core/node-core#statsig-options): ```js theme={null} import { Statsig } from '@statsig/statsig-server-core-node'; import { StatsigAI, StatsigAIOptions } from '@statsig/statsig-ai'; const statsig = new Statsig('YOUR_SERVER_SECRET_KEY', { environment: 'production', }); await statsig.initialize(); await statsigAI.initialize(); ``` ## Using the SDK ### Getting a Prompt Statsig can act as the control plane for your LLM prompts, allowing you to version and change them without deploying code. For more information, see the [Prompts](/ai-evals/prompts) documentation. ```js theme={null} import { StatsigUser } from '@statsig/statsig-ai'; // Create a user object const user = new StatsigUser({ userID: 'a-user' }); // Get the prompt const myPrompt = statsigAI.getPrompt(user, 'my_prompt'); // Use the live version of the prompt const liveVersion = myPrompt.getLive(); // Get the candidate versions of the prompt const candidateVersions = myPrompt.getCandidates(); // Use the live version of the prompt in a completion const response = await openai.chat.completions.create({ model: liveVersion.getModel({ fallback: 'gpt-4' }), // optional fallback temperature: liveVersion.getTemperature(), max_tokens: liveVersion.getMaxTokens(), messages: [{ role: 'user', content: 'Your prompt here' }], }); ``` ### Logging Eval Results When running an [online eval](/ai-evals/online-evals), you can log results back to Statsig for analysis. Provide a score between 0 and 1, along with the grader name and any useful metadata (e.g., session IDs). Currently, you must provide the grader manually — future releases will support automated grading options. ```js theme={null} import { StatsigUser } from '@statsig/statsig-ai'; const livePromptVersion = statsigAI.getPrompt(user, 'my_prompt').getLive(); // Create a user object const user = new StatsigUser({ userID: 'a-user' }); // Log the results of the eval statsigAI.logEvalGrade(user, livePromptVersion, 0.5, 'my_grader', { session_id: '1234567890', }); // flush eval grade events to statsig await statsigAI.flush(); ``` ### Programmatic Evaluation Programmatic evaluation allows you to run evaluations on datasets programmatically, automatically scoring outputs and sending results to Statsig for analysis. With programmatic evaluation, you can: * **Run evaluations on datasets**: Process arrays, iterators, or async generators of input/expected pairs * **Define custom tasks**: Create functions that generate outputs from inputs (supports both sync and async) * **Score outputs**: Use single or multiple named scorer functions to evaluate outputs (supports boolean, numeric, or metadata-rich scores) * **Use parameters**: Pass dynamic parameters to tasks using Zod schemas (Node) or dictionaries (Python) * **Categorize data**: Group evaluation records by categories for better analysis * **Compute summary scores**: Aggregate results across all records with custom summary functions * **Handle errors gracefully**: Task and scorer errors are caught and reported without stopping the evaluation The evaluation automatically sends results to Statsig, where you can view them in the console alongside your other eval data. Tasks and scorers can be async functions. Data can also be provided as async functions, promises, or async iterators. The `expected` field in data records is optional; scorers can evaluate outputs without expected values. Task and scorer errors are automatically caught and reported in the results. ```js theme={null} import { Eval } from '@statsig/statsig-ai'; import { z } from 'zod'; // Basic evaluation with a single scorer const result = await Eval('greeting_task', { data: [ { input: 'world', expected: 'Hello world' }, { input: 'test', expected: 'Hello test' }, ], task: (input: string) => `Hello ${input}`, scorer: ({ output, expected }) => output === expected, evalRunName: 'run-123', }); // Multiple named scorers const result2 = await Eval('multi_scorer_task', { data: [ { input: 'world', expected: 'Hello world' }, { input: 'test', expected: 'Hello test' }, ], task: (input: string) => `Hello ${input}`, scorer: { correctness: ({ output, expected }) => output === expected, startsWithHello: ({ output }) => output.startsWith('Hello'), lengthCheck: ({ output }) => output.length > 5, }, }); // Using parameters with Zod schemas const result3 = await Eval('parameterized_task', { data: [ { input: 'world', expected: 'Hi world' }, ], task: (input: string, hooks) => { const prefix = hooks.parameters.name || 'Hello'; return `${prefix} ${input}`; }, scorer: ({ output, expected }) => output === expected, parameters: { name: z.string().default('Hi'), }, }); // Extras: Categories and summary scores const result4 = await Eval('categorized_with_summary', { data: [ { input: 'world', expected: 'Hello world', category: 'greeting' }, { input: 'test', expected: 'Hello test', category: ['greeting', 'test'] }, { input: 'foo', expected: 'Goodbye foo', category: 'farewell' }, ], task: (input: string) => `Hello ${input}`, scorer: { correctness: ({ output, expected }) => output === expected, }, summaryScoresFn: (results) => { const correct = results.filter(r => r.scores.correctness === 1).length; return { accuracy: correct / results.length, total: results.length, }; }, }); ``` ### OpenTelemetry (OTEL) The AI SDK works with OpenTelemetry for sending telemetry to Statsig. You can enable OTel tracing by calling the `initializeTracing` function. You can also provide a custom `TracerProvider` to the `initializeTracing` function if you want to customize the tracing behavior. More advanced OTel configuration and exporter support are on the way. The simplest way to start tracing with Statsig and OTel is to call `initializeTracing()` at the root of your application. ```js theme={null} // instrumentation.{js,ts} import { initializeTracing } from '@statsig/statsig-ai/otel'; initializeTracing({ // optional: enables the global trace provider registration // so that you can create spans without having to create a new trace provider enableGlobalTraceProviderRegistration: true, }); ``` If you already have your own OTel setup with `NodeSDK`, you only need to initialize Statsig's OTel tracing and use the processor created by `initializeTracing()`. ```js theme={null} // instrumentation.{js,ts} import { NodeSDK } from '@opentelemetry/sdk-node'; import { PeriodicExportingMetricReader, ConsoleMetricExporter, } from '@opentelemetry/sdk-metrics'; import { initializeTracing } from '@statsig/statsig-ai/otel'; // when you have your own otel setup and don't want to use the global trace provider // you can disable it with the options below const { processor } = initializeTracing({ // prevents creating a global context manager skipGlobalContextManagerSetup: true, exporterOptions: { sdkKey: process.env.STATSIG_SDK_KEY!, }, }); const sdk = new NodeSDK({ // IMPORTANT: use the processor created by initializeTracing // to make sure that spans are exported to Statsig spanProcessors: [processor], metricReader: new PeriodicExportingMetricReader({ exporter: new ConsoleMetricExporter(), }), // ... other node sdk options like autoInstrumentations }); sdk.start(); export { sdk }; ``` The `initializeOTel` function accepts the below options for setting up tracing with OTel. ```ts theme={null} type InitializeOptions = { /** An optional global context manager to use. If not provided, one will be created and set as the global context manager unless `skipGlobalContextManagerSetup` is true. */ globalContextManager?: ContextManager; /** If true, will not attempt to set up a global context manager automatically. */ skipGlobalContextManagerSetup?: boolean; /** If true, will register the trace provider globally. */ enableGlobalTraceProviderRegistration?: boolean; /** An optional global trace provider to use. If not provided, a new BasicTracerProvider will be created and optionally registered globally */ globalTraceProvider?: TracerProvider; /** Options to pass to the StatsigOTLPTraceExporter */ exporterOptions?: StatsigOTLPTraceExporterOptions; // resource options serviceName?: string; version?: string; environment?: string; }; ``` For more examples see the [Statsig AI Node SDK](https://github.com/statsig-io/statsig-ai-node/tree/main/examples/otel). ### Wrapping OpenAI The Statsig OpenAI Wrapper automatically adds tracing and log events to your OpenAI SDK usage, giving you in-console visibility with minimal setup. ```js theme={null} import { wrapOpenAI, StatsigAI } from '@statsig/statsig-ai'; import { OpenAI } from 'openai'; // if you have your own otel, you do not need an statsigAI instance here. // But if you want to use the default Otel on statsigAI, you need to initialize the SDK. statsigAI = new StatsigAI({"YOUR_SERVER_SECRET_KEY"}); await statsigAI.initialize(); const client = wrapOpenAI( new OpenAI({ apiKey: process.env.OPENAI_API_KEY, }) ); const response = await client.chat.completions.create({ model: "gpt-4", messages: [{ role: "user", content: "Hello, world!" }], }); ``` ## Using other SDK methods Whether you passed in a Statsig instance or not, you can access the Statsig instance from the statsigAI instance, and use its many methods: ```javascript theme={null} // Check a gate value const gate = statsigAI.getStatsig().checkGate(statsigUser, 'my_gate'); // Log an event statsigAI.getStatsig().logEvent(statsigUser, 'my_event', { value: 1 }); ``` Refer to the [Statsig Node SDK](/server-core/node-core) docs for more information on how to use the Core Statsig SDK methods, plus information on advanced setup + singleton usage. # Offline Evals Source: https://docs.statsig.com/ai-evals/offline-evals Run offline AI evaluations in Statsig to grade model outputs against fixed test sets and catch regressions before exposing changes to real users. ## What are Offline Evals Offline evals offer a quick, automated grading of model outputs on a fixed test set. They catch wins / regressions early—before any real users are exposed. e.g. compare a new support‑bot’s replies to gold (human curated) answers to decide if it is good enough to ship. Steps to do this on Statsig - 1. Create a Prompt. This contains the prompt for your task (e.g. Classify tickets as high, medium or low urgency based on ticket text) 2. Upload a sample dataset - with example inputs and ideal answers (e.g. Ticket1 text, High; Ticket2 text, Low) 3. Run your AI on that dataset to produce output. (e.g. classify each ticket in this example) 4. Grade or score the outputs. You can do this by comparing ideal answer in the dataset with the output your AI generated. 5. Create multiple versions of your prompts. Compare scores across versions and promote the best one to be Live. ## Create/analyze an offline eval in 10 minutes **1. Create a Prompt within Statsig** This captures the instruction you provide to an LLM to accomplish your task. You can now use the Statsig [Node](/server-core/node-core#getting-a-prompt) or [Python](/server-core/python-core/#getting-a-prompt) Server Core SDKs to retrieve this prompt within your app and use it. You can create multiple versions of the prompt as you iterate, and choose which one is "live" (retrieved by the SDK). Statsig prompt editor listing live and candidate versions with messages **2. Create a dataset you can use to evaluate LLM completions for your prompt** For the example above, this might be a list of words, along side known good translations in French. Small lists can be entered (or upload a CSV). Dataset creation table with translation pairs for offline evaluation **3. Create a grader that will grade LLM completions for your prompt** Configure a grader that compares the LLM completion text with the reference output. You can use one of the out of box string evaluators, or even configure an LLM-as-a-Judge evaluator that mimics a human's grading rubric. Grader configuration form comparing model output against reference answers **3. Run evaluation** Run an evaluation on a version of the prompt. You should see results in a few minutes that look like this. You can click into any row of the dataset to understand more about the evaluation for that row. Offline evaluation results table showing prompt version scores You can categorize your dataset, and break scores out by category. Category breakdown chart splitting evaluation scores by dataset segments If you have scores for multiple versions, you can compare them to see what changed between versions. Comparison view charting multiple prompt versions across graders # Online Evals Source: https://docs.statsig.com/ai-evals/online-evals Run online AI evaluations in Statsig to grade model outputs in production on real traffic, including shadow runs for candidate prompts and models. ## What are Online Evals Online evals let you grade your model output in production on real world use cases. You can run the "live" version of a prompt, but can also shadow run "candidate" versions of a prompt, without exposing users to them. Grading works directly on the model output, and has to work without a ground truth to compare against. Steps to do this in Statsig - 1. Create a Prompt. This contains the prompt for your task (e.g. Summarize ticket content. Don't include email addresses or credit cards in the summary). Create a v2 prompt that improves on this. 2. In your app, use and produce model output using the v1 and v2 prompts. The output from v1 is rendered to the user; the output from v1 and v2 are judged by an LLM-as-a-judge. 3. The grades from v1 and v2 are logged back to Statsig and can be compared there. Online Evals is currently in beta. We are no longer accepting new beta customers at this time. ## Create/analyze an online eval in 15 minutes **1. Identify the prompts you want to serve** In Prompts, there are four prompt types: Live, Candidate, Draft and Archive. Before starting an online evaluation, it’s important to organize your prompt versions into these categories: * **Live** prompt is the version actively served to users. * **Candidate** prompts are not shown to users but still served to your code. The user’s input is still processed against them, and their outputs are logged and graded alongside the live version. * **Draft** prompts are the offline prompts you will iterate on in console, before deciding that you want to serve them. In order to start serving them, you should promote them to "Candidate" or "Live" * **Archive** prompts are inactive versions that are not iterated on and kept offline. Prompts that you can access in code will comprise of the Live version and Candidate versions. Prompt versions list showing live, candidate, draft, and archived prompts with setup form **2. Load your prompts in code and run completions on user input** In the example below, we demonstrate how to integrate your prompts in your application using the [Statsig AI SDKs](/ai-evals/node). Once you grab your Live or Candidate prompts, pass in the approriate values to replace your macros in your prompt (\{\{input}} should be replaced by the user input). You can then run completions on each of these prompts. ```js theme={null} const prompts = statsigAI.getPrompt(user, "ai_config_name"); // get the live prompt const livePrompt = prompts.getLive(); // get the candidate prompts const candidatePrompts = prompts.getCandidates(); // get the live prompt messages const livePromptMessages = livePrompt.getPromptMessages({ input: userInput }); // run completions on your live prompt and show the output to the user const liveOutput = client.completions.create( my_model, livePromptMessages, (temperature = livePrompt.getTemperature()) ); // simulateneously run completions on the candidate prompts to get their output ``` **3. Score your output using graders** Once you have a completion’s output, it should be evaluated using a grader—either one created in Statsig or a custom grader of your choice. The resulting score should always fall within the range of 0 to 1. **4. Log Eval Results to Statsig** You can log your scores as events in Statsig to see the results in your console ```js theme={null} // Log the results of the eval statsigAI.logEvalGrade(user, livePromptVersion, 0.5, "my_grader", { session_id: "1234567890", }); ``` **5. View Results in Statsig** You can now view these results in Statsig! Select the version you want to evaluate and the versions you want to compare it against. This end to end online eval helps you iterate on your prompts and gain valuable insights. Online eval results dashboard comparing prompt versions with cumulative events chart and grader deltas # AI Evals Overview Source: https://docs.statsig.com/ai-evals/overview Overview of Statsig AI Evals for evaluating prompts and models with offline and online graders, currently available in private beta for AI applications. AI Evals are currently in beta. We are no longer accepting new beta customers at this time. ## What are AI Evals? Statsig AI Evals have a few core components to help iterate and serve your LLM apps in production. 1. **[Prompts](/ai-evals/prompts)**: Prompts are a way to represent your LLM prompt (and associated LLM config like Model Provider, Model, Temperature etc). This typically represents a task you're getting the LLM to do (e.g. "Classify this ticket to a triage queue" or "Summarize this text"). You can version prompts, choose in Statsig which version is currently live and retrieve and use this prompt in Production using the Statsig server SDKs. It is possible to use Prompts as the control pane for your LLM apps without using the rest of the Evals product suite. 2. **[Offline Evals](/ai-evals/offline-evals)**: Offline evals offer a quick, automated grading of model outputs on a fixed test set. They catch wins / regressions early—before any real users are exposed. e.g. compare a new support‑bot’s replies to gold (human curated) answers to decide if it is good enough to ship. It is possible to grade output even without a golden dataset (e.g. if you're having an LLM validate English to French translation). 3. **[Online Evals](/ai-evals/online-evals)**: Online evals let you grade your model output in production on real world use cases. You can run the "live" version of a prompt, but can also shadow run "candidate" versions of a prompt, without exposing users to them. Grading works directly on the model output, and has to work without a ground truth to compare against. ## What about Gates, Experiments and Analytics? The standard suite of Statsig product building capabilities are also available for use here. For example, you can target an LLM feature at a set of people that meet some criteria with a Feature Gate, or choose to roll out a new prompt version as an Experiment and understand impact on metrics, similar to any other experiment. ### LLM as a Judge Some grading can use heuristics (e.g. check if the AI generated output matches the ideal answer in the dataset when the output is as simple as High, Medium or Low). Some grading can't - you're trying to decide if "Your ticket has been escalated" and "This ticket has been escalated" mean the same thing. LLM-as-a-judge lets you quickly and cheaply evaluate AI outputs at scale without needing tons of human reviewers. It mimics how a human would assess quality — and while not perfect, it's fast, consistent, and good enough to compare different versions of your model or prompt. In this example, we could write an LLM-as-a-judge prompt "Score how close this answer is to the ideal one on a scale between 0-1.0". # Prompts & Graders Source: https://docs.statsig.com/ai-evals/prompts Manage AI prompts and graders in Statsig to evaluate, version, and roll out prompts in production without deploying code, similar to dynamic configs. ## What is a Prompt in Statsig? A Prompt is a way to represent an LLM prompt or a task in Statsig, with it's config. Prompts are similar to Dynamic Configs, and allow you to evaluate and roll out prompts in production without deploying code. You can use the Statsig [Node](/server-core/node-core#getting-a-prompt) or [Python](/server-core/python-core/#getting-a-prompt) Server Core SDKs to retrieve this prompt within your app at runtime and use it. With Prompts, you can * Manage your prompt configuration outside of your application code. You can update model, configuration or prompt at runtime. * Team mates who have access to Statsig can collaborate and iterate on prompts, while benefitting from Statsig's production change control processes and versioning. * Add configuration for a new model, model provider and progressively shift production traffic to this while comparing costs, user satisfaction or any metric of interest. * Support advanced use cases such as * retrieval-augmented generation (RAG) and * evaluation in production. Prompt creation screen showing model selection and message list Code example calling statsigAI.getPrompt to fetch live prompt version Prompt results page summarizing grader scores per version ## What is a Grader? A grader is the evaluation component that scores or judges the output of an AI system against a desired standard. Think of it as the core evaluation unit in the workflow: Inputs: The grader takes in the AI model’s response (and sometimes the “ideal” or ground-truth answer if one exists). Process: It applies a scoring method. This could be: Rule-based (exact string match, regex check, cosine similarity) or LLM-as-a-Judge (using another model to evaluate correctness, relevance, style, or safety). Outputs: It produces a score - ideally 0 (Fail) or Pass (1). This score feeds into the overall Statsig experiment or eval framework to determine performance across datasets, experiments, or model versions. ## What is a Critical Grader? A critical grader is a must-pass evaluation in Statsig AI Evals: if the AI output fails this grader, the entire run is marked as failed. It enforces non-negotiable requirements, acting as a hard gate before results are considered valid. When it does not fail, it acts like a normal grader. ### Use Case For example, in a financial support chatbot, a critical grader could check that the model never fabricates account balances. Even if the answer is otherwise helpful, a single failure here blocks the model from being promoted. # Python AI SDK Source: https://docs.statsig.com/ai-evals/python Statsig's Python SDK for AI Application Configuration & Telemetry Python AI SDK on Github, PyPI Package The Statsig AI Python SDK is currently in beta. We are no longer accepting new beta customers at this time. ## Overview The Statsig Python AI SDK lets you manage your prompts, online and offline evals, and debug your LLM applications in production. It depends upon the [Statsig Python Server SDK](/server-core/python-core), but provides convenient hooks for AI-specific functionality. ```python pip theme={null} pip install statsig-ai ``` ```python poetry theme={null} poetry add statsig-ai ``` ```python pipenv theme={null} pipenv install statsig-ai ``` For initialization requirements in forking and WSGI servers, see the [Statsig Python Server SDK](/server-core/python-core) docs. If you already have a Statsig instance, you can pass it into the SDK. Otherwise, we'll create an instance for you internally. Initialize the AI SDK with a Server Secret Key from the Statsig console. Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. ```python theme={null} from statsig_ai import StatsigAI, StatsigCreateConfig statsig_ai = StatsigAI(statsig_source=StatsigCreateConfig(server_secret_key='YOUR_SERVER_SECRET_KEY')) statsig_ai.initialize(). ``` Optionally, you can configure [StatsigOptions](/server-core/python-core#statsig-options) for your Statsig instance: ```python theme={null} from statsig_ai import StatsigAI from statsig_python_core import StatsigOptions # if you want to configure any statsig options, this is optional: statsig_options = StatsigOptions() statsig_options.environment = 'production' statsig_ai_options.statsig_options = statsig_options statsig_ai = StatsigAI(statsig_source=StatsigCreateConfig(server_secret_key='YOUR_SERVER_SECRET_KEY', statsig_options=statsig_options)) statsig_ai.initialize() # if you would like to use any statsig methods, you can access the statsig instance from the statsig_ai instance: gate = statsig_ai.get_statsig().check_gate(statsig_user, 'my_gate') ``` After installation, initialize the SDK with a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. If you initialize this way, the AI SDK won’t handle initialization, flushing, or shutdown. ```python theme={null} from statsig_python_core import Statsig from statsig_ai import StatsigAI, StatsigAttachConfig statsig = Statsig('YOUR_SERVER_SECRET_KEY') statsig.initialize() statsig_ai = StatsigAI(statsig_source=StatsigAttachConfig(statsig=statsig)) statsig_ai.initialize() ``` Optionally, you can configure [StatsigOptions](/server-core/python-core#statsig-options): ```python theme={null} from statsig_python_core import Statsig, StatsigOptions from statsig_ai import StatsigAI, StatsigAttachConfig options = StatsigOptions() options.environment = 'production' statsig = Statsig('YOUR_SERVER_SECRET_KEY', options) statsig.initialize() statsig_ai = StatsigAI(statsig_source=StatsigAttachConfig(statsig=statsig)) statsig_ai.initialize() ``` ## Using the SDK ### Getting a Prompt Statsig can act as the control plane for your LLM prompts, allowing you to version and change them without deploying code. For more information, see the [Prompts](/ai-evals/prompts) documentation. ```python theme={null} from statsig_ai import StatsigUser # Create a user object user = StatsigUser(user_id='a-user') # Get the prompt my_prompt = statsig_ai.get_prompt(user, 'my_prompt') # Use the live version of the prompt live_version = my_prompt.get_live() # Get the candidate versions of the prompt candidate_versions = my_prompt.get_candidates() # Use the live version of the prompt in a completion response = openai.chat.completions.create( model=live_version.get_model(fallback='gpt-4'), # optional fallback temperature=live_version.get_temperature(), max_tokens=live_version.get_max_tokens(), messages=[{'role': 'user', 'content': 'Your prompt here'}], ) ``` ### Logging Eval Results When running an [online eval](/ai-evals/online-evals), you can log results back to Statsig for analysis. Provide a score between 0 and 1, along with the grader name and any useful metadata (e.g., session IDs). Currently, you must provide the grader manually — future releases will support automated grading options. ```python theme={null} from statsig_ai import StatsigUser live_prompt_version = statsig_ai.get_prompt(user, 'my_prompt').get_live() # Create a user object user = StatsigUser(user_id='a-user') # Log the results of the eval statsig_ai.log_eval_grade(user, live_prompt_version, 0.5, 'my_grader', { 'session_id': '1234567890', }) # flush eval grade events to statsig statsig_ai.flush().wait() ``` ### Programmatic Evaluation Programmatic evaluation allows you to run evaluations on datasets programmatically, automatically scoring outputs and sending results to Statsig for analysis. With programmatic evaluation, you can: * **Run evaluations on datasets**: Process arrays, iterators, or async generators of input/expected pairs * **Define custom tasks**: Create functions that generate outputs from inputs (supports both sync and async) * **Score outputs**: Use single or multiple named scorer functions to evaluate outputs (supports boolean, numeric, or metadata-rich scores) * **Use parameters**: Pass dynamic parameters to tasks using Zod schemas (Node) or dictionaries (Python) * **Categorize data**: Group evaluation records by categories for better analysis * **Compute summary scores**: Aggregate results across all records with custom summary functions * **Handle errors gracefully**: Task and scorer errors are caught and reported without stopping the evaluation The evaluation automatically sends results to Statsig, where you can view them in the console alongside your other eval data. Tasks and scorers can be async functions. Data can also be provided as async functions, promises, or async iterators. The `expected` field in data records is optional; scorers can evaluate outputs without expected values. Task and scorer errors are automatically caught and reported in the results. ```python theme={null} from statsig_ai import Eval, EvalScorerArgs, EvalDataRecord, EvalHook # Basic evaluation with a single scorer result = Eval( name='greeting_task', data=[ {'input': 'world', 'expected': 'Hello world'}, {'input': 'test', 'expected': 'Hello test'}, ], task=lambda input: f'Hello {input}', scorer=lambda args: args.output == args.expected, eval_run_name='run-123', ) # Multiple named scorers result2 = Eval( name='multi_scorer_task', data=[ {'input': 'world', 'expected': 'Hello world'}, {'input': 'test', 'expected': 'Hello test'}, ], task=lambda input: f'Hello {input}', scorer={ 'correctness': lambda args: args.output == args.expected, 'starts_with_hello': lambda args: args.output.startswith('Hello'), 'length_check': lambda args: len(args.output) > 5, }, ) # Using parameters def task_with_params(input: str, hook: EvalHook) -> str: prefix = hook.parameters.get('prefix', 'Hello') return f'{prefix} {input}' result3 = Eval( name='parameterized_task', data=[ {'input': 'world', 'expected': 'Hi world'}, ], task=task_with_params, scorer=lambda args: args.output == args.expected, parameters={'prefix': 'Hi', 'suffix': '!', 'number': 123}, ) # Extras: Categories and summary scores def summary_scorer(results): correct = sum(1 for r in results if r.scores.get('correctness', 0.0) == 1.0) return { 'accuracy': correct / len(results) if results else 0.0, 'total': len(results), } result4 = Eval( name='categorized_with_summary', data=[ {'input': 'world', 'expected': 'Hello world', 'category': 'greeting'}, {'input': 'test', 'expected': 'Hello test', 'category': ['greeting', 'test']}, {'input': 'foo', 'expected': 'Goodbye foo', 'category': 'farewell'}, ], task=lambda input: f'Hello {input}', scorer={ 'correctness': lambda args: args.output == args.expected, }, summary_score_fn=summary_scorer, ) # Using EvalDataRecord dataclass result5 = Eval( name='dataclass_records', data=[ EvalDataRecord(input='world', expected='Hello world'), EvalDataRecord(input='test', expected='Hello test'), ], task=lambda input: f'Hello {input}', scorer=lambda args: args.output == args.expected, ) ``` ### OpenTelemetry (OTEL) The AI SDK works with OpenTelemetry for sending telemetry to Statsig. You can enable OTel tracing by calling the `initializeTracing` function. You can also provide a custom `TracerProvider` to the `initializeTracing` function if you want to customize the tracing behavior. More advanced OTel configuration and exporter support are on the way. Otel is not supported in the Python AI SDK yet. Coming soon! ### Wrapping OpenAI The Statsig OpenAI Wrapper automatically adds tracing and log events to your OpenAI SDK usage, giving you in-console visibility with minimal setup. OpenAI wrapper is not supported in the Python AI SDK yet. Coming soon! ## Using other SDK methods Whether you passed in a Statsig instance or not, you can access the Statsig instance from the statsig\_ai instance, and use its many methods: ```python theme={null} # Check a gate value gate = statsig_ai.get_statsig().check_gate(statsig_user, 'my_gate') # Log an event statsig_ai.get_statsig().log_event(statsig_user, 'my_event', value=1) ``` Refer to the [Statsig Python SDK](/server-core/python-core) docs for more information on how to use the Core Statsig SDK methods, plus information on advanced setup + singleton usage. # List Topline Alert Events Source: https://docs.statsig.com/api-reference/alerts/list-topline-alert-events https://api.statsig.com/openapi/20240601.json get /console/v1/alerts/{id}/events # List Topline Alerts Source: https://docs.statsig.com/api-reference/alerts/list-topline-alerts https://api.statsig.com/openapi/20240601.json get /console/v1/alerts # Read Topline Alert Source: https://docs.statsig.com/api-reference/alerts/read-topline-alert https://api.statsig.com/openapi/20240601.json get /console/v1/alerts/{id} # Read Topline Alert Event Source: https://docs.statsig.com/api-reference/alerts/read-topline-alert-event https://api.statsig.com/openapi/20240601.json get /console/v1/alerts/{id}/events/{eventId} # List Audit Logs Source: https://docs.statsig.com/api-reference/audit-logs/list-audit-logs https://api.statsig.com/openapi/20240601.json get /console/v1/audit_logs # Get Ranked List for Contextual Bandit Source: https://docs.statsig.com/api-reference/autotune/get-ranked-list-for-contextual-bandit /http-api/httpopenapi.json post /v1/get_ranked_list Returns a ranked list of variants for a contextual multi-armed bandit (autotune) experiment. The ranking is based on predicted performance. # Create Autotune Source: https://docs.statsig.com/api-reference/autotunes/create-autotune https://api.statsig.com/openapi/20240601.json post /console/v1/autotunes # Delete Autotune Source: https://docs.statsig.com/api-reference/autotunes/delete-autotune https://api.statsig.com/openapi/20240601.json delete /console/v1/autotunes/{id} # Finish Experiment Early Source: https://docs.statsig.com/api-reference/autotunes/finish-experiment-early https://api.statsig.com/openapi/20240601.json put /console/v1/autotunes/{id}/make_decision # Fully Update Autotune Source: https://docs.statsig.com/api-reference/autotunes/fully-update-autotune https://api.statsig.com/openapi/20240601.json post /console/v1/autotunes/{id} Update all properties of the experiment # List Autotune Source: https://docs.statsig.com/api-reference/autotunes/list-autotune https://api.statsig.com/openapi/20240601.json get /console/v1/autotunes # Partially Update Autotune Source: https://docs.statsig.com/api-reference/autotunes/partially-update-autotune https://api.statsig.com/openapi/20240601.json patch /console/v1/autotunes/{id} Update selected properties of the experiment # Read Autotune Source: https://docs.statsig.com/api-reference/autotunes/read-autotune https://api.statsig.com/openapi/20240601.json get /console/v1/autotunes/{id} # Reset Experiment Source: https://docs.statsig.com/api-reference/autotunes/reset-experiment https://api.statsig.com/openapi/20240601.json put /console/v1/autotunes/{id}/reset # Start Autotune Experiment Source: https://docs.statsig.com/api-reference/autotunes/start-autotune-experiment https://api.statsig.com/openapi/20240601.json put /console/v1/autotunes/{id}/start # Change Validation Source: https://docs.statsig.com/api-reference/change-validation/change-validation https://api.statsig.com/openapi/20240601.json post /console/v1/change_validation See how change validation works [here](https://docs.statsig.com/guides/setting-up-reviews#configuring-custom-approval-workflows-pre-commit-webhooks) # Update change validation message Source: https://docs.statsig.com/api-reference/change-validation/update-change-validation-message https://api.statsig.com/openapi/20240601.json patch /console/v1/change_validation/message # Get Company Info Source: https://docs.statsig.com/api-reference/company/get-company-info https://api.statsig.com/openapi/20240601.json get /console/v1/company # Read Exposure Event Count Source: https://docs.statsig.com/api-reference/configs/read-exposure-event-count https://api.statsig.com/openapi/20240601.json get /console/v1/exposure_count Get the count of exposure events recently received by Statsig. # Add Widgets to Dashboard Source: https://docs.statsig.com/api-reference/dashboards/add-widgets-to-dashboard https://api.statsig.com/openapi/20240601.json post /console/v1/dashboards/{id}/widgets # Create Dashboard Source: https://docs.statsig.com/api-reference/dashboards/create-dashboard https://api.statsig.com/openapi/20240601.json post /console/v1/dashboards # List Dashboards Source: https://docs.statsig.com/api-reference/dashboards/list-dashboards https://api.statsig.com/openapi/20240601.json get /console/v1/dashboards # Read Dashboard Source: https://docs.statsig.com/api-reference/dashboards/read-dashboard https://api.statsig.com/openapi/20240601.json get /console/v1/dashboards/{id} # Read Dashboard Widget Results Source: https://docs.statsig.com/api-reference/dashboards/read-dashboard-widget-results https://api.statsig.com/openapi/20240601.json get /console/v1/dashboards/{id}/widgets/{widgetId}/results # Replace Widgets on Dashboard Source: https://docs.statsig.com/api-reference/dashboards/replace-widgets-on-dashboard https://api.statsig.com/openapi/20240601.json put /console/v1/dashboards/{id}/widgets # Archive Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/archive-dynamic-config https://api.statsig.com/openapi/20240601.json put /console/v1/dynamic_configs/{id}/archive # Commit Dynamic Config Review Source: https://docs.statsig.com/api-reference/dynamic-configs/commit-dynamic-config-review https://api.statsig.com/openapi/20240601.json put /console/v1/dynamic_configs/{id}/reviews/{reviewID}/commit # Create Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/create-dynamic-config https://api.statsig.com/openapi/20240601.json post /console/v1/dynamic_configs # Delete Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/delete-dynamic-config https://api.statsig.com/openapi/20240601.json delete /console/v1/dynamic_configs/{id} # Delete Dynamic Config Rule Source: https://docs.statsig.com/api-reference/dynamic-configs/delete-dynamic-config-rule https://api.statsig.com/openapi/20240601.json delete /console/v1/dynamic_configs/{id}/rule/{ruleId} # Disable Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/disable-dynamic-config https://api.statsig.com/openapi/20240601.json put /console/v1/dynamic_configs/{id}/disable # Enable Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/enable-dynamic-config https://api.statsig.com/openapi/20240601.json put /console/v1/dynamic_configs/{id}/enable # Fully Update Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/fully-update-dynamic-config https://api.statsig.com/openapi/20240601.json post /console/v1/dynamic_configs/{id} # Get Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/get-dynamic-config https://api.statsig.com/openapi/20240601.json get /console/v1/dynamic_configs/{id} # Get Dynamic Config or Experiment Source: https://docs.statsig.com/api-reference/dynamic-configs/get-dynamic-config-or-experiment /http-api/httpopenapi.json post /v1/get_config Fetches configuration values for a dynamic config or experiment. Works for both types - the system automatically determines which type based on the name. Automatically logs exposure events. # Get Dynamic Config Rules Source: https://docs.statsig.com/api-reference/dynamic-configs/get-dynamic-config-rules https://api.statsig.com/openapi/20240601.json get /console/v1/dynamic_configs/{id}/rules # Get Specific Dynamic Config Rule Source: https://docs.statsig.com/api-reference/dynamic-configs/get-specific-dynamic-config-rule https://api.statsig.com/openapi/20240601.json get /console/v1/dynamic_configs/{id}/rule/{ruleId} # List Dynamic Config Versions Source: https://docs.statsig.com/api-reference/dynamic-configs/list-dynamic-config-versions https://api.statsig.com/openapi/20240601.json get /console/v1/dynamic_configs/{id}/versions # List Dynamic Configs Source: https://docs.statsig.com/api-reference/dynamic-configs/list-dynamic-configs https://api.statsig.com/openapi/20240601.json get /console/v1/dynamic_configs # Partially Update Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/partially-update-dynamic-config https://api.statsig.com/openapi/20240601.json patch /console/v1/dynamic_configs/{id} # Unarchive Dynamic Config Source: https://docs.statsig.com/api-reference/dynamic-configs/unarchive-dynamic-config https://api.statsig.com/openapi/20240601.json put /console/v1/dynamic_configs/{id}/unarchive # Update Dynamic Config Rule By Id Source: https://docs.statsig.com/api-reference/dynamic-configs/update-dynamic-config-rule-by-id https://api.statsig.com/openapi/20240601.json patch /console/v1/dynamic_configs/{id}/rule/{ruleId} # Get Environments Source: https://docs.statsig.com/api-reference/environments/get-environments https://api.statsig.com/openapi/20240601.json get /console/v1/environments # Update Environments Source: https://docs.statsig.com/api-reference/environments/update-environments https://api.statsig.com/openapi/20240601.json post /console/v1/environments # Get metrics using event name Source: https://docs.statsig.com/api-reference/events/get-metrics-using-event-name https://api.statsig.com/openapi/20240601.json get /console/v1/events/{eventName}/metrics # Get specific events Source: https://docs.statsig.com/api-reference/events/get-specific-events https://api.statsig.com/openapi/20240601.json get /console/v1/events/{eventName} # List Events Source: https://docs.statsig.com/api-reference/events/list-events https://api.statsig.com/openapi/20240601.json get /console/v1/events # Log Custom Events Source: https://docs.statsig.com/api-reference/events/log-custom-events /http-api/httpopenapi.json post /v1/log_event Logs one or more custom events for analytics and metric calculation. Events are used to measure experiment outcomes and user behavior. # Log Custom Exposure Events Source: https://docs.statsig.com/api-reference/events/log-custom-exposure-events /http-api/httpopenapi.json post /v1/log_custom_exposure Manually logs exposure events for experiments or feature gates. Useful for analytics-only experiments, delayed exposure logging, or when automatic exposure logging is disabled. # Create Qualifying Event Source: https://docs.statsig.com/api-reference/experiments-warehouse-native/create-qualifying-event https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/qualifying_events # Delete Qualifying Event Source: https://docs.statsig.com/api-reference/experiments-warehouse-native/delete-qualifying-event https://api.statsig.com/openapi/20240601.json delete /console/v1/experiments/qualifying_events/{name} # List qualifying event Source: https://docs.statsig.com/api-reference/experiments-warehouse-native/list-qualifying-event https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/qualifying_events # Read Qualifying Event Source: https://docs.statsig.com/api-reference/experiments-warehouse-native/read-qualifying-event https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/qualifying_events/{name} # Update Qualifying Event Source: https://docs.statsig.com/api-reference/experiments-warehouse-native/update-qualifying-event https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/qualifying_events/{name} # Abandon Experiment Source: https://docs.statsig.com/api-reference/experiments/abandon-experiment https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/abandon # Archive Experiment Source: https://docs.statsig.com/api-reference/experiments/archive-experiment https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/archive # Cancel Pulse Load (Warehouse Native) Source: https://docs.statsig.com/api-reference/experiments/cancel-pulse-load-warehouse-native https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/pulse_load_history/{dagID}/cancel # Commit Experiment Review Source: https://docs.statsig.com/api-reference/experiments/commit-experiment-review https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/reviews/{reviewID}/commit # Conclude Experiment & Defer Decision Source: https://docs.statsig.com/api-reference/experiments/conclude-experiment-&-defer-decision https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/defer_decision # Create Assignment Source Source: https://docs.statsig.com/api-reference/experiments/create-assignment-source https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/assignment_sources # Create Entity Property Source Source: https://docs.statsig.com/api-reference/experiments/create-entity-property-source https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/entity_properties # Create Experiment Source: https://docs.statsig.com/api-reference/experiments/create-experiment https://api.statsig.com/openapi/20240601.json post /console/v1/experiments # Delete Assignment Source Source: https://docs.statsig.com/api-reference/experiments/delete-assignment-source https://api.statsig.com/openapi/20240601.json delete /console/v1/experiments/assignment_source/{name} # Delete Entity Property Source Source: https://docs.statsig.com/api-reference/experiments/delete-entity-property-source https://api.statsig.com/openapi/20240601.json delete /console/v1/experiments/entity_property/{name} # Delete Experiment Overrides Source: https://docs.statsig.com/api-reference/experiments/delete-experiment-overrides https://api.statsig.com/openapi/20240601.json delete /console/v1/experiments/{id}/overrides # Deleted Experiment Source: https://docs.statsig.com/api-reference/experiments/deleted-experiment https://api.statsig.com/openapi/20240601.json delete /console/v1/experiments/{id} # Disable Experiment Groups Source: https://docs.statsig.com/api-reference/experiments/disable-experiment-groups https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/disable_groups # Enable Experiment Groups Source: https://docs.statsig.com/api-reference/experiments/enable-experiment-groups https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/enable_groups # Finish Experiment Early Source: https://docs.statsig.com/api-reference/experiments/finish-experiment-early https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/make_decision # Fully Update Experiment Source: https://docs.statsig.com/api-reference/experiments/fully-update-experiment https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id} # Get Entity Property Source Source: https://docs.statsig.com/api-reference/experiments/get-entity-property-source https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/entity_property/{name} # Get Experiment Source: https://docs.statsig.com/api-reference/experiments/get-experiment https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id} # Get Experiment Context Source: https://docs.statsig.com/api-reference/experiments/get-experiment-context https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/context # Get Experiment Guardrail Alert Statuses Source: https://docs.statsig.com/api-reference/experiments/get-experiment-guardrail-alert-statuses https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/alerts # Get Experiment Overrides Source: https://docs.statsig.com/api-reference/experiments/get-experiment-overrides https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/overrides # Get Pulse Load History Details (Warehouse Native) Source: https://docs.statsig.com/api-reference/experiments/get-pulse-load-history-details-warehouse-native https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/pulse_load_history/{dagID} # List Assignment Sources Source: https://docs.statsig.com/api-reference/experiments/list-assignment-sources https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/assignment_sources # List Entity Property Sources Source: https://docs.statsig.com/api-reference/experiments/list-entity-property-sources https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/entity_properties # List Experiment Versions Source: https://docs.statsig.com/api-reference/experiments/list-experiment-versions https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/versions # List Experiments Source: https://docs.statsig.com/api-reference/experiments/list-experiments https://api.statsig.com/openapi/20240601.json get /console/v1/experiments # Load Pulse (Warehouse Native) Source: https://docs.statsig.com/api-reference/experiments/load-pulse-warehouse-native https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/load_pulse # Partially Update Experiment Source: https://docs.statsig.com/api-reference/experiments/partially-update-experiment https://api.statsig.com/openapi/20240601.json patch /console/v1/experiments/{id} # Partially Update Experiment Overrides Source: https://docs.statsig.com/api-reference/experiments/partially-update-experiment-overrides https://api.statsig.com/openapi/20240601.json patch /console/v1/experiments/{id}/overrides # Patch Assignment Source Source: https://docs.statsig.com/api-reference/experiments/patch-assignment-source https://api.statsig.com/openapi/20240601.json patch /console/v1/experiments/assignment_source/{name} # Patch Entity Property Source Source: https://docs.statsig.com/api-reference/experiments/patch-entity-property-source https://api.statsig.com/openapi/20240601.json patch /console/v1/experiments/entity_property/{name} # Post Assignment Source Source: https://docs.statsig.com/api-reference/experiments/post-assignment-source https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/assignment_source/{name} # Post Entity Property Source Source: https://docs.statsig.com/api-reference/experiments/post-entity-property-source https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/entity_property/{name} # Pulse Load History (Warehouse Native) Source: https://docs.statsig.com/api-reference/experiments/pulse-load-history-warehouse-native https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/pulse_load_history # Reset Experiment Source: https://docs.statsig.com/api-reference/experiments/reset-experiment https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/reset # Resolve Metric Rollout Alert Source: https://docs.statsig.com/api-reference/experiments/resolve-metric-rollout-alert https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/alerts/{metricId}/resolve # Restart As New Experiment Source: https://docs.statsig.com/api-reference/experiments/restart-as-new-experiment https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/restart_as_new # Retrieve cumulative exposures Source: https://docs.statsig.com/api-reference/experiments/retrieve-cumulative-exposures https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/cumulative_exposures # Retrieve Experiment Checks Diagnostics Source: https://docs.statsig.com/api-reference/experiments/retrieve-experiment-checks-diagnostics https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/diagnostics_checks # Retrieve Experiment Summary Charts (Beta) Source: https://docs.statsig.com/api-reference/experiments/retrieve-experiment-summary-charts-beta https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/summary_charts # Retrieve Exposures By Dimension Source: https://docs.statsig.com/api-reference/experiments/retrieve-exposures-by-dimension https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/dimensional_exposures # Retrieve Pulse Metric Result Source: https://docs.statsig.com/api-reference/experiments/retrieve-pulse-metric-result https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/pulse_metric_result # Retrieve Pulse Results (Beta) Source: https://docs.statsig.com/api-reference/experiments/retrieve-pulse-results-beta https://api.statsig.com/openapi/20240601.json get /console/v1/experiments/{id}/pulse_results # Schedule Experiment Start Source: https://docs.statsig.com/api-reference/experiments/schedule-experiment-start https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/schedule_start # Start Experiment Source: https://docs.statsig.com/api-reference/experiments/start-experiment https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/start # Start Experiment Code Cleanup Source: https://docs.statsig.com/api-reference/experiments/start-experiment-code-cleanup https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/code_cleanup # Unarchive Experiment Source: https://docs.statsig.com/api-reference/experiments/unarchive-experiment https://api.statsig.com/openapi/20240601.json put /console/v1/experiments/{id}/unarchive # Update Experiment Overrides Source: https://docs.statsig.com/api-reference/experiments/update-experiment-overrides https://api.statsig.com/openapi/20240601.json post /console/v1/experiments/{id}/overrides # Check Feature Gate(s) Source: https://docs.statsig.com/api-reference/feature-gates/check-feature-gates /http-api/httpopenapi.json post /v1/check_gate Evaluates one or more feature gates for a user and returns pass/fail results. Automatically logs exposure events for analytics. # Add Gate Overrides Source: https://docs.statsig.com/api-reference/gates/add-gate-overrides https://api.statsig.com/openapi/20240601.json patch /console/v1/gates/{id}/overrides # Add Gate Rule Source: https://docs.statsig.com/api-reference/gates/add-gate-rule https://api.statsig.com/openapi/20240601.json post /console/v1/gates/{id}/rule # Archive Gate Source: https://docs.statsig.com/api-reference/gates/archive-gate https://api.statsig.com/openapi/20240601.json put /console/v1/gates/{id}/archive # Commit Gate Review Source: https://docs.statsig.com/api-reference/gates/commit-gate-review https://api.statsig.com/openapi/20240601.json put /console/v1/gates/{id}/reviews/{reviewID}/commit # Create Gate Source: https://docs.statsig.com/api-reference/gates/create-gate https://api.statsig.com/openapi/20240601.json post /console/v1/gates # Delete Gate Overrides Source: https://docs.statsig.com/api-reference/gates/delete-gate-overrides https://api.statsig.com/openapi/20240601.json delete /console/v1/gates/{id}/overrides # Delete Gate Rule Source: https://docs.statsig.com/api-reference/gates/delete-gate-rule https://api.statsig.com/openapi/20240601.json delete /console/v1/gates/{id}/rules/{ruleID} # Delete Gates Source: https://docs.statsig.com/api-reference/gates/delete-gates https://api.statsig.com/openapi/20240601.json delete /console/v1/gates/{id} # Disable Gate Source: https://docs.statsig.com/api-reference/gates/disable-gate https://api.statsig.com/openapi/20240601.json put /console/v1/gates/{id}/disable # Enable Gate Source: https://docs.statsig.com/api-reference/gates/enable-gate https://api.statsig.com/openapi/20240601.json put /console/v1/gates/{id}/enable # Fully Update Gates Source: https://docs.statsig.com/api-reference/gates/fully-update-gates https://api.statsig.com/openapi/20240601.json post /console/v1/gates/{id} # Get Gate Override Source: https://docs.statsig.com/api-reference/gates/get-gate-override https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/overrides # Launch Gate Source: https://docs.statsig.com/api-reference/gates/launch-gate https://api.statsig.com/openapi/20240601.json put /console/v1/gates/{id}/launch # List Dynamic Config References Source: https://docs.statsig.com/api-reference/gates/list-dynamic-config-references https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/dynamic_config_references # List Experiment References Source: https://docs.statsig.com/api-reference/gates/list-experiment-references https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/experiment_references # List Gate References Source: https://docs.statsig.com/api-reference/gates/list-gate-references https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/gate_references # List Gate Versions Source: https://docs.statsig.com/api-reference/gates/list-gate-versions https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/versions # List Gates Source: https://docs.statsig.com/api-reference/gates/list-gates https://api.statsig.com/openapi/20240601.json get /console/v1/gates # Load Pulse Gate Source: https://docs.statsig.com/api-reference/gates/load-pulse-gate https://api.statsig.com/openapi/20240601.json post /console/v1/gates/{id}/load_pulse # Partially Update Gates Source: https://docs.statsig.com/api-reference/gates/partially-update-gates https://api.statsig.com/openapi/20240601.json patch /console/v1/gates/{id} # Pulse Load History (Warehouse Native) Source: https://docs.statsig.com/api-reference/gates/pulse-load-history-warehouse-native https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/rules/{ruleID}/pulse_load_history # Read Gate Source: https://docs.statsig.com/api-reference/gates/read-gate https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id} # Read Gate Checks Source: https://docs.statsig.com/api-reference/gates/read-gate-checks https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/checks # Read Gate Rules Source: https://docs.statsig.com/api-reference/gates/read-gate-rules https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/rules # Resolve Metric Rollout Alert Source: https://docs.statsig.com/api-reference/gates/resolve-metric-rollout-alert https://api.statsig.com/openapi/20240601.json post /console/v1/gates/{id}/alerts/{metricId}/resolve # Retrieve Pulse Results Source: https://docs.statsig.com/api-reference/gates/retrieve-pulse-results https://api.statsig.com/openapi/20240601.json get /console/v1/gates/{id}/rules/{ruleID}/pulse_results # Start Gate Code Cleanup Source: https://docs.statsig.com/api-reference/gates/start-gate-code-cleanup https://api.statsig.com/openapi/20240601.json post /console/v1/gates/{id}/code_cleanup # Unarchive Gate Source: https://docs.statsig.com/api-reference/gates/unarchive-gate https://api.statsig.com/openapi/20240601.json put /console/v1/gates/{id}/unarchive # Update Gate Overrides Source: https://docs.statsig.com/api-reference/gates/update-gate-overrides https://api.statsig.com/openapi/20240601.json post /console/v1/gates/{id}/overrides # Update Gate Rules Source: https://docs.statsig.com/api-reference/gates/update-gate-rules https://api.statsig.com/openapi/20240601.json patch /console/v1/gates/{id}/rules/{ruleID} Update all given rules. It does NOT create or delete if you add more rules and remove rules in the rules object. # Add Holdout Overrides Source: https://docs.statsig.com/api-reference/holdouts/add-holdout-overrides https://api.statsig.com/openapi/20240601.json patch /console/v1/holdouts/{id}/overrides # Create holdout Source: https://docs.statsig.com/api-reference/holdouts/create-holdout https://api.statsig.com/openapi/20240601.json post /console/v1/holdouts # Delete holdout by id Source: https://docs.statsig.com/api-reference/holdouts/delete-holdout-by-id https://api.statsig.com/openapi/20240601.json delete /console/v1/holdouts/{id} # Get holdout by id Source: https://docs.statsig.com/api-reference/holdouts/get-holdout-by-id https://api.statsig.com/openapi/20240601.json get /console/v1/holdouts/{id} # List Holdouts Source: https://docs.statsig.com/api-reference/holdouts/list-holdouts https://api.statsig.com/openapi/20240601.json get /console/v1/holdouts # Partially update holdout by id Source: https://docs.statsig.com/api-reference/holdouts/partially-update-holdout-by-id https://api.statsig.com/openapi/20240601.json patch /console/v1/holdouts/{id} You can pass in only the data you want to update. # Read Holdout Overrides Source: https://docs.statsig.com/api-reference/holdouts/read-holdout-overrides https://api.statsig.com/openapi/20240601.json get /console/v1/holdouts/{id}/overrides # Remove Holdout Overrides Source: https://docs.statsig.com/api-reference/holdouts/remove-holdout-overrides https://api.statsig.com/openapi/20240601.json delete /console/v1/holdouts/{id}/overrides Remove selected ids from an id list # Retrieve Pulse Results Source: https://docs.statsig.com/api-reference/holdouts/retrieve-pulse-results https://api.statsig.com/openapi/20240601.json get /console/v1/holdouts/{id}/pulse_results # Update holdout by id Source: https://docs.statsig.com/api-reference/holdouts/update-holdout-by-id https://api.statsig.com/openapi/20240601.json post /console/v1/holdouts/{id} # Update Holdout Overrides Source: https://docs.statsig.com/api-reference/holdouts/update-holdout-overrides https://api.statsig.com/openapi/20240601.json post /console/v1/holdouts/{id}/overrides # Backfill Ingestion Source: https://docs.statsig.com/api-reference/ingestions/backfill-ingestion https://api.statsig.com/openapi/20240601.json post /console/v1/ingestion/backfill # Create Ingestion Databricks Source: https://docs.statsig.com/api-reference/ingestions/create-ingestion-databricks https://api.statsig.com/openapi/20240601.json post /console/v1/ingestion/connection/databricks # Create Ingestion Source Source: https://docs.statsig.com/api-reference/ingestions/create-ingestion-source https://api.statsig.com/openapi/20240601.json post /console/v1/ingestion # Delete Ingestion Source Source: https://docs.statsig.com/api-reference/ingestions/delete-ingestion-source https://api.statsig.com/openapi/20240601.json delete /console/v1/ingestion # Get Ingestion Event Count Source: https://docs.statsig.com/api-reference/ingestions/get-ingestion-event-count https://api.statsig.com/openapi/20240601.json get /console/v1/ingestion/events/count # Get Ingestion Event Delta Ledger Source: https://docs.statsig.com/api-reference/ingestions/get-ingestion-event-delta-ledger https://api.statsig.com/openapi/20240601.json get /console/v1/ingestion/events/delta # List Ingestion Runs Source: https://docs.statsig.com/api-reference/ingestions/list-ingestion-runs https://api.statsig.com/openapi/20240601.json get /console/v1/ingestion/runs # List Ingestions Status Source: https://docs.statsig.com/api-reference/ingestions/list-ingestions-status https://api.statsig.com/openapi/20240601.json get /console/v1/ingestion/status # Read Ingestion Source: https://docs.statsig.com/api-reference/ingestions/read-ingestion https://api.statsig.com/openapi/20240601.json get /console/v1/ingestion # Read Ingestion Run Source: https://docs.statsig.com/api-reference/ingestions/read-ingestion-run https://api.statsig.com/openapi/20240601.json get /console/v1/ingestion/runs/{id} # Read Ingestion Schedule Source: https://docs.statsig.com/api-reference/ingestions/read-ingestion-schedule https://api.statsig.com/openapi/20240601.json get /console/v1/ingestion/schedule # Update Ingestion Schedule Source: https://docs.statsig.com/api-reference/ingestions/update-ingestion-schedule https://api.statsig.com/openapi/20240601.json post /console/v1/ingestion/schedule # Update Ingestion Source Source: https://docs.statsig.com/api-reference/ingestions/update-ingestion-source https://api.statsig.com/openapi/20240601.json patch /console/v1/ingestion # Create Key Source: https://docs.statsig.com/api-reference/keys/create-key https://api.statsig.com/openapi/20240601.json post /console/v1/keys # Deactivate Key Source: https://docs.statsig.com/api-reference/keys/deactivate-key https://api.statsig.com/openapi/20240601.json patch /console/v1/keys/{id}/deactivate # Delete Key Source: https://docs.statsig.com/api-reference/keys/delete-key https://api.statsig.com/openapi/20240601.json delete /console/v1/keys/{id} # List Keys Source: https://docs.statsig.com/api-reference/keys/list-keys https://api.statsig.com/openapi/20240601.json get /console/v1/keys # Read Key Source: https://docs.statsig.com/api-reference/keys/read-key https://api.statsig.com/openapi/20240601.json get /console/v1/keys/{id} # Rotate Key Source: https://docs.statsig.com/api-reference/keys/rotate-key https://api.statsig.com/openapi/20240601.json patch /console/v1/keys/{id}/rotate # Update Key Source: https://docs.statsig.com/api-reference/keys/update-key https://api.statsig.com/openapi/20240601.json patch /console/v1/keys/{id} # Add Layer Overrides Source: https://docs.statsig.com/api-reference/layers/add-layer-overrides https://api.statsig.com/openapi/20240601.json patch /console/v1/layers/{id}/overrides # Create a Layer Source: https://docs.statsig.com/api-reference/layers/create-a-layer https://api.statsig.com/openapi/20240601.json post /console/v1/layers # Delete a layer Source: https://docs.statsig.com/api-reference/layers/delete-a-layer https://api.statsig.com/openapi/20240601.json delete /console/v1/layers/{id} # Delete Layer Overrides Source: https://docs.statsig.com/api-reference/layers/delete-layer-overrides https://api.statsig.com/openapi/20240601.json delete /console/v1/layers/{id}/overrides # Get Layer Overrides Source: https://docs.statsig.com/api-reference/layers/get-layer-overrides https://api.statsig.com/openapi/20240601.json get /console/v1/layers/{id}/overrides # Get Layer Parameters Source: https://docs.statsig.com/api-reference/layers/get-layer-parameters /http-api/httpopenapi.json post /v1/get_layer Fetches parameter values from a layer. Layers allow you to share parameters across multiple experiments. Automatically logs exposure events. # Get Layers Source: https://docs.statsig.com/api-reference/layers/get-layers https://api.statsig.com/openapi/20240601.json get /console/v1/layers # Get one layer Source: https://docs.statsig.com/api-reference/layers/get-one-layer https://api.statsig.com/openapi/20240601.json get /console/v1/layers/{id} # Lineage: List Experiment related to Layer Source: https://docs.statsig.com/api-reference/layers/lineage:-list-experiment-related-to-layer https://api.statsig.com/openapi/20240601.json get /console/v1/layers/{id}/experiments # Partially update a layer Source: https://docs.statsig.com/api-reference/layers/partially-update-a-layer https://api.statsig.com/openapi/20240601.json patch /console/v1/layers/{id} # Update a layer Source: https://docs.statsig.com/api-reference/layers/update-a-layer https://api.statsig.com/openapi/20240601.json post /console/v1/layers/{id} # Update Layer Overrides Source: https://docs.statsig.com/api-reference/layers/update-layer-overrides https://api.statsig.com/openapi/20240601.json post /console/v1/layers/{id}/overrides # Cancel archive a metric Source: https://docs.statsig.com/api-reference/metrics/cancel-archive-a-metric https://api.statsig.com/openapi/20240601.json put /console/v1/metrics/{id}/cancel_archive # Create Metric Source: https://docs.statsig.com/api-reference/metrics/create-metric https://api.statsig.com/openapi/20240601.json post /console/v1/metrics # Create Metric Source Source: https://docs.statsig.com/api-reference/metrics/create-metric-source https://api.statsig.com/openapi/20240601.json post /console/v1/metrics/metric_source # Delete a metric Source: https://docs.statsig.com/api-reference/metrics/delete-a-metric https://api.statsig.com/openapi/20240601.json delete /console/v1/metrics/{id} # Delete Metric Source Source: https://docs.statsig.com/api-reference/metrics/delete-metric-source https://api.statsig.com/openapi/20240601.json delete /console/v1/metrics/metric_source/{name} # Get SQL for a metric Source: https://docs.statsig.com/api-reference/metrics/get-sql-for-a-metric https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/{id}/sql # Lineage: List experiments related to Metric Source: https://docs.statsig.com/api-reference/metrics/lineage:-list-experiments-related-to-metric https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/{id}/experiments # List All Metric Values Source: https://docs.statsig.com/api-reference/metrics/list-all-metric-values https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/values List all metric values # List all Metrics Source: https://docs.statsig.com/api-reference/metrics/list-all-metrics https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/list # List metric source Source: https://docs.statsig.com/api-reference/metrics/list-metric-source https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/metric_source/list # Read Metric Definition Source: https://docs.statsig.com/api-reference/metrics/read-metric-definition https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/{id} # Read Metric Definition by Name Source: https://docs.statsig.com/api-reference/metrics/read-metric-definition-by-name https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/{name}/{type} # Read Metric Source Source: https://docs.statsig.com/api-reference/metrics/read-metric-source https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/metric_source/{name} # Read Metric Source Metrics Source: https://docs.statsig.com/api-reference/metrics/read-metric-source-metrics https://api.statsig.com/openapi/20240601.json get /console/v1/metrics/metric_source/{name}/metrics # Read Single Metric Value Source: https://docs.statsig.com/api-reference/metrics/read-single-metric-value https://api.statsig.com/openapi/20240601.json get /console/v1/metrics # Reload metric data Source: https://docs.statsig.com/api-reference/metrics/reload-metric-data https://api.statsig.com/openapi/20240601.json post /console/v1/metrics/{id}/reload # Schedule a metric archive Source: https://docs.statsig.com/api-reference/metrics/schedule-a-metric-archive https://api.statsig.com/openapi/20240601.json put /console/v1/metrics/{id}/schedule_archive # Unarchive a metric Source: https://docs.statsig.com/api-reference/metrics/unarchive-a-metric https://api.statsig.com/openapi/20240601.json put /console/v1/metrics/{id}/unarchive # Update a metric Source: https://docs.statsig.com/api-reference/metrics/update-a-metric https://api.statsig.com/openapi/20240601.json post /console/v1/metrics/{id} # Update Metric Source Source: https://docs.statsig.com/api-reference/metrics/update-metric-source https://api.statsig.com/openapi/20240601.json post /console/v1/metrics/metric_source/{name} # Create Param Store Source: https://docs.statsig.com/api-reference/param-store/create-param-store https://api.statsig.com/openapi/20240601.json post /console/v1/param_stores # Delete Param Store Source: https://docs.statsig.com/api-reference/param-store/delete-param-store https://api.statsig.com/openapi/20240601.json delete /console/v1/param_stores/{name} # Get Param Store Source: https://docs.statsig.com/api-reference/param-store/get-param-store https://api.statsig.com/openapi/20240601.json get /console/v1/param_stores/{name} # List Param Stores Source: https://docs.statsig.com/api-reference/param-store/list-param-stores https://api.statsig.com/openapi/20240601.json get /console/v1/param_stores # Update Param Store Source: https://docs.statsig.com/api-reference/param-store/update-param-store https://api.statsig.com/openapi/20240601.json post /console/v1/param_stores/{name} # Get Project Info Source: https://docs.statsig.com/api-reference/project/get-project-info https://api.statsig.com/openapi/20240601.json get /console/v1/project # Create Prompt Source: https://docs.statsig.com/api-reference/prompts/create-prompt https://api.statsig.com/openapi/20240601.json post /console/v1/prompts # Create Prompt Version Source: https://docs.statsig.com/api-reference/prompts/create-prompt-version https://api.statsig.com/openapi/20240601.json post /console/v1/prompts/{id}/versions # Get Prompt Source: https://docs.statsig.com/api-reference/prompts/get-prompt https://api.statsig.com/openapi/20240601.json get /console/v1/prompts/{id} # List Prompts Source: https://docs.statsig.com/api-reference/prompts/list-prompts https://api.statsig.com/openapi/20240601.json get /console/v1/prompts # Start Prompt Version Evaluation Job Source: https://docs.statsig.com/api-reference/prompts/start-prompt-version-evaluation-job https://api.statsig.com/openapi/20240601.json post /console/v1/prompts/{id}/versions/{versionId}/start_evals # Update Prompt (partial) Source: https://docs.statsig.com/api-reference/prompts/update-prompt-partial https://api.statsig.com/openapi/20240601.json patch /console/v1/prompts/{id} # Abort Pipeline Trigger Source: https://docs.statsig.com/api-reference/release-pipelines/abort-pipeline-trigger https://api.statsig.com/openapi/20240601.json put /console/v1/release_pipeline_triggers/{id}/abort # Approve Pipeline Trigger Phase Source: https://docs.statsig.com/api-reference/release-pipelines/approve-pipeline-trigger-phase https://api.statsig.com/openapi/20240601.json put /console/v1/release_pipeline_triggers/{id}/approve # Create Pipeline Source: https://docs.statsig.com/api-reference/release-pipelines/create-pipeline https://api.statsig.com/openapi/20240601.json post /console/v1/release_pipelines # Delete Pipeline Source: https://docs.statsig.com/api-reference/release-pipelines/delete-pipeline https://api.statsig.com/openapi/20240601.json delete /console/v1/release_pipelines/{id} # Fully Roll Out Pipeline Trigger Source: https://docs.statsig.com/api-reference/release-pipelines/fully-roll-out-pipeline-trigger https://api.statsig.com/openapi/20240601.json put /console/v1/release_pipeline_triggers/{id}/rollout # Get Pipeline Source: https://docs.statsig.com/api-reference/release-pipelines/get-pipeline https://api.statsig.com/openapi/20240601.json get /console/v1/release_pipelines/{id} # Get Pipeline Trigger Source: https://docs.statsig.com/api-reference/release-pipelines/get-pipeline-trigger https://api.statsig.com/openapi/20240601.json get /console/v1/release_pipeline_triggers/{id} # List Pipeline Triggers Source: https://docs.statsig.com/api-reference/release-pipelines/list-pipeline-triggers https://api.statsig.com/openapi/20240601.json get /console/v1/release_pipeline_triggers # List Pipelines Source: https://docs.statsig.com/api-reference/release-pipelines/list-pipelines https://api.statsig.com/openapi/20240601.json get /console/v1/release_pipelines # Pause Pipeline Trigger Source: https://docs.statsig.com/api-reference/release-pipelines/pause-pipeline-trigger https://api.statsig.com/openapi/20240601.json put /console/v1/release_pipeline_triggers/{id}/pause # Skip to Pipeline Trigger Phase Source: https://docs.statsig.com/api-reference/release-pipelines/skip-to-pipeline-trigger-phase https://api.statsig.com/openapi/20240601.json put /console/v1/release_pipeline_triggers/{id}/skip # Unpause Pipeline Trigger Source: https://docs.statsig.com/api-reference/release-pipelines/unpause-pipeline-trigger https://api.statsig.com/openapi/20240601.json put /console/v1/release_pipeline_triggers/{id}/unpause # Update Pipeline Source: https://docs.statsig.com/api-reference/release-pipelines/update-pipeline https://api.statsig.com/openapi/20240601.json post /console/v1/release_pipelines/{id} # Get Reports Source: https://docs.statsig.com/api-reference/reports/get-reports https://api.statsig.com/openapi/20240601.json get /console/v1/reports # Create Role Source: https://docs.statsig.com/api-reference/roles/create-role https://api.statsig.com/openapi/20240601.json post /console/v1/roles # Delete Role Source: https://docs.statsig.com/api-reference/roles/delete-role https://api.statsig.com/openapi/20240601.json delete /console/v1/roles/{id} # Get Role Source: https://docs.statsig.com/api-reference/roles/get-role https://api.statsig.com/openapi/20240601.json get /console/v1/roles/{id} # List Roles Source: https://docs.statsig.com/api-reference/roles/list-roles https://api.statsig.com/openapi/20240601.json get /console/v1/roles # Update Role Source: https://docs.statsig.com/api-reference/roles/update-role https://api.statsig.com/openapi/20240601.json patch /console/v1/roles/{id} # Delete scimgroups Source: https://docs.statsig.com/api-reference/scim-groups/delete-scimgroups /access-management/scim/scimopenapi.json delete /scim/Groups/{id} # Get scimgroups Source: https://docs.statsig.com/api-reference/scim-groups/get-scimgroups /access-management/scim/scimopenapi.json get /scim/Groups # Get scimgroups 1 Source: https://docs.statsig.com/api-reference/scim-groups/get-scimgroups-1 /access-management/scim/scimopenapi.json get /scim/Groups/{id} # Patch scimgroups Source: https://docs.statsig.com/api-reference/scim-groups/patch-scimgroups /access-management/scim/scimopenapi.json patch /scim/Groups/{id} # Post scimgroups Source: https://docs.statsig.com/api-reference/scim-groups/post-scimgroups /access-management/scim/scimopenapi.json post /scim/Groups # Put scimgroups Source: https://docs.statsig.com/api-reference/scim-groups/put-scimgroups /access-management/scim/scimopenapi.json put /scim/Groups/{id} # Get scimresourcetypes Source: https://docs.statsig.com/api-reference/scim-settings/get-scimresourcetypes /access-management/scim/scimopenapi.json get /scim/ResourceTypes # Get scimschemas Source: https://docs.statsig.com/api-reference/scim-settings/get-scimschemas /access-management/scim/scimopenapi.json get /scim/Schemas # Get scimserviceproviderconfig Source: https://docs.statsig.com/api-reference/scim-settings/get-scimserviceproviderconfig /access-management/scim/scimopenapi.json get /scim/ServiceProviderConfig # Get scimusers Source: https://docs.statsig.com/api-reference/scim-users/get-scimusers /access-management/scim/scimopenapi.json get /scim/Users # Get scimusers 1 Source: https://docs.statsig.com/api-reference/scim-users/get-scimusers-1 /access-management/scim/scimopenapi.json get /scim/Users/{id} # Patch scimusers Source: https://docs.statsig.com/api-reference/scim-users/patch-scimusers /access-management/scim/scimopenapi.json patch /scim/Users/{id} # Post scimusers Source: https://docs.statsig.com/api-reference/scim-users/post-scimusers /access-management/scim/scimopenapi.json post /scim/Users # Put scimusers Source: https://docs.statsig.com/api-reference/scim-users/put-scimusers /access-management/scim/scimopenapi.json put /scim/Users/{id} # Add IDs to Segment Source: https://docs.statsig.com/api-reference/segments/add-ids-to-segment https://api.statsig.com/openapi/20240601.json patch /console/v1/segments/{id}/id_list This endpoint is rate limited to 900 requests /15m or 12 requests /10s. # Add IDs to User Store ID List Source: https://docs.statsig.com/api-reference/segments/add-ids-to-user-store-id-list https://api.statsig.com/openapi/20240601.json patch /console/v1/segments/{id}/add_ids This endpoint has a limit of 1000 ids per request # Archive Segment Source: https://docs.statsig.com/api-reference/segments/archive-segment https://api.statsig.com/openapi/20240601.json put /console/v1/segments/{id}/archive # Commit Segment Review Source: https://docs.statsig.com/api-reference/segments/commit-segment-review https://api.statsig.com/openapi/20240601.json put /console/v1/segments/{id}/reviews/{reviewID}/commit # Create Segment Source: https://docs.statsig.com/api-reference/segments/create-segment https://api.statsig.com/openapi/20240601.json post /console/v1/segments # Delete Segment Source: https://docs.statsig.com/api-reference/segments/delete-segment https://api.statsig.com/openapi/20240601.json delete /console/v1/segments/{id} # Get ID List Metadata Source: https://docs.statsig.com/api-reference/segments/get-id-list-metadata https://api.statsig.com/openapi/20240601.json get /console/v1/segments/{id}/idlist_metadata This endpoint gets the metadata of the ID List. # Get IDs in a Segment Source: https://docs.statsig.com/api-reference/segments/get-ids-in-a-segment https://api.statsig.com/openapi/20240601.json get /console/v1/segments/{id}/id_list This endpoint is rate limited to 100 # Get Segment Source: https://docs.statsig.com/api-reference/segments/get-segment https://api.statsig.com/openapi/20240601.json get /console/v1/segments/{id} # List Segments Source: https://docs.statsig.com/api-reference/segments/list-segments https://api.statsig.com/openapi/20240601.json get /console/v1/segments # Remove IDs from Segment Source: https://docs.statsig.com/api-reference/segments/remove-ids-from-segment https://api.statsig.com/openapi/20240601.json delete /console/v1/segments/{id}/id_list This endpoint is rate limited to 900 requests /15m or 12 requests /10s. # Remove IDs from User Store ID List Source: https://docs.statsig.com/api-reference/segments/remove-ids-from-user-store-id-list https://api.statsig.com/openapi/20240601.json patch /console/v1/segments/{id}/remove_ids This endpoint has a limit of 1000 ids per request # Reset ID List Segment Source: https://docs.statsig.com/api-reference/segments/reset-id-list-segment https://api.statsig.com/openapi/20240601.json post /console/v1/segments/{id}/id_list/reset Warning: Not atomic for big ID lists (> 1000) # Update Segment Rules Source: https://docs.statsig.com/api-reference/segments/update-segment-rules https://api.statsig.com/openapi/20240601.json post /console/v1/segments/{id}/conditional # Get Project Settings Source: https://docs.statsig.com/api-reference/settings/get-project-settings https://api.statsig.com/openapi/20240601.json get /console/v1/settings/project # Get Reviews Settings Source: https://docs.statsig.com/api-reference/settings/get-reviews-settings https://api.statsig.com/openapi/20240601.json get /console/v1/settings/reviews # Get Roles Settings Source: https://docs.statsig.com/api-reference/settings/get-roles-settings https://api.statsig.com/openapi/20240601.json get /console/v1/settings/roles # Get Teams Settings Source: https://docs.statsig.com/api-reference/settings/get-teams-settings https://api.statsig.com/openapi/20240601.json get /console/v1/settings/teams # Update Project Settings Source: https://docs.statsig.com/api-reference/settings/update-project-settings https://api.statsig.com/openapi/20240601.json post /console/v1/settings/project # Update Reviews Settings Source: https://docs.statsig.com/api-reference/settings/update-reviews-settings https://api.statsig.com/openapi/20240601.json post /console/v1/settings/reviews # Update Roles Settings Source: https://docs.statsig.com/api-reference/settings/update-roles-settings https://api.statsig.com/openapi/20240601.json post /console/v1/settings/roles # Update Teams Settings Source: https://docs.statsig.com/api-reference/settings/update-teams-settings https://api.statsig.com/openapi/20240601.json post /console/v1/settings/teams # Create Tag Source: https://docs.statsig.com/api-reference/tags/create-tag https://api.statsig.com/openapi/20240601.json post /console/v1/tags # Delete Tag Source: https://docs.statsig.com/api-reference/tags/delete-tag https://api.statsig.com/openapi/20240601.json delete /console/v1/tags/{id} # List Tags Source: https://docs.statsig.com/api-reference/tags/list-tags https://api.statsig.com/openapi/20240601.json get /console/v1/tags # Read Tag Source: https://docs.statsig.com/api-reference/tags/read-tag https://api.statsig.com/openapi/20240601.json get /console/v1/tags/{id} # Update Tag Source: https://docs.statsig.com/api-reference/tags/update-tag https://api.statsig.com/openapi/20240601.json patch /console/v1/tags/{id} # Bulk Assign Target Apps Source: https://docs.statsig.com/api-reference/target-app/bulk-assign-target-apps https://api.statsig.com/openapi/20240601.json patch /console/v1/target_app # Create Target App Source: https://docs.statsig.com/api-reference/target-app/create-target-app https://api.statsig.com/openapi/20240601.json post /console/v1/target_app # Delete Target App Source: https://docs.statsig.com/api-reference/target-app/delete-target-app https://api.statsig.com/openapi/20240601.json delete /console/v1/target_app/{id} # List Target Apps Source: https://docs.statsig.com/api-reference/target-app/list-target-apps https://api.statsig.com/openapi/20240601.json get /console/v1/target_app # Read Target App Source: https://docs.statsig.com/api-reference/target-app/read-target-app https://api.statsig.com/openapi/20240601.json get /console/v1/target_app/{id} # Update Target App Source: https://docs.statsig.com/api-reference/target-app/update-target-app https://api.statsig.com/openapi/20240601.json patch /console/v1/target_app/{id} # Create Unit ID Type Source: https://docs.statsig.com/api-reference/unit-id-types/create-unit-id-type https://api.statsig.com/openapi/20240601.json post /console/v1/unit_id_types # Delete Unit ID Type Source: https://docs.statsig.com/api-reference/unit-id-types/delete-unit-id-type https://api.statsig.com/openapi/20240601.json delete /console/v1/unit_id_types/{id} # Get Unit ID Type Source: https://docs.statsig.com/api-reference/unit-id-types/get-unit-id-type https://api.statsig.com/openapi/20240601.json get /console/v1/unit_id_types/{id} # List Unit ID Types Source: https://docs.statsig.com/api-reference/unit-id-types/list-unit-id-types https://api.statsig.com/openapi/20240601.json get /console/v1/unit_id_types # Update Unit ID Type Source: https://docs.statsig.com/api-reference/unit-id-types/update-unit-id-type https://api.statsig.com/openapi/20240601.json patch /console/v1/unit_id_types/{id} # Get Report in CSV format Source: https://docs.statsig.com/api-reference/usage/get-report-in-csv-format https://api.statsig.com/openapi/20240601.json get /console/v1/project/usage_billing/report # Create Team Source: https://docs.statsig.com/api-reference/users/create-team https://api.statsig.com/openapi/20240601.json post /console/v1/users/teams # Delete Team Source: https://docs.statsig.com/api-reference/users/delete-team https://api.statsig.com/openapi/20240601.json delete /console/v1/users/teams/{id} # Get Team Source: https://docs.statsig.com/api-reference/users/get-team https://api.statsig.com/openapi/20240601.json get /console/v1/users/teams/{id} # Get user by email Source: https://docs.statsig.com/api-reference/users/get-user-by-email https://api.statsig.com/openapi/20240601.json get /console/v1/users/{email} # Get user by ID Source: https://docs.statsig.com/api-reference/users/get-user-by-id https://api.statsig.com/openapi/20240601.json get /console/v1/users/id/{id} # Invite users Source: https://docs.statsig.com/api-reference/users/invite-users https://api.statsig.com/openapi/20240601.json post /console/v1/users/invite Invite a list of users with assigned roles and teams. To avoid spamming, invitation emails are not sent. Invitee will see invitation notification in-app after logging in. # List Teams Source: https://docs.statsig.com/api-reference/users/list-teams https://api.statsig.com/openapi/20240601.json get /console/v1/users/teams # List Users Source: https://docs.statsig.com/api-reference/users/list-users https://api.statsig.com/openapi/20240601.json get /console/v1/users # Update team Source: https://docs.statsig.com/api-reference/users/update-team https://api.statsig.com/openapi/20240601.json patch /console/v1/users/teams/{id} Replace the team definition. Use GET to fetch current data if you intend to add fields without overwriting existing values. # Update user Source: https://docs.statsig.com/api-reference/users/update-user https://api.statsig.com/openapi/20240601.json post /console/v1/users/{email} # Update Warehouse Connection Parameters Source: https://docs.statsig.com/api-reference/warehouse-connections/update-warehouse-connection-parameters https://api.statsig.com/openapi/20240601.json patch /console/v1/wh_connections # Bandit FAQs Source: https://docs.statsig.com/autotune/bandit-faq Frequently asked questions about Statsig Autotune multi-armed bandits, including methodology, traffic allocation, and how to interpret bandit results. ### When should I see data show up in a bandit? You will see diagnostic data appear in near-real time in the logstream on your bandit. Data on the models/results section will depend on your settings. We do not consider units enrolled until their attribution window has elapsed, so if you have a 24 hour attribution window it will take 24 hours for you to see your first units and model updates. Models are trained hourly, and there can be a few minutes of data delay, so there may be 1-2 hours of additional delay beyond that period. Generally, this is not a super time-sensitive operation and accordingly does not have a strict SLA on Statsig's side. There may be delays due to data quality checks or other investigations to avoid bad model training data polluting your production environment. ### What do I put into the Variant JSON? The variants return this JSON as a config, just like dynamic configs or experiments. You can just put an identifier for the variant, or specify a set of attributes that will be accessed in code to avoid needing to write conditional statements to fetch the corresponding attributes. ### Why use a linked experiment? Bandits optimize for a single target metric or event. This doesn't give a super-rich analysis of how the bandit is impacting your users on its own. Wrapping a bandit in an experiment where the control group gets a default experience and the variant group gets served an experience by the bandit gives you the ability to get deeper insights on how it is impacting your end users. This step is highly recommended! ### Where is my Autotune data available? Autotune data is downloaded hourly into the `statsig_first_exposures` metric source. Click into the history modal on the autotune itself to see the SQL and tables generated for an individual autotune. ### Why was a "losing" variant chosen? Autotune makes the assumption that a metric is consistent across time; in other words, if a metric has strong seasonality (e.g. a metric which tanks on the weekends) it is not a good candidate for usage in an Autotune experiment. With seasonality, this case is possible: * One variant which consistently performs worse than other variants gets less and less traffic, until it has essentially zero traffic * Other variants, which are still receiving traffic, see their metric tank when the season effects kick in * The variant with zero traffic doesn't see that metric tank (because no traffic is being allocated to it), which causes it to be chosen as the winning variant This may sometimes cause a "losing" variant to be chosen as the winner. # Get Started with Autotune AI Source: https://docs.statsig.com/autotune/contextual/getting-started Get started with Statsig Contextual Bandits to personalize variant selection per user based on context features and a single goal metric. Getting started with Autotune AI can be done very quickly. Statsig supports contextual autotune in all Client SDKs, but only in the following server SDKs: * [Node](/server-core/node-core) * [Python](/server-core/python-core) * [Java](/server-core/java-core) * [Elixir](/server-core/elixir-core) * [Rust](/server-core/rust-core) * [Php](/server-core/php-core) * [Ruby](/server/rubySDK) # Set Up Your Contextual Autotune The first thing you will do is configure your contextual autotune in the Statsig console. This can also be configured programmatically via the Statsig console API. ## Make the Contextual Autotune Log into your Statsig console, and navigate to Autotune under Experiments. Statsig console navigation with Autotune section highlighted Click create. You need to name your contextual autotune, and optionally specify the goal so other users can understand the motivation behind it. Create contextual autotune dialog with name and goal fields Set your autotune type to Contextual Autotune type selector set to Contextual ## Configure Optimization If optimizing for * Users clicking * Checkouts * Actions Choose event occurring /outcome. If optimizing for a continuous output like * Revenue * Latency Choose Event Value, and set the directionality. You will also choose the field from your log or metric source (warehouse native) that you want to use for the value. For warehouse native customers, specify the metric source and optional filters for your target event. Optimization settings screen choosing event outcome and metric source We highly recommend wrapping contextual autotunes in an experiment, but it is not required. You can set this up before your contextual autotune, and after. This experiment will wrap autotune calls in code, and can be used to measure the topline impact of using this contextual autotune in your project. ## Training Settings The other settings can use defaults, but you may want to tune them as well: * Exploration window: how long to serve traffic randomly to bootstrap the exploration of the bandit * Attribution window: how long, after a user sees your variant, to count outcome events for. If set to 1 hour, a user has 1 hour to click after seeing our experience in the example above * Exploration rate: controls how much the bandit favors explore vs. exploit. 0 would not use confidence intervals at all and would just use the best prediction. 1 would use close to the 99.9% CI instead of the 95% CI for exploration * Long-term exploration allocation %: how much traffic will always get randomly assigned? Use higher values if you plan to run this contextual autotune for a long time to help avoid drift * Feature list: provide a list of features Statsig should use to train the model. This is just a mask/filter, and if not set Statsig will read every custom attribute. The main use case is fetching this via CAPI to understand which features a given contextual autotune requires for evaluation when they're fetched on-demand (e.g. introducing latency) Training settings configuration showing exploration and attribution options Set up your variants. These are configurations that you will fetch in code. For example, the group below would send this configuration to your codebase and the "red" value could be passed to the color setting on a button. Variants list showing configuration keys and values for contextual autotune Variant detail editor specifying contextual parameters like button color # Use the Contextual Autotune in Code (python example) We assume you have your server secret key for the following code. Before running python, you'll need the SDK: `pip install statsig-python-core` First, import and initialize Statsig: ``` from statsig_python_core import Statsig, StatsigUser key = autotune_name = statsig = Statsig(key) statsig.initialize().wait() ``` Then, create a user object and fetch your config: ``` user = StatsigUser('user_id', custom={'key1': 'value1', 'key2': 'value2'}) cfg = statsig.get_experiment(user, autotune_name) ``` Now you have your cfg and can apply it! ``` color = cfg.get_string("color", "default color") print(f"Going to use {color} for my color now") ``` You should be able to: * see that you fetched a value from one of your variants * go to the diagnostics page of your autotune, and see a log of the userID along with the corresponding variant That's it! Your code is now serving personalized variants to your users. ## Notes Statsig requires a few hundred units to train a model, and will also not start training until those units' attribution window has elapsed. If you want to test the functionality, we highly recommend "faking a test" to confirm things work like you expect - use logic like ``` fetch_autotune_value() if(user country == 'us'): log_click() ``` to conditionally send events and make sure the model picks up on the conditional behavior # Contextual Bandit (Autotune AI) Source: https://docs.statsig.com/autotune/contextual/introduction Introduction to Statsig Contextual Bandits, which choose the best variant per user based on context features and continuous learning from outcomes. Contextual Multi-Armed Bandits are a subset of Multi-Armed-Bandits which use context about a user to personalize their experience. This is generally achieved by predicting outcomes from among the variants, and picking the best outcome while factoring for uncertainty. Specifically, they will tend to prefer a slightly worse prediction that has a lot of uncertainty, thereby exploring that variant. ## Use Cases Contextual bandits bridge the gap between un-personalized solutions and fully fledged ranking solutions. The main limitation is that contextual bandits: * Have a fixed output set of variants they can show * Have limited ability to account for complex context on the "object" being seen/predict for novel content (e.g. video ranking) At the same time, their simplicity comes with advantages. Statsig's Autotune AI evaluates in near-real-time on both the server and client, taking a few milliseconds or less to return the ideal experience for a given user context. They're also simple to set up and test; you can set up a test in less than an hour, start getting model results the next hour, and start to see experiment results the next in the Statsig console. See our [blog](https://www.statsig.com/blog/statsig-autotune-contextual-bandits-personalization) for more discussion on use cases and motivations. ## Methodology Statsig's autotune AI uses a LinUCB based approach. We think this paper is a good introduction to the topic: [Li, Chu, Langford, Schapire](https://arxiv.org/pdf/1003.0146). For coverage of regret analysis, we think these lecture notes from [Jain from the University of Washington](https://courses.cs.washington.edu/courses/cse599i/18wi/resources/lecture10/lecture10.pdf) are a useful resource. Autotune AI works with categorical and numerical features. Whatever key-value pairs attached to the custom object on the Statsig user will be converted into categorical/numerical features based on their data type. Categorical features will be one-hot-encoded. You should not need to build complex training pipelines, though many customers will pass pre-evaluated user attributes or predictions as context objects. There is support for specifying features up front in Statsig's console. This can be helpful for you to know what features you need to fetch for the bandit in cases that there's an expensive/live lookup. For warehouse native customers, there is planned work around allowing you to join entity properties during the analysis phase so that you can plug in your own feature store to autotune AI analysis, similar to our approach with [CURE](/experiments-plus/cure). Algorithmically, we will choose the best model (e.g. Ridge, Logistic regressions) under the hood based on your data types and performance, and generate a model from your data. The estimated standard error of the model is used to generate a prediction confidence interval. During evaluation, user context is used to predict an outcome for each Variant, and the corresponding confidence interval is applied to that prediction. The best variant is chosen as the one with the highest upper end of a 95% confidence interval. This interval size can be tuned by modifying the exploration parameter in the Autotune setup page. For deeper discussion, refer to the [Methodology](/autotune/contextual/methodology) page. You can also fetch a ranked list from Statsig and then manually expose those you show to the user, for use cases where you have client-side filtering or want to show multiple options; refer to [Advanced Usage](/autotune/using-bandits). ## Drawbacks Because Statsig manages the models, we cannot guarantee perfect tuning of models, or provide more advanced models like Neural Networks. This means that if recommendations are a critical business problem for your team, this feature can be a stepping stone but is not an appropriate end-state solution. In terms of approach, we believe our approach offers a good balance of simplicity, speed, and regret; but if you have specific use cases in mind (e.g. real-time updates) they may not be fully served by the current approach. Lastly, given our models generally assume linearity, we may not be able to capture complex user interactions; this is best for broad-strokes effects, though feature interactions terms can generate decent power in cases of "if-and-then" relationships between predictors and outcomes. ## Outcome Types Autotune AI has a few different model types under the hood - this means it can be used for both classification use cases (will this user click a button) and for continuous outcomes (how much time will the user spend reading articles). This means that you can optimize for both "outcomes" and "metrics"; additionally, you can toggle off higher is better for metrics to try to drive down an outcome like latency. For classification cases, Autotune AI will identify if any outcome occurs within its attribution window. For continuous cases, Autotune AI requires an event name and field name, and will use the numerical value associated with that field. ## Training Training pipelines are run hourly. For Warehouse Native customers, data is processed in your warehouse, and an anonymized feature set is used to train the models. Exposures will be exported on-demand for each load up to the first million, and in daily batches thereafter. Log events sent to statsig are exported hourly if you are using statsig to log outcomes, or you can plug into metric sources from your warehouse for outcome tracking. For cloud customers, the data is processed and trained entirely in Statsig's servers. ## SDK Support Statsig supports contextual autotune in all Client SDKs, but only in the following server SDKs: * [Node](/server-core/node-core) * [Python](/server-core/python-core) * [Java](/server-core/java-core) * [Elixir](/server-core/elixir-core) * [Rust](/server-core/rust-core) * [Go](/server/golangSDK) v1.39.0+ * [Ruby](/server/rubySDK) v2.4.0+ # Contextual Bandit Methodology Source: https://docs.statsig.com/autotune/contextual/methodology Methodology behind Statsig Contextual Bandits, including the contextual algorithm, exploration strategy, model retraining, and reward attribution. ## Methodology This page covers the high level approach that Statsig takes to running contextual bandits across cloud and warehouse native. Specifics of implementation change frequently as we experiment and optimize our approaches, so this documentation is deliberately high level. ## Core Approach What is implemented in Statsig is close to the disjoint model methodology from [Li, Chu, Langford, Schapire](https://arxiv.org/pdf/1003.0146). In short, several models are trained - one per variant - and their estimate CIs modelled. When the contextual autotune is triggered, the latest version of the model is used to estimate the user's outcome, plus the upper end of the 95% CI for the estimate. methodology workflow In the case of categorical outcomes, this is modelled as a logistic regression with l2 penalty. In the case of a continuous outcome space, this is modelled as a multivariate ridge regression. ## Training Data and Sampling To keep data relevant, contextual autotune data is preferentially upsampled for recent dates. Specifically there are two mechanisms for sampling: * A flat number of samples, preferring most recent records, are selected * Per day, in the last two weeks, samples are chosen to prefer more recent records * Samples from the explore dataset are strictly preferred, but non-explore data may be chosen after to satisfy sample requirements. Records are then prioritized by a unit-ID hash to maintain stability in the training set between subsequent runs and avoid major jitter * A sample set is chosen per-variant using this methodology to avoid bias from a model dominating the space and therefore being overrepresented in training data This means that if a model has very low volume, it will have low representation in the training data; this causes higher CIs and makes the sample more likely to be selected due to the upper bound being increased by merit of its rarity and acts as a mechanism for bounce-back. ## Model and Feature Updates Models are updated hourly. If a model definition (features or target outcome change), the data will be fully reset, functionally resetting and retraining the model to match the new definition on the next hourly update. The pre-training data pipeline is available in the history view on the results page, showing the SQL used and caching tables where data is stored. This allows users to easily use the same data to validate or explore modeling approaches. ## Feature Encoding Features with numerical-only values are treated as continuous RVs. All others are string-encoded and one-hot-encoded into binary variables for regression, using the top 25 levels available in the data which have >1% coverage in the dataset. This does mean that using an array of categories or tags does not work, or will encode the most common tagsets. The tagset needs to be provided as individual key-value pairs in the user custom object. ## Model Monitoring Currently, there are not diagnostics for model characteristics over time. The model coefficients are visible in the results tab, and a view comparing performance between naive, random traffic and the targeted traffic is available for understanding if the model's performance relative to blind allocation improves or degrades over time. # Monitoring your Contextual MAB Source: https://docs.statsig.com/autotune/contextual/monitoring Monitor the health and performance of Statsig Contextual Bandits, including traffic allocation, reward signals, model drift, and exploration coverage. There are three primary ways we recommend you monitor autotune performance. # Linked Experiments The best way to evaluate if a bandit is working is seeing if it drives more of the targeted behavior via baseline experience. You can easily set up and link an a/b test in Statsig to evaluate this, and this will also let you monitor other user behaviors and guardrail metrics. This is the gold standard of measurement and is highly encouraged. Standard practice is to wrap the autotune in a experiment with a binary parameter, either as 50/50 or a 90/10 holdback. You can link the experiment to the autotune to get the results on the autotune page. In code, this might look like: ``` experiment_value = statsig.get_experiment('wrapping_experiment').get('flag') default_param = '..." if(experiment_value): param = statsig.get_experiment('autotune').get('param_name') else: param = default_param # use param in code ``` You would start this experiment at the same time that you launch your autotune. # Success Rate Statsig will track the cumulative and daily success rate of your variants over time. This can be tricky to interpret. It may be the case that variant A has lower CTR, but the users being served variant A *would have had* even lower CTR on other variants. We generally recommend using this for tracking and for understanding, and seeing if there are good or bad outliers in your variants. # Traffic Allocation Traffic allocation shows you where Statsig is sending users who see your Autotune. This is a helpful debug view and can help you identify if a variant is dominating traffic or is not receiving any traffic. # Model Features Statsig tracks and surfaces coefficients and feature importance; this can be very useful for understanding which features might be worth further study, or which populations may have unmet needs in your product. * Importance is an estimate of the influence of a feature on the outcome; in layman's terms, this is "how important" the feature is to the prediction * A positive coefficient means that feature leads to an outcome being more likely (or for continuous outcome spaces is associated with a higher outcome). A negative coefficient means the outcome is less likely, or is associated with a lower continuous outcome. # Autotune Source: https://docs.statsig.com/autotune/monitoring Monitor Statsig Autotune experiments to track variant performance, reward signals, exploration rate, and traffic allocation over time. ## How to monitor your Autotune Test ### Computations and Traffic Allocation Updating The results tab within Autotune provides a view of your ongoing and completed Autotune tests. Autotune is computed hourly with metrics and traffic allocation updated throughout the day. Results from exposures are ingested once the attribution window is complete. For example, an exposure at 1pm with a 6 hr attribution window is not counted until 7 pm. ### Result Autotune test results summary This section shows you a summary of your Autotune test. The top bars show a 95% Bayesian Credible interval for the estimated conversion rate (exposure -> success event). There is a 95% chance that the real value is within this interval. The table shows the number of exposures, success events, and overall success rate for each variation across the duration of the test. We also provide a plain English text to describe the current state of the test. ### Details Autotune test detailed analytics charts There are several charts provided: 1. Probability of Best - shows the progress of the Autotune test, and which variant is currently winning. 2. Cumulative Success Rate - Shows the overall success rate (exposure -> success) to date. 3. Daily Success Rate - Shows the success rate for each variation per day. 4. Traffic Allocation - Shows the daily number of exposures allocated to each variation on a given day. ### Linked Experiments The best way to evaluate if a bandit is working is seeing if it drives more of the targeted behavior via baseline experience. You can easily set up and link an a/b test in Statsig to evaluate this, and this will also let you monitor other user behaviors and guardrail metrics. This is the gold standard of measurement and is highly encouraged. Standard practice is to wrap the autotune in a experiment with a binary parameter, either as 50/50 or a 90/10 holdback. You can link the experiment to the autotune to get the results on the autotune page. In code, this might look like: ``` experiment_value = statsig.get_experiment('wrapping_experiment').get('flag') default_param = '..." if(experiment_value): param = statsig.get_experiment('autotune').get('param_name') else: param = default_param # use param in code ``` You would start this experiment at the same time that you launch your autotune. # Methodology Source: https://docs.statsig.com/autotune/multi-armed-bandit How multi-armed bandits work in Statsig Autotune to automatically allocate traffic to the best-performing variant based on a single goal metric. ## Model The base Autotune implementation uses a Thompson Sampling (Bayesian) algorithm to estimate each variant's probability of being the best variant and allocate a proportional amount of traffic. For example, if a given variant has a 60% probability of being the best, Autotune will provide it 60% of the traffic. At a high level, the multi-armed bandit algorithm works by adding more users to a treatment as soon as it recognizes that it is clearly better in maximizing the reward (the target metric). Throughout the process, higher performing treatments are allocated more traffic whereas underperforming treatments are allocated less traffic. When the winning treatment beats the second best treatment by a specified margin, the process terminates. Some helpful references: * [Statsig Blog](https://www.statsig.com/blog/introducing-autotune) * [Goyal and Agrawal (Microsoft Research)](https://proceedings.mlr.press/v23/agrawal12/agrawal12.pdf) Regret Analysis * [Doordash Engineering](https://doordash.engineering/2022/03/15/using-a-multi-armed-bandit-with-thompson-sampling-to-identify-responsive-dashers/) Summary Blog ## Advantages The major advantage of the base Multi-Armed Bandit over a contextual bandit is its ability to converge and identify a best variant. In cases where there is a "one-size-fits-all" solution, this is an elegant way to efficiently allocate traffic and determine a correct long-term solution while minimizing "regret" - the outcome of a high number of users being exposed to the worse variant like in an a/b test. ## Disadvantages The major disadvantages of a Multi-Armed Bandit as compared to a contextual bandit is its inability to personalize; in cases where there's interactions between a user's attributes and variants, vanilla Autotune can identify a global maxima that is worse than serving users their individual best variant. You can see this in a toy example; even if the "US Flag" variant had the highest overall CTR, it would be a bad choice for CA users. In this example, both groups will converge to a sub-optimal variant. | | A/B/n Test | Multi-Armed Bandit (Autotune) | Contextual Bandit (Autotune AI) | Ranking Engine | | -------------------------- | ---------- | ----------------------------- | -------------------------------- | ---------------------------------------- | | Typical # Variants | 2-3 | 4-8 | 4-8 | Arbitrary # | | Personalization Factor | None | None | Moderate | High | | Input Data Required | None | Very Little (100+ samples) | Little - generally 1000+ samples | Tens of thousands to millions of samples | | Model Efficacy | None | Basic | Moderate | High | | Identifies Best Variant | Yes | Yes | No | No | | Consistent User Assignment | Yes | No | No | No | # Autotune (Bandits) Source: https://docs.statsig.com/autotune/overview Statsig Autotune and Autotune AI use multi-armed bandits to weigh explore versus exploit and ship the best-performing variant for a single goal metric. Autotune and Autotune AI are Statsig's Multi-Armed Bandit solutions that automatically find the best variant among a group of candidates, while dynamically allocating traffic to optimize for a single target metric. Autotune, the Multi-Armed Bandit solution, allocates traffic towards high-performing variants and can eventually identify a winning variant. ## How Autotune works Autotune is Statsig's [Bayesian Multi-Armed Bandit](./multi-armed-bandit). It tests and measures different variations and their effect on a target outcome. The multi-armed bandit continuously adjusts traffic towards the best performing variations until it can confidently pick the best variation. The winning variation will then receive 100% of traffic. Bandits seek to balance the "explore"/"exploit" problem - balancing between "exploiting" the current best known solution versus "exploring" to get more information about other solutions. Our blog posts on [Multi-Armed Bandits](/autotune) and [Contextual Bandits](https://www.statsig.com/blog/statsig-autotune-contextual-bandits-personalization) go into depth on use cases and considerations. | | A/B/n Test | Multi-Armed Bandit (Autotune) | Contextual Bandit (Autotune AI) | Ranking Engine | | -------------------------- | ---------- | ----------------------------- | -------------------------------- | ---------------------------------------- | | Typical # Variants | 2-3 | 4-8 | 4-8 | Arbitrary # | | Personalization Factor | None | None | Moderate | High | | Input Data Required | None | Very Little (100+ samples) | Little - generally 1000+ samples | Tens of thousands to millions of samples | | Model Efficacy | None | Basic | Moderate | High | | Identifies Best Variant | Yes | Yes | No | No | | Consistent User Assignment | Yes | No | No | No | ## Implementing Autotune Implementing an Autotune is as simple as checking an experiment in Statsig. After initialization, or on server SDKs, this comes with sub-millisecond latency. Autotune will have a JSON config associated with each variant, which will be returned by the SDK and can be used to modify elements of your webpage (e.g. an image URL or button color), or simply identify the variant so that you know which code to use. ## When to use Autotune Autotune has two major differences from A/B testing (Statsig Experiments): 1. The traffic split isn't fixed over the duration of the test. This allows Autotune to divert more traffic to the winner, fewer from the losers while making fewer mistakes. However, this means the user experience may not be consistent upon repeated visits. 2. Autotune can only optimize for a single metric. Autotune can't accurately measure a collection of metrics, and isn't a great way to understand secondary effects of your changes. Because of this, it works best when the metric is well-understood, has a direct and immediate relationship to the change being tested Because of these differences, Statsig recommends Autotune in the following scenarios: 1. The cost of exposing users to a losing treatment is high. For example, sending new users to a landing page that is inferior may result in lost revenue or churn. While this may be a one-time loss, testing two user registration flows may result in users that never sign up. In this case, Autotune avoids permanently losing users since it can quickly adapt to feedback unlike a static A/B test. 2. You want the decision to be automated. Because Autotune automatically selects the winner, it requires no human decision-making. This is great for launching dozens of simultaneous tests, or for running a long-term unmonitored test. 3. When it's okay for users to be exposed to different experiences upon return visits. For example, changing text or recommendation algorithms. 4. When you have one simple metric to optimize for (eg. click-through rate) that has is an immediate effect of the test. 5. When you want to test multiple variations. Autotune can quickly rules out really poor performers while focusing traffic on the best variants. Autotune should be avoided in the following scenarios: 1. When you have a complex ecosystem and want to understand secondary effects, tradeoffs between variants, and user behavior. 2. When you are optimizing for complex metrics or delayed effects. For these cases, we recommend A/B testing with [Experiments\*](/experiments-plus). In general, it is also a best practice to run Autotune within an experiment with a small group of users that doesn't get the Autotune to measure the impact of the Autotune. # Get Started With Autotune Source: https://docs.statsig.com/autotune/setup Set up a Statsig Autotune experiment, including selecting a goal metric, defining variants, configuring traffic allocation, and launching to users. ## How to set up Autotune 1. To create a new Autotune experiment, navigate to the [Autotune section on the Statsig console](https://console.statsig.com/autotune). 2. Click the Create button and enter the name and description of the Autotune experiment that you want to create. 3. Provide the variants that you want to test in the Autotune experiment. Each variant needs a name, and a corresponding JSON value. The variant that's listed as Control/Default will be returned when the Autotune experiment is not running. Autotune experiment variant configuration interface 4. Select the success event to optimize for as shown below. You can further specify an optional [event value](/guides/logging-events). Autotune success event selection interface There are a few parameters you can specify: * Exploration Window - The initial time period where Autotune will equally split the traffic. This is useful for noisy or temporal metrics where hourly swings in data can bias Autotune's initial measurements. * Attribution Window - The maximum duration between the exposure and success event that counts as a success. We recommend 1 hr for most applications, but adjust accordingly if you expect the success event to lag the exposure event by several hours. * Winner Threshold - The "probability of best" threshold a variant needs to achieve for Autotune to declare it the winner, stop collecting data, and direct all traffic. Setting a lower number will result in faster decisions but increases the probability of making suboptimal decisions (picking the wrong winner). Click "Create" to finalize the setup. 6. Similar to Feature Gates and Experiments, you can find a code snippet for the exposure check event to add to your code. Don't forget to click "Start" when you're ready to launch your Autotune test. Autotune code snippet and launch interface # Advanced Bandit Usage Source: https://docs.statsig.com/autotune/using-bandits When and how to use Statsig multi-armed bandits to optimize for a single goal metric, including ideal use cases, limitations, and result interpretation. Both contextual and non-contextual bandits are managed on Statsig's console, or through Statsig's [console API](../console-api/autotunes.mdx) for programmatic creation. Both bandit types use a common, streamlined API, making it easy to explore either use case without significant changes from using experiments. Statsig's bandits are designed to provide the power of complex bandits with a simple interface - an observation we made in development is that many of these systems are stymied by implementation issues. There's no additional steps beyond a regular experiment check to use a bandit, though for contextual multi-armed bandits you will want to make sure that you attach any context to your user object (usually at initialization) before making the experiment check. This can easily be confirmed in the diagnostics tab of your bandit in console. ### Checking Bandit Check bandits using your standard experiment call. For example, in react this is as simple as configuring a bandit's variant json like: ``` { "text": "", ... } ``` and accessing it in code using the same pattern as [experiments](../guides/abn-tests.mdx): ``` const banditText = useExperiment('contextual_bandit').config.get('text'); ``` This is separated from a bandit's linked experiment call to reduce excessive logging; if you have a linked wrapping experiment, you can just wrap the call after checking the experiment. Statsig will automatically detect and differentiate categorical and numerical features based on cardinality. ### Getting a List It's extremely common that users want to get a ranked list of the variants for a bandit; there may be client-side filtering or other reasons to not show one variant. Getting a list of scored outcomes is supported in Statsig, but requires using manual exposure logging in order to let Statsig know which variant you displayed to the user. ### Outcomes In Statsig Cloud, you will log an outcome event (as a flag, or with a value) to Statsig and Statsig handles connecting exposures to outcome events within your configured time periods. In Statsig Warehouse Native, you can choose which approach to take. Logging events with Statsig will export events to your warehouse in near-real-time for use in Bandits, or you can provide your own outcome dataset as a metric source. One note is that it is worth being mindful of data latency; if your output dataset has >24h lag, bandit exploration can be slowed down significantly. # Android Client SDK Source: https://docs.statsig.com/client/Android Install and use the Statsig Android SDK in Java and Kotlin apps to run experiments, evaluate feature flags, and log analytics events on Android devices. Source code: statsig-io/android-sdk ## Setup the SDK v4.37.1 and higher are published to only [Maven Central](https://central.sonatype.com/artifact/com.statsig/android-sdk). To install the SDK, set the Maven Central repository in your build.gradle. ```java theme={null} dependencies { implementation "com.statsig:android-sdk:4.37.1" } ``` Legacy versions (\<=V4.37.0) can be installed with [Jitpack](https://jitpack.io/#statsig-io/android-sdk). Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. ```java MainActivity.java theme={null} import com.statsig.androidsdk.*; ... public class MainActivity extends AppCompatActivity implements IStatsigCallback { ... StatsigOptions options = new StatsigOptions(); options.setTier(Tier.PRODUCTION); StatsigUser user = new StatsigUser("UUID"); Statsig.initializeAsync(app, "client-key", user, this, options); ... // SDK is usable, but values will be from the cache or defaults (false for gates, {} for configs) // Once onStatsigInitialize fires, then @Override public void onStatsigInitialize() { // SDK is initialized and has the most up to date values } @Override public void onStatsigUpdateUser() { // User has been updated and values have been refetched for the new user } } ``` ```kotlin MainActivity.kt theme={null} import com.statsig.androidsdk.* ... async { Statsig.initialize( this.application, "my_client_sdk_key", StatsigUser("user_id"), ) }.await() ``` ## Use the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```java Java theme={null} DynamicConfig config = Statsig.getConfig("awesome_product_details"); // The 2nd parameter is the default value to be used in case the given parameter name does not exist on // the Dynamic Config object. This can happen when there is a typo, or when the user is offline and the // value has not been cached on the client. String itemName = config.getString("product_name", "Awesome Product v1"); Double price = config.getDouble("price", 10.0); Boolean shouldDiscount = config.getBoolean("discount", false); ``` ```kotlin Kotlin theme={null} val config = Statsig.getConfig("awesome_product_details") // The 2nd parameter is the default value to be used in case the given parameter name does not exist on // the Dynamic Config object. This can happen when there is a typo, or when the user is offline and the // value has not been cached on the client. val itemName = config.getString("product_name", "Awesome Product v1") val price = config.getDouble("price", 10.0) val shouldDiscount = config.getBoolean("discount", false) ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```java Java theme={null} if (Statsig.checkGate("new_homepage_design")) { // Gate is on, show new home page } else { // Gate is off, show old home page } ``` ```kotlin Kotlin theme={null} if (Statsig.checkGate("new_homepage_design")) { // Gate is on, show new home page } else { // Gate is off, show old home page } ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```java Java theme={null} // Values via getLayer Layer layer = Statsig.getLayer("user_promo_experiments") String promoTitle = layer.getString("title", "Welcome to Statsig!"); Double discount = layer.getDouble("discount", 0.1); // or, via getExperiment DynamicConfig titleExperiment = Statsig.getExperiment("new_user_promo_title"); DynamicConfig priceExperiment = Statsig.getExperiment("new_user_promo_price"); String promoTitle = titleExperiment.getString("title", "Welcome to Statsig!"); Double discount = priceExperiment.getDouble("discount", 0.1); ... Double price = msrp * (1 - discount); ``` ```kotlin Kotlin theme={null} // Values via getLayer val layer = Statsig.getLayer("user_promo_experiments") val promoTitle = layer.getString("title", "Welcome to Statsig!") val discount = layer.getDouble("discount", 0.1) // or, via getExperiment val titleExperiment = Statsig.getExperiment("new_user_promo_title") val priceExperiment = Statsig.getExperiment("new_user_promo_price") val promoTitle = titleExperiment.getString("title", "Welcome to Statsig!") val discount = priceExperiment.getDouble("discount", 0.1) ... val price = msrp * (1 - discount); ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```java Java theme={null} Statsig.logEvent("purchase", 2.99, Map.of("item_name", "remove_ads")); ``` ```kotlin Kotlin theme={null} Statsig.logEvent("purchase", 2.99, Map.of("item_name" to "remove_ads")) ``` ## Parameter Stores Parameter Stores hold a set of parameters for your mobile app. These parameters can be remapped on-the-fly from a static value to a Statsig entity (Feature Gates, Experiments, and Layers), so you can decouple your code from the configuration in Statsig. Read more about Param Stores [here](/client/concepts/parameter-stores). ### Getting a Parameter Store To fetch a set of parameters, use the following api: ```java Java theme={null} ParameterStore homepageStore = Statsig.getParameterStore("homepage"); ``` ```kotlin Kotlin theme={null} val homepageStore = Statsig.getParameterStore("homepage") ``` ### Getting a parameter You can then access parameters like this: ```java Java theme={null} String title = homepageStore.getString( "title", //parameter name "Welcome" // default value ); boolean shouldShowUpsell = homePageStore.getBoolean("upsell_upgrade_now", false); ``` ```kotlin Kotlin theme={null} val title = homepageStore.getString( "title", // parameter name "Welcome", // default value ) val shouldShowUpsell = homepageStore.getBoolean("upsell_upgrade_now", false) ``` ## Statsig User You need to provide a StatsigUser object to check/get your configurations. You should pass as much information as possible in order to take advantage of advanced gate and config conditions. Most of the time, the `userID` field is needed in order to provide a consistent experience for a given user (see [logged-out experiments](/guides/first-device-level-experiment) to understand how to correctly run experiments for logged-out users). Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Once the user logs in or has an update/changed, make sure to call `updateUser` with the updated `userID` and/or any other updated user attributes: ```java Java theme={null} StatsigUser newUser = new StatsigUser("new_user_id"); Statsig.updateUserAsync(newUser, this); // this must implement IStatsigCallback ... @Override public void onStatsigUpdateUser() { // User has been updated and values have been refetched for the new user } ``` ```kotlin Kotlin theme={null} Statsig.updateUser(StatsigUser("new_user_id")) ``` ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` and `user` during initialization to customize the Statsig client. Default endpoint for all SDK network requests. Do not override unless you implement the Statsig API elsewhere. Include the current top-level activity on logged events by default. Set to `true` to disable. Deprecated. Previously prevented the SDK from sending diagnostic information. Milliseconds to wait for the initial request before completing. Set to `0` to wait indefinitely. Periodically fetch updated values for the current user when enabled. Frequency (in minutes) for auto value refresh. Minimum is `1` minute. Override the SDK-generated `stableID` for the user. Whether the SDK should block on loading saved values from disk. Provide the initialize response directly to bootstrap the client synchronously. See NodeJS Server SDK for generating values and the Bootstrap docs. When `true`, gate/config/experiment names are not hashed and remain readable. Requires special authorization from Statsig. Override how the cache key is generated for stored values when the default does not fit your needs. Callback invoked whenever a gate, config, experiment, or layer is checked. Receives the evaluated `BaseConfig`. Callbacks that may trigger multiple times over the lifetime of the client SDK. They will be called on the main thread. * `onValuesUpdated()` - called whenever the fired when new values are received and available for use. May be called after `Statsig.updateUser()`, `Statsig.updateUserAsync()`, or auto value updates (see `enableAutoValueUpdate` above) #### Methods * **setTier | setEnvironmentParameter | getEnvironment** * used to signal the environment tier the user is currently in. * `setTier` can be PRODUCTION, STAGING or DEVELOPMENT. e.g. passing in a value of `Tier.STAGING` will allow your users to pass any condition that pass for the staging environment tier, and fail any condition that only passes for other environment tiers. * `setEnvironmentParameter` can be used for custom tiers, eg `options.setEnvironmentParameter("tier", "test")` #### Runtime Options Starting in `V4.43.0`, a subset of options can be set during initialization and later updated while the Statsig client is running. These options are defined in `StatsigRuntimeMutableOptions` (which `StatsigOptions` extends) and are detailed below. Call `Statsig.updateRuntimeOptions(runtimeMutableOptions: StatsigRuntimeMutableOptions)` or the corresponding method in `StatsigClient` to update the Statsig client with new values. * **loggingEnabled**: `Boolean`, default `true` * Setting this value to `false` will prevent the Statsig client from sending logging events over the network or saving events to its on-disk cache. The 1000 most recent events will be queued in memory. They can be logged to network (or cached) if `loggingEnabled` is set to `true` later during that session. * Calling `Statsig.flush()` after setting `loggingEnabled` to `true` will immediately clear the queue and minimize loss of older log events * This can be useful for cases where it is necessary for users to grant permission before events should be logged, or in any other cases where logging should not be enabled ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```java Java theme={null} Statsig.shutdown(); ``` ```kotlin Kotlin theme={null} Statsig.shutdown() ``` ## Using Persistent Evaluations If you want to ensure that a user's variant stays consistent while an experiment is running, regardless of changes to allocation or targeting, you can use persistent storage. The Android SDK supports a minimal implementation using the keepDeviceValues flag, see the [Client Persistent Assignment Doc](/client/concepts/persistent_assignment#support-in-ios-and-android-sdks) for more info. ## Local Overrides If you want to locally override gates/configs/experiments/layers for testing, Statsig offers convenient methods for a quick local override. Unless you call the remove method, these will be persisted session-to-session on the client's device. Note that these overrides only apply locally - they don't impact definitions in the console or elsewhere. ```kotlin theme={null} // Overrides the given gate to the specified value overrideGate(gateName: String, value: Boolean) // Overrides the given config (dynamic config or experiment) to the provided value overrideConfig(configName: String, value: Map) // Removes any overrides associated with the provided gate/config/experiment name removeOverride(name: String) // Removes all overrides removeAllOverrides() // Returns the set of gate and config overrides currently in place on the client getAllOverrides(): StatsigOverrides class StatsigOverrides( @SerializedName("gates") val gates: MutableMap, @SerializedName("configs") val configs: MutableMap> ) {} ``` ## Manual Exposures Manual logging is error-prone and can often introduce issues like uneven exposures, which compromise experiment results. You can query your gates/experiments without triggering an exposure, and manually log the exposures later: To check a gate without an exposure being logged: ```kotlin Kotlin theme={null} val result = Statsig.checkGateWithExposureLoggingDisabled("a_gate_name") ``` ```java Java theme={null} boolean result = Statsig.checkGateWithExposureLoggingDisabled("a_gate_name"); ``` Later, to manually log the gate exposure: ```kotlin Kotlin theme={null} Statsig.manuallyLogGateExposure("a_gate_name") ``` ```java Java theme={null} Statsig.manuallyLogGateExposure("a_gate_name"); ``` To get a dynamic config without an exposure being logged: ```kotlin Kotlin theme={null} val config = Statsig.getConfigWithExposureLoggingDisabled("a_config_name") ``` ```java Java theme={null} DynamicConfig config = Statsig.getConfigWithExposureLoggingDisabled("a_config_name"); ``` Later, to manually log the config exposure: ```kotlin Kotlin theme={null} Statsig.manuallyLogConfigExposure("a_config_name") ``` ```java Java theme={null} Statsig.manuallyLogConfigExposure("a_config_name"); ``` To get an experiment without an exposure being logged: ```kotlin Kotlin theme={null} val experiment = Statsig.getExperimentWithExposureLoggingDisabled("an_experiment_name") ``` ```java Java theme={null} DynamicConfig experiment = Statsig.getExperimentWithExposureLoggingDisabled("an_experiment_name"); ``` Later, to manually log the experiment exposure: ```kotlin Kotlin theme={null} Statsig.manuallyLogExperimentExposure("an_experiment_name", false) ``` ```java Java theme={null} Statsig.manuallyLogExperimentExposure("an_experiment_name", false); ``` To get a layer parameter without an exposure being logged: ```kotlin Kotlin theme={null} val layer = Statsig.getLayerWithExposureLoggingDisabled("a_layer_name", false) val result = layer.getString("a_parameter_name", "fallback") ``` ```java Java theme={null} Layer layer = Statsig.getLayerWithExposureLoggingDisabled("a_layer_name"); String result = layer.getString("a_parameter_name", "fallback"); ``` Later, to manually log the layer parameter exposure: ```kotlin Kotlin theme={null} Statsig.manuallyLogLayerParameterExposure("a_layer_name", "a_parameter_name", false) ``` ```java Java theme={null} Statsig.manuallyLogLayerParameterExposure("a_layer_name", "a_parameter_name", false); ``` ## StableID Each client SDK has the notion of stableID, a devive-level identifier that is generated the first time the SDK is initialized and is stored locally for all future sessions. Unless storage is wiped (or app deleted), the stableID will not change. This allows us to run device level experiments and experiments when other user identifiable information is unavailable (Logged out users). ```kotlin theme={null} // Retrieve the StableID Statsig.getStableID(); // Override the StableID before initializing, if you have something you'd prefer to use instead val opts = StatsigOptions(overrideStableID = "my_stable_id") Statsig.initialize(app, "client-xyx", options = opts) ``` ## Using multiple instances of the SDK Up to this point, we've used the SDK's singleton. We also support creating multiples instances of the SDK - the `Statsig` singleton wraps a single instance of the SDK (typically called a `StatsigClient`) that you can instantiate. You must use a different SDK key for each sdk instance you create for this to work. Various functionality of the Statsig client is keyed on the SDK key being used. Using the same key will lead to collisions. All top level static methods from the singleton carry over as instance methods. To create an instance of the Statsig sdk: ```java Java theme={null} StatsigClient client = new StatsigClient(); client.initializeAsync(application, sdkKey, user, callback, options); ``` ```kotlin Kotlin theme={null} var client: StatsigClient = StatsigClient() client.initialize(application, sdkKey, user, options) ``` ## Initialize Response The SDK provides a method to access the raw values that are used internally for gate, config, and layer value. This can be useful for debugging or for advanced use cases where you need to access the underlying data. For example, you can use these values to bootstrap another SDK, like the javascript SDK when you open an in-app browser. The `getInitializeResponseJson` method returns an `ExternalInitializeResponse` object that contains: 1. A JSON string representation of the initialize response values 2. Evaluation details that provide metadata about how the values were obtained (network, cache, etc.) ```java Java theme={null} // Get the raw values that the SDK is using internally to provide gate/config/layer results ExternalInitializeResponse response = Statsig.getInitializeResponseJson(); // Get the JSON string representation of the initialize response String jsonValues = response.getInitializeResponseJSON(); // Get the evaluation details EvaluationDetails details = response.getEvaluationDetails(); ``` ```kotlin Kotlin theme={null} // Get the raw values that the SDK is using internally to provide gate/config/layer results val response = Statsig.getInitializeResponseJson() // Get the JSON string representation of the initialize response val jsonValues = response.getInitializeResponseJSON() // Get the evaluation details val details = response.getEvaluationDetails() ``` # Statsig in Angular Source: https://docs.statsig.com/client/Angular Integrate the Statsig Angular SDK to run experiments, evaluate feature flags, and capture analytics events from Angular single page applications. Source code: statsig-io/js-client-monorepo ## Set Up the SDK ```bash npm theme={null} npm install @statsig/angular-bindings ``` ```bash yarn theme={null} yarn add @statsig/angular-bindings ``` Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. The Statsig Angular bindings package provides a `StatsigService` that can be injected into your components. The way you provide and inject this service can vary depending on how you structure your app. ```ts theme={null} import { STATSIG_INIT_CONFIG } from '@statsig/angular-bindings'; const StatsigConfig = { sdkKey: "client-KEY", user: {}, // initial user object options: {...} // optional } ``` ### using app config ```ts theme={null} // app.config.ts import { ApplicationConfig } from '@angular/core'; import { STATSIG_INIT_CONFIG } from '@statsig/angular-bindings'; export const appConfig: ApplicationConfig = { providers: [ { provide: STATSIG_INIT_CONFIG, useValue: StatsigConfig, }, ], }; //main.ts import { AppComponent } from './app/app.component'; import { appConfig } from './app/app.config'; bootstrapApplication(AppComponent, appConfig).catch((err) => console.error(err), ); ``` ### using app module ```ts theme={null} // app.module.ts import { StatsigService } from '@statsig/angular-bindings'; import { AppComponent } from './app.component'; @NgModule({ declarations: [AppComponent], imports: [], providers: [ { provide: STATSIG_INIT_CONFIG, useValue: StatsigConfig, }, ], bootstrap: [AppComponent], }) export class AppModule {} ``` ## Use the SDK Once you have provided the statsig config token, you can now inject the service into a component or another service and use it. ```ts theme={null} // example.component.ts import { Component } from '@angular/core'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ selector: 'app-example', template: `...`, }) export class ExampleComponent { constructor(private statsigService: StatsigService) {} } ``` ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```ts theme={null} // feature-gate.component.ts import { CommonModule } from '@angular/common'; import { Component, OnInit } from '@angular/core'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ standalone: true, selector: 'app-feature-gate', imports: [CommonModule], template: `
Feature is enabled!
`, }) export class FeatureGateComponent implements OnInit { isFeatureEnabled = false; constructor(private statsigService: StatsigService) {} ngOnInit(): void { this.isFeatureEnabled = this.statsigService.checkGate('feature_gate_name'); } } ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```ts theme={null} // dynamic-config.component.ts import { Component, OnInit } from '@angular/core'; import { CommonModule } from '@angular/common'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ standalone: true, selector: 'app-dynamic-config', imports: [CommonModule], template: `
Config Value: {{ configValue }}
`, }) export class DynamicConfigComponent implements OnInit { configValue: string | null = null; constructor(private statsigService: StatsigService) {} ngOnInit(): void { const dynamicConfig = this.statsigService.getDynamicConfig('config_name'); this.configValue = dynamicConfig.get('key', 'default_value'); } } ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```ts theme={null} // experiment.component.ts import { Component, OnInit } from '@angular/core'; import { CommonModule } from '@angular/common'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ standalone: true, selector: 'app-experiment', imports: [CommonModule], template: `
Experiment Value: {{ experimentValue }}
`, }) export class ExperimentComponent implements OnInit { experimentValue: string | null = null; constructor(private statsigService: StatsigService) {} ngOnInit(): void { const experiment = this.statsigService.getExperiment('experiment_name'); this.experimentValue = experiment.get('experiment_key', 'default'); } } ``` ```ts theme={null} // layer.component.ts import { Component, OnInit } from '@angular/core'; import { CommonModule } from '@angular/common'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ standalone: true, selector: 'app-layer', imports: [CommonModule], template: `
Layer Value: {{ layerValue }}
`, }) export class LayerComponent implements OnInit { layerValue: string | null = null; constructor(private statsigService: StatsigService) {} ngOnInit(): void { const layer = this.statsigService.getLayer('layer_name'); this.layerValue = layer.get('layer_key', 'default_layer_value'); } } ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```ts theme={null} import { Component, OnInit } from '@angular/core'; import { CommonModule } from '@angular/common'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ standalone: true, selector: 'app-log-event', imports: [CommonModule], template: `

{{ message }}

`, }) export class LogEventComponent { message: string | null = null; constructor(private statsigService: StatsigService) {} logUserAction(): void { this.statsigService.logEvent('UserClickedButton', 1, { buttonColor: 'blue', buttonText: 'Click Me', }); this.message = 'Event logged: UserClickedButton'; } } ``` ### Flushing Logged Events `flush()` sends queued events immediately. Use `shutdown()` when your app is exiting. ```ts theme={null} import { Component } from '@angular/core'; import { CommonModule } from '@angular/common'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ standalone: true, selector: 'app-flush-events', imports: [CommonModule], template: `

{{ message }}

`, }) export class FlushEventsComponent { message: string | null = null; constructor(private statsigService: StatsigService) {} async flushEvents(): Promise { await this.statsigService.getClient().flush(); this.message = 'Queued events flushed'; } } ``` ## Updating user properties (e.g., Login) Sometimes you'll need to update user properties, say when the user logs in and a userID is assigned, or a set of new properties have been identified. This would require Statsig to go fetch new values for all the gates, experiments and config evaluations. This is achieved by the calling `updateUserAsync` from the service: ```ts theme={null} import { Component } from '@angular/core'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ selector: 'app-user-update', template: ` `, }) export class UserUpdateComponent { constructor(private statsigService: StatsigService) {} updateUser(): void { const user = { userID: 'user-1234', email: 'user@example.com', // Add other user properties here }; this.statsigService.updateUserAsync(user) .then(() => { console.log('User updated successfully'); }) .catch((error) => { console.error('Error updating user:', error); }); } } ``` ## Loading State Dependent on your setup, you may want to wait for the latest values before checking a gate or experiment. You can use the `isLoading` observable to determine if the SDK is still loading and display a loading state. ```ts theme={null} import { CommonModule } from '@angular/common'; import { Component, OnInit } from '@angular/core'; import { StatsigService } from '@statsig/angular-bindings'; @Component({ selector: 'app-loading-state', imports: [CommonModule], template: `

Loading...

Content loaded successfully!

`, }) export class LoadingStateComponent implements OnInit { isLoading = this.statsigService.isLoading$; constructor(private statsigService: StatsigService) {} ngOnInit(): void { // other initialization logic here } } ``` ## Lifecycle & Advanced Usage ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```ts theme={null} import { StatsigService } from '@statsig/angular-bindings'; export class ShutdownStatsigComponent { constructor(private statsigService: StatsigService) {} ngOnDestroy(): void { void this.statsigService.getClient().shutdown(); } } ``` ## Angular Directives ### Statsig Module To use the directives, you need to import the `StatsigModule` in your Angular module. ```ts theme={null} // app.module.ts import { NgModule } from '@angular/core'; import { BrowserModule } from '@angular/platform-browser'; import { AppComponent } from './app.component'; import { StatsigModule } from '@statsig/angular-bindings'; @NgModule({ declarations: [AppComponent], imports: [BrowserModule, StatsigModule], bootstrap: [AppComponent] }) export class AppModule {} ``` ### Check gate directive The stgCheckGate directive allows you to conditionally display content in your Angular templates based on the evaluation of a Feature Gate from Statsig. It listens for updates to the feature gate's value and dynamically adds or removes content based on whether the gate is enabled or not. ```ts theme={null} // feature-demo.component.ts import { Component } from '@angular/core'; @Component({ selector: 'app-feature-demo', template: `

This content will only show if the 'new_feature_gate' is enabled.

`, }) export class FeatureDemoComponent {} ``` ## Session Replay By including the [`@statsig/session-replay`](https://www.npmjs.com/package/@statsig/session-replay) package in your project, you can automatically capture and log user sessions as videos. This feature is useful for debugging and understanding user behavior. Read more about [Session Replay](/session-replay/overview). ```ts theme={null} import { STATSIG_INIT_CONFIG } from '@statsig/angular-bindings'; import { StatsigSessionReplayPlugin } from '@statsig/session-replay'; const StatsigConfig = { sdkKey: "client-KEY", user: { userID: 'a-user' }, options: { plugins: [ new StatsigSessionReplayPlugin() ] } } ``` ## Web Analytics / Auto Capture By including the [`@statsig/web-analytics`](https://www.npmjs.com/package/@statsig/web-analytics) package in your project, you can automatically capture common web events like clicks and page views. For more information on filtering events, enabling console log capture, and other configuration options available in web analytics, see the [Web Analytics Configuration](/webanalytics/overview#event-filtering-and-console-configuration) documentation. ```ts theme={null} import { STATSIG_INIT_CONFIG } from '@statsig/angular-bindings'; import { StatsigAutoCapturePlugin } from '@statsig/web-analytics'; const StatsigConfig = { sdkKey: "client-KEY", user: { userID: 'a-user' }, options: { plugins: [ new StatsigAutoCapturePlugin() ] } } ``` # C++ Client SDK Source: https://docs.statsig.com/client/CPP Install and use the Statsig C++ client SDK to evaluate feature flags, run experiments, and log analytics events from native desktop and game applications. Source code: statsig-io/cpp-client-sdk ## Set Up the SDK ```cpp theme={null} add_subdirectory(path/to/downloaded/statsig_sdk) target_link_libraries(${PROJECT_NAME} StatsigClientSDK) ``` ```cpp theme={null} include(FetchContent) FetchContent_Declare(statsig GIT_REPOSITORY https://github.com/statsig-io/cpp-client-sdk.git GIT_TAG main ) FetchContent_MakeAvailable(statsig) target_link_libraries(${PROJECT_NAME} StatsigClientSDK) ``` Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. ```cpp theme={null} #include using namespace statsig; StatsigUser user; user.user_id = "a-user"; user.custom_ids = { {"employeeID", "an-employee"} }; // Create your own instance StatsigClient client; // Initialize synchronously using cached values from the previous session client.InitializeSync("client-{YOUR_CLIENT_SDK_KEY}", user); // or, Initialize asynchronously from network client.InitializeAsync( "client-{YOUR_CLIENT_SDK_KEY}", [](StatsigResultCode result) { // completion callback }, user ); ``` **Synchronous** initialization will leverage cache (if available), returning immediately. Data for subsequent sessions will then be fetched in the background. **Asynchronous** initialization, on the other hand provides a callback, allowing you to wait for the most current data to be fetched. For convenience, there is also a singleton instance that can be accessed via `StatsigClient::Shared()`. ```cpp theme={null} // Initialize synchronously using cached values from the previous session StatsigClient::Shared().InitializeSync("client-{YOUR_CLIENT_SDK_KEY}", user); // or, Initialize asynchronously from network StatsigClient::Shared().InitializeAsync( "client-{YOUR_CLIENT_SDK_KEY}", [](StatsigResultCode result) { // completion callback }, user ); ``` **Optional** - Configuration via StatsigOptions It is possible to adjust certain aspects of how the SDK works via a [StatsigOptions](#statsig-options) struct. Just pass in a StatsigOptions struct during initialization. ```cpp theme={null} StatsigOptions options; options.environment = StatsigEnvironment{"staging"}; client.InitializeSync(..., options); // or client.InitializeAsync(..., options); ``` ## Use the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```cpp theme={null} if (client.CheckGate("a_gate")) { // show new feature } // or, use the shared instance if (StatsigClient::Shared().CheckGate("a_gate")) { // show new feature } ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```cpp theme={null} DynamicConfig config = client.GetDynamicConfig("a_config"); // or, use the shared instance DynamicConfig config = StatsigClient::Shared().GetDynamicConfig("a_config"); // then access the params std::cout << config.GetValue()["a_string_param"] << std::endl; ``` DynamicConfig.GetValue returns JsonObj which is an unordered map of string to `nlohmann/json`. See [https://github.com/nlohmann/json](https://github.com/nlohmann/json) ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```cpp theme={null} // Values via getLayer Layer layer = StatsigClient::Shared().GetLayer("name"); std::string promoTitle = layer.GetValue("title").get(); double discount = layer.GetValue("discount").get(); // or, via getExperiment Experiment titleExperiment = StatsigClient::Shared().GetExperiment("new_user_promo_title"); Experiment priceExperiment = StatsigClient::Shared().GetExperiment("new_user_promo_price"); std::string promoTitle = titleExperiment.GetValue()["title"].get(); double discount = priceExperiment.GetValue()["discount"].get(); ``` Layer.GetValue and Experiment.GetValue return JsonObj which are unordered maps of string to `nlohmann/json`. See [https://github.com/nlohmann/json](https://github.com/nlohmann/json) ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```cpp theme={null} std::unordered_map metadata{ { "price", "9.99" }, { "item_name", "some_great_product" } }; StatsigEvent event("add_to_cart", "SKU_12345", metadata); StatsigClient::Shared().LogEvent(event); // Then, at some point later, you need to "flush" the events StatsigClient::Shared().Flush(); ``` ## Statsig User You need to provide a StatsigUser object to check/get your configurations. You should pass as much information as possible in order to take advantage of advanced gate and config conditions. Most of the time, the `userID` field is needed in order to provide a consistent experience for a given user (see [logged-out experiments](/guides/first-device-level-experiment) to understand how to correctly run experiments for logged-out users). Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Once the user logs in or has an update/changed, make sure to call `updateUser` with the updated `userID` and/or any other updated user attributes: ```cpp theme={null} StatsigUser user; user.user_id = "a-user"; user.email = "developer@statsig.com"; user.custom_ids = { {"employeeID", "an-employee"} }; ``` ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ### Updating Users At some point, your user might need to change. To make Statsig aware of this new user, you will need to make a call to an UpdateUser function. ```cpp theme={null} client.UpdateUserSync(user); // or, use the shared instance StatsigClient::Shared().UpdateUserSync(user); ``` If you want to ensure you have the latest values for an update (Say you are transition from logged out to logged in). You can use the Asynchronous update function. ```cpp theme={null} {client or StatsigClient::Shared()}.UpdateUserAsync( user, [](StatsigResultCode result) { if (result == StatsigResultCode::Ok) { // do something now that the latest values have been fetched } else { // error state } } ); ``` Asynchronous vs Synchronous behaviors are the same as the Initialize functions. ## Statsig Options `StatsigClient::Initialize`, in addition to `sdk_key` and `user`, takes an optional parameter `options` that you can provide to customize the StatsigClient. Here are the current options and we are always adding more to the list: The API to use for all SDK network requests. You should not need to override this unless you have another API that implements the Statsig API endpoints. Array of EvaluationsDataProvider, used to customize the initialization and update behavior. ## Lifecycle & Advanced Usage ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```cpp theme={null} client.Shutdown(); // or, use the shared instance StatsigClient::Shared().Shutdown(); ``` #### How do I run experiments for logged out users?​ See the guide on [device level experiments](/guides/first-device-level-experiment) # Dart Client SDK Source: https://docs.statsig.com/client/Dart Use the Statsig Dart and Flutter SDK to evaluate feature gates, run experiments, and capture analytics events from cross-platform mobile and web apps. Source code: statsig-io/dart-sdk ## Set Up the SDK With Dart: ```bash theme={null} dart pub add statsig ``` With Flutter: ```bash theme={null} flutter pub add statsig ``` If you are using **Flutter**, be sure to add Statsig as part of the [app lifecycle](#flutter-lifecycle-hooks) to avoid losing events. Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. ```dart theme={null} import 'package:statsig/statsig.dart'; await Statsig.initialize('client-sdk-key', StatsigUser(userId: "a-user-id")); ``` ## Use the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```dart theme={null} if (Statsig.checkGate("new_homepage_design")) { // Gate is on, show new home page } else { // Gate is off, show old home page } ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```dart theme={null} var config = Statsig.getConfig("awesome_product_details"); // The 2nd parameter is the default value to be used in case the given parameter name does not exist on // the Dynamic Config object. This can happen when there is a typo, or when the user is offline and the // value has not been cached on the client. var itemName = config.get("product_name", "Awesome Product v1"); var price = config.get("price", 10.0); var shouldDiscount = config.get("discount", false); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```dart theme={null} // Values via getLayer var layer = Statsig.getLayer("user_promo_experiments"); var promoTitle = layer.getString("title", "Welcome to Statsig!"); var discount = layer.getDouble("discount", 0.1); // or, via getExperiment var titleExperiment = Statsig.getExperiment("new_user_promo_title"); var priceExperiment = Statsig.getExperiment("new_user_promo_price"); var promoTitle = titleExperiment.get("title", "Welcome to Statsig!"); var discount = priceExperiment.get("discount", 0.1); ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```dart theme={null} // Provide a doubleValue argument for number values Statsig.logEvent("purchase", doubleValue: 2.99, metadata: {"item_name": "remove_ads"}); // or provide a stringValue argument for string values Statsig.logEvent("login", stringValue: "a.user@mail.com"); ``` ## Parameter Stores Parameter Stores hold a set of parameters for your mobile app. These parameters can be remapped on-the-fly from a static value to a Statsig entity (Feature Gates, Experiments, and Layers), so you can decouple your code from the configuration in Statsig. Read more about Param Stores [here](/client/concepts/parameter-stores). ```dart theme={null} var homepageStore = Statsig.getParameterStore("homepage"); var title = homepageStore.get("title", "Welcome"); var shouldShowUpsell = homepageStore.get("upsell_upgrade_now", false); ``` ## Statsig User You need to provide a StatsigUser object to check/get your configurations. You should pass as much information as possible in order to take advantage of advanced gate and config conditions. Most of the time, the `userID` field is needed in order to provide a consistent experience for a given user (see [logged-out experiments](/guides/first-device-level-experiment) to understand how to correctly run experiments for logged-out users). Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Once the user logs in or has an update/changed, make sure to call `updateUser` with the updated `userID` and/or any other updated user attributes: ```dart theme={null} await Statsig.updateUser(StatsigUser("a_new_user")); ``` ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options Used to decide how long (in seconds) the Statsig client waits for the initial network request to respond before calling the completion block. The Statsig client will return either cached values (if any) or default values if checkGate/getConfig/getExperiment is called before the initial network request completes. If you always want to wait for the latest values fetched from Statsig server, you should set this to 0 so we do not timeout the network request. The endpoint to use for all SDK network requests. You should not override this (unless you have another API that implements the Statsig API endpoints). The environment tier to evaluate rules for the current user. Default empty, which is the same as "production." On non-production tiers, events will not make it in to downstream pulse results. ## Lifecycle & Advanced Usage ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```dart theme={null} await Statsig.shutdown(); ``` ## Flutter Lifecycle Hooks Due to the nature of mobile development, apps can be closed by the operating system when they are no longer in the foreground. To be sure that all events are logged before an app is closed by the operating system, we recommend adding Statsig to the app lifecycle events. This way we can flush all pending events when an app state change is detected. Something like the following: ```dart theme={null} // An example App Lifecycle Observer class StatsigLifecycleObserver extends WidgetsBindingObserver { @override void didChangeAppLifecycleState(AppLifecycleState state) async { switch (state) { case AppLifecycleState.resumed: await Statsig.initialize('client-sdk-key'); break; case AppLifecycleState.paused: await Statsig.shutdown(); break; } } } ``` Then in your app code, add this observer to the WidgetsBinding instance. ```dart theme={null} @override void initState() { super.initState(); WidgetsBinding.instance.addObserver(StatsigLifecycleObserver()); } ``` ## FAQs ### How do I run experiments for logged out users? See the guide on [device level experiments](/client/concepts/persistent_assignment). # .NET Client SDK Source: https://docs.statsig.com/client/DotNet Install the Statsig .NET client SDK to evaluate feature flags, run experiments, and log analytics events from C#, WPF, MAUI, and Windows applications. Source code: statsig-io/dotnet-sdk ## Setup the SDK The package is hosted on [Nuget](https://www.nuget.org/packages/Statsig/). You can either install it from your Visual Studio's Nuget package manager, or through the NuGet CLI: ```shell theme={null} nuget install Statsig ``` Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. ```csharp theme={null} using Statsig; using Statsig.Client; await StatsigClient.Initialize( "client-sdk-key", new StatsigUser { UserID = "some_user_id", Email = "user@email.com" }, new StatsigOptions(new StatsigEnvironment(EnvironmentTier.Development)) // optional, use when needed to customize certain behaviors ); ``` ## Use the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```csharp theme={null} if (StatsigClient.CheckGate("new_homepage_design")) { // Gate is on, show new home page } else { // Gate is off, show old home page } ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```csharp theme={null} DynamicConfig config = StatsigClient.GetConfig("awesome_product_details"); // The 2nd parameter is the default value to be used in case the given parameter name does not exist on // the Dynamic Config object. This can happen when there is a typo, or when the user is offline and the // value has not been cached on the client. string itemName = config.Get("product_name", "Awesome Product v1"); double price = config.Get("price", 10.0); bool shouldDiscount = config.Get("discount", false); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```csharp theme={null} // Values via getLayer Layer layer = StatsigClient.GetLayer("user_promo_experiments"); var promoTitle = layer.Get("title", "Welcome to Statsig!"); var discount = layer.Get("discount", 0.1); // or, via getExperiment DynamicConfig titleExperiment = StatsigClient.GetExperiment("new_user_promo_title"); DynamicConfig priceExperiment = StatsigClient.GetExperiment("new_user_promo_price"); var promoTitle = titleExperiment.Get("title", "Welcome to Statsig!"); var discount = priceExperiment.Get("discount", 0.1); ... double price = msrp * (1 - discount); ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```csharp theme={null} StatsigClient.LogEvent( "add_to_cart", "SKU_12345", new Dictionary() { { "price", "9.99" }, { "item_name", "diet_coke_48_pack" } } ); ``` ## Statsig User You need to provide a StatsigUser object to check/get your configurations. You should pass as much information as possible in order to take advantage of advanced gate and config conditions. Most of the time, the `userID` field is needed in order to provide a consistent experience for a given user (see [logged-out experiments](/guides/first-device-level-experiment) to understand how to correctly run experiments for logged-out users). Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Once the user logs in or has an update/changed, make sure to call `updateUser` with the updated `userID` and/or any other updated user attributes: ```csharp theme={null} // if you want to update the existing user, or change to a different user, call updateUser await StatsigClient.UpdateUser( new StatsigUser { UserID = "new_user_id", Email = "new_user@email.com" }, ); ``` ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options `Initialize()` takes an optional parameter `options` in addition to `sdkKey` and `user` that you can provide to customize the Statsig client. Set environment variables that apply to all users in the session for targeting purposes. Commonly used to set the environment tier, e.g. `new StatsigEnvironment(EnvironmentTier.Staging)`. Maximum milliseconds to wait for `/initialize` before proceeding with cached/default values. Directory path for persistent storage of cached values and logs. ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```csharp theme={null} StatsigClient.Shutdown(); ``` ## FAQs #### How do I run experiments for logged out users? See the guide on [device level experiments](/client/concepts/user#device-level-experiments) # Expo Client SDK Source: https://docs.statsig.com/client/Expo Install the Statsig Expo SDK to evaluate feature gates, run A/B experiments, and log analytics events from Expo and React Native managed workflow apps. Source code: statsig-io/js-client-monorepo ## Setup the SDK ```shell theme={null} npx expo install @statsig/expo-bindings ``` ### Peer Dependencies The `@statsig/expo-bindings` package has peer dependencies which may also need to be installed if they are not already in your project. ```shell theme={null} npx expo install expo-device expo-application @react-native-async-storage/async-storage ``` Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. The setup for an Expo environment is very similar to a plain [React environment](/client/React). The only difference is that you need to use the Expo specific `StatsigProviderExpo`. This automatically switches out the storage layer used by the SDK, utilizing [AsyncStorage](https://github.com/react-native-async-storage) instead of LocalStorage (which isn't available in RN environments). ```tsx theme={null} import { StatsigProviderExpo, useFeatureGate } from "@statsig/expo-bindings"; function Content() { const gate = useFeatureGate("a_gate"); return Reason: {gate.details.reason}; // Reason: Network or NetworkNotModified } function App() { return ( Loading...} > ); } ``` ## Use the SDK You can get an instance of the StatsigClient to check gates, experiments, dynamic configs, layers, and log events. ```jsx theme={null} import { useStatsigClient } from "@statsig/expo-bindings"; const { client } = useStatsigClient(); ``` See the methods you can call on the client below. ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. You can evaluate a gate by getting the client with the `useStatsigClient` hook, and then calling `checkGate` ```tsx theme={null} const { client } = useStatsigClient(); return (
Gate is {client.checkGate('check_user') ? 'passing' : 'failing'}.
); ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: You can get a DynamicConfig value by getting the client with the `useStatsigClient` hook, and then calling `getConfig` ```tsx theme={null} const { client } = useStatsigClient(); const config = client.getConfig('app_properties'); return (
{config.get('title', 'Default Title')}
); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. You can access the experiment variant and parameters for the user by getting the client with the `useStatsigClient` hook, and then calling `getExperiment`. ```tsx theme={null} const { client } = useStatsigClient(); const experiment = client.getExperiment('headline_test'); return (
Headline Parameter: {experiment.get('headline', 'Default')}.
); ``` You can access layers and layer parameters for the user by getting the client with the `useStatsigClient` hook, and then calling `getLayer`. ```tsx theme={null} const { client } = useStatsigClient(); const layer = client.getLayer('homepage_layer'); return (
Headline Parameter: {layer.get('hero_text', 'Welcome')}.
); ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: You can get the client with the `useStatsigClient` hook, and then call `logEvent` ```tsx theme={null} const { client } = useStatsigClient(); return ``` ### Flushing Logged Events `flush()` sends queued events immediately. Use `shutdown()` when your app is exiting. ```tsx theme={null} import { Button } from 'react-native'; import { useStatsigClient } from '@statsig/expo-bindings'; const { client } = useStatsigClient(); return ( ); ``` ## Loading State Dependent on your setup, you may want to wait for the latest values before checking a gate or experiment. If you are using `StatsigProviderExpo`, you can pass in a `loadingComponent` prop to display a loading state while the SDK is initializing. If you are using the `useClientAsyncInitExpo` hook, you can check the `isLoading` prop to determine if the SDK is still loading. ```tsx theme={null} export function App() { const loadingComponent =
Loading...
; return ( ); } ```
```tsx theme={null} export function App() { const { client, isLoading } = useClientAsyncInitExpo(...); if (isLoading) { return
Loading...
; } return ( ); } ```
## Lifecycle & Advanced Usage ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```tsx theme={null} import { useEffect } from 'react'; import { useStatsigClient } from '@statsig/expo-bindings'; const { client } = useStatsigClient(); useEffect(() => { return () => { void client.shutdown(); }; }, [client]); ``` ## Advanced ### Expo Without React In some scenarios, you may need to use the `StatsigClient` when you are not in the React component tree. Things like background tasks or handling notifications. For these, you can use the Expo specific `StatsigClientExpo`. ```tsx theme={null} import { StatsigClientExpo } from '@statsig/expo-bindings'; const myClient = new StatsigClientExpo( YOUR_CLIENT_KEY, { userID: "a-user" } ); await myClient.initializeAsync(); if (myClient.checkGate("my_gate")) { // do something cool } ``` ### Synchronous Storage with MMKV If you are utilizing [MMKV](https://github.com/mrousavy/react-native-mmkv) in your project, and would prefer to use that instead of the default ([AsyncStorage](https://github.com/react-native-async-storage)). You can provide you own `StorageProvider` via `StatsigOptions`. Something like: ```tsx theme={null} import { MMKV } from "react-native-mmkv"; import { StorageProvider } from "@statsig/client-core"; import { StatsigProviderExpo } from "@statsig/expo-bindings"; function App() { const [storageProvider] = useState(() => { const mmkv = new MMKV(); return { isReady: () => true, isReadyResolver: () => null, getProviderName: () => "MMKV", getAllKeys: () => mmkv.getAllKeys(), getItem: (key: string) => mmkv.getString(key) ?? null, setItem: (key: string, value: string) => mmkv.set(key, value), removeItem: (key: string) => mmkv.delete(key), }; }); return ( ... ); } ``` ## Debugging ### Network Issues Some users have reported a `ERROR: A networking error occured during POST request` messages when first initializing Statsig. This issue is solved in releases of the SDK after 3.1.0, so upgrading your SDK should solve the issue. If it persists, reach out to us in [Slack](https://statsig.com/slack). # Next.js Client SDK Source: https://docs.statsig.com/client/Next Set up the Statsig Next.js SDK to evaluate feature gates, run experiments, and capture analytics events in App Router and Pages Router applications. Source code: statsig-io/js-client-monorepo ## Set Up the SDK ### AI-powered Setup Setup Statsig in 90 seconds by copying this AI prompt into your IDE: ```text expandable Prompt theme={null} # Statsig SDK Integration for Next.js You are a frontend engineer integrating the Statsig SDK into a **Next.js application**. Follow all steps below one by one: --- ## Full Integration Instructions 1. **Detect the package manager** by checking for: - `package-lock.json` → use `npm` - `yarn.lock` → use `yarn` - `pnpm-lock.yaml` → use `pnpm` 2. **Detect the Next.js router type** by checking for: - `app/` directory → **App Router** - `pages/` directory → **Pages Router** 3. **Install the Statsig package** using the correct package manager: # For npm npm install @statsig/react-bindings @statsig/session-replay @statsig/web-analytics # For yarn yarn add @statsig/react-bindings @statsig/session-replay @statsig/web-analytics # For pnpm pnpm add @statsig/react-bindings @statsig/session-replay @statsig/web-analytics 4. Add your Statsig client key to .env.local: NEXT_PUBLIC_STATSIG_CLIENT_KEY=ask the user for their CLIENT KEY and use that input 5. Integrate Statsig into the app (auto-detect router type): ### If the project uses the App Router (has an app/ directory): // Create app/my-statsig.tsx "use client"; import React from "react"; import { LogLevel, StatsigProvider } from "@statsig/react-bindings"; export default function MyStatsig({ children }: { children: React.ReactNode }) { const id = typeof userID !== "undefined" ? userID : "a-user"; const user = { userID: id, // Optional additional fields: // email: 'user@example.com', // customIDs: { internalID: 'internal-123' }, // custom: { plan: 'premium' } }; return ( {children} ); } // Update app/layout.tsx to wrap children with MyStatsig import MyStatsig from "./my-statsig"; export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode }>) { return ( {children} {/* Preserve all existing layout content */} ); } ### If the project uses the Pages Router (has a pages/ directory): // Update pages/_app.tsx import type { AppProps } from "next/app"; import { LogLevel, StatsigProvider } from "@statsig/react-bindings"; export default function App({ Component, pageProps }: AppProps) { const id = typeof userID !== "undefined" ? userID : "a-user"; const user = { userID: id, // Optional additional fields: // email: 'user@example.com', // customIDs: { internalID: 'internal-123' }, // custom: { plan: 'premium' } }; return ( {/* Preserve all existing pages */} ); } ### Final Notes - The system must **detect the router type** and **apply the correct integration automatically**. - **Do not remove or change any existing JSX or layout structure** — only wrap the app with `StatsigProvider`. - **Preserve the file's language**: if it's TypeScript (`.tsx`), keep it as TypeScript; if it's JavaScript (`.jsx`), keep it as JavaScript. - After these steps, Statsig will be integrated across the entire app, with **feature gates, configs, and experiments** available everywhere. ``` ### Manual Setup Statsig supports both [Page Router](https://nextjs.org/docs/pages) & [App Router](https://nextjs.org/docs/app), with some differences in integration patterns. Add the keys to your .env.local file: ```bash .env.local theme={null} # the NEXT_PUBLIC_ prefix is required for this to be available on the client side NEXT_PUBLIC_STATSIG_CLIENT_KEY=client- STATSIG_SERVER_KEY=secret- ``` For App Router, install the @statsig/next package: ```bash NPM theme={null} npm i @statsig/next ``` ```bash Yarn theme={null} yarn add @statsig/next ``` ```bash PNPM theme={null} pnpm add @statsig/next ``` The \ creates both a Statsig Client and Server instance under the hood, and ["bootstraps"](/client/concepts/initialize#2-bootstrap-initialization) the client so it can render each page without a blocking network request. this will keep your app speedy and is recommended for most users. If you need more control over your setup, our [Bootstrapping](#client-bootstrapping-recommended) and [React](/client/React) docs can provide more guidance. Add this component around the content in your root `layout.tsx` file: ```tsx app/layout.tsx theme={null} import { StatsigBootstrapProvider } from "@statsig/next" export default function RootLayout({ children, }: Readonly<{ children: React.ReactNode; }>) { const user = { userID: "user-123", // add additional parameters as needed }; return ( {children} ); } ``` Add the keys to your .env.local file: ```bash .env.local theme={null} # the NEXT_PUBLIC_ prefix is required for this to be available on the client side NEXT_PUBLIC_STATSIG_CLIENT_KEY=client- ``` Install the @statsig/react-bindings package: ```bash NPM theme={null} npm i @statsig/react-bindings @statsig/web-analytics ``` ```bash Yarn theme={null} yarn add @statsig/react-bindings @statsig/web-analytics ``` ```bash PNPM theme={null} pnpm add @statsig/react-bindings @statsig/web-analytics ``` To integrate Statsig into your Page Router app you can add the `StatsigProvider` to your `_app.tsx` file. There is a [full example](https://github.com/statsig-io/js-client-monorepo/tree/main/samples/next-js-pages-router-sample) in the samples directory of the javascript sdk. ```tsx pages/_app.tsx theme={null} import type { AppProps } from "next/app"; import { LogLevel, StatsigProvider, } from "@statsig/react-bindings"; import { StatsigAutoCapturePlugin } from '@statsig/web-analytics'; export default function App({ Component, pageProps }: AppProps) { return ( ); } ``` See the [User (StatsigUser)](/concepts/user) doc for more info on the user property. From here, you're ready to start checking gates & experiments and sending events in any sub-file of layout.tsx! ## Use the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```tsx theme={null} 'use client'; import { useGateValue } from "@statsig/react-bindings"; export default function Home() { const gate = useGateValue("my_gate"); return (
Gate Value: {gate ? 'PASSED' : 'FAILED'}
); } ``` In an App Router app, you need to use the [`use client` directive](https://nextjs.org/docs/app/building-your-application/rendering/client-components) to ensure your logic runs on the frontend.
```tsx theme={null} import { useGateValue } from "@statsig/react-bindings"; export default function Home() { const gate = useGateValue("my_gate"); return (
Gate Value: {gate ? 'PASSED' : 'FAILED'}
); } ```
### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```tsx theme={null} 'use client'; import { useDynamicConfig } from "@statsig/react-bindings"; export default function Home() { const config = useDynamicConfig("my_dynamic_config"); return (
Title: {config.get('title', 'Fallback Title')}
); } ``` In an App Router app, you need to use the [`use client` directive](https://nextjs.org/docs/app/building-your-application/rendering/client-components) to ensure your logic runs on the frontend.
```tsx theme={null} import { useDynamicConfig } from "@statsig/react-bindings"; export default function Home() { const config = useDynamicConfig("my_dynamic_config"); return (
Title: {config.get('title', 'Fallback Title')}
); } ```
### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```tsx theme={null} 'use client'; import { useExperiment, useLayer} from "@statsig/react-bindings"; export default function Home() { const layer = useLayer("my_experiment_layer"); // or const experiment = useExperiment("my_experiment"); return (
Title: {layer.get('title', 'Fallback Title')} {/* or */} Title: {experiment.get('title', 'Fallback Title')}
); } ``` In an App Router app, you need to use the [`use client` directive](https://nextjs.org/docs/app/building-your-application/rendering/client-components) to ensure your logic runs on the frontend.
```tsx theme={null} import { useExperiment, useLayer} from "@statsig/react-bindings"; export default function Home() { const layer = useLayer("my_experiment_layer"); // or const experiment = useExperiment("my_experiment"); return (
Title: {layer.get('title', 'Fallback Title')} {/* or */} Title: {experiment.get('title', 'Fallback Title')}
); } ```
## Parameter Stores Parameter Stores hold a set of parameters for your mobile app. These parameters can be remapped on-the-fly from a static value to a Statsig entity (Feature Gates, Experiments, and Layers), so you can decouple your code from the configuration in Statsig. Read more about Param Stores [here](/client/concepts/parameter-stores). ```tsx theme={null} 'use client'; import { useParameterStore} from "@statsig/react-bindings"; export default function Home() { const store = useParameterStore("my_param_store"); return (
Title: {store.get('title', 'Fallback Title')}
); } ``` In an App Router app, you need to use the [`use client` directive](https://nextjs.org/docs/app/building-your-application/rendering/client-components) to ensure your logic runs on the frontend.
```tsx theme={null} import { useParameterStore} from "@statsig/react-bindings"; export default function Home() { const store = useParameterStore("my_param_store"); return (
Title: {store.get('title', 'Fallback Title')}
); } ```
### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```tsx theme={null} 'use client'; import { useStatsigClient } from "@statsig/react-bindings"; export default function Home() { const { client } = useStatsigClient(); return (
); } ``` In an App Router app, you need to use the [`use client` directive](https://nextjs.org/docs/app/building-your-application/rendering/client-components) to ensure your logic runs on the frontend.
```tsx theme={null} import { useStatsigClient } from "@statsig/react-bindings"; export default function Home() { const { client } = useStatsigClient(); return (
); } ```
### Flushing Logged Events `flush()` sends queued events immediately. Use `shutdown()` when your app is exiting. ```tsx theme={null} 'use client'; import { useStatsigClient } from "@statsig/react-bindings"; export default function Home() { const { client } = useStatsigClient(); return (
); } ``` In an App Router app, you need to use the [`use client` directive](https://nextjs.org/docs/app/building-your-application/rendering/client-components) to ensure your logic runs on the frontend.
```tsx theme={null} import { useStatsigClient } from "@statsig/react-bindings"; export default function Home() { const { client } = useStatsigClient(); return (
); } ```
## Session Replay ```tsx theme={null} 'use client'; import { StatsigProvider } from '@statsig/react-bindings'; import { StatsigSessionReplayPlugin } from '@statsig/session-replay'; export default function App({ children }: { children: React.ReactNode }) { return ( {children} ); } ``` ```tsx theme={null} import { StatsigProvider } from '@statsig/react-bindings'; import { StatsigSessionReplayPlugin } from '@statsig/session-replay'; export default function App({ Component, pageProps }) { return ( ); } ``` ## Web Analytics / Auto Capture By including the [`@statsig/web-analytics`](https://www.npmjs.com/package/@statsig/web-analytics) package in your project, you can automatically capture common web events like clicks and page views. For more information on filtering events, enabling console log capture, and other configuration options available in web analytics, see the [Web Analytics Configuration](/webanalytics/overview#event-filtering-and-console-configuration) documentation. ```tsx theme={null} 'use client'; import { StatsigProvider } from '@statsig/react-bindings'; import { StatsigAutoCapturePlugin } from '@statsig/web-analytics'; export default function App({ children }: { children: React.ReactNode }) { return ( {children} ); } ``` ```tsx theme={null} import { StatsigProvider } from '@statsig/react-bindings'; import { StatsigAutoCapturePlugin } from '@statsig/web-analytics'; export default function App({ Component, pageProps }) { return ( ); } ``` ## Stable ID Stable ID provides a consistent device identifier. It lets you run [logged-out experiments](/guides/first-device-level-experiment) and target gates at the device level. ### How Stable ID Works * On first initialization the SDK generates a Stable ID and stores it in `localStorage` under `statsig.stable_id.`. * Subsequent sessions reuse the stored value. Each client SDK key has its own Stable ID entry. * Local storage is scoped per domain, so cross-domain usage requires sharing the value manually (see below). ### Reading the Stable ID ```tsx theme={null} const context = client.getContext(); console.log('Statsig StableID:', context.stableID); ``` ```tsx theme={null} import { useStatsigClient } from '@statsig/react-bindings'; function MyComponent() { const { client } = useStatsigClient(); const context = client.getContext(); return
{context.stableID}
; } ```
### Overriding the Stable ID Provide a custom Stable ID through `StatsigUser.customIDs.stableID` if you already manage a durable device identifier. ```tsx theme={null} import { StatsigClient, StatsigUser } from '@statsig/js-client'; const userWithStableID: StatsigUser = { customIDs: { stableID: 'my-custom-stable-id', }, }; const client = new StatsigClient('client-xyz', userWithStableID); await client.updateUserAsync(userWithStableID); ``` ```tsx theme={null} import { StatsigProvider, useStatsigClient } from '@statsig/react-bindings'; function App() { return (
Your App
); } function MyComponent() { const { client } = useStatsigClient(); useEffect(() => { client.updateUserAsync({ customIDs: { stableID: 'my-custom-stable-id' }, }); }, [client]); } ```
When you override the Stable ID it is persisted to local storage, so subsequent sessions reuse your custom value. ### Sharing Stable ID Across Subdomains Add this helper script before initializing the SDK and then copy the stored value onto your user object. ```html theme={null} ``` (Use this script at your discretion and test thoroughly.) ### Aligning Stable ID Between Client and Server To share Stable ID with a backend Statsig SDK, send the value with requests and persist it server-side when missing. The server can bootstrap the client with the same Stable ID. ```tsx theme={null} // Server: ensure Stable ID exists, then return initialize response for the client const values = Statsig.getClientInitializeResponse(user, YOUR_CLIENT_KEY, { hash: 'djb2', }); // Client: apply the server-provided values and initialize synchronously const { values, user: verifiedUser } = await fetch('/init-statsig-client', { method: 'POST', body: loadUserData(), }).then((res) => res.json()); const myClient = new StatsigClient(YOUR_CLIENT_KEY, verifiedUser); myClient.dataAdapter.setData(values); myClient.initializeSync(); ``` ## Lifecycle & Advanced Usage ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```tsx theme={null} 'use client'; import { useEffect } from "react"; import { useStatsigClient } from "@statsig/react-bindings"; export default function Home() { const { client } = useStatsigClient(); useEffect(() => { return () => { void client.shutdown(); }; }, [client]); return null; } ``` In an App Router app, you need to use the [`use client` directive](https://nextjs.org/docs/app/building-your-application/rendering/client-components) to ensure your logic runs on the frontend. ```tsx theme={null} import { useEffect } from "react"; import { useStatsigClient } from "@statsig/react-bindings"; export default function Home() { const { client } = useStatsigClient(); useEffect(() => { return () => { void client.shutdown(); }; }, [client]); return null; } ``` ## Advanced Setup ### Client Bootstrapping (Recommended) ```ts app/api/statsig-bootstrap/route.ts theme={null} import { Statsig, StatsigUser } from '@statsig/statsig-node-core'; export async function POST(request: Request): Promise { const body = await request.json(); const user = new StatsigUser(body?.user ?? {}); // Ensure server SDK is initialized at startup // await Statsig.initialize(process.env.STATSIG_SERVER_KEY!); const values = Statsig.getClientInitializeResponse(user, { hashAlgorithm: 'djb2', }); return new Response(JSON.stringify(values), { status: 200 }); } ``` ```tsx app/layout.tsx theme={null} import { StatsigBootstrapProvider } from '@statsig/next'; export default function RootLayout({ children }: { children: React.ReactNode }) { const user = { userID: 'user-123' }; return ( {children} ); } ``` ```ts pages/api/statsig-bootstrap.ts theme={null} import type { NextApiRequest, NextApiResponse } from 'next'; import { Statsig, StatsigUser } from 'statsig-node'; // legacy Node SDK for pages router export default async function handler( req: NextApiRequest, res: NextApiResponse, ) { if (req.method !== 'POST') { res.status(400).send('/statsig-bootstrap only supports POST'); return; } // Ensure server SDK is initialized at startup // await Statsig.initialize(process.env.STATSIG_SERVER_KEY!); const { user } = JSON.parse(req.body) as { user: StatsigUser }; const values = Statsig.getClientInitializeResponse(user, { hash: 'djb2' }); res.status(200).send(JSON.stringify(values)); } ``` ```tsx pages/_app.tsx theme={null} import type { AppProps } from 'next/app'; import { StatsigProvider } from '@statsig/react-bindings'; import { StatsigClient } from '@statsig/js-client'; import { useEffect, useMemo, useState } from 'react'; export default function App({ Component, pageProps }: AppProps) { const user = useMemo(() => ({ userID: 'a-user' }), []); const [client, setClient] = useState(null); useEffect(() => { (async () => { const res = await fetch('/api/statsig-bootstrap', { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ user }), }); const initializeValues = await res.json(); const inst = new StatsigClient( process.env.NEXT_PUBLIC_STATSIG_CLIENT_KEY!, user, { initializeValues }, ); await inst.initializeAsync(); setClient(inst); })(); }, [user]); if (!client) { return null; } return ( ); } ``` ### Proxying Network Traffic (Optional) ```ts app/proxy/initialize/route.ts theme={null} // Note: Using generic path names like "proxy" instead of "statsig-proxy" // to prevent ad blockers from blocking these requests import { generateBootstrapValues } from './statsig-backend'; export async function POST(request: Request): Promise { const json = await request.json(); if (!json || typeof json !== 'object') { return new Response(null, { status: 400 }); } const data = await generateBootstrapValues(); return new Response(data); } ``` ```ts app/proxy/search.ts theme={null} // Note: Using generic path names like "search" instead of "log_event" or "events" // to prevent ad blockers from blocking these requests type ExtendedRequestInit = RequestInit & { duplex?: 'half' | 'full' }; export async function POST(request: Request): Promise { const tail = request.url.split('?').pop(); const logEventUrl = `https://events.statsigapi.net/v1/log_event?${tail}`; const fetchOptions: ExtendedRequestInit = { method: 'POST', body: request.body, headers: request.headers, duplex: 'half', }; return fetch(logEventUrl, fetchOptions); } ``` ```ts pages/api/proxy/initialize.ts theme={null} // Note: Using generic path names like "proxy" instead of "statsig-proxy" // to prevent ad blockers from blocking these requests import type { NextApiRequest, NextApiResponse } from 'next'; import { StatsigUser } from 'statsig-node'; import { getStatsigValues } from '@/pages/statsig-backend'; export default async function handler( req: NextApiRequest, res: NextApiResponse, ) { if (req.method !== 'POST') { res.status(400).send('/initialize only supports POST'); return; } const { user } = JSON.parse(req.body) as { user: StatsigUser }; const values = await getStatsigValues(user); res.status(200).send(values); } ``` ```ts pages/api/proxy/search.ts theme={null} // Note: Using generic path names like "search" instead of "log_event" or "events" // to prevent ad blockers from blocking these requests import type { NextApiRequest, NextApiResponse } from 'next'; type ExtendedRequestInit = RequestInit & { duplex?: 'half' | 'full' }; export default async function handler( req: NextApiRequest, res: NextApiResponse, ): Promise { if (req.method !== 'POST') { res.status(400).send('/search only supports POST'); return; } let logEventUrl = `https://events.statsigapi.net/v1/log_event`; const queryParams = [] as string[]; for (const [key, value] of Object.entries(req.query)) { queryParams.push(`${key}=${value}`); } if (queryParams.length > 0) { logEventUrl += '?' + queryParams.join('&'); } const fetchOptions: ExtendedRequestInit = { method: 'POST', body: req.body as BodyInit, headers: req.headers as HeadersInit, duplex: 'half', }; try { const response = await fetch(logEventUrl, fetchOptions); if (!response.ok) { res.status(500).send('Failed to log event'); return; } const body = await response.text(); res.status(response.status).send(body); } catch (err) { res.status(500).send('Failed to log event: ' + err); } } ``` ```ts theme={null} // Assign URLs when creating the client const inst = new StatsigClient(clientSdkKey, user, { networkConfig: { logEventUrl: '/api/proxy/search', initializeUrl: '/api/proxy/initialize', logEventCompressionMode: 'Forced', }, disableCompression: true, disableStatsigEncoding: true, }); ``` ## Statsig Site Generation (SSG) Vercel's Static Site Generation renders HTML at build time. Because static HTML can't be responsive to per-user values, experimenting on SSG content requires one of these patterns: * Use Vercel Edge Middleware with Statsig's Edge Config Adapter for zero-latency redirects. * Isolate Statsig usage to hydrated client components only. ```tsx theme={null} // Create a single client and share it across multiple StatsigProviders const myStatsigClient = new StatsigClient(YOUR_SDK_KEY, user, options); await myStatsigClient.initializeAsync(); ``` ## Statsig Options Controls logging behavior. * `browser-only` (default): log events from browser environments. * `disabled`: never send events. * `always`: log in every environment, including non-browser contexts. Use `loggingEnabled: 'disabled'` instead. Skip generating a device-level Stable ID. Recompute every evaluation instead of using the memoized result. Override the generated session ID. Persist Stable ID in cookies for cross-domain tracking. Prevent any local storage writes (disables caching). Override network endpoints per request type. Base URL for all requests. The SDK appends endpoint paths like `/initialize` and `/rgstr`; append `/v1` when your proxy expects it. Endpoint for initialization requests only. Takes precedence over `api` for `/initialize`. Fallback endpoints for initialization requests only. This does not create a generic fallback for `api`. Endpoint for event uploads. Fallback endpoints for event uploads only. This does not create a generic fallback for `api`. Request timeout in milliseconds. Disable all outbound requests; combine with `loggingEnabled: 'disabled'` to silence log warnings. Provide custom transport (e.g., Axios). Set environment-wide defaults (for example `{ tier: 'staging' }`). Console verbosity. Max events per log batch. Interval between automatic flushes. Modify evaluations before returning them. Attach the current page URL to logged events. Send requests without Statsig-specific encoding. Control compression for batched events. Use `logEventCompressionMode` instead. Provide a custom data adapter to control caching/fetching. Override cache key generation for stored evaluations. ## Additional Resources * [JavaScript Client SDK](/client/javascript-sdk) * [React Client SDK](/client/React) * [Initialization Concepts](/client/concepts/initialize) # React Client SDK Source: https://docs.statsig.com/client/React Use the Statsig React SDK with hooks and providers to evaluate feature flags, run experiments, and add optional session replay and autocapture plugins. Source code: statsig-io/js-client-monorepo ## Set Up the SDK If you need a starter project, follow the official React quickstart. Looking for Next.js instead? See the Next.js SDK docs. ### AI-powered Setup Setup Statsig in 90 seconds by copying this AI prompt into your IDE: ```text expandable theme={null} You are a frontend engineer integrating the Statsig SDK into a React app. Follow these instructions carefully: 1. Install the required Statsig packages: npm install @statsig/react-bindings @statsig/session-replay @statsig/web-analytics 2. In the main component file (`App.jsx` or `App.tsx`): - Import `StatsigProvider` and `useClientAsyncInit` from `@statsig/react-bindings` - Import `StatsigAutoCapturePlugin` from `@statsig/web-analytics` and `StatsigSessionReplayPlugin` from `@statsig/session-replay` - Initialize the SDK using your client key: 'YOUR-CLIENT-API-KEY' - Use `userID` from an existing variable if it's already declared in the file; otherwise, default to `'a-user'` - Wrap the existing app content inside ``, using `
Loading...
` as the `loadingComponent` 3. DO NOT remove any existing JSX content from the component. Just wrap it. 4. Here is what the final file structure should look like: import { StatsigProvider, useClientAsyncInit } from '@statsig/react-bindings'; import { StatsigAutoCapturePlugin } from '@statsig/web-analytics'; import { StatsigSessionReplayPlugin } from '@statsig/session-replay'; import YourApp from './YourApp'; function App() { const id = typeof userID !== 'undefined' ? userID : 'a-user'; const { client } = useClientAsyncInit( 'YOUR-CLIENT-API-KEY', { userID: id }, { plugins: [new StatsigAutoCapturePlugin(), new StatsigSessionReplayPlugin()] } ); return ( Loading...}> ); } 5. Ask the user to provide their CLIENT-API-KEY and insert it where prompted above. ``` ### Install Packages ```bash theme={null} npm install @statsig/react-bindings ``` ```bash theme={null} yarn add @statsig/react-bindings ``` Add `@statsig/session-replay` and `@statsig/web-analytics` if you plan to enable Session Replay or Auto Capture.
Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. ### Wrap Your App With `StatsigProvider` Provide your client SDK key and initial user when you render the provider. ```tsx theme={null} import { StatsigProvider } from '@statsig/react-bindings'; function App() { return (
Hello world
); } ``` ### Typical Project Structure Most projects render a root component inside the provider. ```tsx theme={null} // App.tsx import RootPage from './RootPage'; import { StatsigProvider } from '@statsig/react-bindings'; export default function App() { return ( ); } ``` ```tsx theme={null} // RootPage.tsx export default function RootPage() { return
Hello World
; } ``` Need to balance startup speed with freshness? Review Initialization Strategies for bootstrap and async options.
## Use the SDK Use `useStatsigClient` inside components to retrieve the client when you need to evaluate something. ```tsx theme={null} import { useStatsigClient } from '@statsig/react-bindings'; const { client } = useStatsigClient(); ``` ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```tsx theme={null} import { useFeatureGate, useGateValue, useStatsigClient, } from '@statsig/react-bindings'; const { checkGate } = useStatsigClient(); const gateValue = useGateValue('my_gate'); const gate = useFeatureGate('my_gate'); return (
{checkGate('my_gate') &&

Passing

} {gateValue &&

Passing

} {gate.value &&

Passing ({gate.details.reason})

}
); ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```tsx theme={null} import { useDynamicConfig, useStatsigClient } from '@statsig/react-bindings'; const config = useDynamicConfig('my_dynamic_config'); const { getDynamicConfig } = useStatsigClient(); return (

Reason: {config.details.reason}

Value: {config.get('a_value', 'fallback_value')}

Another Value: {getDynamicConfig('my_dynamic_config').get('a_bool', false)}

); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```tsx theme={null} import { useExperiment, useStatsigClient } from '@statsig/react-bindings'; const experiment = useExperiment('my_experiment'); const { getExperiment } = useStatsigClient(); return (

Group: {getExperiment('my_experiment').groupName}

Value: {experiment.get('a_value', 'fallback_value')}

); ``` ```tsx theme={null} import { useLayer, useStatsigClient } from '@statsig/react-bindings'; const layer = useLayer('my_layer'); const { getLayer } = useStatsigClient(); return (

Group: {getLayer('my_layer').groupName}

Value: {layer.get('a_value', 'fallback_value')}

); ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```tsx theme={null} import { useStatsigClient } from '@statsig/react-bindings'; const { logEvent } = useStatsigClient(); return ; ``` ### Flushing Logged Events `flush()` sends queued events immediately. Use `shutdown()` when your app is exiting. ```tsx theme={null} import { useStatsigClient } from '@statsig/react-bindings'; const { client } = useStatsigClient(); return ( ); ``` ## Parameter Stores Parameter Stores hold a set of parameters for your mobile app. These parameters can be remapped on-the-fly from a static value to a Statsig entity (Feature Gates, Experiments, and Layers), so you can decouple your code from the configuration in Statsig. Read more about Param Stores [here](/client/concepts/parameter-stores). ## Manage Users ### Updating User Properties Switch identities when a user logs in or when you collect richer attributes. ```tsx theme={null} import { useGateValue, useStatsigUser } from '@statsig/react-bindings'; export default function AccountBanner() { const gateValue = useGateValue('check_user'); const { updateUserAsync } = useStatsigUser(); return (
Gate is {gateValue ? 'passing' : 'failing'}.
); } ``` ## Loading State On a fresh page load with no cached values in `localStorage` (for example, an incognito window or a first visit), `isClientLoading` from `useStatsigClient` stays `true` until the SDK finishes fetching values from the network. The recommended pattern is to initialize with `useClientAsyncInit` and pass the client to `StatsigProvider`, so your app waits for the initial fetch before rendering. Passing a `loadingComponent` directly to `StatsigProvider` is an equivalent shorthand. ```tsx theme={null} import { StatsigProvider, useClientAsyncInit } from '@statsig/react-bindings'; export function App() { const { client, isLoading } = useClientAsyncInit( 'client-KEY', { userID: 'a-user' }, ); if (isLoading) { return
Loading...
; } return ( ); } ```
```tsx theme={null} import { StatsigProvider } from '@statsig/react-bindings'; export function App() { return ( Loading...} > ); } ```
## React Hooks Hooks that read gates, configs, experiments, or layers will log exposures on render. Use `useStatsigClient` to defer checks until you actually change the UI. ### Feature Gate Hooks * Recommended: `useStatsigClient().checkGate` logs when invoked. * `useGateValue` returns the boolean value and logs immediately. * `useFeatureGate` returns the full gate object with details. ```tsx theme={null} import { useFeatureGate, useGateValue, useStatsigClient, } from '@statsig/react-bindings'; const { checkGate } = useStatsigClient(); const gateValue = useGateValue('my_gate'); const gate = useFeatureGate('my_gate'); return (
{checkGate('my_gate') &&

Passing

} {gateValue &&

Passing

} {gate.value &&

Passing ({gate.details.reason})

}
); ``` ### Dynamic Config Hooks * Recommended: `useStatsigClient().getDynamicConfig` defers exposure until called. * `useDynamicConfig` logs on render. ```tsx theme={null} import { useDynamicConfig, useStatsigClient } from '@statsig/react-bindings'; const config = useDynamicConfig('my_dynamic_config'); const { getDynamicConfig } = useStatsigClient(); return (

Reason: {config.details.reason}

Value: {config.get('a_value', 'fallback_value')}

Another Value: {getDynamicConfig('my_dynamic_config').get('a_bool', false)}

); ``` ### Experiment Hooks * Recommended: `useStatsigClient().getExperiment` to control exposures. * `useExperiment` logs on render. ```tsx theme={null} import { useExperiment, useStatsigClient } from '@statsig/react-bindings'; const experiment = useExperiment('my_experiment'); const { getExperiment } = useStatsigClient(); return (

Group: {getExperiment('my_experiment').groupName}

Value: {experiment.get('a_value', 'fallback_value')}

); ``` ### Layer Hooks Layers only log exposures when you call `.get()`. ```tsx theme={null} import { useLayer, useStatsigClient } from '@statsig/react-bindings'; const layer = useLayer('my_layer'); const { getLayer } = useStatsigClient(); return (

Group: {getLayer('my_layer').groupName}

Value: {layer.get('a_value', 'fallback_value')}

); ``` ### Parameter Store Hooks ```tsx theme={null} import { useParameterStore } from '@statsig/react-bindings'; function MyComponent() { const store = useParameterStore('my_parameter_store'); const title = store.get('page_title', 'Default Title'); const maxItems = store.get('max_items', 10); const isEnabled = store.get('feature_enabled', false); const storeNoExposure = useParameterStore('my_parameter_store', { disableExposureLog: true, }); return
{title}
; } ``` ### Log Events From Hooks ```tsx theme={null} import { useStatsigClient } from '@statsig/react-bindings'; const { logEvent } = useStatsigClient(); return ; ``` ### StatsigUser Hook ```tsx theme={null} import { useStatsigUser } from '@statsig/react-bindings'; const { user, updateUserSync } = useStatsigUser(); return (

Current User: {user.userID}

); ``` ### Direct Access to the Client ```tsx theme={null} import { useStatsigClient } from '@statsig/react-bindings'; const { client } = useStatsigClient(); console.log('stableID', client.getContext().stableID); ``` ### Client Initialization Hooks * `useClientAsyncInit` — fetches the latest values before rendering. * `useClientBootstrapInit` — bootstrap from server-provided values. You can also initialize your own client instance manually. See Initialization Strategies for alternatives. ## Statsig Options Controls logging behavior. * `browser-only` (default): log events from browser environments. * `disabled`: never send events. * `always`: log in every environment, including non-browser contexts. Use `loggingEnabled: 'disabled'` instead. Skip generating a device-level Stable ID. Recompute every evaluation instead of using the memoized result. Override the generated session ID. Persist Stable ID in cookies for cross-domain tracking. Prevent any local storage writes (disables caching). Override network endpoints per request type. Base URL for all requests. The SDK appends endpoint paths like `/initialize` and `/rgstr`; append `/v1` when your proxy expects it. Endpoint for initialization requests only. Takes precedence over `api` for `/initialize`. Fallback endpoints for initialization requests only. This does not create a generic fallback for `api`. Endpoint for event uploads. Fallback endpoints for event uploads only. This does not create a generic fallback for `api`. Request timeout in milliseconds. Disable all outbound requests; combine with `loggingEnabled: 'disabled'` to silence log warnings. Provide custom transport (e.g., Axios). Set environment-wide defaults (for example `{ tier: 'staging' }`). Console verbosity. Max events per log batch. Interval between automatic flushes. Modify evaluations before returning them. Attach the current page URL to logged events. Send requests without Statsig-specific encoding. Control compression for batched events. Use `logEventCompressionMode` instead. Provide a custom data adapter to control caching/fetching. Override cache key generation for stored evaluations. ## Testing Mock Statsig hooks in Jest to isolate component logic. ```tsx theme={null} import { StatsigProvider, useFeatureGate, useExperiment } from '@statsig/react-bindings'; function Content() { const gate = useFeatureGate('a_gate'); const experiment = useExperiment('an_experiment'); return (
a_gate: {gate.value ? 'Pass' : 'Fail'}
an_experiment: {experiment.get('my_param', 'fallback')}
); } function App() { return ( ); } ``` ```tsx theme={null} import { render, screen } from '@testing-library/react'; import * as ReactBindings from '@statsig/react-bindings'; jest.mock('@statsig/react-bindings', () => ({ ...jest.requireActual('@statsig/react-bindings'), useFeatureGate: () => ({ value: true }), useExperiment: () => ({ get: () => 'my_value' }), })); test('renders gate pass', async () => { render(); const elem = await screen.findByTestId('gate_test'); expect(elem.textContent).toContain('Pass'); }); test('renders experiment value', async () => { render(); const elem = await screen.findByTestId('exp_test'); expect(elem.textContent).toContain('my_value'); }); ``` ## Lifecycle & Advanced Usage ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```tsx theme={null} import { useEffect } from 'react'; import { useStatsigClient } from '@statsig/react-bindings'; const { client } = useStatsigClient(); useEffect(() => { return () => { void client.shutdown(); }; }, [client]); ``` ## Session Replay Install `@statsig/session-replay` and register the plugin to record user sessions. ```tsx theme={null} import { StatsigProvider, useClientAsyncInit } from '@statsig/react-bindings'; import { StatsigSessionReplayPlugin } from '@statsig/session-replay'; function App() { const { client } = useClientAsyncInit( 'client-KEY', { userID: 'a-user' }, { plugins: [new StatsigSessionReplayPlugin()] }, ); return ( Loading...}>
Hello World
); } ``` ## Web Analytics / Auto Capture By including the [`@statsig/web-analytics`](https://www.npmjs.com/package/@statsig/web-analytics) package in your project, you can automatically capture common web events like clicks and page views. For more information on filtering events, enabling console log capture, and other configuration options available in web analytics, see the [Web Analytics Configuration](/webanalytics/overview#event-filtering-and-console-configuration) documentation. ```tsx theme={null} import { StatsigProvider, useClientAsyncInit } from '@statsig/react-bindings'; import { StatsigAutoCapturePlugin } from '@statsig/web-analytics'; function App() { const { client } = useClientAsyncInit( 'client-KEY', { userID: 'a-user' }, { plugins: [new StatsigAutoCapturePlugin()] }, ); return ( Loading...}>
Hello World
); } ``` ## Using Persistent Evaluations Keep experiment variants stable across rerenders or user transitions by plugging in persistent storage. The React integration mirrors the [JavaScript workflow](/client/javascript-sdk#using-persistent-evaluations) and you can adapt the [Next.js sample](https://github.com/statsig-io/js-client-monorepo/tree/main/samples/next-js/src/app/persisted-user-storage-example) to your setup. Read more in [Client Persistent Assignment](/client/concepts/persistent_assignment). ## Additional Resources * [Initialization Concepts](/client/concepts/initialize) * [JavaScript Client SDK](/client/javascript-sdk) * [Persistent Assignment](/client/concepts/persistent_assignment) # React Native Client SDK Source: https://docs.statsig.com/client/ReactNative Install the Statsig React Native SDK to evaluate feature gates, run experiments, and log analytics events on iOS and Android React Native applications. Source code: statsig-io/js-client-monorepo ## Setup the SDK ## Installation Statsig uses a multi-package strategy, so you will need to install both the Statsig client and the React Native specific bindings. ```shell NPM theme={null} npm install @statsig/react-native-bindings ``` ```shell Yarn theme={null} yarn add @statsig/react-native-bindings ``` ### Peer Dependencies The `@statsig/react-native-bindings` package has peer dependencies which may also need to be installed if they are not already in your project. ```shell NPM theme={null} npm install react-native-device-info @react-native-async-storage/async-storage ``` ```shell Yarn theme={null} yarn add react-native-device-info @react-native-async-storage/async-storage ``` Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. ## React Native + React Specific Setup The setup for a ReactNative environment is very similar to a plain [React environment](/client/React). The only difference is that you need to use the ReactNative specific `StatsigProviderRN`. This automatically switches out the storage layer used by the SDK, utilizing [AsyncStorage](https://github.com/react-native-async-storage) instead of LocalStorage (which isn't available in RN environments). ```tsx theme={null} import { StatsigProviderRN, useFeatureGate, } from "@statsig/react-native-bindings"; function Content() { const gate = useFeatureGate("a_gate"); // Reason: Network or NetworkNotModified return ( Value: {gate.value ? "Pass" : "Fail"} Reason: {gate.details.reason} ); } function App() { return ( Loading...} > ); } ``` ## Use the SDK You can get an instance of the StatsigClient to check gates, experiments, dynamic configs, layers, and log events. ```jsx theme={null} import { useStatsigClient } from "@statsig/react-native-bindings"; const { client } = useStatsigClient(); ``` See the methods you can call on the client below. ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. You can evaluate a gate by getting the client with the `useStatsigClient` hook, and then calling `checkGate` ```tsx theme={null} const { client } = useStatsigClient(); return (
Gate is {client.checkGate('check_user') ? 'passing' : 'failing'}.
); ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: You can get a DynamicConfig value by getting the client with the `useStatsigClient` hook, and then calling `getConfig` ```tsx theme={null} const { client } = useStatsigClient(); const config = client.getConfig('app_properties'); return (
{config.get('title', 'Default Title')}
); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. You can access the experiment variant and parameters for the user by getting the client with the `useStatsigClient` hook, and then calling `getExperiment`. ```tsx theme={null} const { client } = useStatsigClient(); const experiment = client.getExperiment('headline_test'); return (
Headline Parameter: {experiment.get('headline', 'Default')}.
); ``` You can access layers and layer parameters for the user by getting the client with the `useStatsigClient` hook, and then calling `getLayer`. ```tsx theme={null} const { client } = useStatsigClient(); const layer = client.getLayer('homepage_layer'); return (
Headline Parameter: {layer.get('hero_text', 'Welcome')}.
); ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: You can get the client with the `useStatsigClient` hook, and then call `logEvent` ```tsx theme={null} const { client } = useStatsigClient(); return ``` ### Flushing Logged Events `flush()` sends queued events immediately. Use `shutdown()` when your app is exiting. ```tsx theme={null} import { Button } from 'react-native'; import { useStatsigClient } from '@statsig/react-native-bindings'; const { client } = useStatsigClient(); return ( } function App() { const [user, setUser] = useState({ userID: "a-user" }); return ( ); } ``` ```tsx New theme={null} import { StatsigProvider, useClientAsyncInit } from '@statsig/react-bindings'; import { StatsigClient } from '@statsig/js-client'; const client = new StatsigClient('', { userID: 'a-user' }); function UpdateUserButton() { const { client } = useClientAsyncInit(client); const handleUpdate = async () => { await client.updateUserAsync({ userID: 'b-user' }); }; return ; } function App() { return ( ); } ``` View the full example on [GitHub](https://github.com/statsig-io/js-client-monorepo/blob/main/samples/react/src/samples/react-precomp/sample-react-precomp-update-user.tsx) # On Device Client SDKs Source: https://docs.statsig.com/client/onDeviceOverview Overview of Statsig on-device evaluation client SDKs that evaluate feature gates and experiments locally for low-latency rules evaluation on devices. ## On Device SDK Overview Statsig's client-side On-Device Eval SDKs provide an alternate client-side architecture where the definition of each experiment or gate is kept in-memory on the device, allowing for faster evaluation with a frequently changing user object (which would require re-initialization on regular client SDKs), at the expense of the privacy of the config definitions. Generally, we encourage customers to use our traditional client SDKs, unless they have specific requirements making that impossible. If you do choose to use the on-device eval SDKs, we encourage you to speak with our team first and understand the privacy risks. ## Alternatives An alternative approach to On-Device Eval SDKs is our [Local Eval Adapter](/client/concepts/local-eval-adapter) which allows you to evaluate locally for only a subset of your experiments/gates that might need to be available at startup, rather than exposing all configs in your project. # Swift On Device Evaluation SDK Source: https://docs.statsig.com/client/swiftOnDeviceEvaluationSDK Use the Statsig Swift on-device evaluation SDK in iOS, macOS, tvOS, and watchOS apps to evaluate feature gates and experiments locally with low latency. Source code: statsig-io/swift-on-device-evaluations-sdk Statsig's normal (remote evaluation) SDKs are recommended for most client applications. Understand the use case and privacy risks by reading the [On-Device Eval SDK overview](/client/onDevice). On-device evaluation SDKs are for Enterprise & Pro Tier only. These SDKs use a different paradigm than their precomputed counterparts: [JS](/client/javascript-sdk), [Android](/client/Android), [iOS](/client/iosClientSDK), they behave more like Server SDKs. Rather than requiring a user up front, you can check gates/configs/experiments for any set of user properties, because the SDK downloads a complete representation of your project and evaluates checks in real time. ### Pros * No need for a network request when changing user properties - just check the gate/config/experiment locally * Can bring your own CDN or synchronously initialize with a preloaded project definition * Lower latency to download configs cached at the edge, rather than evaluated for a given user (which cannot be cached as much) ### Cons * Entire project definition is available client side - the names and configurations of all experiments and feature flags accessible by your client key are exposed. See our [client key with server permission best practices](/access-management/api-keys#client-keys-with-server-permissions) * Payload size is strictly larger than what is required for the traditional SDKs * Evaluation performance is slightly slower - rather than looking up the value, the SDK must actually evaluate targeting conditions and an allocation decision * Does not support ID list segments with > 1000 IDs * Does not support IP or User Agent based checks (Browser Version/Name, OS Version/Name, IP, Country) ## Set Up the SDK To use the SDK in your project, you must add Statsig as a dependency. ```swift Swift Package Manager theme={null} // In your Xcode, select File > Swift Packages > Add Package Dependency // and enter the URL https://github.com/statsig-io/swift-on-device-evaluations-sdk.git. // // You can also include it directly in your project's Package.swift. // Find out the latest release version on our GitHub page: // https://github.com/statsig-io/swift-on-device-evaluations-sdk/releases dependencies: [ // see the latest version on https://github.com/statsig-io/swift-on-device-evaluations-sdk/releases .package(url: "https://github.com/statsig-io/swift-on-device-evaluations-sdk.git", .upToNextMinor("X.Y.Z")), ], //... targets: [ .target( name: "YOUR_TARGET", dependencies: ["StatsigOnDeviceEvaluations"] ) ], ``` ```ruby CocoaPods theme={null} # If you are using CocoaPods, our pod name is 'StatsigOnDeviceEvaluations', # and you can include the following line to your Podfile: use_frameworks! target 'TargetName' do //... pod 'StatsigOnDeviceEvaluations', '~> X.Y.Z' end # Find the latest versions by searching cocoapods.org or on Github: # https://github.com/statsig-io/swift-on-device-evaluations-sdk/releases ``` Next, initialize the SDK with a client SDK key from the ["API Keys" tab on the Statsig console](https://console.statsig.com/api_keys). These keys are safe to embed in a client application. Along with the key, pass in a [User Object](#statsig-user) with the attributes you'd like to target later on in a gate or experiment. For On-Device Evaluation, you'll need to add the **"Allow Download Config Specs"** scope. Client keys, by default, are not able to download the project definition for on-device evaluation. While client keys are safe to include, Server and Console keys should always be kept private. When creating a new client key, select **"Allow Download Config Specs"** Add DCS Scope to New Key To add the scope to an existing key, under **Project Settings** → **API Keys** → **Client API Keys**, select **Actions** → **Edit Scopes**, and select **"Allow Download Config Specs"**, then **Save**. Add DCS Scope to Existing Key ```swift Async (Swift) theme={null} import StatsigOnDeviceEvaluations // (optional) Configure the SDK if needed let opts = StatsigOptions() opts.environment.tier = "staging" Statsig.shared.initialize("client-sdk-key", options: opts) { err in if let err = err { print("Error \(err)") } } // or, create your own instance let myStatsigInstance = Statsig() myStatsigInstance.initialize("client-sdk-key", options: opts) { err in if let err = err { print("Error \(err)") } } ``` ```objective-c Objective C theme={null} StatsigOptions *options = [StatsigOptions new]; StatsigEnvironment *env = [StatsigEnvironment new]; env.tier = @"staging"; options.environment = env; [[Statsig sharedInstance] initializeWithSDKKey:@"client-sdk-key" options:options completion:^(NSError * _Nullable error) { if (error != nil) { NSLog(@"Error %@", error); } }]; ``` ```swift Synchronous (Swift) theme={null} import StatsigOnDeviceEvaluations // (optional) Configure the SDK if needed let opts = StatsigOptions() opts.environment.tier = "staging" let specs: NSString = "..." // JSON string of your configurations let error = client.initializeSync("client-sdk-key", initialSpecs: specs) if let err = error { print("Error \(err)") } ``` It is possible to configure the SDK to use cached values if they are newer than the local file. This can be useful if you ship your app with a local file, but would like it to only be used for the first session. In the following example, the SDK will only use initialSpecs if there is no cache or if the cache is older than initialSpecs. ```swift theme={null} let options = StatsigOptions() options.useNewerCacheValuesOverProvidedValues = true client.initializeSync( "client-sdk-key", initialSpecs: specs, options: options ) ``` You can get a copy of your current specs data by visiting: `https://api.statsigcdn.com/v1/download_config_specs/client-{YOUR_SDK_KEY}.json` ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's check a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. ```swift Swift theme={null} // Simple Pass/Fail check let isPassing: Bool = Statsig.shared.checkGate("my_gate", user) // or, the verbose FeatureGate check let gate = Statsig.shared.getFeatureGate("my_gate", user) print(gate.evaluationDetails.reason) // "Network" | "Cache" | "Unrecognized" let isPassing: Bool = gate.value ``` ```objective-c Objective C theme={null} BOOL isPassing = [[Statsig sharedInstance] checkGate:@"my_gate" forUser:user]; ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, **Dynamic Configs** can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```swift theme={null} let config = Statsig.shared.getDynamicConfig("my_dynamic_config", user) let name: String? = config.value["product_name"] as? String let price: Double? = config.value["price"] as? Double ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```swift theme={null} // Getting values via getLayer let layer = Statsig.shared.getLayer("my_layer", user) let name: String? = layer.getValue(param: "product_name", fallback: "Unknown") as? String // or, using getExperiment let experiment = Statsig.shared.getExperiment("my_experiment", user) let name: String? = experiment.value["product_name"] as? String let price: Double? = experiment.value["price"] as? Double ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API for the event, and you can additionally provide some value and/or an object of metadata to be logged together with the event: ```swift theme={null} let event = StatsigEvent( eventName: "add_to_cart", value: "SKU_1234", metadata: [ "price": "9.99", "item_name": "CoolProduct" ] ) Statsig.shared.logEvent(event, user) ``` ### Code Examples Working sample apps are available in the repository: * [Swift & Objective C Examples](https://github.com/statsig-io/swift-on-device-evaluations-sdk/tree/main/Sample/App/Examples/OnDeviceEvaluations) Included are both Swift and Objective C uses. ## Statsig User You need to provide a StatsigUser object to check/get your configurations. You should pass as much information as possible in order to take advantage of advanced gate and config conditions. Most of the time, the `userID` field is needed in order to provide a consistent experience for a given user (see [logged-out experiments](/guides/first-device-level-experiment) to understand how to correctly run experiments for logged-out users). Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Once the user logs in or has an update/changed, make sure to call `updateUser` with the updated `userID` and/or any other updated user attributes: ```swift Swift theme={null} let user = StatsigUser( userID: "a-user", customIDs: ["EmployeeID": "an-employee"], email: "user@statsig.io", ip: "58.84.239.246", userAgent: "Mozilla/5.0 (iPad; CPU OS 13_4_1....", country: "NZ", locale: "en_NZ", appVersion: "3.2.1", custom: ["Level": "9001"], privateAttributes: ["SensitiveInfo": "shhh"] ) ``` ```objective-c Objective C theme={null} StatsigUser *user = [StatsigUser userWithUserID:@"a-user"]; user.customIDs = @{ @"EmployeeID": @"an-employee" }; user.email = @"user@statsig.io"; user.ip = @"58.84.239.246"; user.userAgent = @"Mozilla/5.0 (iPad; CPU OS 13_4_1...."; user.country = @"NZ"; user.locale = @"en_NZ"; user.appVersion = @"3.2.1"; [user.custom setString:@"9001" forKey:@"Level"]; [user.privateAttributes setString:@"shhh" forKey:@"SensitiveInfo"]; ``` ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ### Setting a Global User To avoid needing to pass the user object to every single evaluation call, you can set a global user. This user will be used for all evaluations unless otherwise specified. ```swift theme={null} Statsig.shared.setGlobalUser(myGlobalUser) Statsig.shared.checkGate("my_gate") // <- Will use myGlobalUser Statsig.shared.checkGate("my_gate", StatsigUser(userID: "user-123")) // <- Will NOT use myGlobalUser ``` ## Statsig Options You can configure certain aspects of the SDKs behavior by passing a StatsigOptions object during initialization. The maximum number of events to batch before flushing logs to the server. How frequently to flush queued logs. The API where all events are sent. The API used to fetch the latest configurations. An object you can use to set environment variables that apply to all of your users in the same session and will be used for targeting purposes. ## Lifecycle & Advanced Usage ## Shutting Statsig Down In order to save users' data and battery usage, as well as prevent logged events from being dropped, we keep event logs in client cache and flush periodically. Because of this, some events may not have been sent when your app shuts down. To make sure all logged events are properly flushed or saved locally, you should tell Statsig to shutdown when your app is closing: ```swift Swift theme={null} Statsig.shared.shutdown { err in if let err = err { print("An error occurred during Statsig shutdown: \(err)") } else { print("Statsig shutdown successfully") } } ``` ```objective-c Objective C theme={null} [[Statsig sharedInstance] shutdownWithCompletion:^(NSError * _Nullable error) { if (error != nil) { NSLog(@"An error occurred during Statsig shutdown: %@", error); } else { NSLog(@"Statsig shutdown successfully"); } }]; ``` ## Post Init Syncing ### From Network By default, the SDK will only sync during initialization. If you would like to re-sync after initialization, you can call the `Statsig.update` method. This will trigger a network call to fetch the latest changes from the server. ```swift theme={null} Statsig.shared.update { err in if let err = err { print("Statsig update error: \(err)") } } ``` ### From a Local File If you maintain your own copy of the "specs" json, you can pass it in to the update with `Statsig.updateSync()`. This will skip the network call and use the provided specs instead. ```swift theme={null} let result = Statsig.shared.updateSync(updatedSpecs: myJsonData) ``` ### Scheduled Polling If you would like the SDK to regularly poll for updates, you can start the polling task with `Statsig.scheduleBackgroundUpdates()`. This will call `Statsig.update` internally, hitting the network and pulling down the latest changes. ```swift theme={null} let pollingTask = Statsig.shared.scheduleBackgroundUpdates() // Defaults to 1 hour interval // or, specify a custom interval let pollingTask = Statsig.shared.scheduleBackgroundUpdates(intervalSeconds: 300) // and, if you need to cancel it later pollingTask?.cancel() ``` ## Local Overrides It is possible to override the values returned by the Statsig SDK. This can be useful in unit testing or for enabling features for local development. To get setup with local overrides, you can pass an instance of `LocalOverrideAdapter` to the SDK via the `StatsigOptions` object. It is possible to write your own override adapter. You can implement the [`OverrideAdapter`](https://github.com/statsig-io/swift-on-device-evaluations-sdk/blob/main/Sources/StatsigOnDeviceEvaluations/OverrideAdapter.swift) protocol and pass that in instead. ```swift Swift theme={null} let user = StatsigUser(userID: "a-user") let overrides = LocalOverrideAdapter() // Override a gate overrides.setGate(user, FeatureGate.create("local_override_gate", true)) // Override a dynamic config (Similar for Layer and Experiment) overrides.setDynamicConfig(user, DynamicConfig.create("local_override_dynamic_config", ["foo": "bar"])) let opts = StatsigOptions() opts.overrideAdapter = overrides Statsig.shared.initialize(YOUR_SDK_KEY, options: opts) { _ in let gate = Statsig.shared.getFeatureGate("local_override_gate", user) print("Result: \(gate.value) (\(gate.evaluationDetails.reason))") } ``` ```objective-c Objective C theme={null} StatsigUser *user = [StatsigUser userWithUserID:@"a-user"]; LocalOverrideAdapter *overrides = [LocalOverrideAdapter new]; // Override a gate [overrides setGateForUser:user gate:[FeatureGate createWithName:@"local_override_gate" andValue:true]]; // Override a dynamic config (Similar for Layer and Experiment) [_overrides setDynamicConfigForUser:user config:[DynamicConfig createWithName:@"local_override_dynamic_config" andValue:@{@"foo": @"bar"}]]; StatsigOptions *options = [StatsigOptions new]; options.overrideAdapter = overrides; [[Statsig sharedInstance] initializeWithSDKKey:YOUR_SDK_KEY options:options completion:^(NSError * _Nullable error) { FeatureGate *gate = [[Statsig sharedInstance] getFeatureGate:@"local_override_gate" forUser:user options:nil]; NSLog(@"Result: %d (%@)", gate.value, gate.evaluationDetails.reason); }]; ``` ## FAQs See the guide on [device level experiments](/guides/first-device-level-experiment). ## Additional Resources * [On-Device Evaluation SDK Overview](/client/onDevice) * [Client Keys with Server Permissions](/access-management/api-keys#client-keys-with-server-permissions) * [Debugging SDK Evaluations](/sdk/debugging) # AI Governance, Security & Privacy Source: https://docs.statsig.com/compliance/ai_governance_security_privacy Reference for Statsig's AI governance, security, and privacy practices, including data handling, retention, and customer obligations for AI features. Trust, security, and privacy are central to Statsig's operations. Your data remains confidential, secure, and owned by you across the Statsig platform. ## Governance The AI features on the Statsig platform use your data to provide additional insights, analysis, and solutions for your review. Examples of our AI features include Knowledge Graph, hypothesis advisor, and suggested metrics. By design, your data is kept separate in our production environment from other customer data. Statsig does not mix or process data from different customers together. This means we do not expose your data to other customers. Your data is not used to build or develop any AI models unless you provide explicit written consent. You own the data you provide and control which of your internal sources are connected to the Statsig platform. You also control who has access to the Statsig platform within your organization. Information on single sign-on on the Statsig platform can be found [here](/access-management/sso/overview). ## Security Whether you are sending aggregated metrics, custom attributes, or hashed identifiers or connecting parts of your code base, Statsig understands the importance of security. Because security starts at design, Statsig's software development lifecycle ensures we design and build security into our offerings at inception. We embrace zero trust and defense in-depth approaches to guide our overall security program. We have also implemented layered security controls across our endpoints, infrastructure, networks, and applications. Statsig uses strong, industry-standard security practices and cryptography to protect your data. This includes using AES-256 encryption at rest and TLS 1.2 or higher in transit. We use automated alerts and manual investigation processes to address any suspicious activity. Our systems also undergo regular risk assessments and audits, including by independent third parties to ensure adherence to high security standards. In addition, Statsig maintains a SOC 2 Type II certification. For more information on Statsig's security practices, you can visit [Security at Statsig](https://www.statsig.com/trust/security). ## Privacy Statsig's data protection practices are designed to support your compliance with GDPR, CCPA, and other applicable privacy laws across our platform. For cross border data transfers, Statsig complies with the EU-US Data Privacy Framework, the UK Extension to the EU-US Data Privacy Framework, and the Swiss-US Data Privacy Framework. Statsig also provides a [Data Processing Addendum](https://www.statsig.com/legal/online-dpa) to support its customers' data handling requirements. These privacy protections also extend to all AI features. AI features on the Statsig platform utilize third-party large language models (LLMs). Data processed through these LLMs is not retained, accessed, or used by the underlying third-party model providers. By design, your data is also never shared with other customers. Further, Statsig's subprocessors are only permitted to use the data as directed by Statsig and in accordance with our contractual commitments. A list of Statsig's subprocessors is available [here](https://www.statsig.com/legal/subprocessors/). # Data Privacy for Mobile Source: https://docs.statsig.com/compliance/data_privacy_for_mobile Data privacy considerations when using Statsig mobile SDKs, including handling of identifiers, opt-out mechanisms, and platform-specific requirements. ## General ### What data does Statsig collect from users of my app? Statsig collects only the data that you configure to be sent to Statsig. This is typically the occurrence of feature flag evaluations (Feature Gates), experiment exposures, and custom events you log with the SDK. ### Does that data include any personally identifiable information (PII)? By default, Statsig uses randomly generated IDs as described below. You can also augment data sent to Statsig with additional context and meta data, including user names, email addresses, or custom attributes. This data, alone or in combination with other data, may constitute PII if it identifies, directly or indirectly, an individual. ### Does Statsig use the device ID to identify a user? No. Statsig on Mobile doesn't use device ID such as Secure.ANDROID\_ID and advertisingIdentifier on iOS. Instead, it uses a StableID which is randomly generated per device, per app, and per installation, and therefore can't be used to identify a single device across application installations. When an application is reinstalled, a new StableID is generated. Since these IDs are solely used to provide approximate statistical information as part of the application monitoring service, it serves that purpose. It's required, for example, for Crash Free Session and User Rates, as well as to indicate the impact of issues based on number of events vs affected users. ### What does Statsig do with the data it collects? Statsig processes the data you send to it to provide our feature flag management, experimentation, and analytics services to you. ## Apple App Store ### Do I need to disclose the use of Statsig in App Store Connect on the Apple App Store? Yes, Statsig is a third-party partner whose code (SDKs) you integrate in your app that collects data from users of your app. ### What do I need to disclose to Apple? You would need to disclose all types of data you are collecting through your app, including data you are sending to Statsig. This could include "Contact Info" or "Identifiers" if you provide those in the StatsigUser object, but don't forget to include any other categories of data that you are collecting or have configured the SDKs to send to Statsig. ### How does Statsig use my data? The standard data use cases for Statsig would be "Analytics" and "App Functionality", but you would also need to disclose to Apple any other ways in which you or your app use data you are collecting. ### Does Statsig use my data to track users? Statsig does not use your data to track users. However, if you or your other third-party partners are tracking users, you would still need to disclose this to Apple. ### Does Statsig use the Advertising Identifier (IDFA)? No. Statsig doesn't require IDFA. ## Google Play ### Does Statsig collect any PII from children? If your app is targeted to children and you configure Statsig to collect PII, then Statsig would collect the elements that you've designated. You would remain responsible for obtaining any appropriate parental consents with respect to the PII you collect from your users and subsequently send to Statsig. You would also remain responsible for declaring your app's target age group to Google Play. ### Do Statsig SDKs cause my app to contain ads? No. Statsig does not cause your app to contain any ads. ### What do I need to disclose to Google Play? You would need to disclose and/or include in your app privacy policy all types of data you are collecting through your app, including data you are sending to Statsig. ## Privacy Controls ### What device and user metadata does Statsig automatically collect? The Statsig SDKs automatically collect the following metadata for targeting and analytics purposes: **iOS SDK collects:** * appIdentifier: Your app's bundle identifier * appVersion: Your app's version * deviceModel: The device model (e.g., iPhone14,3) * deviceOS: The operating system (iOS) * language: The user's preferred language * locale: The user's locale identifier * sdkType: The type of SDK (ios-client) * sdkVersion: The version of the Statsig SDK * sessionID: A randomly generated UUID for the current session * stableID: A persistent device identifier (see below) * systemVersion: The iOS version * systemName: The system name (iOS) **Android SDK collects:** * appIdentifier: Your app's package name * appVersion: Your app's version name * deviceModel: The device model (e.g., Pixel 7) * deviceOS: The operating system (Android) * locale: The user's locale * language: The user's language * sdkType: The type of SDK (android-client) * sdkVersion: The version of the Statsig SDK * sessionID: A randomly generated UUID for the current session * stableID: A persistent device identifier (see below) * systemVersion: The Android API level * systemName: The system name (Android) ### How can I limit the metadata collected by Statsig? Both iOS and Android SDKs provide the `optOutNonSdkMetadata` option to limit the collection of device-specific information: **iOS SDK:** ```swift theme={null} let options = StatsigOptions() options.optOutNonSdkMetadata = true Statsig.start(sdkKey: "client-sdk-key", options: options) ``` **Android SDK:** ```kotlin theme={null} val options = StatsigOptions(optOutNonSdkMetadata = true) ``` When `optOutNonSdkMetadata` is enabled, only the following core SDK metadata is included: * sdkType: The type of SDK * sdkVersion: The version of the Statsig SDK * sessionID: A randomly generated UUID for the current session * stableID: A persistent device identifier All device-specific information (appIdentifier, appVersion, deviceModel, deviceOS, locale, language, systemVersion, systemName) is excluded from logs and targeting. ### How does StableID work? The StableID is a persistent identifier that Statsig uses to provide consistent user experiences and analytics: **iOS Implementation:** * The StableID is stored in UserDefaults with the key "com.Statsig.InternalStore.stableIDKey" * When first generated, it's created as a random UUID * It persists across app launches but is regenerated when the app is reinstalled * It can be overridden via the StatsigOptions.overrideStableID parameter **Android Implementation:** * The StableID is stored in SharedPreferences * When first generated, it's created as a random UUID * It persists across app launches but is regenerated when the app is reinstalled * It can be overridden via the StatsigOptions.overrideStableID parameter The StableID is not shared across different apps or websites and cannot be used to track users across different applications or platforms. ### How can I prevent sending sensitive user data to Statsig? Use the `privateAttributes` field for sensitive data that should be used for targeting but not logged: **iOS SDK:** ```swift theme={null} let user = StatsigUser( userID: "user-123", email: nil, // Not included at top level to keep private privateAttributes: ["email": "user@example.com"] // Used for evaluation but not logged ) ``` **Android SDK:** ```kotlin theme={null} val user = StatsigUser("user-123") user.privateAttributes = mapOf("email" to "user@example.com") ``` These attributes are sent to Statsig servers during initialization for feature flag and experiment evaluation, but are removed before sending any event logs to Statsig servers. This means the attributes can be used for targeting users with specific features or experiments, but won't appear in your analytics data. For more comprehensive privacy controls, consider using [Client Bootstrapping](/client/concepts/initialize#bootstrapping-overview) to generate all assignments locally on your server, eliminating the need to send any user attributes from the client device to Statsig. # User Data Deletion Requests API Source: https://docs.statsig.com/compliance/user_data_deletion_requests How to submit and process user data deletion requests in Statsig to comply with GDPR, CCPA, and other privacy regulations across SDK and console data. User data deletion requests are for Enterprise contracts only. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you need to enable Enterprise features as you use Statsig. We currently only support deleting data for unit type `user_id`. GDPR and similar laws may require that you delete a user's data when they request it. To support deletion requests on our side, we have built this API for you to call. ### How to use the API All requests must include the STATSIG-API-KEY field in the header. The value should be a SERVER API Key which can be created in the Project Settings on console.statsig.com/api\_keys. ## Sending Requests Data deletion requests take the following parameters: * `unit_type`: The unit type that corresponds to the IDs that you would like to deleted. Currently, we only support `user_id`. Support for other ID types will be added in the future. * `ids`: A comma separated list containing the IDs you would like to delete data for * `delimiter` (optional): In the case that your IDs contain commas, you can use a different delimiter to separate the IDs, and pass that delimiter here * `request_id` (optional): If you store a request ID on your side that you would like us to use, you can pass it in, and it will be used to track the request. If left unset, we will provide an ID for you to use in the response. Note that it must be a unique ID if you are supplying your own, otherwise the request will fail with a 400 code. ```bash theme={null} curl \ --header "statsig-api-key: " \ --header "Content-Type: application/json" \ --request POST \ --data '{"unit_type": "user_id", "ids": "1,2,3", "request_id": "test_request_1"}' \ "https://api.statsig.com/v1/delete_user_data" ``` Response: `{"request_id":"test_request_1"}` ## Checking on Deletion Status Using the request ID, you can check on the status of a deletion. Input: * `request_id`: The ID of the request Output: * `COMPLETE` if the data has been deleted * `PENDING` if the data is still pending deletion * `UNKNOWN` if this is an invalid request ID ```bash theme={null} curl \ --header "statsig-api-key: " \ --header "Content-Type: application/json" \ --request POST \ --data '{"request_id": "test_request_1"}' \ "https://api.statsig.com/v1/get_delete_user_data_request_status" ``` Response: `PENDING` ## SLA Data deletion requests are not handled synchronously. We batch requests and do mass deletions periodically. We guarantee that data will be deleted within 30 days of receiving a request. ## Important Notes * Once a data deletion request has been submitted, it cannot be deleted * Once data has been deleted, we no longer store the set of IDs that we deleted data for (since this is considered personally identifiable information in some cases) * The User Data Deletion Requests API deletes data from the Statsig platform only. To delete data from the Amplitude platform, use Amplitude's [User Privacy API](https://amplitude.com/docs/apis/analytics/user-privacy). # Console API Overview Source: https://docs.statsig.com/console-api/introduction Introduction to the Statsig Console API for programmatically managing feature gates, experiments, dynamic configs, metrics, and project settings. The "Console API" is the CRUD API for performing the actions offered on console.statsig.com without needing to go through the web UI. If you have any feature requests, drop on in to our [slack channel](https://www.statsig.com/slack) and let us know. ## Base URL `https://statsigapi.net` ## Authorization All requests must include the **STATSIG-API-KEY** field in the header. The value should be a **Console API Key** which can be created in the Project Settings on [console.statsig.com/api\_keys](https://console.statsig.com/api_keys) ## Rate Limiting Mutation requests (POST/PATCH/PUT/DELETE) to the Console API are limited to \~ 100 requests / 10 seconds and \~ 900 requests / 15 minutes, per project. ## API Version The Console API is versioned. Each version is guaranteed to not break existing usage; each new version introduces breaking changes. There is only one version: `20240601`. The [OpenAPI spec](https://api.statsig.com/openapi/20240601.json) for this API version is kept up-to-date. Pass the version in the **STATSIG-API-VERSION** field in the header. For now, this is optional; in the future, this will be required. # Control Panel Source: https://docs.statsig.com/control-panel/overview Use the Statsig Control Panel to track, search, and manage feature gates, experiments, dynamic configs, and rollouts across your entire project. Control Panel is a surface to measure and track you and your teams' features that are deployed and measured with Statsig. This surface is similar to the standard gate/experiment tables in Statsig, and enables quickly tabbing between filters on top of the objects used to deploy and measure changes - configs, experiments, and gates - and quickly measure them, make or review ship decisions, and manage development flows like overrides. ## Using Control Panel * Configure Sections in the left hand column. These specify different filters and display settings - e.g. "My Changes", or "My Team's Changes". * View results inline. Configs and experiments have a summary of metric movements, which can be hovered to see a table of all observed metric movements. Gates and Dynamic Configs can be opened to see results per-rule. * Manage configs. You can see or set your override status from control panel, and additionally killswitch features in case there's unexpected behaviors This feature is in Beta. Please reach out if you or your team would like to try it! # Athena Ingestion Source: https://docs.statsig.com/data-warehouse-ingestion/athena Configure Statsig data warehouse ingestion from Amazon Athena, including authentication, scheduled queries, and mapping to events and properties. ## Overview To set up connection with Athena, Statsig needs the following * Region * Granting Athena Access Permissions to a Statsig-owned Service Account In place of granting Athena Access Permissions to a Statsig-owned Service Account, you can also provide the following: * IAM User Access Key * IAM Secret Access Key The above IAM User will need to be given permissions to query from Athena. Here's a sample policy with required permissions to access Athena: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "athena:StartQueryExecution", "athena:GetQueryExecution", "athena:GetQueryResults", "athena:CreatePreparedStatement", "athena:DeletePreparedStatement", "athena:GetPreparedStatement", "athena:GetQueryResultsStream", "s3:GetObject", "s3:ListBucket", "s3:GetBucketLocation", "glue:GetTable", "glue:GetDatabase" ], "Resource": [ "arn:aws:athena:*::workgroup/*", "arn:aws:glue:::*" ] } ] } ``` # BigQuery Source: https://docs.statsig.com/data-warehouse-ingestion/bigquery Configure Statsig data warehouse ingestion from Google BigQuery, including authentication, scheduled queries, and mapping to events and properties. ## Overview To set up connection with BigQuery, we need the following: * Granting Permissions to a Statsig-owned Service Account * Your BigQuery Project ID Start by enabling the BigQuery source under Metrics -> Ingestion -> Add Source. ## Grant Permissions to Statsig's Service Account You need to grant some permissions for Statsig from your Google Cloud console in order for us to access your BigQuery data. 1. In your BigQuery's [IAM & Admin settings](https://console.cloud.google.com/iam-admin/), add the Statsig service account you copied in the Statsig Console as a new principal for your project, and give it the following roles: * `BigQuery User` BigQuery IAM permissions configuration
2. Navigate to your [BigQuery SQL Workspace](https://console.cloud.google.com/bigquery), choose the dataset, click on "+ Sharing" -> "Permissions" -> "Add Principal" to give the same Statsig service account "BigQuery Data Viewer" role on the dataset. BigQuery dataset permissions setup Now the service account should have the required permissions to export data from this dataset. ## BigQuery Project ID Find your BigQuery Project ID below 1. Click on your Project Dropdown inside your Cloud Console. Frame 4 2. Copy and paste relevant Project ID from the modal pop-up. Frame 5 # Data Mapping Source: https://docs.statsig.com/data-warehouse-ingestion/data_mapping Map columns from your data warehouse to Statsig events, user IDs, and properties so ingested data flows into metrics and experiment analysis correctly. ## Overview Statsig requires certain data schema in order for proper processing. We support 3 different types of datasets to be ingested into our platform: 1. Custom Events 2. Precomputed Metrics 3. Exposure Events During setup, we will ask you to map columns in your data output to fields Statsig expects and run a small sample query to make sure that there aren't any basic issues with data types, the mapping, or the base query itself. Please note that we will cast fields into the appropriate type. For example, Statsig accepts string IDs, but it is okay to leave an ID field as an integer. *** ### Custom Events Events that are emitted by your application to measure the ongoing impact of your features and experiments. #### Required | Column | Description | Format/Rules | | ----------- | -------------------------------------- | --------------------------------------------------------------------------------- | | timestamp | The unix time your event was logged at | BIGINT. please cast timestamps into epoch time in seconds | | event\_name | The name of the event | STRING/VARCHAR. Not null. Length \< 128 characters | | unit\_id | Unique unit identifier | STRING/VARCHAR. User ID, Stable ID, etc. The same event row can have multiple IDs | #### Optional | Column | Description | Rules | | --------------- | -------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | event\_value | The value of the event | STRING/VARCHAR. Length \< 128 characters. Statsig will detect numeric values | | event\_metadata | Metadata columns about the event | (MANY) STRING:STRING. Statsig will generate a metadata json field from however many metadata columns you provide | | metadata\_json | Metadata json about the event | JSON STRING. Statsig will unpack this json 1 level deep. Nested values will be stored as strings |
An example dataset for events might look like this: | unit\_id | visit\_id | event | timestamp | value | metadata\_blob | user\_type | | -------- | --------- | -------- | ---------- | ----- | ---------------------------------------------------------- | ----------------- | | 331444 | | click | 1676484875 | | `{"click_target": "exit_details_button"}` | power\_user | | 331444 | | click | 1676484860 | | `{"click_target": "open_details_button"}` | power\_user | | 265113 | | click | 1676484333 | | `{"click_target": "button", "button_color": "green"}` | churn\_risk\_user | | 445332 | aeeer43d | visit | 1676483821 | | `{"page": "landing_page"}` | new\_user | | 224448 | | checkout | 1676482222 | 33.22 | `{"product_id": "11eefj", "product_category": "clothing"}` | power\_user | Note that: * One user can send multiple of the same event, with or without any changes in metadata. Statsig will aggregate these together. * You can send metadata in both of a json-formatted (only one-level deep) string, and/or pull in fields from columns. You can use metadata and values to generate custom metrics in the console, like sum(value) where "product\_category"="clothing". * You can send multiple IDs on a single event. For example, the visit above would could for both user and visit level metrics/experiments. During the mapping flow you tell us which unit types your different IDs correspond to in statsig. ***
### Precomputed Metrics Precomputed metrics are a powerful way to leverage statsig for experiment results. Use these to send complex metrics and metrics that require delays due to attribution windows or long baking periods. Precomputed metrics in statsig are expected to be calculated at a user-day granularity. #### Required | Column | Description | Format/Rules | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | unit\_id | The unique user identifier this metric is for. This might not necessarily be a user\_id - it could be a custom\_id of some kind | STRING | | id\_type | The id\_type the unit\_id represents. | STRING. A valid ID type from your project | | date | Date of the daily metric | DATE or ISOFORMATTED STRING. The date of the metric value for the unit\_id provided | | metric\_name | The name of the metric | STRING (Not null). Length \< 128 characters | | metric\_value | A numeric value for the metric | DOUBLE/NUMERIC. Metric value
OR
Both of numerator/denominator need to be provided for Statsig to process the metric. | | numerator | Numerator for metric calculation | DOUBLE/NUMERIC. If present along with a denominator in any record, the metric will be treated as ratio and only calculated for users with non-null denominators. | | denominator | Denominator for metric calculation | DOUBLE/NUMERIC. If present along with a numerator in any record, the metric will be treated as ratio and only calculated for users with non-null denominators. |
An example dataset for metrics might look like this: | unit\_id | unit\_type | date | metric\_name | metric\_value | numerator | denominator | | -------- | ---------- | ---------- | ------------------------- | ------------- | --------- | ----------- | | 331444 | user\_id | 2023-02-13 | clicks | 2 | | | | aeeer43d | visit\_id | 2023-02-13 | visits | 1 | | | | 224448 | user\_id | 2023-02-13 | checkout\_rate | | 2 | 15 | | 224448 | user\_id | 2023-02-13 | clothing\_checkout\_value | 33.22 | | | Note that: * In this dataset, unit types are in different rows from each other * Metrics can either have a value or a numerator/denominator pair. We will calculate any metric with numerator/denominator pair as a ratio metric. Ratio takes priority over value; if you provide all 3 fields, we will assume it is a ratio metric. * For users with null values, we will infer 0 for metric\_value, and exclude null value users for ratio metrics. ***
### Exposure Events Exposure event import is deprecated. If this is an important use case, see [Statsig Warehouse Native](https://www.statsig.com/blog/announcing-statsig-warehouse-native), available to Enterprise Customers Exposure events are generated by your assignment tool, when it assigns your users to a certain variant of an experiment (e.g., show ad vs. hide ad). #### Required | Column | Description | Format/Rules | | ---------- | ------------------------------------------- | --------------------------------------------------------------------------------- | | timestamp | The unix time your event was logged at | BIGINT. please cast timestamps into timezoneless unix time | | experiment | Your experiment identifier | STRING/VARCHAR. Not null. Length \< 128 characters | | group\_id | Unique identifier for the experiment groups | STRING/VARCHAR. Not null. | | unit\_id | Unique user identifier | STRING/VARCHAR. User ID, Stable ID, etc. The same event row can have multiple IDs | #### Optional | Column | Description | Rules | | -------------- | ----------------------------- | ------------------------------------------------------------------------------------------------ | | metadata\_json | Metadata json about the event | JSON STRING. Statsig will unpack this json 1 level deep. Nested values will be stored as strings | # Databricks Source: https://docs.statsig.com/data-warehouse-ingestion/databricks Configure Statsig data warehouse ingestion from Databricks, including authentication, scheduled queries, and mapping to events and user properties. ## Overview To set up connection with Databricks, Statsig needs the following * API Key * Server Hostname * HTTP Path We can use any cluster in your project to connect to your data, but we recommend using a databricks [SQL warehouse/endpoint](https://docs.databricks.com/sql/admin/sql-endpoints.html) so that the cluster does not need to spin up for every pull. ### API Key You can generate a new API key by going to "User Settings" in your Databricks console. There, you should be able to generate a new token as shown below. databricks info You can also use a personal access token for a service principal. Generate one by following the steps in the doc [here](https://docs.databricks.com/en/administration-guide/users-groups/service-principals.html#manage-personal-access-tokens-for-a-service-principal). ### Server Hostname & HTTP Path You can find your Server Hostname and HTTP Path in your Databricks console by going to your specific cluster, navigating to the "Configuration" tab and expanding the "Advanced options." credentials # FAQ & Troubleshooting Source: https://docs.statsig.com/data-warehouse-ingestion/faq Frequently asked questions about Statsig data warehouse ingestion, including supported warehouses, scheduling, costs, and troubleshooting connection issues. ## What IP addresses will Statsig access data warehouses from? Statsig currently accesses data warehouses from both the Statsig console service and Statsig data pipelines. If your data warehouse is IP protected, please refer to the [Statsig IP range documentation](/infrastructure/statsig_ip_ranges) for IPs to allowlist. Reach out to us on slack if you have any issues. ## Does event data from ingestion count towards User Accounting Metrics? No, event data from ingestions does not count towards Statsig's User Accounting Metrics such as DAU or Retention. Customers typically send Statsig a subset of their events, which could result in multiple competing values for "fact" data such as daily active users in your Statsig project. Statsig recommends sending your own precomputed metric for DAU or as a daily event per user (1 'daily\_active' event if a user was active that day). ## How long does the data take to load? For most customers, data ingestions should take 1-2 hours to materialize in the Statsig console after the ingestion is scheduled. Note that the schedule is in PST, and not PDT, so depending on daylight savings time ingestions may start an hour later or earlier. ## Does Statsig load data incrementally every day? Statsig loads data incrementally every day. Statsig also monitors data over several follow-up windows for up to two weeks, and reloads data for a given day if it has changed more than 1%. ## Can I ingest multiple metrics in the same scheduled ingestion? Yes, you can ingest multiple metrics (and event types) in the same scheduled ingestion. Statsig enables you to run a SQL query against your data warehouse cluster to join multiple tables to generate a view with all your precomputed metrics. You can use this as the source view for your scheduled data ingestion and import multiple metrics at the same time. For example, your dataset could import both `metric-1` and `metric-2`, with `metric-2` including multiple units of analysis, say user\_id and alphabet\_id. ## How does missing metric values affect experiment calculations? If the metric value is unavailable for a given user on a given day, Statsig takes it to be `zero` for additive metrics such as counts and sums. For metrics that depend on a user "participating" in the metric, say conversion rate, the user is excluded. Note that additive metrics typically have a single `metric_value` column in the ingested data, while ratio (participating) metrics typically have separate `numerator` and `denominator` columns. ## Does Statsig notify about ingestion status? Statsig shows the status of your daily ingestion on the console under the **Ingestions** tab. Statsig reports three kinds of ingestion statuses: * ingestion succeeded for a given day * ingestion succeeded for a given day, but no data was detected * ingestion failed for a given day Statsig also sends email notifications with these status updates to the Statsig user who set up the ingestion. This user can also enable Slack direct message (DM) notifications to themselves in their Statsig [Account Settings](https://console.statsig.com/account_notifications). ## Does Statsig automatically backfill data? Statsig looks back at data for 3 days from the initial ingestion to see data has changed (>5% increase in the number of rows) to automatically trigger a backfill. Outside of this window, we expect the customer to trigger backfill for the range of dates. # Data Warehouse Ingestion Source: https://docs.statsig.com/data-warehouse-ingestion/introduction Introduction to Statsig data warehouse ingestion, which imports events and metrics from Snowflake, BigQuery, Redshift, and other warehouses on a schedule. Slide 4_3 - 2 ## Introduction Statsig Cloud can directly ingest data from your Data Warehouse. This lets you send raw events and pre-computed metrics for tracking and experimental measurement. We currently support ingestion from the following providers: We support you making multiple data connections to your project, but only support a single export connection. 1. [BigQuery](/data-warehouse-ingestion/bigquery) 2. [Redshift](/data-warehouse-ingestion/redshift) 3. [Snowflake](/data-warehouse-ingestion/snowflake) 4. [Databricks](/data-warehouse-ingestion/databricks) 5. [Synapse](/data-warehouse-ingestion/synapse) 6. [S3](/data-warehouse-ingestion/s3) 7. [Athena](/data-warehouse-ingestion/athena) Warehouse Native users: You're viewing the Cloud docs for this page. If your project is configured as [Statsig Warehouse Native](/statsig-warehouse-native/introduction/), your data should already be available if you completed the [quickstart](../statsig-warehouse-native/guides/quick-start/). ### How it works In Statsig console, you can: 1. Set up connection to your data warehouse 2. Query your data warehouse for appropriate data 3. Map your data fields to Statsig's expected schema 4. Bulk ingest & schedule future ingestions Ingestion is set up on a daily schedule. Statsig will run a query you provide on your data warehouse, download the result set, and materialize the results into your console the same as those that came in through the SDK. If data lands late or is updated, Statsig will detect this change and reload the data for that day (details below). ### How to Begin Data Ingestion To begin ingestion from a Data Warehouse: 1. Go to your Statsig Console 2. Navigate to Data tab on the side navigation bar 3. Go to the "Ingestion" tab Statsig Ingestions page prompting you to connect a data warehouse You will be required to set up connections with necessary credentials, and map your data fields to the fields Statsig expects to ingest. Please refer to the warehouse-level setup documentation for more information on setup. ### Connection Flow See the docs sidebar to find the documentation for the data warehouse of your choice. Upon connection, you will provide a SQL query to generate a view via data for Statsig to ingest. Query Event Data Columns interface showing SQL editor and expected columns panel ### Data Mapping After connecting and providing a SQL query, you'll map columns in your data output to fields Statsig expects. We'll run a small sample query to ensure there are no basic issues with data types. To process data correctly, Statsig requires each ingestion to include columns for unit\_id, event\_name, timestamp, and metadata. Event data column mapping workflow with required fields for timestamp, event name, and user ID See [here](/data-warehouse-ingestion/data_mapping) for more information. ### Scheduling Ingestion & Backfilling Statsig supports multiple schedules for ingestion. At the scheduled window, we will check if data is present in your warehouse for the latest date, and load if it exists. We will check the underlying source table for changes. For up to 3 days after initial ingestion, we will check for >5% changes in row counts and reload the data. We also support a user-triggered backfill. This could be useful if a specific metric definition has changed, or you want to resync data older than a few days. To change your ingestion schedule or start a backfill, click the ellipses at the end of the data connection and navigate to these menus. Reloading data and backfilling metrics and events is billed as any other [custom event](/metrics/raw-events#billing) Auto-generated **User Accounting Metrics** are not supported today for data warehouse ingestions. ### Troubleshooting Ingestions If any ingestion errors occur, Statsig will notify you in project and direct your to the Ingestions page. You can diagnose an error directly in Statsig by following the step-by-step triage flow. Common errors may include missing permissions and out-of-date credentials. ### API Triggered Ingestion (mark\_data\_ready) Enterprise customers can trigger ingestion for `metrics` or `events` using the statsig API. This will run your daily ingestion immediately after triggering, and can be helpful for companies whose data availability timing may vary day over day and want data to land as soon as possible in Statsig. This can be enabled by selecting "API Triggered" as your ingestion schedule - note that with this enabled, there will not be an automatic ingestion, but we will still re-sync data after the initial ingestion if we observe a change. To trigger ingestion, send a post request to the `https://api.statsig.com/v1/mark_data_ready_dwh` endpoint using your statsig API key. An example would be: ``` curl \ --header "statsig-api-key: " \ --header "Content-Type: application/json" \ --request POST \ --data '{"datestamps": "2023-02-20", "type": "events", "sources":["source1", "source2]}' \ "https://api.statsig.com/v1/mark_data_ready_dwh" ``` Parameters: * datestamps: Refers to the date of the data being triggered. * type: `metrics` or `events` * sources (only for multi-source ingestions): Array of strings representing the sources to trigger This is rate limited to once every two hours, and there may be a few minutes delay after triggering before status updates while compute resources are created. ### Frequently Asked Questions For frequently asked questions, see our [FAQ page](/data-warehouse-ingestion/faq). # Redshift Source: https://docs.statsig.com/data-warehouse-ingestion/redshift Configure Statsig data warehouse ingestion from Amazon Redshift, including authentication, scheduled queries, and mapping to events and properties. ## Overview To set up connection with Redshift, Statsig needs the following * Cluster Endpoint * Admin User Name * Admin User Password SHA256 passwords are not currently supported, please utilize MD5 to avoid issues. You can find this information in your aws console within your specific cluster, as shown in the image below. (Open image in new tab for a bigger image) AWS Redshift cluster details highlighting endpoint and admin user Admin user name and password will be used by Statsig to create a user with restricted access to query from your data warehouse. ## SSH Tunneling For Redshift connections, we also allow users to create an SSH tunnel into their Redshift cluster for a more secure and private access to the database. To enable access, Statsig requires: * SSH Host * SSH Port * SSH User Statsig will use this information to generate an SSH key. Please add this generated key to your `~/.ssh/authorized_keys` file on your SSH proxy machine to enable SSH tunneling. ### Custom User Privileges To create a custom user with specific privileges instead of using an admin user, run the following code in your Redshift cluster with your admin user. Replace `` and `` with your value, which you will copy over into our console. ```sql theme={null} # Create Statsig User CREATE USER WITH PASSWORD SYSLOG ACCESS UNRESTRICTED; # Give access to any Schemas that the Statsig User needs to read from GRANT USAGE ON SCHEMA to ; GRANT SELECT ON ALL TABLES IN SCHEMA to ; # Create a Schema for Statsig User to write temporary data to CREATE SCHEMA IF NOT EXISTS statsig_ingestion_staging; GRANT ALL ON SCHEMA TO ; ``` After running the script, input the `` and `` you created in our console, during Connection Set Up stage under the Advanced settings options. Statsig Redshift connection setup form showing cluster endpoint and credentials fields # S3 Source: https://docs.statsig.com/data-warehouse-ingestion/s3 Configure Statsig data warehouse ingestion from Amazon S3 buckets, including authentication, file format support, and mapping to events and properties. ## Overview To set up connection with S3, Statsig needs the following * Region * Bucket Name * Granting Bucket Read Access Permissions to a Statsig-owned Service Account You can find the regions and bucket name of your S3 bucket in your AWS console within your S3 Buckets overview page, as shown in the image below. (Open image in new tab for a bigger image) AWS S3 console showing bucket regions and names You will be given a Statsig owned IAM user that you'll need to grant S3 bucket permissions to. Statsig IAM user configuration interface The user will need read access permissions to your bucket, you can use the below bucket policy for your convenience, replacing STATSIG\_IAM\_USER and YOUR\_S3\_BUCKET. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::999689269917:user/STATSIG_IAM_USER" }, "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::YOUR_S3_BUCKET" }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::999689269917:user/STATSIG_IAM_USER" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::YOUR_S3_BUCKET/*" } ] } ``` ### S3 bucket Format For each dataset you're ingesting through S3, we expect a top level folder in the S3 bucket matching the name of the dataset (e.g metrics, events), with folders denoting each day of data. In each folder we expect parquet files with data corresponding to that day's import. See the following screenshot for a example folder structure. S3 bucket folder structure example ### S3 Export Permissions For exports, the user will need bucket-level permissions to list the bucket and retrieve its location, as well as object-level permissions to read, write, and delete objects (including managing multipart uploads). You can use the bucket policy below for convenience, replacing STATSIG\_IAM\_USER and YOUR\_S3\_BUCKET as needed. ``` { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::999689269917:user/STATSIG_IAM_USER" }, "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::YOUR_S3_BUCKET" }, { "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::999689269917:user/STATSIG_IAM_USER" }, "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:AbortMultipartUpload", "s3:ListBucketMultipartUploads", "s3:ListMultipartUploadParts" ], "Resource": "arn:aws:s3:::YOUR_S3_BUCKET/*" } ] } ``` # Snowflake Source: https://docs.statsig.com/data-warehouse-ingestion/snowflake Configure Statsig data warehouse ingestion from Snowflake, including authentication, scheduled queries, and mapping to events and user properties. ## Overview To set up connection with Snowflake, Statsig needs the following * Account Name * Database Name * Schema Name * Admin User Name * If authenticating via login credentials: * Admin User Password * If authenticating via key-pair authentication: * Private Key * Private Key Passphrase (Optional) Admin user name and password will be used by Statsig to create a user with restricted access to query from your data warehouse. If you don't want to use this, skip ahead [here](/data-warehouse-ingestion/snowflake#custom-user-privileges) ### Account Name For the Account Name field, please enter in the format of `..`. So, your information may look something like this: `xy12345.us-central1.gcp` To get this information navigate to bottom left in your Snowflake console, as shown in the picture below and copy the link URL: Snowflake console footer showing account identifier link The copied URL will look something like this: `https://xy12345.us-central1.gcp.snowflakecomputing.com` You can extract information from here to get the required fields for Account Name, which for this example would be `xy12345.us-central1.gcp`. Using `-` for Account Name For the Account Name field, you can also enter your Snowflake [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#format-1-preferred-account-name-in-your-organization), which typically takes the form `-`. To find the `` in the Snowflake console, click on your account profile (usually at the bottom left) to view account details as shown below. Snowflake account profile interface ### Database and Schema Name For each data type, provide the database/schema of the table(s) you will be ingesting from. Snowflake database explorer highlighting schema selection ### Key-Pair Authentication To set up key-pair authentication, first follow the [snowflake documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth) to generate the private and public keys, and then set the public key on the service user. The private key can then be provided here Private key authentication configuration interface ### Custom User Privileges To create a custom user with specific privileges instead of using an admin user, run the following code in your Snowflake worksheet that has sysadmin and securityadmin roles. Replace `` and `` with your value, which you will copy over into our console. ```sql theme={null} BEGIN; -- set up variable values to be used in statements later -- make sure to configure user_name and user_password with your own values SET user_name = ''; -- REPLACE WITH YOUR OWN VALUE SET user_password = ''; -- REPLACE WITH YOUR OWN VALUE SET role_name = 'STATSIG_ROLE'; -- CAN BE ANYTHING, BUT THE USER NEEDS TO -- HAVE THIS ROLE AND THE ROLE NEEDS ACCESS TO THE TABLES PER THE GRANTS BELOW -- change role to sysadmin for warehouse / database steps USE ROLE sysadmin; -- create a warehouse, database, schema and tables for Statsig CREATE OR REPLACE WAREHOUSE STATSIG_INGESTION WITH warehouse_size='XSMALL'; CREATE DATABASE IF NOT EXISTS STATSIG_STAGING; -- change current role to securityadmin to create role and user for Statsig's access USE ROLE securityadmin; -- create role for Statsig CREATE ROLE IF NOT EXISTS identifier($role_name); GRANT ROLE identifier($role_name) TO ROLE SYSADMIN; -- create a user for Statsig CREATE USER IF NOT EXISTS identifier($user_name) password = $user_password default_role = $role_name default_warehouse = STATSIG_INGESTION; GRANT ROLE identifier($role_name) TO USER identifier($user_name); -- grant Statsig role access to create warehouse and schema GRANT USAGE ON WAREHOUSE STATSIG_INGESTION TO ROLE identifier($role_name); GRANT CREATE SCHEMA, MONITOR, USAGE ON DATABASE STATSIG_STAGING TO ROLE identifier($role_name); -- grant Statsig role read access to database and schema passed in GRANT USAGE ON DATABASE TO ROLE identifier($role_name); GRANT USAGE ON SCHEMA . TO ROLE identifier($role_name); GRANT SELECT ON ALL TABLES IN DATABASE TO ROLE identifier($role_name); GRANT SELECT ON FUTURE TABLES IN DATABASE TO ROLE identifier($role_name); GRANT SELECT ON ALL VIEWS IN DATABASE TO ROLE identifier($role_name); GRANT SELECT ON FUTURE VIEWS IN DATABASE ACTUAL_DATA TO ROLE identifier($role_name); COMMIT; ``` After running the script, input the `` and `` you created in our console, during Connection Set Up stage under the Advanced settings options. Advanced connection settings for warehouse ingestion showing credential inputs # Synapse Source: https://docs.statsig.com/data-warehouse-ingestion/synapse Configure Statsig data warehouse ingestion from Azure Synapse Analytics, including authentication, scheduled queries, and mapping to events and properties. ## Overview To set up connection with Azure Synapse, Statsig needs the following * Workspace SQL Endpoint * Database Name * Admin User Name * Admin User Password Admin user name and password will be used by Statsig to create a user with restricted access to query from your data warehouse. You can find this information in your Azure console within your Synapse workspace overview page, as shown in the image below. (Open image in new tab for a bigger image) Azure Synapse workspace overview page # Adding Rules Source: https://docs.statsig.com/dynamic-config/add-rule Add targeting rules to Statsig dynamic configs to control which users receive specific JSON values based on user, environment, or custom attributes. ## Add a rule to a dynamic config To add new user targeting rules to a dynamic config, * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Dynamic Configs** * Select the dynamic config where you want to add a rule * Click the **Add Targeting** button * Click the **Add New Rule** button * Select the criteria for identifying the users you want to target: * You can target users based on common attributes such as their operating system as shown below Operating system targeting rule configuration * You can target users in a defined [segment](/segments) as shown below User segment targeting rule configuration * You can target users who are eligible for a specific feature gate as shown below; this ensures that the dynamic config is activated only for users who're exposed to the target feature gate Feature gate targeting rule configuration * To complete the dynamic config, click on the **Edit** link to open the JSON configuration editor. In the editor, type the configuration parameters and values that your application should receive and click **Confirm** JSON configuration editor interface # Create a dynamic config Source: https://docs.statsig.com/dynamic-config/create-new Step-by-step guide to creating a new dynamic config in the Statsig console, including defining schema, default values, and adding initial rules. To create a dynamic config, * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Dynamic Configs** * Click the **Create New** button * Enter a name and description, and click **Create** Dynamic config creation interface # Using a schema Source: https://docs.statsig.com/dynamic-config/enforce-schema Use JSON Schema in Statsig dynamic configs to enforce consistent return value shapes across rules and catch invalid configuration before it ships. Dynamic configs support schemas using [JSON Schema](https://json-schema.org/learn/getting-started-step-by-step) syntax to enforce a common convention between the return values for each rule you'll set. Schemas are only enforced when editing dynamic configs through the console or API, and are not used at code runtime. For example, if you have a dynamic config that returns settings for a site banner, you might have a schema of: ``` { "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { "title": { "type": "string", }, "description": { "type": "string", }, "cta": { "type": "string", } }, "required": ["title", "description", "cta"], } ``` Now, each of your rules must return an object including title, description, and CTA. ## Infer schema from current values You can also infer schema based on the current values in your targeting rule variations. This is useful as a quick starting point for validations or for documenting the expected shape of your config. Infer schema from current values **Important - JSON5 output** Statsig emits the inferred schema in JSON5 (a super set of JSON). JSON5 allows **unquoted object keys** and trailing commas. This might fail other standard JSON Schema validators and typical unit-test tooling. Statsig accepts JSON5 in the Console and API, but if you want strict JSON for validators or tests you should convert the output. # Dynamic Config Source: https://docs.statsig.com/dynamic-config/overview Statsig dynamic configs replace hard-coded values in your application with JSON defined on the server, enabling remote configuration without redeploys. ## What is a Dynamic Config? Dynamic configs replace hard-coded values in your application with JSON defined on the server. Statsig users leverage Dynamic Configs to avoid hard-coding configuration values of any kind in their code - and change it dynamically near real-time. You can also *target* Dynamic Configs, providing different experiences based on user attributes. These configuration parameters can include any property across your client-side or server-side application code, from button colors to ranking configurations. ## When to Use a Dynamic Config Many technology companies use tools such as dynamic config to make their server-side code the source of truth for configurable application properties. For example, Spotify uses [Remote Configuration](https://engineering.atspotify.com/2020/10/29/spotifys-new-experimentation-platform-part-1/) to dynamically update properties of their clients or backend services. ### Examples of Use Cases: * **Feature Toggles**: Dynamically enable or disable features without code changes. * **Gradual Rollouts**: Vary configurations across user segments to rollout more safely. * **Real-Time Updates**: Instantly adjust app properties like UI elements or service thresholds. ### Limitations * There's a 100kb limit on the JSON payload within a dynamic config. ## Next Steps * To get started, follow the guide on creating your [first dynamic config](/guides/first-dynamic-config). * For a deeper dive into how you can work with dynamic config, refer to the detailed tutorial on [working with dynamic config](/dynamic-config/working-with). # Working with Dynamic Config Source: https://docs.statsig.com/dynamic-config/working-with Use Statsig dynamic config parameters in your application to control behavior in near real-time, including reading values and handling parameter changes. A dynamic config allows you to use configuration parameters to control the behavior of your application in near real-time. In the example below, the dynamic config called **localization** allows you to retrieve localized strings for users in different countries. Users in Spanish speaking countries see Spanish strings, while users in French and Korean speaking countries see French and Korean strings respectively. Dynamic config localization interface A sample JSON payload for French speakers is also shown below. JSON payload example for French localization The following tutorials show you how to perform common tasks with dynamic configs. #### Tutorials * [Create a dynamic config](/dynamic-config/create-new) * [Create a rule for a dynamic config](/dynamic-config/add-rule) * [Use a language specific Statsig SDK to implement a dynamic config in your application](/sdks/getting-started) # Bayesian Experiments Source: https://docs.statsig.com/experiments/advanced-setup/bayesian Learn how Bayesian A/B testing works in Statsig experiments, including informative priors, posterior probabilities, and implementation details for analysis. ### Bayesian Testing in Statsig Experiments are frequentist by default. To switch to Bayesian mode, go to Advanced Settings. Bayesian experiment configuration interface The experiment type cannot be modified once the experiment starts. Bayesian experiment type selection interface Deep dive analysis should also reflect Bayesian statistics Bayesian deep dive analysis interface ### Informed Bayesian Bayesian experiments allow you to specify a prior belief on the relative average treatment effect. Statsig will combine the prior distribution with the observed data to display the prior-adjusted results. You can enable this by selecting the option to "use informative priors". Informative priors configuration interface ### Drawing the Correct Prior Distribution From Historical Data If you are using the Bayesian with informative priors, the assumption is that you have a clear understanding of what power the priors have over your experimental results, and your organization has established a reliable prior based on the domain knowledge. With that said, here are some patterns people follow to derive their priors: 1. You can use the $AVG(\text{average treatment effect})$ of past experiments with a similar setup and population as your prior mean. You can use the standard deviation, or a multiple of it, as the prior standard error. 2. You can also use the $AVG(\text{observed standard error})$ as your prior standard error. ### Implementation Details Denote $\mathcal{N}(ATE_{prior}, STE_{prior}^2)$ as the prior distribution, where $ATE_{prior}$ is the average treatment effect and $STE_{prior}$ is the standard error. Similarly, $\mathcal{N}(ATE_{observed}, STE_{observed}^2)$ as the observed distribution. The posterior distribution is then calculated as $$ ATE_{post} = \frac{ \frac{ATE_{prior}}{STE_{prior}^2} + \frac{ATE_{observed}}{STE_{observed}^2} }{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} } $$ $$ STE_{post}^2 = \frac{1}{ \frac{1}{STE_{prior}^2} + \frac{1}{STE_{observed}^2} } $$ If the prior is not specified, the $\mathcal{N}(ATE_{prior}, STE_{prior}^2)$ is represented as $\mathcal{N}(0, \infty)$. ### Bayesian Statistics Bayesian A/B tests have a glossary that are different from the frequentist framework and often believed to be more intuitive in communication to non-technical audience. * Credible Interval: the interval which we believe contains the true parameter at the given probability * Chance to Beat: the probability that the test is better than control * Expected Loss: the average potential risk if you ship test # Frequentist Sequential Testing Source: https://docs.statsig.com/experiments/advanced-setup/sequential-testing Sequential testing in Statsig addresses the peeking problem in A/B tests so you can monitor experiments and make early decisions with statistical rigor. ## What's the problem with looking early in a "standard" A/B test? Traditional A/B testing best practices (t-tests, z-tests, etc.) dictate that the readout of experiment metrics should occur only once, when the target sample size of the experiment has been reached (i.e. your design duration has been reached and you reach the desired sample size). We call this a "Fixed Horizon Test", because when designing an experiment you set the amount of desired units you wish to observe and (ideally) commit to analyzing the results only once this dataset is complete. Continuous experiment monitoring (i.e. "peeking") for the purpose of decision making, however, results in inflated false positive rates (a.k.a. *the peeking problem*) which can be much higher than that expected from your desired significance level. ## How does peeking increase decision error rates? Continuous monitoring leads to inflated false positives because any time you consider ending an experiment early you are at risk of making a conclusion that is incorrect. Remember, at the core of a standard hypothesis test, you are deciding if you should "accept the null" hypothesis or "reject the null" hypothesis and accept the alternative. As an A/B practitioner, you must decide between rejecting or not rejecting the null. Any time you look early and allow the possibility of making a decision early, you are potentially rejecting the null hypothesis even when the null hypothesis is actually correct. ## Why would early results be "wrong"? Metric values and p-values always fluctuate to some extent due to noise during any experiment, and results can transition into and out of statistical significance due to this noise, even when there is no real underlying effect. These noisy fluctuations can be caused by random unit assignment and random human behavior we can't predict, and the effects can't be entirely removed from an experiment. Not every test is subject to the same amount of experimental noise, however, and like so many things it's dependent on what you're testing and who your users are. Tests also vary over time in the amount of noise they see, especially as adding more users and observing them for longer tends to help the random fluctuations even out. Peeking, however well-intentioned, will always introduce some amount of selection bias if we adjust the date of a readout. When an experimenter makes any early decision about results (e.g. "is the result stat-sig, can we ship a variant early?") they've increased the chances that their decision is based on a temporary snapshot of always fluctuating results. They are potentially cherry-picking a stat-sig result that might never be seen if the data were analyzed only once at the full, pre-determined completion of the experiment. Unfortunately, when running frequentist A/B test procedures this early decision can only increase the false positive rate (declaring an experimental effect when there really is none), even when the intention is to make a less-biased decision based on the statistics. ## What is Sequential Testing for an A/B test? In the variety of Sequential Testing on Statsig, the experimental results for each preliminary analysis window are adjusted to compensate for the increased false positive rate associated with peeking. Statsig adjusts your p-values and confidence intervals automatically, and you can see this in the Results tab: Sequential testing results visualization *In this example, the confidence intervals for each metric are expanded using the "wings" or "tabs". This serves as a quick visual indicator that sequential testing is enabled and shows you how much the intervals have been expanded.* Statsig results table highlighting sequential testing adjusted confidence interval *In this real-world example, you can see for the indicated result that the sequential testing adjust makes the difference between declaring the result stat-sig or not.* The goal of Sequential Testing is to enable early decision making when there's sufficiently strong observations that outweigh the random fluctuations while limiting the risk of false positives. While peeking is typically discouraged, regular monitoring of experiments with sequential testing is particularly valuable in some cases. For example: * Unexpected regressions: Sometimes experiments have bugs or unintended consequences that severely impact key metrics. Sequential testing helps identify these regressions early and distinguishes significant effects from random fluctuations. * Opportunity cost: This arises when a significant loss may be incurred by delaying the experiment decision, such as launching a new feature ahead of a major event or fixing a bug. If sequential testing shows an improvement in the key metrics, an early decision could be made. But use caution: An early stat-sig result for certain metrics doesn't guarantee sufficient power to detect regressions in other metrics. Limit this approach to cases where only a small number of metrics are relevant to the decision. Sequential testing can be used anywhere you do an experimental analysis. This includes your main experimental Results page as well as any [custom queries](/pulse/custom-queries/). ## Quick Guides ### Enabling Sequential Testing Results In the **Setup** tab of your experiment, with Frequentist selected as your Analytics Type, you can enable Sequential Testing under the Analysis Settings section. This setting can be toggled at any time during the life of the experiment, and it does not need to be enabled prior to the start of the experiment. Sequential testing configuration interface ### Interpreting Sequential Testing Results Click on Edit at the top of the metrics section in Pulse to toggle Sequential Testing on/off. Pulse metrics sequential testing toggle When enabled, an adjustment is automatically applied to results calculated before the target completion date of the experiment. Sequential testing confidence interval visualization The dashed line represents the expanded confidence interval resulting from the adjustment. The solid bar is the standard confidence interval computed without any adjustments. If the adjusted confidence interval overlaps with zero, this means the metric delta is not stat-sig at the moment, and the experiment should continue its course as planned. Sequential testing is a reliable way to make an early decision, particularly for early detection of regressions. One should be mindful that early decision-making will often result in underpowered lift estimates with a high degree of uncertainty. If making the right decision is important, you can use statistically-significant sequential testing results. If an accurate measurement is important, you should wait for full power as estimated by your pre-experimental power calculation. We do not calculate statistical power on post-hoc experimental results (See section "Post-hoc Power Calculations are Noisy and Misleading" in [Kohavi, Deng, and Vermeer, A/B Testing Intuition Busters](https://bit.ly/ABTestingIntuitionBusters). ## Statsig's Implementation of Sequential Testing ### Two-Sided Tests #### Confidence Intervals Statsig uses mSPRT based on the approach proposed by Zhao et al. in this [paper](https://arxiv.org/pdf/1905.10493.pdf). The two-sided Sequential Testing confidence interval with significance level $\alpha$ is given by: $$ CI^*(\Delta \overline{X}) = \Delta \overline{X} \pm Z^*_{\alpha/2} \cdot \sqrt{V} $$ where * $Z^*_{\alpha/2}$ is the z-critical value, modified for sequential testing: $$ Z^*_{\alpha/2} = \sqrt{\frac{(V+\tau)}{\tau}\left(-2\ln(\alpha/2)-\ln(\frac{V}{V+\tau})\right)} $$ * $V$ is the standard variance of the delta of means when computing [variance](/stats-engine/variance). It can be obtained from the sample variance of the test and control group means: $$ V = var(\Delta \overline X) = var(\overline X_t) + var(\overline X_c) = \frac{var(X_t)}{N_t} + \frac{var(X_c)}{N_c} $$ * $\tau$ is the mixing parameter given by: $$ \tau =(Z_{\alpha/2})^2\cdot\frac{var(X_t)+var(X_c)}{N_t+N_c} $$ * $Z_{\alpha/2}$ is the z-critical value used in the non-sequential test, for the desired significance level (1.96 for the standard $\alpha = 0.05$) We have validated that this parameter satisfies the expected False Positive Rate and provides enough power to detect large effects early. More details on this analysis are available [here](https://www.statsig.com/blog/sequential-testing-on-statsig). #### p-Values It's possible to produce p-values for sequential testing that are consistent with the expanded confidence intervals above by modifying our [p-value methods](/stats-engine/p-value). We want to evaluate the mSPRT test so that our Type I error remains approximately equal to $\alpha$, and so that the sequential testing p-value is consistent with the expanded confidence interval. (I.e. A CI that includes 0.0% should have p-value ≥ $\alpha$, and one that excludes 0.0% should have p-value \< $\alpha$.) Our observed z-statistic (i.e. z-score) remains unchanged. Instead of evaluating $Z$ on a standard-normal distribution $N(0, 1)$ as we usually do, we evaluate against some other normal distribution $N(0, \sigma^2)$ with mean of zero and standard deviation $\sigma$. For a two-sided test, since we want the probability of an observed $Z$ exceeding $Z^*_{\alpha/2}$ (assuming the null hypothesis to be true) to be limited to $\alpha$, we can find the unknown parameter by solving for $\sigma$: $$ \sigma=\frac{Z_{\alpha/2}^*}{\sqrt{2} \cdot erf^{-1}(1-\alpha)} $$ where $erf^{-1}$ is the [inverse error function](https://en.wikipedia.org/wiki/Error_function#Inverse_functions). From here we can compute the two-sided sequential testing p-value as: $$ \text{p-value}^* = 2 \cdot \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{-|Z|} \frac{e^{- \frac{t^2}{{2\sigma^2}}}}{\sigma}dt $$ where $Z$ is the observed z-statistic (i.e. z-score) as usual. ### One-Sided Tests We can modify each step for one-sided sequential testing. $$ CI^*(\Delta \overline{X}) = \begin{cases} \left[\Delta \overline{X} - Z^*_{\alpha} \cdot \sqrt{V}, \quad +\infty \right) & \text{if right-sided test} \\ \\ \left(- \infty, \quad \Delta \overline{X} + Z^*_{\alpha} \cdot \sqrt{V} \:\right] & \text{if left-sided test} \\ \end{cases} $$ $$ \text{p-value}^* = \begin{cases} 1 - \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z} \frac{e^{- \frac{t^2}{{2\sigma^2}}}}{\sigma}dt \quad \text{if right-sided test} \\ \\ \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z} \frac{e^{- \frac{t^2}{{2\sigma^2}}}}{\sigma}dt \quad \text{if left-sided test} \\ \end{cases} $$ where * $Z^*_{\alpha}$ is the one-sided test z-critical value, modified for sequential testing: $$ Z^*_{\alpha} = \sqrt{\frac{(V+\tau)}{\tau}\left(-2\ln(\alpha)-\ln(\frac{V}{V+\tau})\right)} $$ * $V$ is the same as for two-sided tests. * $\tau$ is the mixing parameter given by: $$ \tau =(Z_{\alpha})^2\cdot\frac{var(X_t)+var(X_c)}{N_t+N_c} $$ * $Z_{\alpha}$ is the one-sided z-critical value used in the non-sequential test, for the desired significance level (1.645 for the standard $\alpha = 0.05$) * $\sigma$ is solved via: $$ \sigma = \begin{cases} \frac{Z_{\alpha}^*}{\sqrt{2} \cdot erf^{-1}(1 - 2 \alpha)} & \text{if right-sided test} \\ \\ \frac{- Z_{\alpha}^*}{\sqrt{2} \cdot erf^{-1}(2 \alpha - 1)} & \text{if left-sided test} \end{cases} $$ * $Z$ is the (signed) observed z-statistic as usual (i.e. z-score) # Sequential Probability Ratio Tests Source: https://docs.statsig.com/experiments/advanced-setup/sprt Use SPRT (Sequential Probability Ratio Test) methodology in Statsig for faster A/B test decision making without statistical penalties for peeking at results. ## What is SPRT? The **Sequential Probability Ratio Test** (SPRT) is another, advanced methodology for running AB tests, differing from the traditional Null Hypothesis Significance Test (commonly called [Frequentist](/stats-engine/p-value) analysis). SPRT can meaningfully improve time to decision for your experiments, including detecting unwanted metric regressions much faster. It also tends to be much easier to share results to stakeholders who aren't super familiar with P-values and Significance levels. Lastly, SPRT has no penalties for peeking; there's no need for sequential testing plans, Alpha spending, or CI-penalties as SPRT is built to be a sequential test methodology from the start. SPRT experiment results scorecard ### Concepts SPRT introduces a few key concepts that differ from standard Frequentist tests. At its core, SPRT relies on the **Likelihood Ratio (LR)** and Upper and Lower decision boundaries, **A** and **B**. The Likelihood Ratio estimates the relative difference in the likelihood of two outcomes: * **Numerator**: What you observe is due to an alternative hypothesis (you set) being correct. * **Denominator**: What you observe is due to the null hypothesis being correct. The Upper and Lower decision boundaries are determined by your joint tolerances for Type I and Type II errors. * **A**: If LR exceeds this upper threshold, you should accept the Alternative Hypothesis. * **B**: If LR is less than this lower threshold, you should accept the Null Hypothesis. * When LR falls into the range between these thresholds, no decision can be made and you should continue collecting data.

SPRT Hover Card

An LR of 5.8, for example, indicates that the what you observed is 5.8x more likely under the alternative hypothesis as compared to the null hypothesis. One of the nice things about SPRT is that this Likelihood Ratio is similar to how most people think about comparing options. Rather than reporting P-values and Significance levels, you can now report a result like "*With an LR of 3.5, it's 3.5x more likely that the feature worked*." ## Why SPRT? * **Faster Decisions:** SPRT allows you to reach conclusions more quickly, potentially reducing experiment run time. * **Intuitive Results:** Instead of p-values, SPRT uses the Likelihood Ratio, a more intuitive measure of evidence for or against your hypotheses. * **Sequential Analysis:** Data is continuously evaluated as it is collected, allowing for early stopping when sufficient evidence is reached. There's no penalty for "peeking" in SPRT experiments. * **Clear Outcomes:** SPRT enables you to confidently accept either the Null or Alternative hypothesis, rather than just “rejecting the null.” * **Data-Informed:** Statsig’s implementation uses your past data and power analysis to inform the likelihood calculations and decision thresholds. ## Comparing SPRT to other analysis methods | Category | Frequentist | Bayesian | SPRT | | --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Test Statistic | P-value:

*Probability of observing the results that is as extreme as the sample data if the null hypothesis is true* | Posterior Probability:

*Probability of Test better than Control given the observed data and your prior information* | Likelihood Ratio:

*Comparing the goodness of fit of two competing statistical models* | | Decision Threshold | Alpha (Industry standard 5%) | Posterior Probability and Credible Intervals | Upper & Lower Decision Boundary decided based one the alpha and beta you picked | | Decision Framework | Reject/Fail to Reject the Null based on if p-value > 5% | Whether chance to beat control exceeds the pre-set decision threshold | Accept the Null Hypothesis, Accept Alternative Hypothesis, Or Continue based on the comparison of calculated likelihood ration with Upper/Lower Decision boundary | | Allows Peeking | Yes, but with Sequential Testing Penalties | Yes, Unlimited | Yes, Unlimited | | Requires Pre-Setup | Yes, requires sample size calculation based on historical metric mean and MDE | Optional, but you can define prior distribution per metric if you have previous knowledge which can accelerates the experiment or correct surprising results | Yes, requires historical information about each metric as well as MDE | | Allows 1- and 2-Sided tests | Yes, per metric | Yes, per metric | Yes, per metric | ## How to Use SPRT in Statsig **Enabling SPRT:** Select SPRT as your analysis method when setting up an AB test in the Statsig console. SPRT power analysis configuration interface **Interpreting Results:** The experiment Results tab shows the latest likelihood ratio for each metric in your experiment and indicates when a decision boundary has been reached, allowing you to accept the null or alternative hypothesis with confidence. SPRT experiment results dashboard ## Computing SPRT Results Statsig uses an updated version of Hajnal's two-sample t test ([Schnuerch and Erdfelder](https://martinschnuerch.com/wp-content/uploads/2020/08/Schnuerch_Erdfelder_2020.pdf), as modified by Derek Ho of Atlassian, in our SPRT calculations. The traditional ratio test using t- or F-distributions (Schnuerch and Erdfelder equations 8 & 10) can be shown to simplify to a ratio of standard-normal distributions. $$ \begin{split} {LR} &= \frac {f(t^2| {df}, \Delta)} {f(t^2| {df}, 1)} \\ &= \frac {\phi(|z_{m}|; \theta, 1)} {\phi(|z_{m}|; 0, 1)} \\ \end{split} $$ where: * $ \phi(x; \theta, 1)$ is the PDF of a normal distribution of shape $ \mathcal{N}(\theta, 1)$ evaluated at $ x$ * $ z$ is the observed Z-statistic between the groups $$ z = \frac {\Delta \bar{X}} {\sigma_{\Delta\bar{X}}} = \frac {\bar{X}_B - \bar{X}_A} {\sigma_{\Delta\bar{X}}} $$ $$ \sigma_{\Delta\bar{X}}=\sqrt{\frac{var(X_A)}{N_A}+\frac{var(X_B)}{N_B}} $$ * $ \theta$ is derived from **Cohen's d** set prior to the experiment for the particular metric being considered $$ \theta = \frac {\delta} {\sqrt{ \frac{1}{N_A} + \frac{1}{N_B} }} $$ * $N_A$ and $N_B$ are the number of observed units for each group Since log likelihood ratios are a more convenient scale for reporting, we often take the natural log of the LR value above. When doing so, the equation can actually be simplified further, making evaluation straightforward: $$ \begin{split} LLR &= ln(LR) \\ &= |z_m \cdot \phi_m | - \frac{1}{2}{|\phi_m|}^2 \end{split} $$ ### Power Analysis & Setting Cohen's d SPRT requires that a value of [**Cohen's d**](https://en.wikiversity.org/wiki/Cohen%27s_d) be set prior to the start of the experiment for each metric being evaluated. Setting the parameter requires three components: * **MDE**: An Minimum Detectable Effect desired to be measured, in units of percent * **Mean**: A baseline average value for the metric, $ \overline{X}$ * **Standard Deviation**: A baseline standard deviation for the metric, $ \sigma\_{X}$ With them, it's easy to compute Cohen's d parameter for each metric: $$ \delta = \frac{\text{MDE\%} \cdot \overline{X}}{100 \cdot \sigma_{X}} $$ This process can be automated using Statsig's built-in query tooling. If you have a past experiment that ran on a similar set of units expected in the upcoming experiment, this can be configured as a **Baseline Experiment** and a query will automatically pull the relevant metric parameters for your metrics. Users can also input all 3 parameters by hand if desired. ### Estimating the decision sample size While Cohen's d is used to compute your experimental results after the experiment starts, it can also be used to estimate the duration of an experiment in advance. Given SPRT allows users to look at results as often as desired, this is not the same as a "required sample size" in traditional frequentist testing. The **Decision Sample Size** is an estimate of the number of samples that will be sufficient for SPRT result for a metric to exceed either threshold and accept one of the hypotheses. Given: $$ A=ln\left(\frac{1-\beta}{\alpha}\right) $$ $$ k=\frac{n_{ec}}{n_{et}}=\frac{\text{units expected in control}}{\text{units expected in treatment}}=\frac{\text{\% units expected in control}}{\text{\% units expected in treatment}} $$ $$ n_{et} = \frac{A}{\frac{1}{2}\left(\frac{k}{1+k}\right)\delta^2} $$ Then, the total number of expected units at decision time is: $$ n_e=n_{et}+n_{ec}=n_{et}(1+k) $$ ## References * [Original SPRT Paper (Wald, 1945)](https://projecteuclid.org/journals/annals-of-mathematical-statistics/volume-16/issue-2/Sequential-Tests-of-Statistical-Hypotheses/10.1214/aoms/1177731118.full) * [The Sequential Probability Ratio t Test (Schnuerch & Erdfelder, 2020)](https://martinschnuerch.com/wp-content/uploads/2020/08/Schnuerch_Erdfelder_2020.pdf) * [A two-sample sequential t-test (Hajnal, 1961)](https://www.jstor.org/stable/2333131) ## FAQ **Can I use SPRT for all experiments?**\ SPRT is best suited for experiments where you want faster, sequential decisions and are comfortable with likelihood-based inference. For some experiment types, traditional methods may still be preferable. **How does SPRT affect experiment duration?**\ SPRT can reduce experiment duration, especially when there is strong evidence for or against an effect. However, if the effect is small or data is noisy, the test may run longer. **What are the limitations?**\ SPRT requires careful setup of thresholds and assumptions. It is not a drop-in replacement for all frequentist methods, and may not be suitable for all experiment types. **Is SPRT the same as Sequential Testing?** SPRT is different from our Sequential Testing option. [Sequential Testing](/experiments-plus/sequential-testing) adjusts your Frequentist analysis method to allow repeated looks (i.e. "peeking"). SPRT is a completely separate experimental procedure and decision framework. They both allow for continuous "sequential" looking at experiment results, but otherwise they are separate methods for designing and running an A/B test. # Stratified Sampling Source: https://docs.statsig.com/experiments/advanced-setup/stratified-sampling Use stratified sampling in Statsig experiments to balance treatment assignment across key segments, reduce variance, and detect smaller effects faster. ## What is Stratified Sampling Stratified sampling involves dividing the entire population into homogeneous groups called strata (plural for stratum). Random samples are then selected from each stratum. e.g. If you had XS and XL customers and randomized them into two groups - Control and Test, you'd want both Control and Test to be balanced across XS and XL customers. You can also stratify based on a metric like Revenue/User. With large numbers, randomization typically solves this. However in B2B scenarios and other relatively low volume or high variance scenarios, stratified sampling is useful to ensure this balance. Statsig supports both automated and manual stratified sampling. On tests where a tail-end of power users drive a large portion of an overall metric value, stratified sampling meaningfully reduces false positive rates and makes your results more consistent and trustworthy. In our simulations, we saw around a 50% decrease in the variance of reported results. ## Automated Stratified Sampling ### How it works The Statsig SDKs use a *salt* to randomize or bucket experiment subjects ([learn more](/faq#how-does-bucketing-within-the-statsig-sdks-work)). When you enable stratified sampling, we'll try n different salts (100 for now) and evaluate how "balanced" your groups. We evaluate this balance based on either a metric you pick - or an attribute you give us describing your experiment subjects. We pick the best salt from this set and save this as the salt to use. [Learn more](https://statsig.com/blog/introducing-stratified-sampling). Stratified sampling algorithm diagram The selection space for the salts is sufficiently large - stratifying multiple experiments on the same metric will not result in overlap. In the simulations we ran, the groups were as independent as expected which matched up with the literature here. ### Enabling Stratified Sampling You can enable this on experiment under Advanced Settings on the experiment setup page. There are two ways you can "stratify" on Statsig. If you choose a metric to stratify using, we'll use that to balance the group. Stratified sampling metric selection interface If you instead choose an attribute or a classification (e.g. S, M, L, XL) we'll use that to balance the group. * On Statsig Cloud, you'll upload a CSV (in Beta) * On Statsig Warehouse Native, you'll use Entity Properties Entity properties configuration for stratified sampling Once you press the Stratify button, we'll analyze a set of salts and pick the best one. Stratification analysis results interface ## FAQ and Best Practices * **What population is used when balancing?** * When evaluating salts, Statsig computes balance using pre-experiment data for the entire targeted population of the experiment’s unit type (e.g., all `userID`s or all `customerID`s) over the selected lookback window. There is no filtering on exposure because the experiment has not started yet. * **How are new units handled after stratification?** * Units that were not present in the pre-experiment data are still assigned deterministically by the chosen salt, i.e., effectively at random with respect to the balancing metric. They do not influence the salt selection and may introduce some drift from the initial balance. * **Should I use stratified sampling for every experiment?** * Not necessarily. It’s most useful when you expect imbalance due to heterogeneous units (e.g., “whales”) or skewed metrics. The tradeoff is time/compute cost that scales with the number of units and adds steps before starting an experiment. If you don’t expect meaningful imbalance, a standard random split is generally recommended. * **Does salt evaluation assume 100% allocation? What about running at less than 100%?** * Yes. All candidate salts are evaluated assuming 100% of the targeted population is allocated. If you then run the experiment at an allocation below 100%, random sampling of that subset can reintroduce imbalance (e.g., by chance, some high-impact units may fall disproportionately into one arm). For the period you care most about inference, prefer 100% allocation to preserve the intended balance. Lower allocations are best used briefly for safe rollouts rather than for the full experiment duration. * **Across candidate salts, is it the same set of users being evaluated?** * Yes. Candidate salts are assessed over the same targeted population; only the randomization induced by the salt changes. * **How long does stratification take?** * Duration depends on the number of units and the metric/source being queried. There is no fixed SLA; larger populations take longer. ## Manual assignment for Stratified Sampling When setting up an experiment, you can configure overrides (e.g. force user X or Segment A into Control, force user Y or Segment B into Test). This is meant for testing; overridden users are excluded from experimental analysis in Pulse results. If you do want manual assignment for stratified sampling, you should check the *Include Overrides in Pulse* checkbox. This will include the users you've manually overridden into each variant in all metric lift analyses. You can configure 100% of experiment participants into your test variants manually, or configure some subset of participants into variants manually and randomly assign the rest of your participants. While you can add overrides for an ID type that is different than the ID type of the experiment, those ID evaluations will not be resolved to the id type of the experiment and will not contribute to pulse results. When you use the Statsig SDK for assignment, it takes care of randomization. When you control assignment of users, you're responsible for making sure users are balanced across experiment groups. Manual assignment override configuration ## Additional reading [Morgan and Rubin 2012](https://projecteuclid.org/journals/annals-of-statistics/volume-40/issue-2/Rerandomization-to-improve-covariate-balance-in-experiments/10.1214/12-AOS1008.full) walks through the history, the philosophy, and the proofs of re-randomization, especially how re-randomization reduces the randomization variance of the difference in means. It's worth noting that "Standard asymptotic-based analysis procedures that do not take the re-randomization into account will be statistically conservative" was called out in the paper. However, to maintain consistent and comparable results across different methods, we stay conservative with the t-test. [Lin & Ding 2019](https://arxiv.org/abs/1906.11291) is another interesting read for your reference. # Create an Experiment Source: https://docs.statsig.com/experiments/create-new Step-by-step guide to creating a new experiment in the Statsig console, including hypothesis, allocation, targeting rules, and scorecard metric setup. Metrics and experiments behave differently in Warehouse Native. Read about [Configuring Experiments in Warehouse Native](/statsig-warehouse-native/features/experiment-options). This doc walks through the steps of creating a new experiment in the Statsig console. If you're looking for an end-to-end guide that includes integrating the Statsig SDK, see [Run your first experiment](/guides/abn-tests). ## User-level Experiments To create a user-level experiment, follow these steps: 1. Log into the Statsig console at [https://console.statsig.com/](https://console.statsig.com/) 2. Navigate to **Experiments** in the left-hand navigation panel 3. Click on the **Create** button 4. Enter the name and description for your experiment as shown in the figure below 5. By default, your experiment runs in its own **Layer**. A Layer allows you to manage multiple experiments and feature flags together. If you want to add this experiment to an existing Layer, select **Add Layer** under **Advanced** in the experiment creation modal. You can also create a new Layer by selecting **Create New Layer**. 6. Click **Create** Experiment creation modal interface ## Configure Your Scorecard When running an experiment, it’s common to test a specific hypothesis using a set of key metrics. The **Scorecard** feature makes this easy by letting you enter your hypothesis and select both primary and secondary metrics. * **Primary Metrics** are those you expect to be directly impacted by the experiment. * **Secondary Metrics** are important to monitor to ensure there are no unintended side effects, but they aren’t the primary focus of your experiment. Configuring the Scorecard is a required step when creating an experiment. It provides your team with clear context on what is being tested and how success is measured. You must enter your hypothesis and select at least one primary metric. Metrics added to the Scorecard are computed daily and eligible for advanced treatments like [CUPED](/experiments/statistical-methods/methodologies/cuped) and [Sequential Testing](/experiments/advanced-setup/sequential-testing). For best practices on configuring your Scorecard, read more [here](/experiments/interpreting-results/read-results). Scorecard configuration modal ## Configure Allocation and Targeting This is where most of your experiment configuration happens. ### Allocation For **Allocation**, enter the percentage of users you want to assign to this experiment. You can allocate up to 100% of eligible users, but it’s good practice to start with a smaller percentage, verify the experiment’s stability, and then ramp up the allocation. Experiment allocation configuration interface You can increase the allocation of your experiment anywhere from 0% to 100% at any time after experiment start; however, you cannot decrease allocation without resetting your experiment, as this would cause biases in group allocation and pollute your metric results. ### Targeting To configure **Targeting** criteria, click to edit the **Targeting** section. You can either set new targeting criteria or use an existing **Feature Gate**. This will limit the experiment to only the users who meet the defined conditions. Experiment targeting configuration interface * If your targeting is straightforward, creating it through Inline Targeting works well. (Click "Criteria: Everyone" to get started.) * For more advanced targeting (e.g., progressive rollouts) or if you want to maintain targeting criteria when you launch your experiment, it’s better to reference an existing **Feature Gate**. By default, no targeting criteria are set, so your experiment will include all allocated users within the defined **Layer** or exposed user base. ## Configure Your Groups and Parameters When configuring **Groups and Parameters**, it’s a good idea to define your parameters first. These are the variables that control the behavior of the different experiment variants. * Enter the values the experiment parameter will take for each variant. For more about the difference between **Groups** and **Parameters**, refer to [Groups vs. Parameters](/experiments/implementation/getting-group). You can add additional groups by clicking the "+" next to the existing groups. The user allocation will automatically adjust as you add more groups. Experiment groups and parameters configuration interface In addition, you can name, describe, and even add variant images for each group under the **Groups** section. However, only the parameters and values will affect what users see—group names and descriptions are not used in the experiment code. ## Device-level and Custom ID Experiments By default, experiments randomize users based on **User ID**. If you need to use a different ID type (e.g., device-level), follow steps 1–4 from the "User-level Experiments" section, then: 1. Click the **ID Type** dropdown menu and choose the desired ID type. 2. Click **Create** ID type selection dropdown interface Afterward, continue with the same steps described above to finish configuring the experiment. ## ID Mapping Capabilities When running experiments, you may want to start with one ID type (like stableID for device-level targeting) but analyze results using events from another ID type (like userID for logged-in user metrics). **Warehouse Native**: Supports ID mapping between different identifier types (e.g., stableID to userID) through Entity Property Source configuration. **Cloud**: Currently does not support mapping between different ID types. Experiments started with stableID will only analyze events with stableID, and experiments started with userID will only analyze events with userID. For advanced ID mapping requirements, consider using Statsig Warehouse Native. ## Isolated Experiments If you want to create an experiment that excludes users exposed to other experiments, follow steps 1–4 from the "User-level Experiments" section. Then: 1. Select **Advanced** options. 2. Select an existing **Layer** or create a new one. 3. Click **Create**. Isolated experiment layer configuration interface Now, complete the rest of the experiment setup as described above. ## Reusing Experiment Salts The Statsig SDKs use [deterministic hashing](/sdks/how-evaluation-works) to bucket users. This means that the same user being evaluated for the same experiment will be bucketed identically - no matter where that happens. Every experiment has it's own unique salt, so that each experiment's assignment is random. For advanced use cases - e.g. a series of related experiments that needs to reuse the control and test buckets, we now expose the ability to copy and set the salts used for deterministic hashing. This is meant to be used with care. and is only available to Project Administrators. It is available in the Overflow (...) menu in Experiments. ## Significance Level Adjustments By default, Experiment Results display with 95% confidence intervals and without Bonferroni correction. This can be customized during experiment setup or later when viewing results in Experiment Results. * **Bonferroni Correction:** Apply this to reduce the risk of false positives in experiments with multiple test groups. The significance level (*α*) is divided by the number of test variants. * **Default Confidence Interval:** Choose a lower confidence interval (e.g., 80%) if you prefer faster results with higher tolerance for false positives, or stick with 95% for greater certainty. Significance level adjustment settings interface ## Target Duration Setting a target duration is optional, but it helps ensure that you wait long enough for the experiment to reach full power. You can set the target as either a specific number of days or a number of exposures, and use the [**Power Analysis Calculator**](/experiments-plus/power-analysis) to determine what target works best for your metrics. Target duration setting interface 💡 **Target durations longer than 90 days:** By default, Statsig computes Experiment Results results for the first 90 days, though the experiment itself can run longer. Before setting a duration beyond 90 days, ask yourself if results past that period will still be relevant, and if earlier data might already provide the insights you need. Hypothesis Advisor screenshot Once set, you can track progress against the target duration/exposures in the experiment header. You’ll also receive notifications via email and Slack (if integrated) when the target is reached. *** ## Hypothesis Advisor Writing good experiment hypotheses is key to a strong experimentation culture. Statsig now gives instant feedback on experiment hypotheses—flagging what’s missing. Admins can set custom requirements, which Statsig uses to guide experimenters toward stronger, more complete hypotheses. This Statsig AI feature is default disabled and has to be enabled for your project. Do this from Settings -> Experiment -> Project -> Statsig AI. This is also where you configure any custom requirements you want Hypothesis Advisor to ensure adherence to (e.g. "Strongly recommend that a validation plan be mentioned"). Setting showing how to enable Statsig AI # Abandon an Experiment Source: https://docs.statsig.com/experiments/ending/abandon Abandon a Statsig experiment to stop all user assignments and clear targeting, including implications for downstream metrics and exposure data. When you realize your experiment has an issue or need to stop it for any reason, you can abandon it. Abandoning an experiment will put it into the unstarted state, which will give every user the default experience back. Experiment abandon interface When an experiment is abandoned, the "salt" that the experiment uses to randomize a user's group will also change. This means that when you re-start the experiment, your users will be randomly assigned to a group that is not necessarily the same group they were in prior to the experiment being abandoned. This is important because it makes sure that the new result for the group that was not performing well due to an issue like a bug or bad experience in the previous run, is not negatively affected even after the issue is addressed in the new wrong. # Conclude Experiment & Defer Decision Source: https://docs.statsig.com/experiments/ending/conclude-experiment-defer-decision Conclude a Statsig experiment while deferring the launch decision so you can finalize analysis without immediately shipping a winning variant to all users. ### Overview The "Conclude Experiment and Defer Decision" feature allows you to effectively conclude an ongoing experiment which means stopping any further user allocation and data collection. This feature is particularly beneficial when you want to decouple running an experiment from making a ship decision. ### Benefits * **Focused Decision Making**: Decoupling running experiment from making decision allows you to take the necessary time to evaluate results, weigh options, and review findings with stakeholders across teams. * **Flexibility in Outcomes**: Decide whether to ship the control, implement the test, or abandon/restart the experiment based on comprehensive insights. Conclude Experiment ### Enabling this Option The "Conclude Experiment and Defer Decision" option must first be enabled in Project Settings to show as an option in the Make Decision modal. To enable this option, head to **Settings** --> **Experimentation** in the **Product Configuration** section and toggle on the **Enable decision type - Conclude Experiment and Defer Decision** setting. Screen Shot 2024-12-04 at 12 10 28 PM ### Key Details Things you should keep in mind when concluding an experiment: * Once you conclude the experiment, no new users will be enrolled in the experiment. * All the already exposed users will start receiving the default experience. * Statsig will stop further data collection on the experiment and conclude results. * After concluding the experiment, you can still decide to ship control vs test variant, reset, or abandon the experiment. ### Reloads After Experiment Conclusion You may notice slight changes in exposure and metric data if you trigger another reload after an experiment has been concluded. This does not mean new users are still being exposed. Rather, depending on the timestamp of when the experiment was concluded versus when the latest exposure data landed in your warehouse in the previous reload, additional exposures that occurred during that time window may appear in the refreshed data. ### Conclusion Utilizing the "Conclude Experiment and Defer Decision" feature enhances your ability to make data-driven decisions without the risk of diluting results by continuously adding new participants or new data from existing participants. This structured approach ensures that all stakeholders are aligned before moving forward. # Ending an Experiment Source: https://docs.statsig.com/experiments/ending/ending-experiment Compare options for ending an experiment in Statsig, including abandoning, stopping assignments, deferring decisions, and shipping a winning variant. For one reason or another, your experiment has come to an end: * Your experiment has statsig results, and it's time to ship an experience (control or test) to all your users * Your experiment has a problem, and you need to stop it ## Making a Decision To ship any of the groups in your experiment, including control, to all your users - you should [Make a Decision](/experiments-plus/make-decision) on the experiment. This will ship the parameters of the variant you selected to all users, and if in a layer will update the layer defaults to reflect your shipped experience. ## Stopping an Experiment If your experiment has a problem and you need to stop it, there are two routes you can take. * If the problem means you no longer wish to run this test, you can [Abandon the experiment](/experiments-plus/abandon). This will put the experiment in a finished state, and all users will fall back to defaults in code. You can still Reset this later, if you decide to revisit the experiment. * If one of the groups has a bug you plan to fix, and you wish to later restart the experiment, you can Reset the experiment. Alternatively, you can [disable a group](/experiments/implementation/disable-group) with bug if you intend to keep the other test groups in experiment running. Resetting will put the experiment into an unstarted state. Every user will get the default experience - either the value defined in code or the layer default. Additionally, the "salt" used to randomize a user's group will be changed. This means that when you start the experiment again, your users will be randomly assigned to a group that is not necessarily the same group they were in prior to the experiment being reset. This is important because it makes sure that the new result for the group that was not performing well due to an issue like a bug or bad experience in the previous run, is not negatively affected even after the issue is addressed in the new version. ## Archiving an Experiment After an experiment’s decision has been made, or when it is abandoned or reset, you can archive the experiment. Archiving preserves the experiment’s history and results, and makes the experiment as read-only. If you later decide to make the experiment active again, you can unarchive the experiment, which makes it ediable but does not automatically restart the experiment. If the unarchived experiment was previously part of a layer and you intend to include it again, you’ll need to manually reset allocation from that layer. # Make a Decision Source: https://docs.statsig.com/experiments/ending/make-decision Analyze Statsig experiment results and decide whether to launch, abandon, or continue an experiment based on metric impact and statistical confidence. Making a decision for an experiment enables you to 'ship' the winning group to all your users. After you make the decision, the variant that your users see depends on whether you're using a **targeting gate** for your experiment. The results for your experiment will still be accessible after you make a decision, but they will stop updating. The last day of metrics will be the day you "make a decision" on the experiment. ## Experiments With No Targeting Gate When you ship a group in an experiment with no targeting gate, the parameter values from the shipped group will become the default values for *all* your users going forward. If the experiment happens to use parameters from a layer, the layer's parameters will now take on the shipped group's parameter values as their defaults. These are the values that *all* your users will see going forward. For example, suppose you have a **Demo Layer** that's configured with a parameter, **a\_param**. It's default value is set to *layer\_default* as shown below. Layer configuration showing default parameter value before experiment Say you decide to create an experiment, **Demo Experiment** in **Demo Layer** as shown below. Experiment creation dialog in Demo Layer with control and test groups You set up **Demo Experiment** with two groups: **Control** and **Test**, intending to experiment with new values for the layer-level parameter, **a\_param** as shown below. Experiment parameter table comparing control and test values for a_param Now if you decide to ship the **Control** group for the **Demo Experiment**, **a\_param** will take the value set for the **Control** group as its default: *experiment\_one\_control* Layer defaults updated to use control group value experiment_one_control On the other hand, if you decide to ship the **Test** group, **a\_param** will take the value set for the **Test** group as its default: *experiment\_two\_test* Layer defaults updated to use test group value experiment_two_test ## Experiment With a Targeting Gate / Targeting Rules When you decide to ship a group in an experiment configured with a targeting gate or targeting rules, you can decide whether to continue targeting after shipping. * If you decide to *discontinue* targeting, the parameter values from the shipped group will become the default values for all your users going forward. If the experiment happens to use parameters from a layer, the layer's parameters will now take on the shipped group's parameter values as their defaults. These are the values that your users will see going forward. * If you decide to *continue* targeting with a **targeting gate**, this will add an override to the experiment layer so that: * all users who **pass** the targeting gate will see shipped group's parameter values * all users who **fail** the targeting gate will see the default value (layer-level parameter defaults or the defaults you set for the parameter in your code) * If you decide to *continue* targeting with **targeting rules**, the experiment will only be shipped to users who pass the inline targeting rules as set in the experiment setup. **Shipping with Targeting On** If you decide to continue targeting, shipping a group will not update the default value of any layer parameters. For example, suppose **Demo Experiment** in a **Demo Layer** that has a parameter called **targeted\_layer\_param**, whose default value is set to *targeted\_layer\_default\_value*. When you decide to ship **Demo Experiment**, if you discontinue targeting, **targeted\_layer\_param** will now take on the value from the **Control** group, *targeted\_layer\_control*, as its default. Targeted layer parameter default switching to control value when targeting disabled On the other hand, if you decide to continue targeting, **targeted\_layer\_param** will now acquire an override so that: * all users who **pass** the targeting gate will see the *overridden* value of the parameter * all users who **fail** the targeting gate will see the *default* value of the parameter In this case, the default value of **targeted\_layer\_param** in **Demo Layer** will not change. Also, any users who pass the targeting gate will not be eligible for future experiments run in this layer. For this reason, we do not encourage shipping experiments with targeting on, especially when the experiment is in a layer. Layer override showing targeted users receiving shipped parameter while others keep default ## Rolling Out an Experiment Group Rolling out an experiment group is an option available when you have decided the winning variant, but want to avoid a sudden, large shift of traffic into the winning variant group. You can use automated rollouts to schedule gradual rollout phases, which will increase your shipped group size to the desired percentage by reallocating users from all other groups proportionally. ### Setting up Rollouts To set up rollouts, open the make decision form and select the winning group. Here you will be able to either use automated rollouts, or ship with rollout. Make decision rollout options interface The ship with rollout option allows you to immediately update the shipped group size. Manual rollouts will clear any automated rollout phases. Ship with rollout configuration screen Alternatively, you can set up automated rollouts, which will open the following dialog that you can populate with the rollout phases: Automated rollout phases configuration dialog From here, you can configure each phase of your Scheduled Rollout. To add phases to your rollout, click **Add Phase** and configure as many phases as you want. Each scheduled rollout phase includes- * Rollout date * Rollout time\* * Pass percentage Rollout times are available in 15 minute increments. Additionally, each configured phase represents a discrete increase to the next rollout percentage, not a gradual rollout amortized over the course of the entire phase. Upon saving, you will be able to see a preview of the rollout and commit the schedule: Rollout schedule preview and commit interface ### Resizing Logic During each phase, the rollout group is resized to the desired percentage, and all other groups are scaled proportionally in the following way: You have *n* groups with sizes: G₁, G₂, ..., Gₙ Their total sum is: 100 = G₁ + G₂ + ... + Gₙ. Now, suppose you **set one group** (say the k-th group) to a **new size** Gₖ′. Let Delta = Gₖ′ − Gₖ. Because the **grand total** must remain 100, you need to **adjust the remaining groups** proportionally. Let T = 100 - Gₖ Then, for each group i ≠ k, Gᵢ′ = Gᵢ − (Delta × Gᵢ / T). In other words, each group other than k is decreased (or increased, if Delta \< 0) by its fraction of T. ### A few notes 1. Experiment results will be frozen to a snapshot of when the rollout decision is made. 2. Rolling out a group to 100% does not fully ship the experiment, meaning configurations such as experiment/layer allocation, targeting, overrides, and so on will not change. To fully ship the experiment go through the usual flow without specifying any rollouts. 3. Groups can be rolled out and back, but the rollout % cannot be lower than the group's original size. This is currently a beta feature. ## Shipping with a Holdback Shipping with holdback lets you release an experiment variant (the “shipped” group) to most of your users while keeping a percentage in the control group for ongoing comparisons. Here’s how it works: 1. Make a decision and select a group to ship: From the Make Decision dropdown, choose which variant you want to ship. 2. Turn on the Ship with holdback option and specify the percentage of users you want to keep in the control (holdback) group. Ship with holdback dialog specifying control percentage in Make Decision modal 3. Allocation of users: The control (holdback) group is set to your specified percentage. The remaining users are assigned to the shipped experience. 4. Splitting the shipped group: The shipped group is divided into two segments: Test and Launched. * The size of Test segment will have the same percentage allocation as the holdback group for an [equal sized comparison](/experiments/holdouts-introduction#how-to-read-holdouts) of 50:50. * The Launched segment will no longer appear in the pulse results, but users in this segment will continue to receive the shipped experience. Pulse results display separating test and launched segments when shipping with holdback 5. Continue monitoring the pulse results that evaluates the Test segment vs. Control (holdback) 6. Once you decide to end the holdback, you can make a decision to ship the variant to everyone which will include users in holdback. Some users currently in control will move to the shipped group to achieve the desired allocation. However, any user who has previously been in a test or shipped group will not be reassigned to control. New users—those who have never been exposed to the experiment will be assigned based on the allocation percentages of each group. Note that the new pulse results (Test segment vs. Holdback) will start when you ship with holdback, but your original experiment's results from the point of holdback decision will be retained and remain available to you. From the history of experiment, you will see the new log of experiment decision and a 'View Results Snapshot' button where you can view the read-only snapshot of original results. Snapshot of original experiment when using ship with holdback By using shipping with holdback, you maintain a dedicated, stable control group alongside a representative test segment of the shipped experience, making it easy to measure ongoing performance and user behavior post-launch. This is currently a beta feature. # Stop Assignments Source: https://docs.statsig.com/experiments/ending/stop-assignments Stop enrolling new users into a Statsig experiment while continuing to analyze existing users so results stabilize before making a launch decision. ## Stopping New User Assignments in an Experiment You have been running an experiment for a while. Your users are split across various control and treatment groups. But now you want to stop enrolling more users into your experiment and going forward analyze only the users who have been exposed thus far. With the Stop Assignment option, you can do exactly that. You will need to configure [Persistent Assignment](/server/concepts/persistent_assignment) for this feature to work. If you don't configure this, people already exposed to the treatment groups will no longer get the treatment experience. Persistent Assignment is required even if you use stable identifiers. ## What Stop Assignment Does * Sets experiment allocation to 0% - no new users will be enrolled in the experiment * Preserves existing user assignments - users already exposed continue to receive their control or treatment experience (only if Persistent Assignment is configured) * New users will not be checked against the experiment and instead get the project’s default experience. To ship a specific variant to all new users, make a decision on the experiment or set this in code. ## Enabling Stop Assignment Option The **Stop Assignment** option must first be enabled in Project Settings to show as an option in the **Make Decision** modal. To enable this option, head to **Settings** --> **Project Info** and toggle on the **Enable stop new user assignments for experiments** setting. Project settings toggle to enable stop new user assignments ## How it Works You can stop assignment for an experiment by clicking the Make Decision dropdown as shown below. Make Decision dropdown highlighting Stop Assignment option Things you should keep in mind when stopping assignments for an experiment: * Once you stop assignment, the experiment will stop enrolling new users in it. That is, the experiment will stop performing checks on new users. * Previously exposed users will keep receiving the consistent control vs treatment experience. Make sure to configure [Persistent Assignment](/server/concepts/persistent_assignment) to preserve the user variants. * The analysis will continue as the experiment keeps recording new data points for the already exposed users. Stop Assignment is an irreversible decision. Once you make this decision, you cannot “resume” assigning users into your experiment. # Metric Insights and Aggregated Impact Source: https://docs.statsig.com/experiments/exploring-results/aggregated-impact How Statsig calculates the aggregated business impact of an experiment across metrics so you can quantify total launch impact in one summary view. ## Metric Insights and Aggregated Impact Statsig's Insights page provides a clear view of how experiments and feature gates impact a specific metric of interest. It not only helps answer key questions such as "How much impact have I driven?", but also serves as a powerful tool for diagnosing unexpected changes in metrics. Insights presents a reverse perspective of the [Pulse](/pulse/read-pulse) view. While Pulse measures the impact of a new feature on all your metrics, Insights allows you to focus on a single metric and identify which tests are impacting it the most. This makes it particularly useful for assessing your or your team's impact, as well as setting realistic goals for your team or company. ## How to read Insights 1. Navigate to the Insights section on the Statsig console: [https://console.statsig.com/](https://console.statsig.com/) . It is also available in the insight tab for each metric. 2. Select a metric that you want to observe from the selector drop down at the top of the page. 3. Select the ID type, time window and other filters that you want to observe. 4. Based on the filters you choose, you can see the relative impact, topline impact and projected launch impact for any experiment/gate which has this metric. 5. We also sum up the projected launch impacts, adjust based on false positive risk ('winner's curse') and show as the 'Aggregated Impact Estimate'. Insights dashboard showing aggregated impact estimates for a metric ## How the math works Check how the topline and projected launch impact are calculated in this [doc](/stats-engine/topline-impact/#computing-projected-launch-impact). To estimate false positive risk and calculate Aggregated Impact, we use the methodology in this [paper](https://dl.acm.org/doi/10.1145/3534678.3539160) which is widely adopted across the industry. Specifically: $$ Aggregated Impact=\sum_{i}{(1 - FPR_i) \times Projected Launch Impact_i} $$ Where the [projected launch impact](/stats-engine/topline-impact/) is an estimate of the topline impact assuming a decision is made and the test group is launched to all users; the false positive risk is calculated by the following formula: $$ FPR_i = \frac{\alpha_i \times \pi}{\alpha_i \times \pi + (1 - \beta_i) \times (1 - \pi)} $$ In this formula, $ \alpha_i$ is the significance level for experiment i, $ \beta_i$ is the type II error, and 1 - $ \pi$ is the prior success rate based on historical experiment results. # Differential Impact Detection Source: https://docs.statsig.com/experiments/exploring-results/differential-impact-detection Statsig automatically flags experiments with extreme differential impacts on sub-populations so you can identify segments driving aggregate metric changes. ## What is Differential Impact Detection? Experiments can have interesting effects on sub-populations that are easily missed. They might have a bug that impacts only a certain browser, OS, or country. If the topline impact isn't significant or is canceled out by other changes - these are missed. Statsig will automatically flag experiments when extreme differential impacts are detected for any sub-population you have configured. Once configured, experiments are analyzed for differential impact when Pulse is loaded after Day 1, Day 3 and when the Target Duration is met. Differential impact detection alert in Pulse results ## Enabling this Configure the "Segments of Interest" you want automatically evaluated for Differential Impact Detection. On Statsig Cloud, these are user properties in the [User Object](/concepts/user) you configure when using the Statsig SDK. On Statsig Warehouse Native they can be configured as an [Entity Property](/statsig-warehouse-native/features/entity-properties) too. Segments of Interest configuration interface This feature is also referred to as **Heterogeneous Treatment Effect** or **Segments of Interest**. ## Seeing Differential Impacts If extreme outliers are found for a segment you have configured, Statsig will flag this when you're looking at Pulse results. You will be able to see the data broken out by segments in the Explore section of your Pulse results. Pulse results showing differential impact by segments ## Methodology We use a Welch’s t-test to compare the treatment effect for a particular segment of users to the treatment effect for all other users since we expect to potentially see unequal population variances when comparing user segments. The average treatment effect is calculated as follows: $\overline{TE} = \overline{X_t} - \overline{X_c} $ The variance in treatment effect is calculated as follows: $var(TE) = var(X_t) + var(X_c)$ The n of the treatment effect is calculated as follows: $n_{TE} = min(n_t, n_c)$ With these calculations, we can determine the t-statistic and degrees of freedom as we would for any experiment using [Welch's t-test](/stats-engine/p-value#welchs-t-test). We then use a Bonferroni Correction to adjust our alpha to avoid false positives, setting a threshold where we consider there to be a high likelihood of heterogeneous treatment effect or some likelihood of heterogeneous treatment effect. We consider a specific dimension vs the rest calculation for one metric of one test variant vs the control to be one "comparison" for the sake of the Bonferroni Correction. $\text{high chance of HTE, } \alpha = \frac{0.01}{\text{number of comparisons}}$ $\text{some chance of HTE, } \alpha = \frac{0.05}{\text{number of comparisons}}$ ## Viewing Results You can go to the explore tab of your experiment and filter to the Differential Impact Detection query type to see all historical analyses. ## # Interaction Detection Source: https://docs.statsig.com/experiments/exploring-results/interaction-detection Detect interactions between overlapping Statsig experiments to understand whether concurrent tests are confounding results or affecting metric movements. ## What is Interaction Detection When you run overlapping experiments, it is possible for them to interfere with each other. Interaction Detection lets you pick two experiments and evaluate them for interaction. This helps you understand if people exposed to both experiments behave very differently from people who're exposed to either one of the experiments. Essentially you want to answer this question: **Does being in both experiments change the effect** **on a metric in a way that’s different than just summing the individual effects?** ## Should I worry about it? Our general guidance is to run overlapping experiments. People seeing your landing page should experience multiple experiments at the same time. [Our experience](https://www.statsig.com/blog/embracing-overlapping-a-b-tests-and-the-danger-of-isolating-experiments) is echoed by all avid experimenters ([link](https://www.microsoft.com/en-us/research/articles/a-b-interactions-a-call-to-relax/)). Teams expecting to run conflicting experiments are typically aware of this and can avoid conflicts by making experiments mutually exclusive via [Layers](/layers) (also referred to as Universes). ## How to use it You can initiate an Interaction Detection analysis as a special type of custom query on top of an Experiment (Experiment -> Results -> Explore -> Interaction Effect Detection) Interaction Detection interface in Statsig console You can pick a second experiment to analyze (along with metrics) and we will show you a quick summary. In each section: * By Groups: gives you the unit counts based on the collective slice of group assignment * Metric Summary Results: shows you the estimated intervals of difference in metric lift * Overlapping Unique Users: offers an overview of the general traffic intersection of the two experiments Interaction Detection analysis results dashboard See more examples in this [article](https://www.statsig.com/blog/interaction-effect-detection). ## Methodology Assume you have two experiments A & B, both have two experiment groups control & test. The interaction effect will calcuate the overlapping impact on users who are exposed in both experiment A & experiment B. $$ \Delta_{\text{treatment effect}} = (\overline{X_{testA \cdot testB}} - \overline{X_{testA \cdot controlB}}) - (\overline{X_{controlA \cdot testB}} - \overline{X_{controlA \cdot controlB}}) $$ $$ \Delta_{\text{treatment effect}}{\%} = \frac{\Delta_{\text{treatment effect}}}{(\overline{X_{controlA \cdot testB}} - \overline{X_{controlA \cdot controlB}})} $$ $$ variance = \frac{Var(X_{testA \cdot testB})}{n_{testA \cdot testB}} + \frac{Var(X_{testA \cdot controlB})}{n_{testA \cdot controlB}} + \frac{Var(X_{controlA \cdot testB})}{n_{controlA \cdot testB}} + \frac{Var(X_{controlA \cdot controlB})}{n_{controlA \cdot controlB}} $$ ### Intuition * The part $\overline{X_{testA \cdot testB}} - \overline{X_{testA \cdot controlB}} $ tells you: “If someone is in A’s test, how much does changing B from control→test change the metric?” * The part $\overline{X_{controlA \cdot testB}} - \overline{X_{controlA \cdot controlB}}$ tells you: “If someone is in A’s control, how much does changing B from control→test change the metric?” * By subtracting those two, you see whether the effect of B differs depending on whether you’re in A’s test or control group → that difference is the interaction effect. If that result is not statistically significant, then the effect of B is the same no matter which group you’re in for A (so no significant interaction). If it’s statistically significant, then the effect of B depends on A’s assignment (so there is a significant interaction). ### Directionality The direction (positive or negative) of the interaction effect tells you **whether the combination of the two treatments amplifies or dampens each other’s effects**. Think of the interaction effect like this: $Interaction Effect=(Effect of B when A=Test)−(Effect of B when A=Control)$ So: * **Statistically Positive interaction** → B’s effect is **stronger** when A is also “on” * **Statistically Negative interaction** → B’s effect is **weaker** (or reversed) when A is “on” ### Magnitude The magnitude (absolute size) of the interaction effect indicates how much the combined impact deviates from simple additivity. For example: * Experiment A produces a **+5%** lift on your metric of interest * Experiment B produces a **+5%** lift on your metric * The interaction effect between A and B is **–3%** This means that when both experiments are rolled out together, their overall impact is expected to be **about 3% lower than the sum of their individual effects** — in other words, the features interfere with each other. To obtain the most accurate estimate of the true combined impact, it’s best to run a new experiment that includes both features together. Because experiments A and B may not have run over exactly the same time period or under identical conditions, differences in timing and seasonality can slightly influence the measured interaction magnitude. When an experiment includes more than two groups, interaction effects are evaluated **pairwise between groups**. You can view the interaction effect for specific group combinations by selecting the desired groups in the **“Select Comparison”** dropdown, as shown in the UI above. # Meta-Analysis Source: https://docs.statsig.com/experiments/exploring-results/meta-analysis Combine results from multiple Statsig experiments into a meta-analysis to evaluate the overall impact of a series of related A/B tests over time. ## The Concept As teams run a number of experiments, it is possible to glean learning across these experiments. This is meta-analysis. Examples of learning people seek to derive include * How hard is a metric to move * Are there more sensitive proxies for the metric we care about? * How are teams doing relative to each other? We've worked with multiple companies to get them to thousands of trustworthy experiments a year. Our inspiration here was looking at what they were trying to learn across these tests. We've built this to be useful whether you're running 50 experiments a year or 5000. Feel free to reach out to help influence our roadmap in [Slack](https://statsig.com/slack). ## Experiment Timeline View This view lets you to filter down to experiments a team has run. At a glance you can answer questions like 1. What experiments are running now? 2. When are they expected to end? 3. What % of experiments ship Control vs Test? 4. What is the typical duration? 5. Do experiments run for their planned duration - or much longer or shorter? 6. Do experiments impact key business metrics - or only shallow or team level metrics? 7. How much do they impact key business metrics? Experiment timeline view dashboard ## Metric Impact (Batting Average) The "batting average" view lets you look at how easy or hard a metric is to move. You can filter to a set of shipped experiments and see how many experiments moved a metric by 1% vs 10%. Like with other meta-analysis views, you can filter down to a team, a tag or even if results were statistically significant. Common ways to use this include * Sniff testing whether the claim that the next experiment will move this metric by 15% is a good idea. * Establishing reasonable goals, based on past ability to move this metric Metric batting average analysis chart ## Metric Correlation View This view lets you visualize two metrics (each data point is an experiment) and visually inspect them for correlation. Often the metric you want to move isn't very sensitive and takes a while to measure. It is helpful to find metrics that are more sensitive and faster to measure - and run experiments on this. This view lets you plot two metrics on the same chart - each data point is an experiment's impact on them. You can quickly get a sense for whether the metrics tend to move together - or not. You can also remove outliers, filter down to a team's experiments or download the underlying dataset. In this hypothetical example - "Checkouts" is the metric you want to move, but it's not very sensitive. "AddToCart" correlates well with "Checkouts", while "ViewItemDetail" doesn't. Metric correlation scatter plot Metric correlation analysis interface ## Metric Insights This view lets you pick a metric and see all experiments and feature rollouts that impact this metric. [Learn more](/aggregated-impact). 417923655-430563dc-4794-4d69-a314-36c76a6fcf74 ## Knowledge Bank The KB acts as a searchable repository of experiment learning across teams. It helps you find shipped, healthy experiments and gain context on past effort and generate ideas on new things to try. Make it easy for new team mates to explore and find experiments a team ran, or where a topic was mentioned. Our meta-analysis tools offer more structured means to discover and look across your experiment corpus, but when you do want free text search, this exists. Knowledge bank search interface # Holdouts Source: https://docs.statsig.com/experiments/holdouts-introduction Measure the cumulative impact of multiple features with holdouts, including how Holdout Pulse compares held-out users against a balanced non-holdout group. Holdouts measure the aggregate impact of multiple features. A holdout keeps a group of users back from a set of features for measurement. While each A/B test or experiment compares control and test groups for that feature, a holdout compares the holdout group (Control) against a balanced group of users who were not held out and continued through the normal rollout or experiment behavior for the included features. ## How to use Holdouts 1. To create a new holdout, navigate to the [Holdouts section on the Statsig console](https://console.statsig.com/holdouts) (it is a specialized kind of experiment). 2. Click the Create New button and enter the name, description and unit type of the holdout that you want to create. 3. You can choose to either create a global or a selected holdout. A global holdout is automatically added to any new feature with the same unit type, and is meant to capture the aggregate impact of all features developed after the holdout began (individual features may be opted out as needed). A selected holdout captures the aggregate impact of a specific selection of features that you want to hold off. 4. By default Holdouts apply to a % of all users (Population = Everyone). You can optionally target the Holdout at a subset of users by applying a Targeting Gate (Population = Targeting Gate). e.g. If you wanted an iOS users only Holdout, you could apply a Targeting Gate that only passes iOS users. 5. You must set the percentage of users to be held-out between 1% to 10%. Statsig recommends a small holdout percentage to limit the number of customers who don’t see new features. Holdout creation interface Holdout configuration settings ## How to read Holdouts Holdouts on Statsig use the [same "equal variant" methodology](/feature-flags/view-exposures#gate-exposures) as Feature Gate rollouts, whereby metric lifts are computed by equal sized groups to calculate holdout lift. You can read more about the advantages of this methodology in ["A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments”](https://drive.google.com/file/d/1oK2HpKKXeQLX6gQeQpfEaCGZtNr2kR76/view) by Ron Kohavi, Alex Deng, & Lukas Vermeer. Accordingly, the Cumulative Exposures panel for a given Holdout shows total exposures of the Holdout, broken down into three groups: 1. **In holdout (Control)**: Units that were included in the Holdout and used for analysis. 2. **Not in holdout (Test); used for analysis**: Units that were not included in the Holdout and were selected for comparison against the holdout group. 3. **Not in holdout (Test); not used for analysis**: Units that were not included in the Holdout and were not used in the lift calculation. For units not included in the Holdout, Statsig generates the two "Not in holdout" groups using random sampling. The group used for analysis is sized to balance the comparison against the holdout group. Holdout metric lifts represent the cumulative impact of launched and active experiments on the Holdout group vs. the same percentage of the rest of the population continuing through the normal behavior for the included rollouts and experiments. The "Not in holdout (Test); used for analysis" group is not necessarily made up of users who saw every treatment. Those users follow the normal non-holdout behavior for each included gate or experiment. In the example below, the 1% Holdout compares the metric values of users in Holdout vs. 1% of users not in Holdout. It does not compare the 1% Holdout against the full remaining 99% of users. The launched features are having an overall negative effect on the "Add to Cart" metric. Holdout pulse results showing metric lift comparison between holdout and exposed users ## Best Practices 1. **Size** - Statsig recommends a low single-digit holdout percentage, say 1% – 2%, to limit the number of customers who don’t see new features. 2. **Duration** - Statsig recommends operating holdouts for a period of three to six months, and then releasing the holdout. Prolonging the holdout period may increase the complexity of your software as you’d have to maintain a functioning product with no new features for a longer period. 3. **Back testing** - Occasionally you may want to turn off a set of features that you have already released to measure the effectiveness of those features. Statsig doesn’t recommend this as it turns off features that users are already using and relying on. However, when a "back measurement" is critical, you can use Holdouts to turn off a set of features and automatically compute the impact of this set of features. 4. **Plan holdouts before launch** - Create holdouts before launching the features or experiments they are intended to measure. If you add a holdout to an already-running experiment, users who were previously assigned or exposed can still fall into the holdout on future evaluations, which may change their experience mid-experiment and contaminate measurement. ## Unit ID Types By default, holdouts are based on User ID. To use a different ID type, select it from the drop down menu during the holdout creation. Holdouts can only be applied to Experiments and Feature Gates that use the same randomization unit. If a team plans to run experiments on both User ID and Stable ID, two separate holdouts are required to evaluate the cumulative impact of each type of experiment. Holdout unit ID type selection ## Holdout effects on Gates & Experiments SDK methods Holdout behavior applies on evaluation, including for users who may have previously seen a gate or experiment before it was added to the holdout. ### Feature Flags/Gates * For users in holdout, gates will always return `False`. ### Experiments * For users in holdout, if the experiment *is not in a Layer*, calls to get experiment parameters will always return the "default value" passed in code. * For users in holdout, if the experiment *is in a Layer*, calls to get experiment parameters will return the values defined in the Layer defaults in the Statsig console. When you ship an experiment in a layer - this would normally update the layer defaults, however, users in the holdout will *not* see those defaults, with the layer instead having a new set of default parameters just for held-out users: Layer Holdout Params ## Ending a Holdout To end a holdout and allow users in the holdout group to see all held-out features, you can disable the holdout. Disabling it stops tracking the effects of those features, but the results will still be retained for future reference. Alternatively, you can delete the holdout if it was created by mistake or if you no longer need to keep the results. # Disable a Group Source: https://docs.statsig.com/experiments/implementation/disable-group Disable poorly performing experiment groups in Statsig while keeping other variants running, so users see a safe fallback without stopping the test. ## Disabling a Group in an Experiment Sometimes you start an experiment with multiple test groups, only to find that one of the groups is performing very poorly or creating bad user experiences. The other test groups are fine and you want to keep them running, but stop the bad experiences. For these situations, we allow you to disable an experiment group. ### What to expect when group is disabled * Users already assigned to the disabled group will begin to receive the default experience * New users can be assigned to the disabled group and receive the default experience * Exposures for the disabled group will continue to be tracked * Sample Ratio Mismatch (SRM) should not be triggered from the disabled group * Experiment results will continue to be calculated for the disabled group We keep showing the group and logging exposures so that you can verify that the user experiences have recovered after the group was disabled. ### How To * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Experiments** * Select the experiment where you want to disable a group * Click on the **...** menu in the top right corner * Select the **Disable A Group** option * Select the experiment groups you would like to disable, and hit **Confirm** Experiment overflow menu showing Disable a Group option Disable group dialog listing experiment groups with toggles ## Reenable a Group If you've previously disabled a group but want to revert this change, you can reenable a group by using the **...** menu and selecting **Disable or Enable A Group**. Menu option to disable or enable a group in experiment controls Modal showing reenable group selections Be mindful that users in any reenabled group might have experienced multiple treatment experiences over the course of the experience. This may be reflected in the metric data collected for the group. # Getting the Group Source: https://docs.statsig.com/experiments/implementation/getting-group Why you should read Statsig experiment parameters in code instead of checking group names, with examples for cleaner, more maintainable implementations. One common misconception when working with experiments on Statsig is trying to check the experiment group in code. Checking the experiment group *in code* is actually an anti-pattern. It's not necessary with Statsig, and it will limit your ability to quickly test different variants. Experiment groups are very useful for understanding what a set of parameters represents in the Statsig console. Comparing "Sorted Long List" vs "Default Search Results" is easier to discuss than trying to understand what the `sorted = true, length = 10` parameters represent. That being said, in code, its much more powerful to directly check parameters and their values. It's also simpler to reason about: rather than hard coding in a particular value, your variable is dynamically evaluated by Statsig. In this way, parameters are the building blocks of your experiments when coding, rather than group names. ### Example: Group Names vs Parameters Hard coding experiment group names would be both very fragile and very limiting. Let's see why. In code, if you tried to use experiment groups, your function might end up looking like this: ```ts theme={null} async function getSearchItems(user: StatsigUser, searchTerm: String): String[] { const results = index.get(searchTerm); const experiment = statsig.getExperimentSync(user, 'search_results'); // NOTE - these APIs don't actually exist - this is for the sake of an example if (experiment.groupName === 'Sorted Long List') { return results.sort().slice(10); } else if (experiment.groupName === 'Sorted Short List') { return results.sort().slice(3); } else { return results; } } ``` There are a few problems with this code: 1. Its very fragile. If the group name in code does not match the name in the Statsig console, you won't return the correct experience 2. Its static. I can't easily add another experiment group without changing the code. I can't add an "Unsorted long list" without a code change. So instead, this is what the code would look like using experiment parameters directly: ```ts theme={null} async function getSearchItems(user: StatsigUser, searchTerm: String): String[] { let results = index.get(searchTerm); const experiment = statsig.getExperimentSync(user, 'search_results'); results = experiment.get("sorted", false) ? results.sort() : results; const numItems = experiment.get("length", 0); return numItems > 0 ? results.slice(numItems) : results; } ``` Now, your code is completely decoupled from the names of experiment groups in the statsig console. You are left with a set of dynamic parameters. You can create whichever experiment groups you want out of these building blocks, and the same code will work. Want to test an unsorted list of 5 items against a sorted list of 20 items? Just set it up in the Statsig console (and name it whatever you want). If you were trying to check group names, you would have to go back and add conditions like: ```ts theme={null} } else if (experiment.groupName === 'Unsorted Short List') { return results.slice(5); } else if (experiment.groupName === 'XL Sorted List') { return results.sort().slice(20); } ``` As you can see, using parameters directly when you are coding is much simpler and more flexible. It makes your code dynamic and offloads experimentation setup to the Statsig console. The group names you configure to describe each set of parameters will make it easy to compare one group against the other when it's time to analyze the results of your experiment. ## Rules The diagnostics stream is meant to be used for debugging your integration, and understanding which groups a user is being bucketed into. The following defines the different Rules you will see, and what they mean. | Rule | Meaning | | ------------------------------- | ------------------------------------------------------------------------------------------------------- | | Not started | The experiment has not been started, so the allocation groups are not determined yet | | Holdout | The user is in a holdout that this experiment references, so they are not in the experiment | | Layer Assignment | The user is not allocated to this experiment because they are bucketed to a different part of the Layer | | Targeting Gate/Inline Targeting | The user does not meet the requirements for the targeting gate or inline targeting | | Not Allocated | The user is not allocated to this experiment because they do not meet the rollout % | | \{group name}\{override name} | The user was forcefully bucketed into a given group by an experiment override | | \{group name} | The user was bucketed into this experiment group | | Abandoned | "Make Decision" selected the control group. This experiment has been abandoned. | | \{group name} (Launched) | "Make Decision" selected this group as the launch group, so the user is seeing the launched experience. | # Implement an Experiment Source: https://docs.statsig.com/experiments/implementation/implement Deploy a Statsig experiment in your code: pull configurations from the SDK, log exposure events, test variants locally, and launch to production users. To deploy an experiment, you'll need to: 1. Pull the experiment configurations in your application 2. Log the events you'll want in your experiment results 3. Test your experiment in development or a lower environment 4. Click "Start"! Every experiment needs to expose users into more than one bucket (#1) and log metrics on their behavior after exposure (#2, called "log events"). Statsig automates many of the annoying parts of setting up an experiment, like writing the code you can use to assign buckets, and conducting analysis on the exposures and log events. The experimenter's job is to devise the experiments - and use our SDKs to accomplish #1 and #2. ## Pulling experiment configurations from Statsig In the code snippets below, we illustrate experimenting on a product demo flow, where you might experiment to improve conversion through the funnel to demo completion. For full implementation details, check out the [SDK documentation](/sdks/getting-started) for the language you'll be using, or walkthrough our example guide for [your first a/b test](/guides/abn-tests). ```js theme={null} const user = { userID: loggedInUserID }; const demoConfiguration = statsig.getExperiment(user, "demo_experience"); // use parameters to control the experience if (demoConfiguration.get("show_banner", false) { showBanner(); } const title = demoConfiguration.get("title", "Start Demo"); banner.setTitle(title); ``` You can also look at a code snippet for your particular experiment by clicking into the code snippet button on the experiment page and selecting the right SDK experiment code snippet button ## Logging events for your scorecard In order to get experiment results for the events and metrics you care about, you should instrument the experience with the proper event logging (or set up an event integration/data warehouse import to send events to Statsig experimentation stats engine). If you'd like to use our SDKs, your code might look like this: ``` statsig.logEvent(user, "demo_started"); ... statsig.logEvent(user, "demo_completed"); ``` Just a few simple events can help you measure how people are moving through a certain funnel in your product, and enable you to experiment on those flows to increase conversion. ## Testing in a lower environment Once experiments are launched, you can't edit the groups without restarting the experiment, as users are already being allocated to each group. We therefore recommend testing each experiment in lower environments before starting. You can do this by clicking the "Test" button in the experiment setup page, then selecting "Enable for Environments". These environments should match your [SDK environment setup](/guides/using-environments/#configuring-environments). Testing in a lower environment and [overrides](/experiments-plus/overrides) can help you manually set your experiment "group" to properly test each variant. Experiment test button interface Once the experiment is enabled for a lower environment, the experiment status will shift from “Not Started” to “Testing”. Lower environment resalt In the results section, you can track cumulative exposures and metric results collected from lower environments. Results data will be displayed in aggregate across all lower environments and won't be distinguishable between individual environments. Metric data and exposure data will be retained even when the experiment is repeatedly disabled and re-enabled for the lower environment. You can resalt experiment in lower environments. This is helpful in the situation where a given user is looking to re-test the experiment E2E. Lower environment resalt ## Starting your experiments Once your experiment has metrics, parameters, and a hypothesis - and you've tested it in a lower environment, you're ready to launch! Click the "Start" button and your experiment will be immediately live in Production. # Export Pulse Results to Your Warehouse in Warehouse Native Source: https://docs.statsig.com/experiments/interpreting-results/access-whn Access and interpret Statsig Warehouse Native experiment results in the console, including Pulse, scorecard, drill-down, and exported analysis views. ## How to Access Pulse Data in Warehouse Native WHN lets you access exposures and metric results across all experiments directly in your warehouse through SQL Views defined in your Statsig project through a metric source. ### Exposures Exposures are automatically written to your warehouse to the table configured in your project setup. You can find the table's location by going to Settings > Data Connection. The table should be located at the `{Database Name}.{Schema Name}.{Exposures Forwarding Table Name}`, e.g. `experimentation.statsig.exposures`. ### Results With a SQL View, you have access to values that include experiment metadata like experiment team, experiment tags, target duration, and experiment settings like CUPED and Sequential testing, then each metric’s metadata like metric tags, and all of the metric lifts-same set of results you see on the Console copy. If you want to start using this feature, simply enable it in your project setting Project Settings > Data Connection > Export. Once you have this enabled, we will automatically handle the setup of SQL view in your warehouse as well as the metric source in your Statsig project. We will then automatically export scorecard metric results to your data warehouse each time an experiment is loaded, generating a new copy. You can differentiate different result versions by using the ds column, which has the timestamp the data was written to your warehouse at. Project settings data connection export interface ### Schema of the Results Data Export Table The default table name used is statsig\_daily\_results. When exports are enabled, Statsig also autocreates a metric source with this name in your Statsig project. | Column | Type | Description | | ------------------------------------ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | | ds | timestamp | The time when the data was written at | | experimentName | string | Name of the experiment | | experimentCreator | string | Creator of the experiment | | experimentTeam | string | Team conducting the experiment | | experimentTags | array of strings | Tags associated with the experiment, represented as an array of strings | | experimentStartTs | number | Start timestamp of the experiment, in milliseconds | | experimentEndTs | number | End timestamp of the experiment, in milliseconds | | targetExposures | number | The target number of exposures for the experiment | | targetDuration | number | The target duration of the experiment | | actualDuration | number | The actual duration the experiment ran, from Start Date to Decision Date. For analyze only experiments, this will be TODAY - Configured Start Date | | controlGroupName | string | Name of the control group in the experiment | | testGroupName | string | Name of the test group in the experiment | | useCUPED | boolean | Whether CUPED was applied in the experiment | | useSequential | boolean | Whether sequential testing was applied in the experiment | | metricName | string | Name of the metric being measured in the experiment | | metricType | string | Type of metric | | metricTags | array of strings | Tags associated with the metric, represented as an array of strings | | higherIsBetter | boolean | Whether a higher value of the metric is better | | isVerifiedMetric | boolean | Whether the metric is verified | | metricTeam | string | Team responsible for the metric | | absoluteDelta | number | The absolute change in the metric value between control and test groups | | absoluteDeltaCI | number | Confidence interval for the absolute delta | | relativeDelta | number | The relative change in the metric value between control and test groups | | relativeDeltaCI | number | Confidence interval for the relative delta | | absoluteDeltaPValue | number | P-value associated with the absolute delta metric result | | toplineAbs | number | The absolute topline metric value for the experiment | | toplineAbsCI | number | Confidence interval for the absolute topline metric | | toplineRel | number | The relative topline metric value for the experiment | | toplineRelCI | number | Confidence interval for the relative topline metric | | projectedTopline | number | Projected topline metric value based on current data | | projectedToplineCI | number | Confidence interval for the projected topline metric | | projectedToplineRel | number | Projected relative topline metric value based on current data | | projectedToplineRelCI | number | Confidence interval for the projected relative topline metric | | controlUnits | number | The number of control group units | | testUnits | number | The number of test group units | | controlTotal | number | Total value for the control group metric | | testTotal | number | Total value for the test group metric | | controlMean | number | The mean value for the control group | | testMean | number | The mean value for the test group | | sequentialTestingAbsoluteDeltaCI | number (optional) | Confidence interval for the absolute delta with sequential testing enabled | | sequentialTestingRelativeDeltaCI | number (optional) | Confidence interval for the relative delta with sequential testing enabled | | sequentialTestingAbsoluteDeltaPValue | number (optional) | P-value for the absolute delta with sequential testing enabled | ## Report Types There are three types of exports: 1. Exposures - A table of all exposed users and their first exposures. This is useful for joining on your own internal data, and running custom queries within your own data warehouse. This can also be used to verify who was in the experiment, what group they were assigned to, and when they were first exposed (around 1-25MB). This will contain: 1. `_first_exposures.csv` - contains a list of users and their first exposure to the experiment. 2. Pulse Summary - This provides precomputed summary experimental data for all metrics and test groups including everything that's visible on Pulse (**around 10-100 kb**). This will contain: 1. `_pulse_summary.csv` - contains Pulse aggregate metrics computed over the duration of the experiment. 3. Raw Data - This provides raw exposures and metrics data at the user-day level. This is best used for manually inspecting data, or recomputing your own statistics (**around 10MB-1GB**). This will contain: 1. `_first_exposures.csv` - contains a list of users and their first exposure to the experiment. If this is the only file you are interested in, you can get this by exporting an "Exposures" report which will be much smaller in size. 2. `_user_metrics.csv` - contains a list of experimental users, and their calculated metrics for each day they were enrolled in the experiment. In WHN, only the Pulse Summary may be exported, as the other two types of data are only stored [in your warehouse](/statsig-warehouse-native/pipeline-overview/#artifacts-and-entity-relationships). The availability of these exports are subject to our retention policy. We hold exposures data for up-to 90 days after an experiment is concluded. We hold raw user-level metrics data for 90 days. ### Pulse Summary File Description - For Feature Gates | Column Name | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | Name of the Experiment or Feature Gate | | rule | Name of the Feature Gate Rule. | | metric\_type | Category of the metric. Different metric\_types are computed differently, including how they're computed in Pulse. | | metric\_name | The name of the metric. For event metrics, this is the name of the event. | | metric\_dimension | The subcategory of the metric. For example, if you log value in LogEvent, then value will show up as a subdimension. dimension = !statsig\_topline indicates that this row reflects an aggregate across all dimensions. | | start\_date | The start date for this measurement | | end\_date | The end date for this measurement | | test\_units | The number of users in the test group | | test\_mean | The average value of this metric across test users (or participating units when applicable) | | test\_stderr | The standard error for the estimate of the mean for test users. This can be used to compute confidence intervals. | | ctrl\_units | The number of users in the control group | | ctrl\_mean | The average value of this metric across control users (or participating units when applicable) | | ctrl\_stderr | The standard error for the estimate of the mean for control users. This can be used to compute confidence intervals. | | abs\_delta | The absolute difference between the test and control mean (test\_mean - ctrl\_mean) | | abs\_stderr | The estimated standard error of abs\_delta | | rel\_delta | The relative difference between test and control mean, sometimes referred to as lift (test\_mean - ctrl\_mean)/ctrl\_mean | | rel\_stderr | The estimated standard error of rel\_delta (abs\_delta/ctrl\_mean) | | z\_score | The calculated Z-score | ### Pulse Summary File Description - For Experiments | Column Name | Description | | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | name | Name of the Experiment or Feature Gate | | rule | Name of the Feature Gate Rule. | | experiment\_group | The group of users for which this metric is computed for. For a feature gate, this is pass/fail. For an experiment, this is the variant name. | | metric\_type | Category of the metric. Different metric\_types are computed differently, including how they're computed in Pulse. | | metric\_name | The name of the metric. For event metrics, this is the name of the event. | | metric\_dimension | The subcategory of the metric. For example, if you log value in LogEvent, then value will show up as a subdimension. dimension = !statsig\_topline indicates that this row reflects an aggregate across all dimensions. | | start\_date | The start date for this measurement | | end\_date | The end date for this measurement | | units | The number of users included in this metric estimate. | | mean | The average value of this metric across units (or participating units when applicable) | | stderr | The standard error for the estimate of the mean. This can be used to compute confidence intervals. | ### First Exposures File Description | Column Name | Description | | -------------------------- | --------------------------------------------------------------------------------------------- | | unit\_id | Refers to the unit identifier used in the experiment (eg. user\_id, stable\_id, org\_id) | | name | The name of the gate/experiment | | rule | For gates, this refers to the rule name | | experiment\_group | The group the user was assigned to | | first\_exposure\_utc | The UTC timestamp when the user was first assigned to the experiment | | first\_exposure\_pst\_date | The date in PST when the user was first assigned to the experiment | | as\_of\_pst\_date | The date this data was generated | | user\_dimensions | JSON-formatted key-value pairs describing the user's attributes at the time of first exposure | ### Unit Metrics File Description | Column Name | Description | | ----------------- | -------------------------------------------------------------------------------------------- | | pst\_ds | The 24hr window the data refers to. All dates are anchored from 12:00a -> 11:59p PST. | | unit\_id | Refers to the unit identifier used in the experiment (eg. user\_id, stable\_id, org\_id) | | metric\_type | The category of the metric | | metric\_name | The name of the metric | | metric\_dimension | The name of the metric dimension. '!statsig\_topline' is the overall metric with no slicing. | | metric\_value | The numeric value of the metric | | numerator | For some metrics, we track the numerator | | denominator | For some metrics, we track the denominator | # Best Practices and Avoiding False Positives Source: https://docs.statsig.com/experiments/interpreting-results/best-practices Best practices for interpreting Statsig experiment results, including avoiding common biases, reading lift correctly, and trusting statistical significance. We have some suggestions on how to interpret Pulse in a scientifically-sound way: 1. Have a hypothesis in mind before viewing Pulse. What are the metric(s) you expect to shift due to the change you made? What else could have happened? What are signs it has gone wrong? 2. Establish a small set of key metrics that are directly related to your hypothesis and would most directly establish that the experiment worked. Having more than a handful of key metrics is usually a sign of an ill-defined hypothesis or shotgun experimentation. Examining too many metrics will lead to a higher false positive rate (seeing results when only statistical noise exists). 3. Avoid cherry-picking results. For example, don't selectively pick three metrics that look good, but ignore the two that don't. Also avoid picking "good" or "bad" numbers that have no connection to your hypothesis. Context matters a lot, and statistically-significant results should have a plausible explanation (false positive can be a plausible explanation). 4. Seeing multiple (independent) effects that are consistent with a plausible story lends credibility that the observed effects are real, even with borderline p-values. 5. Expect to see false positives and be suspicious of statistically significant results with borderline p-values. For example, a 95% confidence interval (5% significance level) is expected to turn up one statistically significant metric out of twenty due purely to random chance. This number goes up if you start to include borderline metrics (eg. p = 0.06). 6. Look beyond your hypothesis. What other effects can you find? Are there tradeoffs? Are there unexpected behaviors? These can reveal information about your users and how they interact with your product. They are often the source of follow-up experiments and new ideas. # Custom "Explore" Queries Source: https://docs.statsig.com/experiments/interpreting-results/custom-queries Run custom queries on Statsig experiment results to explore segments, joins, and aggregations beyond the built-in scorecard and drill-down views. Custom queries are a way to run additional custom experiment analyses on your existing data beyond what is in your main Results tab. You may run them to gain deeper insights from your experiments and feature roll-outs, debug interesting results, or scope down your results to interesting sub-groups. Custom queries allow you to filter or group metrics by event or user dimensions, or filter to a specific set of users to see how an experiment or launch has impacted these users' experience. Custom queries are experimental analyses just like in the main Results tab, and all the same statistical procedures apply. Results are computed as p-values and confidence intervals for your metric deltas, and advanced statistical methods like [CUPED](/stats-engine/methodologies/cuped) and [Sequential Testing](/experiments-plus/sequential-testing) can be used. Be careful when drawing your inferences of Custom Queries, especially when grouping by a dimension with lots of options. This can increase your chance of seeing a false-positive statistically significant result. ### Dimension Loading Timing for Precomputed User Dimensions When viewing results for precomputed user dimensions (which are configured and run on a schedule), be aware that these dimensions are loaded through separate asynchronous explore queries. This means: * The main experiment results will appear first * Precomputed dimensions will continue loading in the background and become available within a few minutes * This timing gap is most noticeable immediately after the first reload of the day * If you see "No dimensions available for this time range" for precomputed dimensions, wait a few minutes and refresh to see if dimensions have completed loading This timing behavior only affects precomputed user dimensions that run on a schedule. User-triggered custom queries do not experience this asynchronous loading delay. ### Running a Custom Query To run a Custom Query, navigate to the **Explore** tab within your experiment. Custom query explore tab interface Custom Query fields: * **Metric(s):** The metric(s) you want to analyze. You can select a single metric, a few metrics, or a Metric Tag. Adding a Tag will include all the metrics within that Tag in your Custom Query. There are three "default" metric selections included as shortcuts: * "Scorecard Metrics", all metrics included in your experiment setup's Primary and Secondary Metrics sections * "Primary Metrics" * "Secondary Metrics" * **Metric Filter:** With metrics selected, you can filter metrics by either Event or User dimensions using the "Add Filter" dropdown. For example, if you wanted to look at your experiment results for Canadian users only, you could filter to "Country = CA". Metric filter dropdown selecting Country equals CA * **Group By:** You can group your Custom Query results by either an Event or User dimension. Whereas Custom Query filters can be applied at the *per-metric* level, the Group By action is at the *query* level (so all included metrics will have whatever Group By you select applied to them). * **Time Range for Metric Data:** The date range you're running your analysis on. By default this will be the "Full date range" of your experiment data. * **(Advanced) ID List Segment filters:** You can choose an ID-list based [Segment](/segments), and your results will only be calculated for users who are in that segment. This can be useful if you forgot to log an important user dimension that you want to filter to, or realized that you only care about a sub-population that you've defined in your own data warehouse. * Careful! This option can easily lead to erroneous and biased results. You will need to make sure the segment is defined based on the user's status *before* they were exposed to the experiment or feature gate. * Similarly, you can choose to *exclude* a certain ID list segment, for example if you want to exclude a set of users who have been retroactively identified as bad actors from your lifts analysis. * **(Advanced) Filter by Exposure Date:** You can also filter the results by Exposure Date which can give you more flexibility. You can choose only include or exclude a date range, or in WHN, you can additionally include/exclude users based on when they were first exposed to the experiment. * This is useful when your metrics have novelty effect, delayed impact, or specific scenarios where you only want to filter your results to certain users. Use it cautiously because it can lead to biased results. User groups in experiment results are based off of first-touch attribution. The filters and grouping applied will be based on the user attributes collected at the time of first exposure in the gate/experiment/layer check. Custom query definition form showing selected metrics and filters ### Viewing a Custom Query in Explore These queries take a few minutes to run (don't worry, we'll send you an email once your results are ready in case you want to hop to another task), but once complete the results will be visible in the **Query History** section of the **Explore** interface. All historical queries (across your team) will be stored here. You can also give your query a display name inline for easier future identification. Explore tab query history list ### Scheduling a Custom Query If you want a daily refresh of a given Custom Query, you can schedule your Custom Query directly from the **Explore** tab. To do this, author the Custom Query you wish to schedule, then tap the "..." menu, then **Schedule**. This Custom Query will now run daily and live in the **Scheduled** tab of your Metric Lifts. Scheduled custom queries tab displaying daily runs Scheduled query configuration interface ### Reviewing Custom Query Results Custom query results look a lot like the main Results tab, because the statistical methods are the same. Statsig uses the same experimental analyses practices on your custom analysis as we do on your main Results. One main point of difference, however, is that your custom query result is a snapshot in time. Once run, the analyses results are saved and will not update if more metric data is collected. If you do want to update your results, you can run a new custom query or schedule custom queries to run at a regular cadence. #### Sequential Testing and Custom Queries If [Sequential Testing](/experiments-plus/sequential-testing) is enabled for your experiment, it can be applied to your custom query results as well. How much and whether to adjust your confidence intervals and p-values will depend on the regular rules of sequential testing: if your custom query doesn't satisfy the experiment's target Days or Unique Exposures from your setup, sequential testing adjustments will be made to your results to account for the under-powered state of the experiment. Since custom queries are computed as a snapshot in time, sequential testing adjustments are computed for that specific analysis only. If you run additional custom queries with more or less data (e.g. more days in the analysis, more unique users in the experiment), the sequential testing adjustments will change accordingly. Some custom queries may have no sequential adjustments applied at all if they meet the configured minimum Days or Unique Exposures. Custom query results table displaying sequential testing adjustments # Pulse Source: https://docs.statsig.com/experiments/interpreting-results/drill-down Drill down into Statsig experiment results by user segment, dimension, or time period to understand which sub-populations drive aggregate metric changes. ## Tooltip Overview A tooltip with key statistics and deeper information is shown if you hover over a metric in Pulse. UI for metric hover card in experiments * **Group**: The name of the group of users. For Feature Gates, the "Pass" group is considered the test group while the "Fail" group is the control. In Experiments, these will be the variant names. * **Units**: The number of distinct units included in the metric. E.g.: Distinct users for user\_id experiments, devices for stable\_id experiments, etc. * **Mean**: The average per-unit value of the metric for each group. * **Total**: The total metric value across all units in the group, over the time period of the analysis. ### Calculation details | Metric Type | Total Calculation | Mean | Units | | ----------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | | event\_count | Sum of events (99.9% winsorization) | Average events per user (99.9% winsorization) | All users | | event\_user | Sum of event DAU (distinct user-day pairs) | Average event\_dau value per user per day. Note that we call this "Event Participation Rate" as this can be interpreted as the probability a user is DAU for that event. | All users | | ratio | Overall ratio: sum(numerator values)/sum(denominator values) | Overall ratio | Participating users | | sum | Total sum of values (99.9% winsorization) | Average value per user (99.9% winsorization) | All users | | mean | Overall mean value | Overall mean value | Participating users | | user: dau | sum of daily active users | Average metric value per user per day. The probability that a user is DAU | All users | | user: wau, mau\_28day | Not shown | Average metric value per user per day. The probability that a user is xAU | All users | | user: new\_dau, new\_wau, new\_mau\_28day | Count of distinct users that are new xAU at some point in the experiment | Fraction of users that are new xAU | All users | | user: retention metrics | Overall average retention rate | Overall average retention rate | Participating users | | user: L7, L14, L28 | Not shown | Average L-ness value per user per day | All users | ### p-Value In Null Hypothesis Significance Tests, the p-value is the probability that such an extreme difference can arise by random chance when the experiment or test actually has no effect. A low p-value implies the observed difference is unlikely due to random chance. In hypothesis testing, a p-value threshold is used to determine which results are due to a real effect and which are plausibly due to random chance. ([p-value calculation](/stats-engine/p-value)) ### Reverse Power Reverse power is the smallest effect size that an experiment can reliably detect in its current state (some studies refer to this value as ex-post MDE). It is calculated from the sample size and standard error from the control group. Importantly, reverse power does *not* depend on the observed effect size. In practice, reverse power answers such questions like: given how the test actually played out, what is the smallest effect we have sufficient power (typically 80%) to detect? For a two-sided test, the reverse power for a given metric X is computed using the following equation: $$ Reverse Power = \frac{(Z_{1-\beta} + Z_{1-\alpha/2})}{\overline{X}_{\text{control}}}\times \sqrt{\frac{\mathrm{var}(\Delta \overline{X})}{N_{\text{control}}}} \times 100\% $$ For a one-sided test, the reverse power for a given metric X is computed using the following equation: $$ Reverse Power = \frac{(Z_{1-\beta} + Z_{1-\alpha})}{\overline{X}_{\text{control}}}\times \sqrt{\frac{\mathrm{var}(\Delta \overline{X})}{N_{\text{control}}}} \times 100\% $$ * $\overline{X}_{\text{control}}$ is the mean metric value across control users * $var(Δ\overline{X})$ is the population variance of delta * $N_{\text{control}}$ are the observed number of units in the control group * $Z_{1-\beta}$ is the standard Z-score for the selected power. Typically ${1-\beta}$ = 0.8 and $Z_{1-\beta}$ = 0.84 * $Z_{1-\alpha/2}$ and $Z_{1-\alpha}$ are the standard Z-scores for the selected significance level in a two-sided test and in a one-sided test. You can enable reverse power as an optional feature. To manage it, go to Settings -> Product Configuration -> Experimentation -> Organization, where you can toggle it on or off. ## Detailed View Click on **View Details** to access in depth metric information. The detailed view contains three sections: * **Time Series**: How the metrics evolve over time * **Raw Date**: Group level statistics * **Impact**: How the experiment impacts the metric ### Time Series In this view, select and drag as needed to zoom-in on different time ranges. Three types of time series are available in the drop-down: **Daily**: The metric impact on each calendar day without aggregating days together. This is useful for assessing the variability of the metric day-over-day and the impact of specific events. It's the recommended time series view for Holdouts, since it highlights the impact over time as new features are launched. Daily metric impact visualization interface **Cumulative**: Shows the cumulative metric impact from the start of experiment over time. This is a good way to observe trends and see how your confidence interval changes over time. Cumulative metric lift visualization interface **Days Since Exposure**: Shows the metric impact based on how long a user has been in the experiment. Daily data for each user is aligned by the day they entered the experiment (Day 0, Day 1, ...etc), not by calendar date. This allows you to distinguish early (novelty) from long-term effects. This view also shows pre-experiment data which identifies biases between groups before the experiment started. This can happen due to random chance or by some issue in the random assignment process. Days since exposure metric visualization interface ### Raw Data This view shows the group level statistics needed to compute the metric deltas and confidence interval. Includes Units, Mean, and Total (explained above), as well as the Standard Error of the mean (Std Err). Details on the statistical calculations are available [here](/stats-engine). ### Impact Experiment impact metrics interface * **Experiment Delta (absolute)**: The absolute difference of the Mean between test groups i.e. Test Mean - Control Mean. The p-value is shown to indicate whether the observed absolute difference is statistically significant. * **Experiment Delta (relative)**: Relative difference of the Mean i.e. 100% x (Test Mean – Control Mean) / Control Mean. * **Topline Impact**: The measured effect that experiment is having on the overall topline metric each day, on average. Computed on a daily basis and averaged across days in the analysis window. The absolute value is the net daily increase or decrease in the metric, while the relative value is the daily percentage change. * **Projected Launch Impact**: An estimate of the daily topline impact we expect to see if a decision is made and the test group is launched to all users. This takes into account the layer allocation and the size of the test group. Assumes the targeting gate (if there is one) remains the same after launch. See [here](/stats-engine/topline-impact) for details on the exact calculation for topline and projected impact. **FAQs about topline impact** *Why is the projected launch impact smaller than the relative experiment delta?* Often times, an experiment can impact only a subset of the user base that contributes to a topline metric. So the relative experiment delta that we observe is effectively diluted when measured against the topline metric value. For example: Consider a top-of-funnel experiment on the registration page. Among users that hit this page, the treatment is leading to more sign ups and a 10% lift in daily active users (DAU). However, our topline DAU metric includes other user segments outside of the experiment, such as long term users that don't go to the registration page. So what was a 10% lift in the test vs. control comparison, may amount to only a 1% increase in overall DAU. *How can the topline impact be higher than the experiment delta?* It's possible for the topline impact to be higher or lower than the experiment delta. This is because the two values are computed differently and have different meaning. Experiment deltas are based on the unit-level averages: The mean value of the metric is computed for each user across all days, and then averaged to obtain the group mean. The topline impact is computed daily based on the total pooled effect from all users, and we take the average across days to show the daily impact. We chose to compute topline impacts in this way because most metrics are tracked on a daily basis and the topline value tends to be computed as an aggregation across all users, rather than a user-level average. For experiment analysis, on the other hand, best practice is for the analysis unit to match the randomization unit, so metrics are aggregated at the unit level first before computing experiment deltas. # Exporting Pulse Reports Source: https://docs.statsig.com/experiments/interpreting-results/export Export Statsig experiment results as CSV, share via link, or pipe to a data warehouse for further analysis in BI tools and notebooks. ## How to Export Pulse Data in Statsig Cloud **Warehouse Native users**: You're viewing the Cloud docs for this page. Metrics and experiments behave differently in Warehouse Native. Read [How to Access Pulse Data in Warehouse Native](/pulse/access-whn). Finding Export Report You can export your Pulse Results for Feature Gates and Experiments. Simply navigate to the relevant "Pulse Results" page, and click "Export Report". Exporting results can take up to 10 minutes. A notification and an email will be sent when the report is ready, and a link will be available under under Project Settings -> Reports. You can export results only if your Pulse screen has results. Export Pulse Report Menu # Frequently Asked Questions on Using Pulse Source: https://docs.statsig.com/experiments/interpreting-results/faq Frequently asked questions about interpreting Statsig experiment results, including p-values, confidence intervals, lift, exposures, and SRM warnings. Interpreting statistical results can be tricky, and often people will have similar questions as we ramp up. Here's some answers! ## I had a stat sig result, but it turned negative. How should I interpret this? In general, you should trust the current result, as it's incorporating more information about the users in your experiment. There's a number of reasons this can happen: * Random noise, which gets diluted as your sample size gets larger * Within-week seasonality (e.g. an effect is different on Mondays), which gets normalized with more data * The population that saw the experiment early early on is somehow different than the slower adopters. This happens frequently - a daily user will likely see your experiment before someone who users your product once a month. You can look at the time series view to get more insight on this * There was some sort of novelty effect that made the experiment meaningful early on, but fall off. Imagine changing a button - people might click on it early out of curiosity or novelty, but once that effect goes away they'll behave like before. You can use the days-since-exposure view to get more insight on this Best practice for timing is to pick a readout date when you launch your experiment (based on a [power analysis](/experiments-plus/power-analysis)), and to disregard the statistical interpretation of results until then. This is because reading results multiple times before then dramatically increases the rate at which you'll get false positives. ## How should I start with interpreting results? Start by using your scorecard metrics to understand if you've moved the metrics you thought you would. You should come into pulse with a hypothesis on what your experiment should drive, and that hypothesis should be answered by your primary metrics. The delta displayed is based on the observed difference between a test and control population. The error bars are a visualization of a confidence interval. A confidence interval is a range of probable values for the difference between groups. A future sample's 95% confidence interval will have a 95% chance to contain the true value of the difference. This is a bit of a wonkish distinction which means that (unfortunately) you can't say that your observed 95% CI has a 95% chance of containing the true value. In practice, the CI is a representative range of what the true value might be. Keep in mind that these results are statistical interpretations and not facts: * If a result is not stat sig, this means you don't have sufficient evidence to reject the null hypothesis (i.e., based on your experiment design the observed result is reasonably likely to have happened by chance). * Generally, you should treat these results as a lack of evidence for your hypothesis * Underpowered tests may lead to neutral results even if a true effect exists * If a result is stat sig, this means that you have sufficient evidence to reject the null hypothesis (i.e., the probability that you would observe this result, or one more extreme, if the two groups' results were identical is below the pre-determined threshold you set). * Generally, you should treat this result as evidence for your hypothesis * Multiple comparisons (many metrics, rerunning an experiment, or grouping by dimensions) greatly increase the chance of seeing a stat sig result when there's *not* a true effect. Be wary of interpreting results when you see those behaviors! * A test that was extremely unlikely to succeed (moonshots, etc.) that has stat sig results have a high chance of that result being a false positive. In cases like this, it's a strong signal but you should consider trying to reproduce the result or running a back-test, or consider reducing your significance level. Once you've looked at the results on your scorecard, we encourage you to use the all-metrics tab and custom queries to get more information on your experiment, but you shouldn't necessarily trust that if those see a stat-sig movement it is a statistically sound interpretation; as mentioned above if you increase the number of metrics you are looking at, you increase the chance that you will see a false positive. Use this section to look for unexpected large regressions and to generate follow-up hypotheses. ## Results aren't showing up for some metrics This normally happens when your company is both using the SDK or event imports, and also importing precomputed metrics from your data warehouse. Since these can run at different times, the data availability may differ. You can adjust your analysis date range to get a full view of your data. ## Our external source shows more exposure events than Statsig. Are data missing? Exposures on the last day (the day you made a decision) are not counted as exposures. Please filter out that day when you analyze your external data. The exact hours which define a "day" for your project depend on which timezone your project is assigned. ## We log categorical metadata for a custom event, but Pulse doesn't show these breakouts. What's wrong? Pulse is only able to show experimental results for various sub-groups of your metric (e.g. iOS vs. Android) when you've configured your metadata as a Dimension. [Value Dimensions](/pulse/read-pulse#value-dimensions) are the most common dimension type as their metadata get logged directly with your custom events. However, value dimensions must be defined in your [custom event setup](/metrics/metric-dimensions). ## Why do I see "No dimensions available for this time range"? You may see this error when trying to view precomputed user dimensions, particularly after the first reload of the day. This happens because: * Dimensions load asynchronously in separate explore queries after the main scorecard results load * The main experiment results will appear first, while dimensions continue loading in the background * Typically, dimensions will become available within a few minutes after the main scorecard loads If you encounter this error, wait a few minutes and refresh the page to see if the dimensions have completed loading. # Participating Units Source: https://docs.statsig.com/experiments/interpreting-results/participating-units Understand how Statsig counts participating units in an experiment, including exposure rules, deduplication, and how to interpret unit counts in results. ## Definition In Statsig, Participating Units (sometimes referred to as participating users) are a subset of an experiment's total exposed unit count relevant to a particular metric. They count the units with a non-zero denominator value for metric types that require a numerator and denominator value. These units are the ones actually used in statistical calculations for ratio-based metrics. Participating units diagram showing subset of total exposed users Ratio metrics are computed only for users that have a non-zero value in the denominator, e.g. the user must have triggered the denominator event on a given day to be included in the daily ratio. Users that don't trigger the denominator event during an experiment are not included in the test vs. control comparison of a ratio metric. ## Example An example of this type of metric for an e-commerce company could be "Total Items Purchased per Order", measuring the average basket size. The numerator would be defined as the count of total items bought, and the denominator would be the total number of orders submitted. If an experiment were run on the checkout flow, only users who actually had at least one Order event would be included in the "Total Items Purchased per Order" ratio metric. # How to Read Experiment Results (Formerly "Pulse") Source: https://docs.statsig.com/experiments/interpreting-results/read-results Read and interpret Statsig experiment results, including scorecards, primary metrics, lift, confidence intervals, and statistical significance indicators. ## Read Experiment Results To read the results of your experiment, go to the **Results** tab, where you will see your experiment hypothesis, **exposures**, and **Scorecard**. ### Exposures Exposures chart showing cumulative users per experiment group At the top of the Results page is the Exposures Chart. Exposures are the unique experimental units enrolled in the experiment. This is typically the number of unique users, and for device-level experimentation, this is the number of devices. The timeline shows you when the experiment was started, and how many exposures were enrolled on any given day. You can see the rate at which users were added into each group of the experiment, how many total users were exposed, and confirm the target ratio matches what you configured in experiment setup. ### Scorecard The experiment **Scorecard** shows the metric lifts for all Primary and Secondary metrics you set up at experiment creation. #### Immediately Post-experiment Start For up to the first 24 hours after starting your experiment (before our daily metric results run), the **Scorecard** section is calculated hourly (this only applies to Statsig Cloud, for WHN projects you will need to reload results on demand or set up a daily schedule). This more real-time scorecard is designed to enable you to confirm that exposures and metrics are being calculated as expected and debug your experiment or gate setup if needed. You should **not** make any experiment decisions based on real-time results data in this first 24 hour window after experiment start. Experiments should only be called once the experiment has hit target duration, as set by your primary metric(s) hitting experimental power. Read more about target duration [here](/experiments-plus/create-new#target-duration). Given data during this early post-experiment start window is designed for diagnostic, not decision-making purposes, you will notice a few key differences between this real-time view and the results that will start showing after daily runs have initiated: * Metric lifts do not have confidence intervals * No time-series view of metric trends * No projected topline impact analysis * No option to apply more advanced statistical tactics, such as CUPED or Sequential Testing All of these are available in daily Results, which will start showing in the next daily run. #### Post-first Day Scorecard Experiment scorecard table displaying metric lifts and confidence intervals The experiment Results daily run calculates the difference between the comparable randomization groups (eg. test and control) across your company's suite of metrics, and applies a statistical test to the results. You can read more about Statsig's stats engine [here](/stats-engine). For every metric, we will show you: * The calculated relative difference (Delta %) * The confidence interval * Whether the result is statistically significant * Positive lifts are green * Negative lifts are red * Non-significant results are grey The formula for calculating lift is: Delta(%) = (Test - Control) / Control Confidence intervals are reported at the selected significance level (95% by default). In a typical two-sided Z-test, we show the confidence interval as +/- 1.96 \* standard error. 99.9% winsorization is automatically applied to event\_count, event\_count\_custom, and sum metrics. This caps extreme outlier values to reduce their impact on experiment results. For metrics added to the **Scorecard** or **Monitoring Metrics** sections of your experiment or gate, you can also apply other optional statistical treatments, such as CUPED (pre-experiment bias reduction) and sequential testing adapted confidence intervals. Read more [here](/stats-engine). * **Experiment results are computed for the first 90 days**: By default, Statsig will compute experiment results for your experiment for only the first 90 days of your experiment. You will be notified via e-mail as you approach the 90 days cap, at which point will be able to extend this compute window for another 30 days at a time. If the experiment runs beyond the compute window, new users will stop getting added into the experiment's result, but analysis for existing users who have been exposed to the experiment will continue to run even if the compute window is not extended, until you make a decision on the experiment. This experiment result calculation window only affects whether a user is included in the experiment's analysis, and does not affect the treatment each user would receive. New users would still receive the experience for the group they get randomized into. ### Experiment Results Views There are a few different views to see your Scorecard metric lifts, namely: * **Cumulative results (default view)**: Displays the aggregate difference between experiment groups and visualizes the corresponding confidence intervals * **Table view**: Displays the same data as the cumulative view but in a table format with additional fields * **Daily results**: Shows the difference between experiment groups aggregated based on days since start of experiment * **Days since exposure**: Shows the difference between experiment groups aggregated based on days since exposure to the experiment Cumulative results includes a detailed view on hover, where you can additionally view the raw statistics used in the metric lift calculations, as well as topline impact. Cumulative results view with hover details ### Dimensions There are two ways in which we can breakdown a given Scorecard metric - one is by a **User Dimension**, the other is by an **Event Dimension**. #### User Dimensions User Dimensions refer to user level attributes that are either part of the user object you log, or additional metadata that Statsig extracts. Examples of these user attributes could be operating system, country, and region. You can create [custom "explore" queries](/pulse/custom-queries) to *filter on* or *group by* available user dimensions. For example, you could "See results for users in the US", or "See results for users using iOS, grouped by their country". Go to the "explore" tab to draft a custom query custom queries #### Event dimensions Events Dimensions refer to the value or metadata logged as part of a custom event that is used to define the metric. If you want to see results for a metric broken down by categories that are specific to that metric, [specify the dimension](/metrics/metric-dimensions) you want to break down by in the **value** or **metadata** attributes when you log the source event. For example, when you log a "click" event on your web or mobile application, you may also log the target category using the **value** attribute as shown below. Statsig will automatically generate results for each category in addition to the top level metric. To see breakdowns for all categories within a metric, click on the (+) sign next to the metric. dimension button dimension results view ### Significance Level Settings These settings can be adjusted at any time to view Scorecard results with different significance levels. * **Apply Benjamini-Hochberg Procedure per Variant**: Select this option to apply the procedure to reduce the probability of false positives by adjusting the significance level for multiple comparisons - [read more here](/stats-engine/methodologies/benjamini-hochberg-procedure). * **Confidence Interval**: Changes the confidence interval displayed with the metric deltas. Choose lower confidence intervals (e.g.: 80%) when there's higher tolerance for false positives and fast iteration with directional results is preferred over longer/larger experiments with increased certainty. * **CUPED**: Toggle CUPED on/ off via the inline settings above the metric lifts. Note that this setting can only be toggled for **Scorecard** metrics, as CUPED is not applied to non-Scorecard metrics. * **Sequential Testing**: Applies a correction to the calculate p-values and confidence intervals to reduce false positive rates when evaluating results before the target completion date of the experiment. This helps mitigate the increased false positive rate associated with the "peeking problem". Toggle Sequential Testing on/ off via the inline settings above the metric lifts. Note that this setting is available only for experiments with a set target duration. analysis settings ### Restarting Results Restart results banner If your experiment has stopped computing results, you can resume updates by clicking the Restart button. There are some important facts to be aware of: * A Restart is not a [Reset](/experiments/ending/ending-experiment#stopping-an-experiment) of your experiment. A Restart will not re-salt (i.e. re-randomize) units in your experiment, and all users will continue to receive the same group assignments. * Statsig will begin computing experiment results anew from the restart point, so your metric results will start over. Old results may still be available in timeseries and explore query views, but they will not be carried forward or updated. * Your Cumulative Exposures chart will update based on new exposures, but the duration of the pause in computations will affect if the chart starts over from zero or your exposure count includes past exposures. It's best to avoid having to Restart Results by actively extending experiments while they're running. Be sure to look out for email alerts from Statsig and check in your experiments regularly. # Reconciling Results Between Experimentation Platforms Source: https://docs.statsig.com/experiments/interpreting-results/reconciling-experiment-results Reconcile differences in experiment results between Statsig and other analysis platforms, including common sources of discrepancy and how to debug them. ## Motivation The same data can yield very different interpretations in experiment results due to the wide variety of analysis methodology available. One of the advantages of modern experimentation platforms is ensuring consistency and transparency in experimental analysis within your organization. This paper is a brief guide to common gaps between platforms, as well as how to identify and resolve them. ## General Approach When companies are evaluating an experimentation vendor, it's common to observe differences in results between their in-house platform and the vendor's platform when they run Proof-of-Concept (POC) validations. We've consistently been able to resolve these gaps with the steps in this document. The high level hypothesis will be that one of the following is true: 1. The metric source data is being read or joined to exposure data differently, invalidating downstream steps 2. Some advanced stats features that are available on the vendor side, but not in-house, are 'working as intended', most often reducing the influence of outliers or pre-experiment bias 3. There is a misunderstanding on how a metric definition works, or how an advanced configuration on a metric or experiment behaves By going through these in order, data teams evaluating a platform can quickly understand and address gaps, or understand the gap and make a decision on if the vendor's approach is acceptable to them. ## Joining Data Based on our observational data, differences in experiment results most often stem from how exposure data is joined with metric data. At the end of this section we will cover a basic check for confirming this isn't occurring. ### ID Formats In some cases, people log IDs in different formats to different places. For example, the binary id `4TLCtqzctSqusYcQljJLJE` maps to the UUID `a0fb4ef0-9d9e-11eb-9462-7bfc2b9a6ff2`, so a company might have the binary ID in their production environment and log that, while their data users work with the equivalent UUIDs. This means that the exposures logged using the binary ID would *not* be able to join with the metric data using the UUID, and results would be empty. As suggested on the 'User Metrics not Calculated' health check, you can check samples for both the metric source in question and the assignment source or diagnostic logstream to confirm that the identifiers are in the same format. ID Resolution can be used to bridge ID type gaps, but is not intended to solve for this scenario; ID Resolution helps you connect identifiers across logged-out/logged-in sessions, or other scenarios where users will commingle their identifiers because of switching identifiers during the experiment. ### Timestamps It is important to analyze metric data only after a user has been exposed to the experiment. Pre-experiment data should have no average treatment effect, and therefore its inclusion dilutes results. #### Statsig Cloud Statsig Cloud uses a date-based join between exposures and metric data. Experiments will include metric data from the whole of the first exposure date for each experimental unit. While some pre-experiment metric data can be included, the average treatment effect of this dilution should be null. This looks like the SQL snippet below: ``` WITH metrics as (...), exposures as (...), joined_data as ( SELECT exposures.unit_id, exposures.experiment_id, exposures.group_id, metrics.timestamp, metrics.value FROM exposures JOIN metrics ON ( exposures.unit_id = metrics.unit_id AND metrics.date_id >= exposures.first_date_id ) ) SELECT group_id, SUM(value) as value FROM joined_data GROUP BY group_id; ``` Statsig's exposures are always in UTC; if metric data is in another timezone it will need to be adjusted to avoid filtering on the wrong comparison. Statsig does support timestamp-based joins for some Enterprise Cloud customers. Please reach out to Statsig if you would like to learn more. #### Statsig Warehouse Native Statsig WHN employs a timestamp-based join for this purpose, with an option for a date-based joins for daily data if preferred. This should look like the SQL snippet below: ``` WITH metrics as (...), exposures as (...), joined_data as ( SELECT exposures.unit_id, exposures.experiment_id, exposures.group_id, metrics.timestamp, metrics.value FROM exposures JOIN metrics ON ( exposures.unit_id = metrics.unit_id AND metrics.timestamp >= exposures.first_timestamp ) ) SELECT group_id, SUM(value) as value FROM joined_data GROUP BY group_id; ``` It's also worth noting that timezones can influence this. Timestamps for Statsig's exposures are always in UTC; if metric data is in another timezone it will need to be adjusted to avoid filtering on the wrong comparison. ### Exposure Duplication Exposure data must be de-duplicated before joining to ensure a single record per user. Many vendors further manage crossover users (users present in more than one experiment group), removing them from analysis and/or alerting if this occurs with high frequency. ``` SELECT unit_id, experiment_id, MIN(timestamp) as first_timestamp, COUNT(distinct group_id) as groups FROM GROUP BY unit_id, experiment_id, group_id HAVING COUNT(distinct group_id) = 1; ``` ### Data Availability When comparing a platform analysis to an **existing** experiment analysis that may have been run in the past, it's possible that the underlying data has since fallen out of retention or has been otherwise deleted. To check this, you can compare the table's retention policy to the analysis dates used in your original experiment analysis to make sure the data still exists. Additionally, you should make sure your experiment in the vendor console is configured to analyze the same time range your original analysis used. ### Validation The validation procedure for the initial metric data and join is to use the query provided in [Timestamps](/experiments-plus/reconciling-experiment-results#timestamps) section, modifying it to run on both platforms to evaluate that a target metric has the same totals per group across both platforms. Warehouse Native platforms have an advantage here in that the SQL dialect and source data will generally be the same in both Vendor code and in your in-house code, making comparisons simpler. We recommend picking one metric of interest, validating this data, and resolving any differences before checking in on Statistical/Metric methodologies. ## Statistical Features Choices in statistical methodologies can significantly impact experiment results. The following are common root-causes for gaps in results, but we always recommend that users closely read the queries being run by the vendor to understand any particulars in methodology. ### Winsorization Outlier trimming, or [Winsorization](https://docs.statsig.com/stats-engine/methodologies/winsorization/), can dramatically alter experiment outcomes. Turning off this feature in Statsig metrics is advisable when doing cross-system comparisons unless it is being manually applied. ### CUPED [CUPED](https://docs.statsig.com/stats-engine/methodologies/cuped/) can significantly change variances and observed deltas, especially with high pre- and post-exposure data correlation and/or systematic differences in groups' pre-experiment data. CUPED can be configured at a metric level, but you have the option to turn it off for a pulse result set after running analysis. ### Ratio Metrics For ratio metrics using the delta method, only units with a non-zero denominator are included, affecting the analysis's comprehensiveness. Additionally, Statsig calculates ratios and means as $\bar{u} = \frac{\sum_{i=0}^{n}(numerator_i)}{\sum_{i=0}^{n}(denominator_i)}$ and uses the [delta method](https://docs.statsig.com/stats-engine/variance/#ratio-and-mean-metrics) to correct for the cluster-based nature of these metrics. ## Metrics Often, users misunderstand how a given metric is calculated. We have a comprehensive [guide in our documentation](https://docs.statsig.com/statsig-warehouse-native/configuration/metrics) with more details. ## Conclusion Following this steps should yield an understanding of where gaps - if any - are coming from between two experiment platforms. Statsig puts a high emphasis on providing the intermediate and result datasets it uses, as well as the queries used in its analysis. This should make it easy in practice to understand where gaps arise. All that said, this can be tricky - please feel free to reach out if you get stuck, and we'll be happy to help. We'd also suggest looking at the [Statsig Warehouse Native Documentation](https://docs.statsig.com/statsig-warehouse-native/introduction) and [Statsig Pipeline Overview](https://docs.statsig.com/statsig-warehouse-native/pipeline-overview/) for an overview of experiment pipeline patterns. # Slicing by User Properties Source: https://docs.statsig.com/experiments/interpreting-results/userproperties Break down Statsig experiment results by user properties like country, platform, or subscription tier to identify heterogeneous treatment effects. Statsig let's you slice results by user properties. Common examples of doing this include breaking down results by user's home country, subscription status or engagement level. Pulse results sliced by user properties For Statsig Cloud, these user properties are captured (and frozen) from the properties set on the user's first exposure. Statsig Warehouse Native also adds support for reading them from a warehouse table (Entity Properties). You can always run custom queries on experiments to slice by user properties. ## Pre-Computed User Properties User properties that are frequently used to slice results can be pre-computed when using Statsig Warehouse Native. To do this, you can configure these properties to be pre-computed on the experiment setup page, under the advanced settings. It's also possible to configure team-level defaults for this - or pre-configure it on an experiment template. Experiment setup page with pre-computed user properties configuration Once configured, you can also apply filters to all metrics on your results. Metrics results with user property filters applied # Layers Source: https://docs.statsig.com/experiments/layers-overview Group related Statsig experiments into mutually exclusive layers so they share parameters and traffic allocation without redeploying application code. ## What are Layers? Layers (a.k.a. Universes) allow us to create experiments that are mutually exclusive to each other. Each layer has a logical representation of all your users and can have experiments created "within" this layer. Users that are in one experiment of a layer, cannot also be in another experiment in the same layer. Layer concept diagram showing mutually exclusive experiments You can add experiments to a layer (or create a layer) during experiment creation. Experiment creation modal with layer selection Once you create a layer, you'll be able to manage them on the layer management tab under Experiments. Layers overview tab listing active layers Layer details page showing shared parameters In addition to that, **Layers are key to improving engineering efficiency and iteration velocity** for product teams. In a Layer, parameters exist at the Layer level, and can be shared across experiments within the Layer. Due to this characteristic, we can abstract the concept of "Experiment" away from the SDKs so that users only need to deal with parameters in code, which makes it super easy to run multiple experiments that change the same thing and iterate on the same experiment without any code changes. Let's say your product has an important signup dialog, which contains some text that your team runs a lot of tests on, some of which were run in parallel, and some were iterations of previous experiments. If you work with Experiments directly, your code will look like this over time: ```jsx theme={null} let signUpText = DEFAULT_SIGNUP_TEXT; const signUpTestV1 = statsig.getExperiment("sign_up_dialog_text_test_v1"); const signUpTestV2 = statsig.getExperiment("sign_up_dialog_text_test_v2"); const specialSignUpTest = statsig.getExperiment("sign_up_test_special_offer"); const holidaySignUpTest = statsig.getExperiment("sign_up_test_holiday"); if (signUpTestV1.get("is_in_test", false)) { // original test, added in app version v1.2 signUpText = signUpTestV1.get("dialog_content", DEFAULT_SIGNUP_TEXT); } else if (signUpTestV2.get("is_in_test", false)) { // v2 of the original test, added in app version v1.6 because we wanted to test a new copy but don't want to stop v1 signUpText = signUpTestV2.get("dialog_content", DEFAULT_SIGNUP_TEXT); } else if (specialSignUpTest.get("is_in_test", false)) { // test showing a special offer in the text, added in v2.0 signUpText = specialSignUpTest.get("dialog_content", DEFAULT_SIGNUP_TEXT); } else if (holidaySignUpTest.get("is_in_test", false)) { // test showing some holiday greetings in the dialog, added in v2.1 signUpText = holidaySignUpTest.get("dialog_content", DEFAULT_SIGNUP_TEXT); } // Then we display the text in the dialog ``` Every time you add a new test, you need to change the code and it's only available in a new version. However, things become **A LOT** easier if you work with Layers: ```jsx theme={null} let signUpText = statsig .getLayer("sign_up_tests") .get("sign_up_dialog_text", DEFAULT_SIGNUP_TEXT); // Then we display the text in the dialog ``` That's all the code you ever need! No more code changes and app releases for new tests. Every time you want to add a new test, simply add a new experiment to the same Layer and choose the parameter `sign_up_dialog_text` as a parameter for the new experiment. The SDK takes care of figuring out which value to serve for the user, based on which experiment the user is allocated to. ## getExperiment vs getLayer API Even though layered experiments remain technically accessible via `getExperiment`, that API evaluates only the current experiment. Use `getLayer` so the SDK honors layer-level decisions, mutual exclusion, and shared parameters. ## A Word on Exposures Calling `getLayer(LayerName)` by itself does not log an exposure. A `statsig::layer_exposure` event is logged when you access a specific parameter within the Layer using `getLayer(LayerName).get(Parameter)`. * If the user is assigned to an experiment within the Layer, the `statsig::layer_exposure` event is billable. * If the user is not assigned to an experiment within the Layer, the `statsig::layer_exposure` event is not billable. Repeated reads of the same Layer parameter for the same user within the deduplication window may count as a single billable exposure. Reads of different Layer parameters may count separately. # Monitor an Experiment Source: https://docs.statsig.com/experiments/monitor Track health checks, exposure counts, and diagnostics for active Statsig experiments to catch sample ratio mismatch and instrumentation issues early. Once an experiment launches you can monitor its health and exposure mix directly from the Statsig console. ## Experiment Health Checks 1. Open **Experiments** from the navigation. 2. Select the experiment you want to inspect. 3. Review the **Experiment Health Checks** banner at the top of the scorecard. Experiment health checks showing status icons Hover a status icon to read the summary, then click for full context. Common checks include: * **Checks started** - Verifies the SDK is reporting config checks shortly after launch. * **Checks have valid unit type** - Confirms checks include the configured unit identifier (userID by default). * **Event metrics have data** - Ensures events carry the same unit ID as exposures so Pulse can compute metrics. This often surfaces when downstream tooling (e.g., Segment) omits stableID or custom IDs. * **Pulse metrics available** - Indicates Pulse results have landed (typically the day after launch). * **Exposures are balanced** - Runs a chi-squared test for sample ratio mismatch (SRM). Occasional warnings happen due to randomness, but persistent red alerts point to assignment or logging issues. * p-value between 0.001 and 0.01 -> Warning (yellow). * p-value \< 0.01 with \<0.1% absolute deviation -> Warning (yellow) with low expected impact. * p-value \< 0.001 and >=0.1% deviation -> Alert (red) requiring investigation. * **Crossover units detected** - Flags users exposed to multiple variants. Statsig Cloud keeps these users in both groups (since the SDK rarely produces crossovers) but highlights them so you can address root causes. Reach out if you see rates above 1%. * **Default value type mismatch** - Warns if an experiment's fallback default value type disagrees with the parameter definition. * **Group assignment healthy** - Surfaces unexpected assignment reasons (e.g., `Uninitialized`, `InvalidBootstrap`). Click **View Assignment Reasons** to see the hourly breakdown. Assignment reasons breakdown chart ## Crossover Troubleshooting Crossover warnings usually mean: 1. The request bootstrapped with a different stable ID (`BootstrapStableIDMismatch`). 2. Both client and server SDKs are checking the same gate/experiment without synchronized updates. If you can't pinpoint the cause, ping us in Slack - we're happy to help. ## Exposure Streams Scroll below the health checks to view exposure streams. These tables show every recent check, including the rule that matched and any secondary exposures (holdouts, targeting gates, etc.). They’re handy for validating targeting and confirming ramp progress. ## Cumulative Exposures To track growth per variant: 1. Open the **Results** tab. 2. Locate the **Cumulative Exposures** chart. Cumulative exposures chart The chart highlights how many users have entered each group over time, making it easy to spot ramp issues early. Keeping an eye on these diagnostics helps you resolve issues quickly and keep experiments on track. # Bot Traffic Source: https://docs.statsig.com/experiments/monitoring/bots How Statsig identifies and filters bot traffic out of experiment exposures and metrics so your A/B test results reflect real user behavior, not crawlers. ## Bots & Filtering One common source of frustration when monitoring gate traffic is online bot traffic from sources like search engines and AI scrapers. These can make it harder to see how many "real" users are seeing your changes. Statsig has bot filtering in place to remove known bots from your exposures data, meaning the exposure counts you see and any analytics you do will be clean. You won't have to worry if the data you're looking at is influenced by bots or real users. Bot filtering is done on all types of exposures data, not just feature flags. You can be sure that anytime you're looking at analysis results for feature flags, holdouts, layers, and experiments bots have been filtered out. This ensures that you're looking at results for real users and not web scrapers in your rollouts. For more on Bot Filtering rollout, see the [Statsig Blog](https://www.statsig.com/blog/guide-online-bot-filtering). Once bot data is filtered from your exposures data, it will not be viewable in the Statsig console. We're exploring how to better surface this information in the future. Please reach out via slack support if you have additional questions. ### Controlling Gates and Experiments for Bots By design, Statsig doesn't block bots from getting your feature flags and experiments. We simply filter out their exposures from any analysis data and the count of exposures that you see in Pulse. There are no changes in the API or SDK results for bots, and they will be served configs and variants following your setup. You might, however, want to purposefully restrict what features bots see. For example, you're testing a new homepage variant but you don't want search engines to index it yet. In this case, there is an easy way to do so via Segments: 1. Create a "Known Bots" Segment for your project: Create a new segment, ensuring that it will be a **conditional** segment. New conditional segment setup for capturing known bots Once created, add a new rule to the segment. Set Criteria to "Browser Name". Leave Operator as "Any Of". In the Values field, copy + paste the following string in its entirety. (There is a copy button to the right.) When pasting, Statsig console will take care of splitting the bots up into individual names. ``` Scomplerbot, WincherBot, fixbot, keys-so-bot, MojeekBot, Gulper Web Bot, Mattermost-Bot, SerendeputyBot, uipbot, WebCrawler, HearsayPDFBot, WRTNBot, BublupBot, InsytfulBot, DingTalkBot, uk_ldfc_renderbot, crawlers, ImagesiftBot, idealo-bot, taboolabot, KlaxoonBot, SemrushBot, archiver/3.1.1 +http://www.archive.org/details/archive.org_bot, StractBot, crawler_eb_germany_2, exabot, DocBase Crawler, co Bot, Superfeedr bot, Pokey_Bot, GooglePlusBot, OtherwebBot, PubMatic Crawler Bot, SiteAuditBot, Gensparkbot, wpbot, archive.org_bot, Audisto Crawler, amazon-product-discovery-bot, Atomseobot, Googlebot-Mobile, hubspot crawler, XoviOnpageCrawler, PerplexityBot, QualifiedBot, YodaoBot, BitSightBot, GG PeekBot, SMTBot, amazonproductbot, FAST-WebCrawler, TwitterCommerceBot, WellKnownBot, PAGEFREEZER CRAWLER, dbot, htc_botdugls, RavenCrawler, oBot, notebot, ViberBot, KStandBot, scoopit-crawler, SpeechifyBot, Spider_Bot, txt Crawler, net/bot, BugBountyBot, Letianpai_Robot, by fynd.bot, discobot, LineBotWebhook, ahrefsbot, Veoozbot, tkbot, coccocbot, Googlebot-Video, Streamline3Bot, Zoombot, adbeat_bot, msnbot, Nextdoorbot, node DuckAssistBot, redditbot, Xing Bot, DocSearch Crawler, ; bot, Storebot, playwright-bot, 47_safeAreaBottom, online-webceo-bot, SmarshBot, BeeperBot, ChannelBot, BrightEdge Crawler, //boteden, Quantcastbot, SpringserveBot, IAS Crawler, managr-webcrawler, ) Bot, Dragonbot, crawler4j, tyseobotmobile, IVW-Crawler, SEBot, sap-search-web-crawler, AndersPinkBot, Dcard-link-preview-bot, Mediumbot, Light Crawler, dataforseobot, Better Uptime Bot, CCBot, es_bot, DuckAssistBot, SeznamBot, telegrambot, crawler, Jugendschutzprogramm-Crawler, SeoCherryBot, GroupMeBot, HyperMegaBotGettingOnlyHTMLsFromYourWebsite, our-crawler, Slackbot-LinkExpanding, YandexMobileBot, Web-Crawler, PaperLiBot, Swiftbot, Paqlebot, YandexRenderResourcesBot, MetaJobBot, SynologyChatBot, GenomeCrawlerd, robot, StatusCakeBot, node bitlybot, Your robot, Pharosbot, TSMbot, WalluBot, slackbot, AmazonAdBot, AspiegelBot, EzoicBot, TiggeritoBot, eventseekerBot, AwarioBot, Leikibot, Timpibot, like Gecko) bot, Quora-Bot, JobBot, googlebot, PetalBot, eu bot, LinkArchiver twitter bot, DF Bot, Screaming Frog Wise SEO Spider, Clickagy Intelligence Bot, BLP_bbot, bitlybot, WazzupCrawler, web-crawler, pingbot, yoozBot, triptease-bot, Plesk screenshot bot, Magus Bot, node Screaming Frog SEO Spider, YextBot, seobilitybot, tyseobot, applebot, bingbot, GetLocalBot, TwitterBot, rogerbot, Preview Service; bot, traq-ogp-fetcher-curl-bot, seesawbot, Greppr Web Crawler, ResearchBot, web-bot, iAskBot, JobboerseBot, CriteoBot, FandomOpenGraphBot, com feedbot, Amazon-Advertising-ad-standards-bot, MotoMinerBot, peer39_crawler, Discordbot, DuckDuckGo-Favicons-Bot, KeybaseBot, adsbot, Open Graph Bot, emulate-seobots, bountybot, InfobipCrawler, GoogleBot, macox bot, Google-bot, captify-crawler, Robot, aiHitBot, fedistatsCrawler, ExtendedStayBot, NetpeakCheckerBot, com/bots, Automattic Analytics Crawler, Blog Rssbot, Dubbotbot, Rightlander Crawler, ClarityBot, Cookiebot, UOrgTestingBot, AcademicBotRTU, SEMrushBot, Server Crawler, Diffbot, DiscourseBot, chatbot, VirusTotalBot, SaberBot, TZUnfurlBot, Mail.RU_Bot, Monsidobot, YandexAccessibilityBot, preview service; bot, ecoresearchCrawler, PulsePoint-Crawler, DataForSeoBot, petalbot, Xbot, LinkedInBot, GnowitNewsbot, vebidoobot, Bawaab_bot, Brightbot, ClineCrawler, ; Bot, MSIECrawler, MoodleBot, Testcrawler, AASA-Bot, GPTBot, StrapBot, 5) bot, mj12bot, screaming frog seo spider, HubSpot Crawler, COIBotParser, OcelotBot, com crawler, Pinterestbot, VelenPublicWebCrawler, Firefox superpagesbot2, Parser Robot, GrapeshotCrawler, Mediatoolkitbot, am a bot, semrushbot, SearchAtlas Bot, DiffeoBot, IBM-Crawler, spbot, DatoCmsSearchBot, SISTRIX Crawler, bountybotttt, Summalybot, ID bot, node AppleNewsBot, DotBot, TesseractBotAgent, foundeebot, BadooBot, BacklinksExtendedBot, wowLink Crawler, about-crawlers, find-seo-bot, GraphiteBot, Sidetrade indexer bot, BLEXBot, rc-crawler, FacebookBot, nerdybot, Senutobot, Facebot, //botim, WallabyupBot, TurnitinBot, XBot_Senior, node ZoominfoBot, AhrefsBot, Exabot, PhaverBot, Applebot, TermlyBot, SemjiBot, Space Unfurl Bot, Slackbot, bidswitchbot, //botsin, AdsBot-Google, Morningscore Bot, DuckDuckBot, UptimeRobot, ClaudeBot, naverbookmarkcrawler, PingdomBot, Web Crawler, PlurkBot, node GrowSEOBot, WebExplorerSearchBot, node FullStoryBot, WebwikiBot, bot, policy adbeat_bot, trendictionbot0, ezoicbot, Catrobatbot, AdsTxtCrawlerTP, com/bot, Nigooutbot, PiBot, pinterestbot, http-spiders-bot, Ocarinabot, msnbot-media, AppsFlyerBot, SeobilityBot, Impressumscrawler, SurdotlyBot, cXensebot, Amazonbot, Rankabot, 2ip bot, harsilbot, FullStoryBot, com bot, Rhobot, FreshpingBot, twitterbot, Twitterbot, Caliperbot, Googlebot-Image, osapon ) bot, yandexbot, MJ12bot, Taboolabot, ActiveComplyBot, MixrankBot, 48_safeAreaBottom, compatible; botify, LoomlyBot, Googlebot, ev-crawler, pagefreezer crawler, AwarioSmartBot, iCjobs Stellenangebote, jbot, aixnew_aibot, SemanticScholarBot, Wire LinkPreview Bot, Elastic-Crawler, UCMore Crawler, x28-job-bot, ISSCyberRiskCrawler, AnytypeBot, clever tech bot, LivelapBot, Screaming Frog SEO Spider, RyteBot, SiteGuruCrawler, XBot, SuperBot, TypetalkBot, RepoLookoutBot, obot, TimeTreeBot, siteauditbot, iASD_SpiderBot, semaltbot, PopeTech-ScanBot, SummalyBot, aka-bot, YandexBot, HatenaBlog-bot, Googlebot-News, yacybot, SiteCheckerBotCrawler, node AwarioSmartBot, turbotime, net-Robot, reurl-bot, Google-Display-Ads-Bot, node CCBot, fr_bot, CapterraBot, seo-audit-check-bot, gptbot, Testomatobot, Snap-URL-Preview (bot, robots, ZumBot, hstspreload-bot, Fedicabot, serpstatbot, Synapse (bot, NetSeer crawler, uk_ldfc_bot, PartnerOptimizer-bot, Scrapbox Bot, domainsbot, NE Crawler, startmebot, VelaBot, SeekportBot, Radius Compliance Bot, AdkernelTopicCrawler, Linkbot, AppleNewsBot, Jones Searchbot, DropboxPreviewBot, ZoominfoBot, edansbot, Baiduspider-render, Python Requests, YisouSpider, OAI-SearchBot, AliyunSecBot, Baiduspider, Bytespider, TikTokSpider, Claude-SearchBot, Qwantbot, Thinkbot, ABEvalBot, HanaleiBot, OpenindexSpider, VamosBot, StartmeBot, Bingbot, parisbot, BetaLiveCrawlBot, MetaBot, CharSiuBot, PlagAwareBot, io/bot, ShowUpCrawler, ScraperBot, Monibot, ShellBot, IbouBot, MatchboxBot, CookieYesbot, Stripebot, Clearscopebot, quillbot, Sogou web spider, Checker Spider, ``` Segment rule showing browser name filter populated with bot list This common segment can then be used for all your launches. 2. Apply the Segment to your Gates and Experiments: For Gates, create a new rule that controls the bot experience. Feature gate rule ensuring known bots fail the rollout For experiments, create a Conditional Override that forces units in this segment to receive whatever version you want. Experiment conditional override mapping bot segment to control variant ### Opting Out of Bot Filtering Bot filtering is done at the project level. Admins can opt out of filtering through their console settings. Bot filtering settings interface ### Suggesting New Bots to Statsig If you have discovered bots that Statsig isn't including in our default set, or you have internal bots your company manages that you'd like to be applied to all bot filtering by Statsig, please reach out to us in Slack. # Managing SRM Source: https://docs.statsig.com/experiments/monitoring/srm How Statsig detects and surfaces sample ratio mismatch (SRM) in experiments and how to debug skewed traffic splits before trusting results. ## What is SRM? SRM, or sample ratio mismatch, is a problem with experiments characterized by there being too many units in some groups, and too few in others. The example below is an exposure crosstab of an experiment that has SRM. Even though the group percentages may look similar, if an assignment system is actually splitting traffic evenly, an imbalance this extreme (or more) would have less than a 0.01% chance to occur randomly. srm_example Statsig, and most experiment platforms, normalize metrics per-user; a count metric will be measured as total count divided by unique users in the experiment. This means that, in a vacuum, it's fine to have more users in one group. So, why is SRM problematic? ## Why SRM Is An Issue SRM is an issue because it is usually *non-random*; the extra or missing traffic is *not identical* to the original traffic. How can SRM occur? * A bug causes a users' client or browser to crash before a log for the exposure can be sent. The users who don't come back are not re-exposed and included, but those that do come back are. This leads to bias in measurement * A conditional dependency causes "who is exposed" to be filtered by some characteristic for one or many groups, meaning the groups are not identical to other groups, biasing measurement * A script is bulk-exposing users going through 1 group at a time, and logs are truncated after a certain count - meaning the last group's exposures are truncated. Those are common examples. SRM checks are critical because they can detect these kinds of effects occurring - and even at low rates they can lead to serious inaccuracy in experiment readouts. ## How SRM is detected Detecting SRM is generally done by using a [chi-squared test](https://en.wikipedia.org/wiki/Chi-squared_test), which is a common and accepted way to analyze categorical data to understand if observed frequencies match expected frequencies. For example, in the experiment above we expect an even distribution of 167.85k units per group, and observe \[166.08k, 171.18k, 166.30k]. If the p-value of the test is low, we reject the null hypothesis that the groups are identical and conclude that there is a difference between the groups' observed and intended/expected assignment rates. ## What to do if an experiment has SRM On Statsig, SRM will create a warning or failure state on an experiment's health check when detected - depending on how extreme the SRM is. srm_failure This often causes concern; people don't want to reset their experiment and lose the data they've collected, and additionally the concern is that if there *is* an underlying issue it will just reproduce if the experiment is reset and restarted. Generally, it's recommended to follow the following steps: **1.) Check the time series data** Statsig generates a chart of SRM p-value over time. If this is noisy and bounces around, it's more likely that this was a false positive alert. If it consistently trends down to 0, it's a strong signal that there is truly an assignment issue. Below is an example of a p-value chart that indicates a real issue. srm_bad_timeseries **2.) Understand if there's a clear root cause for SRM** Use Statsig's SRM debugger, or do a data analysis using exported exposures to understand if a certain segment is driving SRM. Often, it's the case that a bug will only exist on a platform like android, or be restricted to users with low internet speeds. If you can find a bug, and fix it, you can restart the experiment safely If it's very clearly isolated, it's also reasonable to filter out that segment from analysis if the experiment was expensive or required a long period of data collection. At this point, you should have a rough idea of how likely it is that there's an issue; now you need to make a call on next steps. **3.) Assess options** It's unfortunate, but in many cases the best bet is to investigate, fix, and restart. In some cases, the SRM may be mild enough, and the experiment low-risk enough, that it's acceptable to make a decision with the tainted data. In general, though, Statsig strongly recommends against this and considers it a best practice to restart the experiment, ideally after investigating any potential SRM vector. # Experiments Overview Source: https://docs.statsig.com/experiments/overview Learn the fundamentals of experimentation with Statsig, including key concepts, randomization units, and statistical significance. **Experimentation** is a powerful tool for making data-driven decisions that improve product outcomes and customer experiences. In this doc, we'll cover key concepts of experimentation such as control variables, randomization units, and statistical significance, helping you understand the science behind A/B testing and multivariate experiments. By the end of this guide, you'll know how to use experiments to validate product changes, discover new opportunities, and drive business impact. Whether you're optimizing existing features or exploring new ideas, these fundamentals will equip you to run effective experiments and iterate faster with confidence. Experiments banner image ## What are experiments? Experiments enable you to run randomized controlled trials (A/B or A/B/n tests) with minimal effort, allowing you to measure the impact of product changes on key metrics. Statsig’s experimentation platform is designed to make it easy to create, manage, and analyze experiments, ensuring you ship features that deliver value to your users and business. Experiments are ideal when you want to: * Test multiple variants (A/B or A/B/n) of a product feature. * Run mutually exclusive experiments in parallel. * Measure the direct impact of changes on product and business metrics. *** ## Why experiment? Controlled experiments are the most scientifically reliable way to establish **causality** between your product changes and their effect on customer behavior. By running experiments, you can: * **Validate Hypotheses**: Only ship features that have been proven to improve the customer experience or drive key business metrics. * **Measure Success**: Measure feature performance post-launch and detect any unexpected side effects. * **Drive Innovation**: Experiments allow teams to iterate faster by providing real-time feedback on product performance. They help you make better, data-driven decisions that accelerate business growth. In comparison, historical metrics may show correlation, but experiments allow you to establish causal relationships. Experiments reduce the influence of uncontrolled external factors, ensuring that observed effects are due to the tested changes. *** ## Key concepts ### Control variables A **control variable** is the variable in an experiment that is manipulated to observe its effect on key metrics. In a simple A/B test, the control variable usually has two values (A and B). More complex experiments may have additional values (e.g., A, B, C, D), known as multivariate experiments. ### Variants A **variant** is a specific version of the product or feature being tested. For example, in an A/B test: * **A (Control)**: Represents the current state of the product or feature. * **B (Treatment)**: Represents the modified state you want to evaluate. Each variant is randomly assigned to users, allowing you to compare their performance. ### Randomization unit The **randomization unit** is the entity (such as a user, device, or session) that is randomly assigned to control or treatment groups in an experiment. Choosing the right randomization unit ensures consistency in user experience and reliable experiment results. This choice is critical to ensure that experiment results reflect real-world user behavior and that data is not skewed by unintentional crossovers between groups. ### Statistical significance Statistical significance is used to determine whether the observed changes in metrics are likely due to the product change or just random variation. Two commonly used methods are: * **p-value**: The p-value measures the probability of observing the results by chance if the variant had no effect. A p-value below 0.05 is typically used to determine statistical significance. * **Confidence Interval**: A confidence interval defines the range in which the true effect of a variant lies, with a given level of confidence (e.g., 95%). If the confidence interval does not overlap zero, the effect is considered statistically significant. These concepts help you interpret the results of your experiment with greater confidence. For more detailed information on designing, monitoring, and analyzing experiments, check out [Product Experimentation Best Practices](https://statsig.com/blog/product-experimentation-best-practices). *** ## Common scenarios for experimentation ### Optimize product growth Use experiments to refine and optimize user experiences, helping you climb toward a local maximum in your product strategy. Common goals include: * Optimizing a specific user journey (e.g., improving onboarding). * Iterating on features to identify high-return opportunities. * Aligning experiments with business-critical metrics and guardrails to prevent negative side effects on fundamental business needs. ### Explore new opportunities Use exploratory experiments to discover entirely new directions. These experiments help you develop new ideas, validate strategies, and uncover long-term opportunities. * Run experiments over longer durations to account for novelty effects and adoption time. * Slowly ramp up experiments to minimize risk and build statistical power. * Test multiple related hypotheses to explore a broader business strategy. *** ## Choosing the right randomization unit The Randomization Unit is the variable you choose that determines how users will be distributed across your groups (e.g., test and control). When you set a variable as the Randomization Unit, any value for that variable will *always* see the same experience. This Unit is also the unit of reference for your metrics. If you choose userID as your Randomization Unit, the userID will deterministically bucket each user, and will be the basis of measurement - your analysis might look at Revenue per userID. Below are some common units of randomization and when to use them. ### User Identifiers The most commonly used randomization unit is the **User ID**. This identifier is generated when a user registers or signs in to your application. Using User IDs ensures a consistent user experience across sessions and devices, as the user is always assigned the same variant, regardless of where or when they access the product. Advantages: * Persistent across sessions and devices. * Independent of client-side cookies, which can be cleared by users. For more details on using User IDs with Statsig, see [Statsig Docs on User Identifiers](/sdks/user). ### Device Identifiers For users who haven't registered or signed in, using **Device IDs** or **Anonymous User IDs** is common. These identifiers track users based on the device they are using and are ideal when you are experimenting with unregistered or guest users. Example: * You can use device IDs to experiment on landing page optimizations aimed at improving user registration rates. **Drawbacks**: * Device-specific: If the same user accesses your app from multiple devices, they may have different experiences. * Shared devices: If multiple users share a device, the experiment may mistakenly treat their behavior as belonging to one individual. Statsig SDKs automatically generate **Stable IDs** for anonymous users, making it easier to manage device-based experiments. For more details, see [Statsig Guide for Device-Level Experiments](/guides/first-device-level-experiment). ### Session Identifiers In certain cases, you may use **Session IDs** as the unit of randomization, especially when testing behavior during a single session (e.g., optimizing a checkout flow). However, session-based randomization assumes each session is independent of others, which can be a risky assumption if users return in multiple sessions. Example: * Session IDs might be used when experimenting with conversion funnels for guest checkouts, which are typically completed within a single session. **Drawbacks**: * Users may remember their experience from one session to another, breaking the assumption of session independence. * If users return in future sessions, there’s a risk they could be placed in different variants, leading to inconsistent user experiences. *** ## Tutorials * [Run your first A/B test](/guides/abn-tests) * [Create an experiment using a userID](/experiments/create-new) * [Create an experiment using a customID](/guides/experiment-on-custom-id-types) * [Use a language specific Statsig SDK to implement an experiment in your application](/experiments/implementation/implement) * [Monitor an experiment](/experiments/monitor) # Power Analysis Source: https://docs.statsig.com/experiments/power-analysis Learn how to use Statsig's Power Analysis Calculator to determine experiment parameters needed for statistically significant results. ### What is Power Analysis? The Statsig Power Analysis Calculator helps pre-determine the experiment parameters needed to reach a statistically significant result. Often, the variable you're optimizing for is duration - how long your experiment runs - but Statsig's power calculator is flexible to other variables. Using the known mean and variance of a metric, and the observed traffic volume, the Power Analysis Calculator estimates the relationship between three variables: * **Minimum detectable effect (MDE)**: The smallest change in the metric that the experiment can reliably detect. For example: An MDE of 1% with Power set to 80% means that if there's a true effect of 1% on our metric, we expect the experiment will have an 80% chance to produce a statistically significant result. If the magnitude of the true effect is smaller than 1%, it will be less likely to produce a statistically significant result (though it can still occur). * **Number of days or exposures**: How long the experiment is active and the number of users enrolled in it. Longer running experiments typically have more observations, leading to tighter confidence intervals and smaller MDE. We use historical data to estimate the number of new users that would be eligible for the experiment each day. * **Allocation**: The percentage of traffic that participates in the experiment. Larger allocation leads to smaller MDE, so it's often desirable to allocate as many users as possible to get faster or more sensitive results. When there's a risk of negative impact or a need for mutually exclusive experiments however, it's useful to know the smallest allocation that can achieve the desired MDE. Many experimentation practitioners use the Power Analysis calculator during setup of every experiment they run. ## Using the Tool Power Analysis Calculator can be accessed from the tools menu. It's also linked on the experiment setup page, underneath the Experiment Duration field. Power Analysis entry points 1. Select the population used to determine the metric mean and variance and to estimate the number of exposures over time. * **Everyone**: Analysis is based on the entire user base. * **Targeting gate**: Analysis is scoped to the set of users who pass the selected feature gate, which must have been active for at least 7 days. Choose this option when you plan to use a targeting gate for the experiment. * **Past experiment**: Analysis is based on data collected from in a previous experiment. Use this option when the new experiment will impact a similar user base or part of the product as the previous one. * **Qualifying event**: Analysis is scoped to the set of users who logged the event specified. 2. Select a metric of interest (or multiple metrics for a targeting gate analysis) 3. Click on **Run Power Analysis** to calculate results Power analysis form inputs Your past power analysis calculations will be available to view in the "Past Analyses" tab. Past power analyses list If you want to attach an existing power analysis to an experiment, that option is available via the dropdown menu on an existing power analysis. Power analysis attachment interface ## Population Types The population selected directly impacts the inputs of the analysis (mean, variance, number of users). To obtain reliable power analysis estimates, the metric values of the selected population should roughly match those of the users you'll be targeting in the experiment. ### Example Say we want to test a change in the checkout flow and we want to know our expected MDE for total\_purchases. Let's assume that only \~10% of our daily users reach the checkout page. If we use the *Everyone* population for our analysis, we're likely to: * Overestimate the number of users that the experiment will get. * Underestimate the mean value of the total\_purchases metric. The 90% of user that don't reach the checkout page have a value of zero, but in practice they won't be in our experiment and won't contribute to the metric. * Incorrectly estimate the variance in the total\_purchases metric. The distribution of metric values is different if we include the 90% of users that have 0 purchases because they never reached the checkout page. Thus, in cases when the experiment only includes a biased subset of users, it's possible the MDE and duration obtained by the power analysis won't be a good estimate. One way to address this is to use data from a past experiment to estimate the power of a new, similar experiment (coming soon!). In our example, if we had a prior experiment that was also targeting the checkout page, we could use it to get better estimates of traffic volumes and metrics for this part of the product. ### Inputs by Population Type This is how the various inputs for the power analysis are obtained from the different population types: | Population | Mean and Variance Calculation | Total Exposures by Week Estimate | | ---------------- | ---------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Everyone | Mean and variance across all users, estimated for 1, 2, 3, and 4 week rollups | Total count of users seen in the past 1, 2, 3, and 4 weeks | | Targeting Gate | Mean and variance for users that pass the targeting gate, computed for 1, 2, 3, 4 week rollups | Total users that passed the targeting gate after 1, 2, 3, 4 weeks | | Past Experiment | Cumulative mean and variance for the control group at 1, 2, 3, and 4 weeks | Total experiment exposures after 1, 2, 3, and 4 weeks, adjusted according to the past experiment's allocation and the desired allocation for the new experiment. | | Qualifying Event | Mean and variance for users who logged a specified event, computed for 1, 2, 3, 4 week rollups | Total users who logged a specified event after 1, 2, 3, 4 weeks | ## Analysis Types When looking at the results, you can modify certain inputs like "# of Groups", "Control Group %", and Analysis Types to update your power analysis results based on your updated inputs. Power analysis results showing adjustable parameters ### Fixed Allocation Analysis If you already know the available allocation, fixed allocation allows you to understand how the length of the experiment impacts the MDE. The example below shows how the MDE for a page load metric shrinks over time in an experiment with 100% allocation. After 1 week we expect 5200 users per group and an MDE of 21.6%, by week 4 the number of users per group should increase to \~48k and the MDE is reduced to 7%. Fixed allocation analysis results chart ### Fixed MDE Analysis If you already know the effect size you want to measure, fixed MDE analysis allows you to understand the allocation and duration needed for desired MDE. Enter the desired MDE as a percentage of the current metric value. For example: If a website currently gets 1,000 page loads per day, an MDE of 10% means we can detect a change of 100 or more page loads per day. The results show the minimum number of weeks needed to reach this MDE for different allocation percentages. In the example below, the experiment should run for at least 2 weeks with 65% allocation or 4 weeks with 50% allocation. There's no way achieve the desired MDE in 1 week, as this would require more than 100% allocation (more users than we expect to see in one week). Fixed MDE analysis results table ## Advanced Options Advanced power analysis options interface Advanced settings to customize the analysis: * **Number of Experiment Groups**: The total number of groups in the experiment, including control. * **Control Group %**: What percent of users will be in the control group, e.g. 50% if half of all users will be control. * **Fixed Allocation or Fixed MDE Analysis**: Different types of analyses you want to run. See [analysis types](/experiments/power-analysis#analysis-types) for more details. * **One-sided or Two-sided test**: Toggle this setting to select the type of z-test to use for the analysis. * **Significant Level (α)** * **Power (1-β)** * **Bonferroni Correction Per Variant**: Whether to include α correction for multiple tests in power analysis. ## Calculation Details The relative percentage MDE for a given metric *X* is computed using the following equation: MDE calculation formula * *X-bar* is the mean metric value across all users * *var(X)* is the population variance of the metric * *Ntest* and *Ncontrol* are the estimated number of users in the test and control group. These are based on historical active user data along with experiment allocation and group size. * *Z1-β* is the standard Z-score for the selected power. Typically *1-β = 0.8* and *Z1-β = 0.84* * *Z1-α/2* is the standard Z-score for the selected significance level in a 2-sided test. Typically *α = 0.05* and *Z1-α/2 = 1.96* This calculation relies on statistics computed across the entire user base of the project. It does not account for the fact that experiments targeting only a subset of users may have different summary statistics for their key metrics. For example, the metric mean and variance can be different in an experiment that targets only Android users or one that exposes users at the lower part of an acquisition funnel. # Experiment Overrides Source: https://docs.statsig.com/experiments/setup/overrides Override Statsig experiment group allocation for specific users or environments during development and QA, so testers see a chosen variant deterministically. ## Override group allocation for an Experiment During development, it can be useful to explicitly state which experiment group a user should fall into. For example, if a developer is testing that each of the experiment conditions work as expected, they may want to force themselves into each of the different conditions to test. Overrides can be added based on Feature Gates and Segments, and will allow you to create rules that force all users that pass a given Feature Gate or Segment to be allocated to a given group. When fetching parameters for an experiment, if the user matches any of the overrides, the overridden result will be returned immediately. **Note**: * For Experiments that are in a Layer, the overrides must be set at the Layer level. A user can only be overridden into one Experiment within that Layer (since layers always make experiments within them mutually exclusive). * It is best to add overrides to your experiment before it starts or at least before a user's first exposure. Users whose first exposures are controlled by overrides are excluded from Pulse results. Since these users are not randomized, we purposely exclude them from experimental results. Adding a large number of users to the override can affect the reliability of your experimental results. * **Warning**: if you add overrides after a user has already been exposed in an experiment, that user will not be excluded from Pulse results. That user's events and metrics will continue to be attributed to the group they were first assigned to. However, the override will be applied and honored once you define it. This can cause your experiment results to be diluted or polluted since the user's actions may be attributed to one group while they were actually exposed to a different group. The overall impact from this issue will depend on how many users fall into this category. * ID Overrides are evaluated first, then conditional overrides (from top to bottom). * ID type used for ID Overrides may or may not match the experiment’s ID type. overrides modal ### Adding an Override * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Experiments** * Select the experiment where you want to add an Override, and navigate to the Setup tab * Click the **Manage Override** button, configure the override, and hit **Save** overrides modal ### Deleting an Override If you add an override but later decide it is no longer needed. You can remove it so the rules will be evaluated as normal. * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Experiments** * Select the Experiment or Layer from which you want to delete the Override * Click the **Manage Overrides** button * Click on the trash can icon next to the override you would like to delete, and hit **Save** ### Testing an Override Once your override has been added, you can test it in the "Check Group for a User" window by navigating to the **Diagnostics** tab, and then adding the necessary properties that would be required to make a user pass the gate or segment that was added as an override. # Experiment Quality Score Source: https://docs.statsig.com/experiments/setup/quality-score Use Statsig's experiment quality score to assess and improve the trustworthiness of your A/B tests across allocation, exposures, and metric health. ## Introduction The Experiment Quality Score is a metric designed to get a quick understanding of the quality - and trustworthiness - of an experiment configured within Statsig. This helps experimenters and their peers across an organization quickly identify potential issues in experiment setup, execution, and data collection, ensuring more confident decision-making. Measuring this score over all experiments can help teams to discover systematic issues in their program, and point out opportunities to mature their experimentation program over time. ## Configuration Experimentation quality score can be enabled in the project settings under Settings -> Experimentation -> Experiment Quality Score. There is a list of pre-defined assessment criteria used. The weights of each criteria can be customized based on an organization's needs, though Statsig provides default values. Experiment quality score configuration interface ## Advanced Configuration For more sophisticated customers there may be additional checks desired, different product teams with different needs, or it may be the case that the default thresholds aren't correct for their use case. For example, maybe hypotheses should be at least 200 characters, AND contain a link to an external planning doc. This can be managed easily through the Statsig console API. By running a POST or PATCH on the `console/v1/experiments` endpoint, one can update the individual scores on any given experiment. Targeting the existing set of scores allows overriding weights (usually to 0), meaning the list can be just the custom set desired. For example, running patch on an experiment with this payload: ``` { "manualQualityScores": [ { "criteriaName": "HYPOTHESIS_LENGTH", "criteriaDescription": "Check passed", "status": "PASSED", "score": 0, "weight": 0 }, { "criteriaName": "MyCompany\'s Hypothesis Check", "criteriaDescription": "Has Internal URL and > 200 Chars", "status": "PASSED", "score": 100, "weight": 100 }, { "criteriaName": "Naming", "criteriaDescription": "Experiment prefixed with team name", "status": "FAILED", "score": 0, "weight": 100 } ] } ``` Would: * Drop the original HYPOTHESIS\_LENGTH check * Keep the other original checks, with their weights * Add a new check, `MyCompany's Hypothesis Check`, for custom logic on the hypothesis * Add a new check, `Naming`, for custom logic on the name Note that the other weights would be normalized. If the original HYPOTHESIS\_LENGTH had a weight of 10, the total weight would now be 290 and scores normalized accordingly. If all of the non-custom checks were passing, the score would be 190/290 or \~66% The general flow for using this approach is to: * Use Console API's `experiments/get` to pull all experiments * For Each Experiment * Run custom logic * Patch results ## Calculation Notes Checks which are in an unready state will be skipped during evaluation, and the other weights will be renormalized to 100%. For example, if the experiment has not started, the `Balanced Exposures` component will be in an unready state and ignored. Checks with a weight of 0 will be omitted entirely from the card. ## Viewing Quality Scores When enabled, Quality Scores will show up in the details tab of an experiment. Applicable checks will be evaluated and contribute to the number shown. The score will be color-coded based on the % threshold it is at. * \>= 85% corresponds to passing/green * \>= 50% corresponds to warning/yellow * \< 50% corresponds to error/red. Experiment quality score display with color-coded status Quality scores are also available through the console API. This allows bulk scrapes of the data for easy analysis. # Confidence Intervals Source: https://docs.statsig.com/experiments/statistical-methods/confidence-intervals How Statsig calculates confidence intervals for experiment metrics, including the formulas, assumptions, and how to interpret intervals in scorecards. Confidence intervals are an intuitive way to quantify the uncertainty in the observed metric deltas. A 95% confidence interval should contain the true effect 95% of the time. This means that if we ran an experiment 100 times, the true value of the metric delta should be inside the confidence intervals 95 times. Confidence interval visualization showing statistical significance In practical terms, a 95% confidence interval that doesn't contain zero (the green bar above) represents a statistically significant result (with *α = 0.05*). This is not always the case, though: There are cases when the p-value of the difference between test and control is statistically significant, but due to uncertainty in the control, a relative delta confidence interval may cross zero (using [The Delta Method](/experiments/statistical-methods/methodologies/delta-method)) or be represented as a point estimate (using [Fieller Intervals](/experiments/statistical-methods/methodologies/fieller-intervals) ) while the absolute difference's p-value is statistically significant. Only 5% of the time would we expect to see the confidence interval exclude zero if the true effect was zero (a.k.a. a false positive). Larger confidence intervals imply less certainty in the exact size of the effect with a larger range of likely values. ## Computing Confidence Intervals Confidence intervals in Statsig are calculated using a two-sample z-test. This test requires knowledge of the variance in the metric delta we're measuring, which is derived differently depending on the type of metric (details [here](/stats-engine/variance)). Once we've established the variance of the delta, it's straightforward to compute the confidence intervals. ### Two-Sided Tests For the **absolute metric delta**, the confidence interval is given by: $$ CI(\Delta \overline{X}) = \Delta \overline{X} \pm Z_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}} $$ where: * $Z_{\alpha/2}$ is the z-critical value for the desired significance level (1.96 for the standard $\alpha=0.05$ and 95% confidence interval) and we run a two-sided test * $var(\Delta \overline{X})$ is the variance of the absolute delta (details [here](/stats-engine/variance)) The confidence interval for the **relative metric delta** can use one of two methods, [Fieller Intervals](/stats-engine/methodologies/fieller-intervals) and the [Delta Method](/stats-engine/methodologies/delta-method). While customers can opt for either method to be used in their Statsig console, Statsig recommends and automatically opts-in all new customers to use Fieller Intervals by default. When using Fieller Intervals, the relative metric delta CI can be computed using: $$ CI(\% \Delta \overline{X} ) = \frac{1}{1-g} ( \frac{\overline{X_T}}{\overline{X_C}} - 1 \pm \frac{Z_{\alpha/2}}{\sqrt{n_C} \cdot \overline{X_C}} \sqrt{(1-g) \cdot \frac{var(X_T)}{n_T(n_T-1)} + \frac{\overline{X_T} var(X_C)}{\overline{X_C} n_C (n_C-1)}}) $$ When using the Delta Method, the confidence interval is: $$ \begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}}\\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot\sqrt{(\frac{\overline X_t}{\overline X_c})^{2} \cdot (\frac{var(X_c)}{n_c \cdot \overline X_c^2} + \frac{var(X_t)}{n_t \cdot \overline X_t^2})} \cdot 100\% \end{split} $$ If using the Delta Method and the control mean is not significantly away from zero, then it's simplified to: $$ \begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}} \\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot \frac{\sqrt{{var\left(\Delta \overline X\right)}}}{\overline X_c} \cdot 100\% \end{split} $$ ### One-Sided Tests When running one-sided tests, the form of the confidence interval calculation changes slightly to account for a redistribution of desired false positive rate when looking for increases or decreases in the metric: $$ CI(\Delta \overline{X}) = \begin{cases} \left[\Delta \overline{X} - Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}}, \quad +\infty \right) & \text{if right-hand test}\\ \\ \left(-\infty, \quad \Delta \overline{X} + Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}} \: \right] & \text{if left-hand test} \end{cases} $$ where: * $Z_{\alpha}$ is the z-critical value for the desired significance level (1.645 for the standard $\alpha=0.05$ and 95% confidence interval) and we run a one-sided test * $var(\Delta \overline{X})$ is the same as for two-sided tests * the choice of confidence interval depends on if the one-sided test is looking for increases or decreases in the metric ## Welch's T-test for Small Sample Sizes For small sample sizes, we use Welch's t-test instead of a standard z-test. This statistical test is a better choice for handling samples of unequal size or variance without increasing the false positive rate. The structure of the confidence interval calculation remains the same as above (depending on 1- or 2-sided test), replacing the z-critical value with the t-critical value with degrees of freedom $\nu$. For a two-sided test, the confidence interval is therefore: $$ CI(\Delta \overline{X}) = \Delta \overline{X} \pm t_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}} $$ $$ \nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}\ = \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}} $$ Where $N_t$ and $N_c$ are the number of users in the test and control groups, respectively. Note that for large number of degrees of freedom, the t-statistic converges with the z-statistic. Therefore, Welch's t-test is used only when $\nu < 100$. ## Comparing Experiment Data to a Fixed Baseline: One-sample T-test Sometimes we want to answer questions like "Does my test variant lead to a click through rate higher than 0.5?". You can define a fixed-baseline comparison when adding metrics to the experiment. The confidence interval is calculated by $$ CI(\Delta \overline X) = (\overline X_{group} - fixed \ value) \pm Z \cdot\sqrt{{var( \overline X_{group})}} $$ # Benjamini–Hochberg Procedure Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/benjamini-hochberg-procedure How Statsig applies the Benjamini-Hochberg procedure to control the false discovery rate when analyzing many metrics in an experiment scorecard. ## What it is The Benjamini-Hochberg Procedure ("BH" procedure) is a statistical method that reduces the probability of false positives by adjusting the significance level for multiple comparisons. It is not as extreme as a [Bonferroni Correction](/stats-engine/methodologies/bonferroni-correction), because instead of controlling the chance of at least one false positive (Family Wise Error Rate), BH controls the expected value of false positives when the null hypothesis has been rejected (False Discovery Rate). Like with many other analysis settings, you can enable BH procedure for individual experiments (or configure global Experiment Settings to default it). Benjamini-Hochberg procedure configuration interface ## Methodology The [Benjamini-Hochberg Procedure](https://www.statisticshowto.com/benjamini-hochberg-procedure/) updates the significance level to be used (modifying your pre-set $\alpha$). The new significance level to be applied is calculated by sorting the p-values of metrics in ascending order and comparing with a paired threshold. Each p-value’s paired threshold is the desired False Discovery Rate divided by the number of comparisons being evaluated multiplied by what rank a p-value is in the ordered list. The largest threshold value which is higher than its corresponding p-value is our new significance level ($\alpha$). The Benjamini-Hochberg Procedure can be applied based on: * The number of test groups (multiple treatment hypotheses). For each metric aggregate the list of p-values from each variant and complete the Benjamini-Hochberg procedure. * The number of metrics in the scorecard. For each variant aggregate the list of p-values from each metric and complete the Benjamini-Hochberg procedure. * Both the number of test groups and number of metrics in the scorecard. All p-values are aggregated to complete the Benjamini-Hochberg procedure. Statsig does not apply BH procedure when evaluating the p-values of any event-dimension or user-property experiment metric results. Only the top-line metric results are compared to the new significance level. ## How do my experiment metrics look now? In the experiment scorecard section, confidence intervals will be derived from (1 - adjusted α) for applicable metrics. If you hover over a confidence interval, the adjusted α will be displayed alongside other relevant metric details. In the experiment explore section, if applicable to the selections you make, a new adjusted α will be calculated and these confidence intervals will use (1 - adjusted α). # Bonferroni Correction Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/bonferroni-correction How Statsig applies the Bonferroni correction to adjust p-values when testing multiple metrics or comparisons in an experiment to control false positives. ## What is Bonferroni Correction? A Bonferroni Correction is a statistical method that reduces the probability of false positives by adjusting the significance level for multiple comparisons. If you run a tests with α = 0.05, the probability of a false positive will be 5%. If you run more comparisons at the same significance level, the chance of at least one false positive goes up because each comparison is an additional opportunity for false positive. Bonferroni corrections are an optional feature on Statsig experiments that reduces the probability of Type I errors (false positives) by adjusting the significance level (α). The significance level is divided by the number of comparisons being evaluated. You can choose to apply these based on one or both of the following: * The number of test groups (multiple treatment hypotheses). The significance level is divided by the number of variants being compared against control. * The number of metrics in the scorecard. Here you may select what percentage of your total α is divided evenly among the Primary Metrics, and the remaining α is split equally among Secondary Metrics. For example: * Significance level of 0.05 * 2 Primary Metrics and 4 Secondary Metrics * 60% of α applied to Primary Metrics * Each Primary Metric is calculated with α = 0.6 \* 0.05 / 2 = 0.015 * Each Secondary Metric is calculated with α = 0.4 \* 0.05 / 4 = 0.005 * If both corrections are selected, they're applied on top of each other. In the example above, if we also wanted to correct for having 2 tests groups, we would further divide each α by 2. When analyzing dimensions, if correction for metrics is enabled, it's applied separately for the dimensional breakdown. We use the number of dimensions as the total metric count to correct for *in the dimensional analysis*, but it does not impact topline metrics. Bonferroni correction configuration interface ## How do my experiment metrics look now? In the experiment scorecard section, confidence intervals will be derived from (1 - adjusted α) for applicable metrics. If you hover over a confidence interval, the adjusted α will be displayed alongside other relevant metric details. In the experiment explore section, if applicable to the selections you make, a new adjusted α will be calculated and these confidence intervals will use (1 - adjusted α). # CUPED Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/cuped How Statsig uses CUPED variance reduction to improve experiment sensitivity by adjusting for pre-experiment user behavior on metric values. ## CUPED - Controlled-experiment Using Pre-Existing Data CUPED (short for Controlled-experiment Using Pre-Existing Data) is a technique which leverages user information from before an experiment to reduce the variance, and increase confidence in experimental metrics. At Statsig, this pre-experiment data is defined as the 7 days before each user's exposure rather than a fixed window before the experiment starts for all users. This can help to debias experiments which have meaningful pre-exposure bias (e.g. the groups were randomly different before any treatment was applied). Our Cloud product uses a 7-day window for CUPED calculation. For Warehouse Native customers, a 7-day window is recommended, but you have the flexibility to customize it to any length. See more at the [Variance Reduction](/experiments/statistical-methods/variance-reduction) page. To dive deep into the methodology, see CURE by Statsig. ## CUPED for Simple Aggregations The methodology for simple aggregations is described in the original [Microsoft paper](https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf), as well as our in-depth [article](https://www.statsig.com/blog/cuped) on the technique. The Cloud product uses stratification alongside CUPED to account for users who may not have pre-experiment data. Users are grouped into strata based on available pre-experimentation information. Treatment and control effects are first estimated within each stratum, then aggregated to produce an overall result. We then apply the standard difference-in-means and variance estimation. This approach allows us to retain users with missing pre-data while still benefiting from variance reduction where applicable. ## CUPED for Ratio Metrics The Microsoft paper also gives details on how to implement CUPED for those with a different analysis unit (Appendix B). On Statsig, we extend it to work for our ratio metrics, where each experiment unit is represented by a numerator and a denominator. The variance reduction process is performed by finding the variance of experiment data, pre-experiment data, and the covariance between the two. Denote the numerator, denominator, pre-experiment numerator, and pre-experiment denominator of a unit as $Y$, $N$, $X$, and $M$, respectively. Using the CUPED-reduced variance formula, $$ Var(\frac{Y_{cv}}{N_{cv}})=Var(\frac{Y}{N})+\theta^2 Var(\frac{X}{M})-2\theta Cov(\frac{Y}{N}, \frac{X}{M}) $$ where optimal $\theta$ is found as $$ \frac{Cov(\frac{Y}{N}, \frac{X}{M})}{Var(\frac{X}{M})} $$ expanded to \\ $$ \frac{Cov(\frac{Y}{\mu_N}-\frac{\mu_Y N}{\mu^2_N}, \frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})}{Var(\frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})} $$ At this point, we have $$ \frac{\hat{Y_{c}}}{\hat{N_{c}}}=\frac{Y_{c}}{N_{c}}-\theta( \frac{X_{c}}{M_{c}} - \mathbb{E}[R]) $$ $$ \frac{\hat{Y_{t}}}{\hat{N_{t}}}=\frac{Y_{t}}{N_{t}}-\theta( \frac{X_{t}}{M_{t}} - \mathbb{E}[R]) $$ While $\mathbb{E}[R]$ is hard to deduct, we recognized that the expectation term is the same for both group. We decided to substitute $\mathbb{E}[R]$ with $\frac{X_{c}}{M_{c}}$ so the formulas above are transformed to these following two: $$ \frac{Y_{cv}(control)}{N_{cv}(control)}=\frac{Y(control)}{N(control)} $$ $$ \frac{Y_{cv}(test)}{N_{cv}(test)} \\ :=\frac{Y(control)}{N(control)} - (\frac{Y(control)}{N(control)} - \theta \frac{X(control)}{M(control)}) + (\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)}) \\ :=\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)} + \theta \frac{X(control)}{M(control)} $$ Using the optimal $\theta$, we are hoping to reduce group-level variance by plugging the parameter back in to calculate the adjustment. Please note that across-group $\theta$ does not necessarily reduce variance for one group, or the sum of variances of all groups, but in most cases it does. Our simulation shows that 98.3% of metrics saw a decrease by CUPED. Statsig will use CUPED variance when all of the following are met: * Core assumptions of the CUPED model are satisfied; this can be violated due to rounding error or other data artifacts * E(X\_hat) = E(X) * The pooled variance of the adjusted population across groups is \< the variance of the unadjusted population * Enough units have pre-experiment values (> 100) * Enough percentage of units have pre-experiment values (> 5%) # Delta Method Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/delta-method How Statsig applies the delta method to compute variances for ratio metrics and other non-linear functions of user-level metric values in experiments. ## Delta Method for Ratio Metics Statsig uses the delta method when calculating the variance for variables that have a numerator and denominator. The variance of ratio and mean metrics depends upon the numerator and denominator variables, which are typically correlated. For example, consider a *clicks per session* metric. The number of clicks and the number of sessions are two sets of observations coming from the same group of users, so they are not independent from each other. To properly account for this correlation, the variance of a ratio metric *R* is obtained using the delta method: Delta method variance formula where the variance of the numerator and denominator means are computed in the same way as detailed above for count metrics, and the covariance is Covariance calculation formula ## Delta Method for Relative Lifts Statsig may also use the delta method when calculating the confidence interval for relative lifts. The other methodology for calculating confidence intervals for relative lifts is [Fieller Intervals](/experiments/statistical-methods/methodologies/fieller-intervals) - the delta method is a heuristic for Fieller Intervals which converges with a large population. # Fieller Intervals Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/fieller-intervals How Statsig uses Fieller intervals to construct confidence intervals for ratio metrics in experiment analysis, with formulas and interpretation notes. ## Fieller Intervals Organizations can choose to use Fieller Intervals as the methodology of calculation for the confidence intervals for the relative change between test and control group. The Delta Method is an approximation for the variance of a ratio between two variables that is then used to establish a confidence interval, while Fieller Intervals are an exact solution for the confidence interval. In most cases though, Fieller Interval results are very similar to results from the Delta Method. Since Fieller Intervals are more accurate, we recommend that you opt into using this methodology! ## Calculation ### 1: Determine if a Fieller Interval is Well-Defined Before proceeding to applying Fieller’s Theorem, we need to check that the denominator of the relative lift metric $ \overline{X_C}$ is significantly distinct from 0. We do this by calculating the parameter $ g$: $$ g = \frac{Z_{\alpha/2}^2 \cdot \mathrm{var}(X_C)}{(n_C-1) \cdot \overline{X_C}^2} $$ Where: $ Z_{\alpha/2}$ is the critical value associated with the desired confidence level $ \mathrm{var}(X_C)$ is the variance of the control group metric values $ n_C$ is the number of units in the control group $ \overline{X_C}$ is the mean of the control group metric values When $ g$ \< 1, the control mean is significantly different from 0, and we can use Fieller intervals. ### 2A: Apply Fieller Interval Formula Since the control and test group results are independent of each other, covariance terms in Fieller's Theorem can be dropped. $$ CI(\% \Delta \overline{X} ) = \frac{1}{1-g} \left( \frac{\overline{X_T}}{\overline{X_C}} \pm \frac{Z_{\alpha/2}}{\overline{X_C}} \sqrt{ \frac{\overline{X_T}^2}{\overline{X_C}^2} \cdot \frac{\mathrm{var}(X_C)}{n_C-1} + (1-g)\frac{\mathrm{var}(X_T)}{n_T-1} } \right) - 1 $$ ### 2B: Edge Case: Control Mean not Statistically Distinct from Zero In rare cases (less than 5% of observed metric comparisons on Statsig), g $\geq$ 1, which means that the control group’s mean is not statistically distinguishable from 0. When $\overline{X_C}$ is not statistically different from zero, the denominator of our relative lift calculation is unstable. This means that the confidence interval for the percent difference between test and control is unbounded. When this happens, we surface the relative lift observed during the experiment. $$ \% \Delta \overline{X} = \frac{\overline{X_T}-\overline{X_C}}{\overline{X_C}} $$ ## Enabling on Statsig Controlling which relative confidence interval methodology you use is available in your Experimentation Settings at the Organization level, and changing this setting only impacts experiments created after the setting change. Experimentation settings configuration interface In many cases, the results will be effectively the same as using the [Delta Method](/stats-engine/methodologies/delta-method), but especially if you’re running experiments with small sample sizes or noisy denominators, Fieller Intervals are more reliable. Thus, we'd strongly recommend using Fieller Intervals. In the experiment scorecard, Fieller Intervals will look like this Experiment scorecard with Fieller intervals # One-Sample Test Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/one-sample-test How Statsig uses one-sample tests to compare an experiment group's metric against a fixed reference value rather than against a control group. ## One-Sample Tests (aka Fixed-Value Test) A one-sample test compares a single sample of data against a known or hypothesized value to determine if there is a statistically significant difference. Unlike A/B tests that compare two groups, one-sample tests evaluate whether a single group differs from a specific benchmark, target, or historical baseline. ## When to Use One-Sample Tests One-sample tests are useful for comparing a single group against a known value: * **Single Group Events**: When only one group can trigger certain events (e.g., feature usage, error types), compare against expected baseline * **Algorithm Testing**: Test if an algorithm performs better than random (e.g., testing if success rate differs from 50%) ## Statistical Considerations One-sample tests provide a way to make statistical inferences about whether your observed data differs significantly from a hypothesized value. The test helps determine if any observed difference is due to random variation or represents a true change in the underlying process. ## How to Enable the Feature 1. Go to the setup page of an experiment Experiment setup screen highlighting metrics section 2. Click the metric name Metric name dropdown showing configure options 3. Select Use Fixed Baseline as Control Fixed baseline control modal for one-sample test configuration # One-Sided Test Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/one-sided-test How Statsig uses one-sided hypothesis tests in experiments to detect changes in a pre-specified direction with higher statistical power. ## One-Sided Tests (aka One-Tailed Test, Non-Inferiority Test) A one-sided test lets you test for a metric moving in only one direction which you specify in advance. This trade off gives you additional sensitivity (or power). It differs from the standard Pulse results which show two-sided results by default. Use cases for one-sided testing include detecting regressions in guardrail metrics and testing for a change in which only one direction has any meaningful business impact. For example, you may be less interested in detecting if a new feature reduces crash rates, but are very interested in learning if the new feature increases crash rates. In this example, you are willing to forgo detecting the former in favor of better detecting the latter. One-sided tests completely disregard the possibility of detecting the metric moving in the direction that isn't specified, but they give you higher sensitivity in the direction you are looking (which allots all your alpha to testing statistical significance in the one direction of interest). This results in one-sided confidence intervals (CIs) that are narrower in the direction of interest than their two-sided counterparts. ## How to enable this When setting up an experiment and identifying metrics to measure, the default setting is to run a two-sided test. If you want to modify this, simply click on the metric name on the experiment setup screen. This will open a popup where you can modify the test type and indicate a desired direction you seek to measure. Our V1 doesn't support Bayesian testing yet. One-sided test configuration interface ## How to read this Metrics using one-sided tests will show up in Pulse very similarly to two-sided tests. The only difference will be that we show a one-sided CI rather than a two-sided CI. Reading one-sided CIs can be a bit confusing at first. They either extend to infinity or negative infinity, which is a bit unusual, but this is entirely expected since we only detect changes in the other direction. As usual with CIs, they indicate that the real mean value of the metric likely falls into this range. Since the CI for the one-sided metric analyzed is so wide, it can be equally useful to read the results as having high confidence the mean value does not fall in the range outside the CI. One-sided confidence interval visualization ## FAQ #### Why can't I just run two one-sided tests? It could be tempting to do this, but in reality it would result in less powerful test. One-sided tests work by allocating the entirety of our Type I error (alpha/significance) to one direction. If we were to add an additional one-sided test in the other direction, we would be re-introducing chance of making a Type I error in that direction. Doing so would mean we claim tighter confidence intervals around metric results that would actually result in higher rates of decision error than the specified confidence level (default: 95%). #### Why use a one-sided tests rather than a two-sided test? This comes down to your use case, metric of interest, and business impact of any decision. Selecting one-sided vs. two-sided tests depends on how you plan to interpret any change in the metric and whether detecting changes in either direction are equally valuable. # Winsorization Source: https://docs.statsig.com/experiments/statistical-methods/methodologies/winsorization How Statsig applies winsorization to cap extreme metric values, reducing variance and stabilizing experiment results against influential outliers. Winsorization is a common technique for removing noise in experiment results, specifically from outliers. Winsorization refers to the practice of measuring the percentile Px of a metric and setting all values over Px to Px. Statsig computes the Px value using all non-zero and non-null unit-level values of the metric; metrics are aggregated from rows or events, and then the Px'th unit's value is used as the threshold to adjust other units' values. At Statsig, the default percentile for winsorization is 99.9%. This reduces the influence of extreme outliers caused by factors such as logging errors or bad actors. Winsorization is applied to sum, event count, mean, ratio, and funnel metrics, including imported metrics. Winsorization will not be applied to Participation or User Accounting metrics. Statsig Warehouse Native lets you configure this per metric - and choose explicitly the upper and/or lower bounds to apply. Winsorization configuration interface Winsorization is applied to sum, event count, mean and ratio metrics. ## Metric Capping This is a very simple, but effective technique to handle outliers. With this capability, you can define max values for a metric for whatever unit type(s) are configured for this metric. Any value surpassing the set cap will automatically be adjusted downward to match it. For instance, if you determine that purchases greater than \$10,000 per day on your E-commerce platform should not skew analysis, any transaction exceeding this threshold will be adjusted downward to this limit, ensuring the integrity of your experiment analysis. Capped metrics are available for Event Count and Aggregation (sum) metric types. # Metric Deltas Source: https://docs.statsig.com/experiments/statistical-methods/metric-deltas How Statsig computes metric deltas to compare absolute and relative differences between experiment groups, including formulas and interpretation guidance. ## Computing Metric Deltas A metric delta refers to the difference in metric values between two groups, by default the test and control groups. This is usually the impact we care about when looking at experiment results. To account for the different number of users (or units) that constitute each group, we compare the mean metric value per user, not the total. **Selecting Groups** We define all deltas here to be the difference between a "treatment" group as compared to a presumably unchanged "control" group. However, Statsig enables comparison between any two groups as desired. Two different metric deltas are available in Pulse. The **absolute delta** is simply the difference between the two means: $$ \Delta \overline{X}=\overline{X}_t-\overline{X}_c $$ It's often helpful to understand the impact relative to the baseline value of the metric. For example, an absolute delta of +1 clicks/user has different meanings with a baseline value of 1 (+100% increase) vs. a baseline value of 100 (+1% increase). The **relative delta** is computed using the mean of the control group as the baseline: $$ \Delta \overline{X} \%=\frac{\overline{X}_t-\overline{X}_c}{\overline{X}_c} \times 100 \% $$ If you reverse the order of group comparison in Pulse to be "control" vs "treatment", then all deltas will be reversed and the direction of change will be inverted. ## Computing Means Properly computing the groups means is critical for obtained meaningful metric deltas. The exact methodology for calculating the metric means depends on the type of metric. ### Event Count and Sum Metrics These metrics represent totals: Number of times an event occurs, sum of time spent, total purchase amount, etc. The mean is the average user-level total during the time period of the analysis. The mean value of the metric $X$ for a group is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} X_{i, d} $$ where: * $N$ is the number of users in the group * $n_i$ is the number of days during the analysis period that user $i$ was the experiment * $X_{i,d}$ is the metric value for user $i$ on day $d$ Only user metrics recorded after a user has been exposed to the experiment are included in the group mean. ### User Accounting and Event User Metrics (and legacy Event DAU) Event User metrics set to "Daily Participation Rate" capture the number of distinct users that have the event each day. In Pulse results, they are normalized by the number of days the user is in the experiment. This represents the probability that a user is daily active for that event, i.e. the daily participation rate. In the terms defined above, the group mean is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N \frac{1}{n_i} \sum_{d=0}^{n_i} X_{i, d} $$ where: * $X_{i,d}$ takes value 0 or 1 depending on if user $i$ has the event on a given day $d$. The following user accounting metrics are computed in the same may: *DAU, WAU, MAU\_28day, L7, L14, L28* For new user accounting (*new\_DAU, new\_WAU, new\_MAU\_28day*) we count users that are new xAU at some point during the analysis window. So the group mean is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N \max \left(X_i\right) $$ Where $\max(X_i)$ is the maximum value of the new xAU metric for user $i$. **event\_dau** metrics are now in legacy support only and are no longer created for new events. Existing event\_dau metrics will continue to be available for any of your new experiments and will continue to be computed daily. For all new events, you should create an event\_user metric to measure daily active users. ### Custom Ratios, Means, Retention and Stickiness Metrics These are metrics such as click through rate, average purchase value, sessions per user, etc. They're obtained by diving a numerator value, $X$, by a denominator value, $Y$. The mean value of a ratio metric $R$ for an experiment group is given by: $$ \overline{R}=\frac{\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} X_{i, d}}{\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} Y_{i, d}}=\frac{\overline{X}}{\overline{Y}} $$ Where $N$ is the number of users in the experiment group that participate in the metric, i.e. have a non-zero denominator value. $X_{i,d}$ and $Y_{i,d}$ are the $X$ and $Y$ values for user $i$ on day $d$. Different approaches exist for dealing with ratio metrics in experiments. This implementation was selected because it's statistically sound as well as interpretable, given that: * $R$ is the ratio of two means of independent observations: A set of user-level $X$ values and a set of user-level $Y$ values. This means the central limit theorem can be used to separately obtain the summary statistics of $X$ and $Y$. * The group means are computed in the same way as the topline metric value, making it easier to interpret the means and relate them to the topline metric. ### Event User One-Time Event For custom **event\_user** metrics with "One-Time Event" selected, statsig computes how many users have the event at any time after the user has been in the experiment. This result is not normalized by the number of days a user is in the experiment. The group mean is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N X_{i} $$ where: * $N$ is the number of users in the group * $X_{i}$ takes value 0 or 1 depending on if user $i$ has the event at any point after entering the experiment # p-Value Calculation Source: https://docs.statsig.com/experiments/statistical-methods/p-value What p-values mean in Statsig experiments, how they are computed, and how to interpret them alongside confidence intervals and lift estimates. In Null Hypothesis Significance Tests, the p-value is the probability of observing an effect larger than or equal to the measured metric delta, under the assumption that the null hypothesis is true. In practice, a p-value that's lower than your pre-defined Type I Error threshold ($\alpha$) is treated as evidence for there being a true effect. The methodology used for p-value calculation depends on the number of degrees of freedom ($\nu$). A two-sample z-test is appropriate for most experiments. Welch's t-test is used for smaller experiments with $\nu < 100$. In both cases, the p-value depends on the metric [mean](/stats-engine/metric-deltas) and [variance](/stats-engine/variance) computed for the test and control groups. Typically, a p-value that indicates statistical significance (below the pre-determined threshold $\alpha$) could only occur with a confidence interval that does not cross 0. However, this phenomenon can occur in the Statsig UI, due to cases when the p-value of the difference between test and control is statsig, but due to uncertainty in the control, a relative delta confidence interval may cross zero (using [The Delta Method](/experiments/statistical-methods/methodologies/delta-method)) or be represented as a point estimate (using [Fieller Intervals](/experiments/statistical-methods/methodologies/fieller-intervals) ) while the absolute difference's p-value is statistically significant. ## Two-Sample Tests ### Two-Sided z-Test The z-statistic (a.k.a. z-score) of a two-sample z-test can be computed in multiple equivalent formats: $$ \begin{split} Z &= \frac{\overline X_t - \overline X_c}{\sqrt{var(\overline X_t)+ var(\overline X_c)}} \\ &= \frac{\overline X_t - \overline X_c}{\sqrt{var(\Delta \overline{X})}} \\ &= \frac{\overline X_t - \overline X_c}{\sqrt{\sigma_{\overline{X}_t}^2 + \sigma_{\overline{X}_c}^2}} \end{split} $$ where: * $Z$ is the observed z-statistic (not the z-critical value $Z_{\alpha/s}$) * $var(\Delta \overline{X})$ is the variance of the absolute delta of means * $var(\overline{X}_i)$ is the variance of sample means either control or treatment group (details [here](/stats-engine/variance)) * $\sigma_{\overline{X}_t}$ is the standard error of the mean of either control or treatment group (these are the terms you can find in Pulse under the Statistics tab of a metric) The two-sided p-value is obtained from the standard normal cumulative distribution function: $$ p-value = 2 \cdot \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{-|Z|}{e^{-t^2/2}dt} $$ ### Welch's t-test For smaller sample sizes, Welch's t-test is the preferred statistical test for lower false positive rates in cases of unequal sizes and variances. In Pulse, Welch's t-test is automatically applied when the degrees of freedom $\nu < 100$. We compute the t-statistic (a.k.a. t-score) identically as the two-sample z-statistic above. Additionally, we compute the degrees of freedom $\nu$ using: $$ \nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}\ := \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}} $$ The p-value is then obtained from the t-distribution with $\nu$ degrees of freedom. ### One-Sided Z-Test The procedure for a one-sided z-test computes the z-statistic $Z$ in the same way as a two-sided test above. The one-sided p-value is obtained from the standard normal cumulative distribution function as well, but with slight differences: $$ p-value = \begin{cases} 1 - \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if right-hand test}\\ \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if left-hand test} \end{cases} $$ where: * $Z$ is computed above in the two-sided test. Note that this uses the signed z-statistic, not the absolute value of the z-statistic as in the two-sided p-value. # Pre-Experiment Bias Source: https://docs.statsig.com/experiments/statistical-methods/pre-experiment-bias How Statsig detects and corrects for pre-experiment bias caused by uneven user distributions between treatment and control groups before exposure. In some cases, users in two experiment groups can have meaningfully different average behaviors before your experiment applies any intervention to them. If this difference is maintained after your experiment starts, it's possible that experiment analysis will attribute that pre-existing difference to your intervention. This can make a result seem more or less "good" or "bad" than it really is. [CUPED](/experiments/statistical-methods/methodologies/cuped) is helpful in addressing this bias, but can't totally account for it. Additionally, some metrics like retention are not viable candidates for CUPED and can't be easily adjusted. Statsig proactively measures the pre-experiment values of all scorecard metrics for all experiment groups, and determines if the values are significantly different and could cause misinterpretations. If bias is detected, users are notified and a warning is placed on relevant Pulse results. ### How it works Statsig provides a "Days Since Exposure" view to help identify novelty effects and existing pre-experiment effects. For example, the test group of the experiment below had a consistently higher mean than the control group in the week before exposure for this metric Pre-experiment bias visualization showing test group with consistently higher mean than control group Statsig detects this bias by running the standard [pulse](/pulse/read-pulse) calculation on the pre-experiment term (looking back one week in cloud, and your configured CUPED lookback window in Warehouse Native), and calculating the p-value for the null hypothesis that the groups are identical. Relevant results will be flagged according to logic which balances awareness and false positives stemming from high numbers of scorecard metrics or groups. ### What to Do Pre-experiment bias can occur by chance and is not always a major issue. * If the total delta is small, it may not meaningfully influence your interpretation of results * If CUPED can account for the bias, then the bias should not impact your results In many cases, you can just use this warning as that - a warning - and proceed while treating impacted metrics with a grain of salt. This is often the correct path forward if the metric is not critical to the experiment, or if you care more about the directional movement than the exact number. Additionally, more time may alleviate the bias if there's no systemic source (which is generally the case), as the bias will be diluted by additional new users. However, if the metric is critical to your analysis and you care about the exact numerical value, you may want to consider resalting and restarting this experiment. # Topline and Projected Impact Source: https://docs.statsig.com/experiments/statistical-methods/topline-impact How Statsig estimates the topline impact of an experiment on company metrics by scaling experiment lift to your total addressable user base. The **topline impact** is the average daily effect that an experiment has on the overall metric value as compared between two groups. This is the real daily impact to a metric resulting from running the experiment, measured amongst the two groups being evaluated. The **projected launch impact** is an estimate of the daily impact we expect to see in the metric measured globally if a decision is made and the test group is launched to all users (beyond just those in the experiment). This impact is computed relative to the expected baseline value of the metric if the experiment wasn't running at all. Topline Impact and Projected Launch Impact are shown in both absolute and relative units. Topline Impact and Projected Launch Impact currently do not use CUPED when measuring any impact induced by your experiment. The reason is that CUPED already adjusts for pre-exposure data, which is what the topline metrics change from - mixing the two would double-count that adjustment. **Example**: Take a simple example experiment with a Control group of 1000 users and a Test group of another 1000 users, which ran for 30 days. For an **event\_count** metric, we observed an Experiment Delta of +1.0 events per user (abs). The Topline Impact for this metric would be +33.33 events per day (abs). ## Computing Topline Impact The topline impact is computed over the total duration of the experiment. This gives the most accurate estimate and tight confidence interval. The exact calculation depends on whether the metric represents an absolute quantity or a ratio: ### Count and Sum Metrics (event\_count, sum) The absolute topline impact is derived directly from the experiment results. It depends on the difference in means between test and control, and the average number of users in the test group per day. $$ Impact_{abs}=(X_t-X_c) \cdot N_t / n_{days} $$ Knowing the absolute impact and the overall metric value (as seen in the [metrics dashboard](/metrics/console)), we can compute the relative impact. This is the percentage change in the overall metric value over the rollup window that is attributed to the active experiment. $$ Impact_{rel}=\frac{Impact_{abs}}{Topline\_Value-Impact_{abs}} \times 100\% $$ ### Ratio and Mean Metrics To properly derive the topline impact on a ratio metric we must understand the impact on the numerator (*X*) and denominator (*Y*) separately. The topline impact is the current value of the ratio metric minus the baseline value we obtain by subtracting the numerator and denominator impacts: $$ Impact_{abs}=\frac{Topline\_X}{Topline\_Y}-Baseline\_Value $$ Where the baseline value is the expected value of the topline metric if the experiment wasn't running: $$ Baseline\_Value=\frac{Topline\_X-(\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline\_Y-(\bar{Y_t}-\bar{Y_c}) \cdot N_t} $$ The relative impact for ratio metrics is obtained by dividing the absolute impact by the baseline value: $$ Impact_{rel}=\frac{Impact_{abs}}{Baseline\_Value} \times 100\% $$ ## Computing Projected Launch Impact The layer allocation of the experiment and the size of the test group are used to estimate a scaling factor *m*, which represents the increase in absolute impact expected when a decision is made to launch the test group. The launch factor over a rollup window is calculated as $$ m_{rollup}=\frac{1}{\sum_{1}^{rollup}{layer\_alloc \times group\_pct}} \times rollup $$ to accommodate changes in allocation during the experiment. The targeting gate isn't factored in. The projected impact calculation assumes that the target gate remains the same after the experiment is launched. ### Count and Sum Metrics (event\_count, event\_dau, sum) For count and sum metrics, the projected absolute impact is simply the current topline impact scaled by a factor of *m*. For example: Consider an experiment running with 50% layers allocation and 50/50 test/control split, so that 25% of all users are in the test group. If the allocation has been changing during this experiment, we will use a weighted average based on historical allocations. If the topline impact is currently +10 events per day, then launching the experiment would lead to +40 events per day. $$ Projected_{abs}=Impact_{abs} \times m $$ The relative projected impact is expected percentage change in the topline metric, relative to the baseline value of the metric without the experiment running. $$ Projected_{rel}=\frac{Projected_{abs}}{Topline\_Value-Impact_{abs}} \times 100\% = Impact_{rel} \times m $$ ### Ratio and Mean Metrics Similar to the topline impact calculation above, the projected impact of ratio metrics depends on the numerator and denominator impacts. We use the same scaling factor *m* to obtain the projected impact for each term: $$ Projected_{abs}=\frac{Topline\_X+(m-1) \cdot (\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline\_Y+(m-1) \cdot (\bar{Y_t}-\bar{Y_c}) \cdot N_t} - Baseline\_Value $$ Where the first term represents the projected metric value after launch. Finally, the projected relative impact of a ratio metric is the projected absolute impact divided by the baseline value of the ratio: $$ Projected_{rel}=(\frac{Projected_{abs}}{Baseline\_Value}) \times 100\% $$ ## Confidence intervals The confidence intervals for topline and projected impact are computed in the same way as the [confidence intervals](/stats-engine/confidence-intervals) for experiment deltas. $$ CI(Impact) = Impact \pm Z \cdot \sqrt{var(Impact)} $$ In the case of absolute impact of count and sum metrics, the variance calculation is simply a linear combination of the test and control variances: $$ var(Impact_{abs})=[var(\bar{X_t})+var({\bar{X_c}})] \cdot N_t^2 $$ And for projected launch impact we get: $$ var(Projected_{abs})=var(Impact_{abs}) \cdot m^2 $$ For ratio metrics and relative impacts, the variance is calculated using the Delta method. This properly accounts for the correlation between the various numerator and denominator terms, leveraging Taylor expansion to linearize expressions containing non-linear combinations of experiment variables. For example, the variance in the relative impact of a count metric is given by: $$ var(Impact_{rel})=var(Impact_{abs}) \cdot \frac{(Topline\_Value - 2 \cdot Impact_{abs})^2}{(Topline\_Value - Impact_{abs})^4} $$ # Standard Error & Mean Variance Source: https://docs.statsig.com/experiments/statistical-methods/variance How Statsig computes variance for experiment metrics, including handling of ratio metrics, clustered data, and user-level aggregation across exposures. The standard error (sometimes denoted "SE" or "std err") of the mean of each group is required for computing the confidence interval and p-value of a metric delta between those groups. The standard error of the mean can be obtained by dividing the sample standard deviation of $X$ by the square root of the number of users in the group. $$ \sigma_{\overline X} = \frac{\sigma_{X}}{\sqrt{N}} = \sqrt{\frac{var(X)}{N}} = \sqrt{var(\overline{X})} $$ Note that standard deviation is the square root of the variance. Since variances are easier to manipulate algebraically, here we derive the variance for each metric type and then take the square root to obtain the confidence intervals. Pulse displays the standard error of the mean of each group alongside the units and mean of each group. ## Computing Variance The variance of the absolute metric delta is simply the sum of the variances of the test and control means: $$ var(\Delta \overline X) =var(\overline X_t - \overline X_c) = var(\overline X_t) + var(\overline X_c) $$ In other words, it comes down to correctly calculating the variance of the means for each group. ### Count and Sum Metrics For count and sum metrics, the variance of the sample mean for a given group is obtained directly from the sample variance: $$ var(\overline{X}) = \frac{var(X)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline{X})^2}{N} $$ Where: * $N$ is the number of users in the group * $X_i$ is the metric value for user $i$ * $\overline{X}$ is the user-level average of $X$ for users in that group ### Ratio and Mean Metrics Some metrics like ratio and mean metrics combines multiple variables $X$ and $Y$ instead of a single variable $X$. The variance of these metrics depends upon both the numerator and denominator variables, which are typically correlated. We'll call the metric of interest $R$ and we can compute the group mean $\overline{R}$ and group variance of the mean $var(\overline{R}$). For example, consider a *clicks per session* metric. The number of clicks and the number of sessions are two sets of observations coming from the same group of users, so they are not independent from each other. To properly account for any correlation, the variance of the mean of a ratio metric $R$ is obtained using the delta method: $$ var(\overline R) = var\left(\frac{\overline X}{\overline Y}\right) := \left(\frac{\overline X}{\overline Y}\right)^2 \cdot \left(\frac{var(\overline X)}{\overline X^2} + \frac{var(\overline Y)}{\overline Y^2} - 2 \cdot \frac{covar(\overline X, \overline Y)}{\overline X\cdot \overline Y} \right) $$ where the variance of the numerator and denominator means are computed in the same way as detailed above for count metrics, and the covariance is $$ covar(\overline X, \overline Y) = \frac{covar(X, Y)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline X)\cdot (Y_i-\overline Y)}{N} $$ # Variance Reduction Source: https://docs.statsig.com/experiments/statistical-methods/variance-reduction Overview of variance reduction techniques in Statsig experiments, including CUPED, stratified sampling, and regression adjustment for higher sensitivity. ## Variance Reduction [Variance](/stats-engine/variance) is a measurement of dispersion which measures the amount of "noise" in a metric or experiment results. Higher variance is associated with larger confidence intervals, and leads to experiments requiring more sample size to consistently observe a statistically significant result on the same effect size. Reducing variance can lead to shorter experiment run times due to the lower sample required. Because of this, techniques have been developed to reduce the variance in experiment results in order to reduce run times and increase confidence. At Statsig, we use a form of CUPED based on a [2013 Microsoft paper](https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf) (Deng, Xu, Kohavi, & Walker). This is automatically applied to experiments at Statsig, and is run for the topline results on key metrics in Pulse. This observably leads to significant variance reduction in the large majority of metrics where CUPED can be applied. Refer to our [launch post for CUPED](https://blog.statsig.com/cuped-on-statsig-d57f23122d0e) for more details. ## CUPED - Controlled-experiment Using Pre-Existing Data CUPED (short for Controlled-experiment Using Pre-Existing Data) is a technique which leverages user information from before an experiment to reduce the variance, and increase confidence in experimental metrics. At Statsig, this pre-experiment data is defined as the 7 days before each user's exposure rather than a fixed window before the experiment starts for all users. This can help to debias experiments which have meaningful pre-exposure bias (e.g. the groups were randomly different before any treatment was applied). The Cloud product uses stratification alongside CUPED to account for users who may not have pre-experiment data. Users are grouped into strata based on available pre-experimentation information. Treatment and control effects are first estimated within each stratum, then aggregated to produce an overall result. We then apply the standard difference-in-means and variance estimation. This approach allows us to retain users with missing pre-data while still benefiting from variance reduction where applicable. ## Winsorization Another common technique for reducing noise is Winsorization, which is a way to manage the influence of outliers. Winsorization refers to the practice of measuring the percentile *Px* of a metric and setting all values over *Px* to *Px*.This reduces the influence of extreme outliers caused by factors such as logging errors or bad actors. ## Metric Selection The metrics you use can dramatically influence the sensitivity of your analysis. The transformations above, in addition to techniques like creating threshold-based flags, can let you trade-off exact numbers for significantly more power. Please refer to our [blog post](https://www.statsig.com/blog/understanding-and-reducing-variance-and-standard-deviation) on the topic for more information. ## Literature Here's a short list of useful content for understanding more about these techniques and its applications * [Deng, Xu, Kohavi, & Walker](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf) is the seminal paper on using this technique for online controlled experiments * [Booking.com](https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d) has an excellent blog post on the theory and practice of CUPED * [Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix](https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf) # Decision Framework Source: https://docs.statsig.com/experiments/templates/decision-framework Use a structured decision framework in Statsig to evaluate experiment results, including success criteria, ship versus iterate logic, and reviewer sign-off. A Decision Framework for experiment templates allows teams to standardize their interpretation of results and decision-making process. Once configured, it provides clear recommendations for which group to ship based on experimental outcomes. While the framework highlights recommended actions based on metric movements, it does not enforce any actions. ## Setup To configure a Decision Framework, find your desired experiment template under **Settings** → [**Templates**](https://console.statsig.com/templates), then navigate to the **Decision Framework** tab. By selecting one primary metric and one or more guardrail metrics, you can choose a recommended action based on metric performance. Team-level Templates Settings Three different types of recommendations can be configured: * **Rollout Winning Group**: An icon appears next to the winning group in the Make Decision button * **Discuss**: A message appears on the experiment page recommending discussion before making a ship decision * **Do Not Roll Out**: An icon appears next to the control group in the Make Decision button recommendation ## Reviews for shipping negative results Reviewers can be set up when a shipping decision doesn't align with the recommendations configured in the decision framework. All it takes is to toggle on "Require reviews when decision opposes recommended decision" and add reviewers in the following dialogue box. recommendation Once set up, a reviewer will be required in the following situations: * **Rollout Winning Group**: Shipping a treatment group that is not recommended * **Discuss**: Shipping any treatment group * **Do Not Roll Out**: No review will be required recoomendation # Templates Source: https://docs.statsig.com/experiments/templates/templates Use Statsig experiment templates to standardize hypothesis, metrics, and review workflows across your experimentation program for higher quality tests. ## Overview Templates enable you to create a blueprint for gates/dynamic configs/ experiments to enable standardization and reusability across your Project. Templates can help enforce a rollout sequence, or make it easy for new experimenters to get up & running with your standard team settings for experimentation. Templates can be enforced at the Org (via [Organization Settings](/org-admin/organization_policies) and [Role-based Access Controls](/access-management/projects)) or at the [Team-level](/access-management/teams). We will detail the various levels of controls and permissions you can enable for templates below. ## Creating Templates There are two primary ways to create a new template- 1. From the [**Templates**](http://console.statsig.com/templates) page under **Product Configuration** 2. From a gate/ experiment you want to turn into a new template ### Creating Templates from Project Settings To create a new template from Project Settings, navigate to **Settings** -> **Project Configuration** -> **Templates** and click the **+Create New** CTA, then select whether you want to create a gate, experiment, or dynamic config template. Template Creation ### Converting an Existing Config into a Template If you create a new config that you think would be useful to more folks on the team as a template, you can convert an existing (or currently being created) config into a template. To do this, in the config you wish to convert to a template, tap the "..." menu and select **Save as Template**. This will prompt you to name your template and add a description before saving. Save as Template 1 Save as Template 2 ## Managing Templates Templates can be managed via the **Templates** setting under **Product Configuration**. Permissions for who can create or modify templates are managed via Statsig's Role-based Access Controls in **People** -> **Roles**. By default Org and Project Admins can modify or delete any Template. Templates ## Creating Configs from Templates To create a config using a template, at the time of config creation, select a template to apply from the template selector. If templates are required at the Organization or Team-level, the user will be blocked from proceeding with config creation until they've selected an approved template. The list of template options in the drop-down is configured at the team level (see below). Template Selection ## Managing Templates- Settings & Permissions There are a few key layers of settings governing templates, namely at the- 1. Org level 2. Team level ### Org-level Templates Settings Within Experiment and Gate Policies (**Settings** -> **Project Configuration** -> **Feature Management**/ **Experimentation** - **Organization Tab**), you can enforce that a template is used for any new gate/ dynamic config/ experiment creation. Only organization admins can configure this setting. You must create at least 1 experiment/ gate template for users to choose if you toggle on this setting, otherwise they will be blocked in creating new configs. Org-level Feature Gate Templates Settings ### Team-level Templates Settings At the team-level, you can configure which templates members of that team can choose from at the time of config creation. You can also choose whether to require use of a template or not at the team level This setting only applies if templates aren't already required at the organization-level, in which case that overrides any team-level configuration). To configure team-specific templates, navigate to **Settings** -> **People** -> **Teams** -> choose a team -> **Settings** and then choose which templates are allowed for gates, dynamic configs, and experiments. Team-level Templates Settings ### Notes * Experiment templates cannot be renamed. An effective work-around for this is to create a new experiment using the template, then make a new template from that experiment with the desired name. * Templates cannot be modified via Console API. # Running an A/A Test Source: https://docs.statsig.com/experiments/types/aa-test Run A/A tests in Statsig to validate your experimentation setup, confirm metrics are configured correctly, and check for sample ratio mismatch issues. In this guide, you will create and implement an A/A test on your product in Statsig from end to end. This is commonly used to validate a new experimentation engine you may be integrating with. For new users just getting started with Statsig, we often recommend running an A/A test to provide a “low-stakes” first test environment to ensure that you have your metrics set up correctly and are seeing exposures flowing through as expected before kicking off your first real A/B test. By the end of this tutorial, you will have: * Created a new **Feature Gate** in the Statsig console, set up as an "A/A test" ## Prerequisites 1. You already have a [Statsig account](https://console.statsig.com/sign_up) 2. You already [integrated the Statsig Client SDK](/sdks/quickstart) into an existing application ## Step 1: Create a feature gate in the console The easiest way to run an A/A test in Statsig is by leveraging a [Feature Gate](/feature-flags/overview). You can also leverage an [Experiment](/guides/abn-tests) to run an A/A, but we chose to use a Feature Gate for this tutorial for simplicity. Log into the Statsig console at [https://console.statsig.com/](https://console.statsig.com/) and navigate to **Feature Gates** in the left-hand navigation panel. Click on the **Create** button and enter the name and (optional) description for your feature gate. We will call our feature gate “aatest\_example”. Click **Create**. create_new_fg_empty In the Setup tab, define the rules for this feature gate. Tap **+ Add New Rule**. While you could run an A/A test on a specific user-group, platform, etc. the easiest setup is to simply divide all of your traffic 50/50 and deliver the same experience (your default product experience) to each group. add_new_rule_empty To do this, under **Criteria** select **Everyone** (you may need to scroll up), name your rule, and then change the **Pass Percentage** to 50%. Click **Add Rule** and that’s it! Tap **Save Changes** in the upper right-hand corner. add_new_rule_filled Your feature gate setup should now look as follows- aa_rule_filled_out Check that it is working as expected by typing in some dummy user IDs into the console- roughly 50% of the time your IDs should pass, and 50% of the time they should fail. check_rule_pass check_rule_fail ## Step 2: Check the feature gate in your code Copy the code snippet in the upper right hand corner of your feature gate page under the **\< >** symbol and drop it into your application at the point you want to call the A/A check. ```jsx theme={null} statsig.checkGate("aatest_example") ``` Now when a user renders this page in their client application, you will automatically start to see a live log stream in the Statsig console when you navigate to the **Diagnostics** tab for your feature gate. logstream ## Step 3: Review A/A test results Within 24 hours of starting your experiment, you'll see the cumulative exposures in the **Pulse Results** tab of your feature gate. cumulative_exposures This will break down your logged exposures (as well as the distribution of the logged exposures). If something looks off, check the **Diagnostics** tab for more granular, day-by-day exposure breakdowns at both the Checks and User level. In the **Metric Lifts** panel, you can see the full picture of how all your tagged metrics are performing. pulse_results_empty What should you expect to see? * **Exposures**- make sure you’re seeing exposures flowing through as expected from your product. If you’re not seeing exposures, use the **Diagnostics** tab and the **Exposure Stream** to debug * **Pulse results**- roughly 5% of your metrics in Pulse should be showing a statistically significant change due to the 95% confidence interval of Statsig’s stats engine We recommend running your A/A long enough to reach most of your weekly active users, or at least a week. ## Simulated A/A Tests We’ve made running A/A tests at scale easy by setting up simulated A/A tests that run every day in the background, for every company on the platform. An A/A test is like an A/B test - but both groups get the same experience. A/A tests help build trust in your experimentation platform (and your metrics!) A/A tests can be Online or Offline. An [Online A/A test](/guides/aa-test) is run on real users. An engineer instruments your app with the Statsig SDK to check for experiment assignment. Assignment is logged, but there's no difference in experience to the user. Since there is no effect, you expect to only see statistical noise. When using 95% confidence intervals, only \~1 in 20 metrics will show a stat-sig difference between control and test. ### Offline A/A tests A single request runs on one unit type, and an offline A/A test works by 1. Querying a representative sample of your data 2. Randomly assigning subjects to Test or Control 3. Computing relevant metrics for Test vs Control and running them through the stats engine 4. You're looking for the % of false positives. If your p-value cutoff is 0.05 (typical), you'd expect a \~5% false positive rate. You can download the running history of your simulated A/A test performance via the “Tools” menu in your Statsig Console. We run 100 tests per request. ### File Description | Column Name | Description | | ------------------------------------ | ------------------------------------------------------------------- | | metric\_name | Name of the Metric | | metric\_type | Type of Metric | | unit\_type | The unit used to randomize (e.g. userID) | | n\_tests | The number of tests run | | pct\_ss\_95\_pct\_confidence | The percentage of tests that have a stat-sig result for this metric | | avg\_units\_per\_test | The number of units (often users) sampled into the A/A test | | avg\_participating\_units\_per\_test | The number of units in the test with a value for this metric | A/A test results table showing statistical significance percentages # SEO Experimentation with Statsig Source: https://docs.statsig.com/experiments/types/seo-testing Run SEO experiments in Statsig to test landing page designs and content variants while preserving organic traffic and avoiding cloaking penalties from Google. In late 2017, Airbnb's growth team faced a deceptively simple question: > Would their new "Magic Carpet" landing page design drive more organic traffic than their existing search results page? With over 100,000 unique URLs spanning different cities, any template change would cascade across their entire search footprint. Traditional A/B testing couldn’t solve this puzzle because Google’s crawlers needed consistent page versions, making user-level randomization impossible. Every marketplace with thousands of templated pages faces the same dilemma: measuring how changes to your template actually impact how Google ranks your pages. This is true whether you’re dealing with physical goods at **Amazon** or **eBay**, or more virtual things at **ZipRecruiter** or **Eventbrite**. Over the past few years at **Statsig**, I’ve helped different marketplaces navigate exactly this problem. What I’ve learned is that companies can ship the same rigorous framework Airbnb developed in **hours, not months**, and see results in the same dashboards they already use for product experiments. *** ## 1. Select a Deterministic Page Bucket * Crawlers must see a consistent version of each URL during the test window, so we **can’t randomize by user**. * Instead, we hash the **canonical URL** into buckets. * In Statsig, you formalize this by adding `page_url` as a **Custom Unit ID**. Steps: 1. From **Project Settings**, navigate to **Custom Unit IDs**. 2. Provide a name and description (it then immediately becomes available to experiments, gates, and dynamic configs). Statsig can now deterministically hash your pages into **Control vs Test** in experiments and keep this assignment stable across sessions. Strip out `http` vs `https` and query params, leaving only the stable base URL, so that is what is hashed deterministically. *** ## 2. Define Metrics Before Shipping Make sure the metrics you’ll want to measure are in your data warehouse, keyed on `page_url`.\ Register these with Statsig’s **metric catalog**. Because the same pipeline powers feature experiments, your existing CUPED or stratified-sampling settings automatically apply. ### Example Metrics | Layer | Metric Source | Why it Matters | | ----------------- | ----------------------------------------------------- | ------------------------------------ | | Indexing lag | Impressions, average position (Search Console) | Early signal during re-crawling | | Primary KPI | Organic sessions keyed by `page_url` (Statsig Events) | Measures traffic that actually lands | | Quality guardrail | Conversion, bounce, read-depth, revenue | Ensures traffic is useful | *** ## 3. Implement the Change Behind an Experiment * Create an experiment called `seo_title_test` in Statsig Docs. * Target on the **Custom Unit ID `page_url`** with a 50/50 split across Control and Test. * Expose the variant in the template renderer or CDN edge function. *** ## 4. Ship, Monitor, Decide * Use **Power Analysis** to determine how long your experiment should run based on traffic volume. * Expect first signals in 2–7 days; wait for re-indexing to plateau before things stabilize. * Merge the winner into your template and archive the test; experiment summaries remain searchable forever. ### SEO-Specific Guardrails | Guardrail | What to Watch | Why it Protects You | | --------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------- | | Indexation Δ | `indexed_pages` vs baseline | A template tweak that blocks crawl (robots, canonicals, noindex) will show a sharp drop long before traffic falls | | Cannibalization ratio | Avg. URLs served per query | Multiple pages newly ranking for the same query dilute CTR and can tank combined traffic | | HTTP response mix | % 410 vs 301 vs 200 | A bulk 410 (gone) or mis-configured 301 can wipe out long-tail pages | | Core Web Vitals drift | LCP & CLS p75 | Page-speed regressions may hurt rankings silently | | Crawl budget | Avg. TTFB + bytes/page | Slow/bloated pages decrease crawl rate | *** ## 5. Concrete Page-Level Changes Worth A/B Testing | Theme | Why It Might Move Organic Traffic | Typical Implementation Knob | | ----------------------- | --------------------------------- | ----------------------------------------------------------------------- | | Title & meta variants | Query-matching, CTR uplift | Add/remove brand suffix, noun → verb phrasing, insert dynamic price | | Structured data | Rich-result eligibility | Inject FAQ, HowTo, Breadcrumb, or Product schema blocks | | Internal-link blocks | Crawl priority & PageRank flow | Swap “related articles” widget ordering; test link count caps | | Content snippets | Relevance & long-tail keywords | Auto-generate 50-word intro vs. none; expand FAQ length | | Canonical/hreflang tags | Duplicate-content handling | Toggle self-canonical vs. cluster canonical; add `hreflang="x-default"` | | Media handling | CLS/LCP scores influence rankings | Defer off-screen images; inline critical hero image; switch to AVIF | | Pagination model | Crawl depth & index coverage | Classic `?page=` URLs vs. `rel="next/prev"` vs. load-more buttons | | Performance budgets | Core Web Vitals ranking factor | 200 ms JS chunk split vs. baseline; CSS purge + inline critical-CSS | | Ad layout | CLS penalties, user engagement | Reserve fixed ad slots vs. dynamic; lazy-load below first viewport | | Schema position | Parser friendliness | Move JSON-LD block to `<head>` vs. end of `<body>` | *** ## 6. Is SEO Experimentation Right for You? | Great Fit | Maybe Not Yet | | ---------------------------------------------------------------- | ----------------------------------------------------------- | | Large page surface (10k+ URLs): marketplaces, docs, publishers | Marketing sites with \<1k pages or sporadic organic traffic | | Teams already shipping **weekly** and want proof before rollouts | Heavy paid-ads model where SEO is \<5% of acquisition | | Companies with engineering bandwidth to template page changes | Sites on locked-down CMSs that forbid code/tag edits | *** ## 7. Key Takeaways * **Segment by page, not user.** Use a Statsig Custom ID for deterministic hashing into Control/Test. * **Measure beyond clicks.** Pair Search Console data with product analytics for full-funnel insight. * **Move fast, break nothing.** Statsig’s sequential engine and guardrails catch bad bets early. * **One platform for every test.** Product, pricing, UX, and SEO experiments in a single, trusted workflow. *** Statsig also supports other experiment types such as **switchback testing** and **geo-testing**. Geo-testing is particularly useful for measuring the **incrementality of ad spend**, which is hard to measure with traditional experiments due to privacy requirements. # Switchback Tests Source: https://docs.statsig.com/experiments/types/switchback-tests Learn about switchback testing methodology and how to set up switchback experiments for marketplaces and network effect scenarios. # What is Switchback Testing? Switchback tests are an alternative experiment form, whereby an entire population is "switched" back and forth between test and control treatments on a set cadence vs. being split and evenly divided between test and control for the duration of the experiment. Switchback tests are particularly common in marketplaces, whereby running a traditional A/B on one side- or a small %- of the marketplace would have an unintended consequence on the rest of the marketplace due to network effects, ultimately impacting experiment results. Another common use case for switchbacks occurs when applying different variants to different users is infeasible for fairness, legal, or logistical reasons. Switchback tests are often carried out across multiple "buckets", typically regions or other defined groups that are flipped between test and control treatments over the course of the experiment. ### Example Let's say you are a rideshare platform and want to test pricing. You initially consider splitting your riders into two groups, one with the higher price and one with a lower price. However, you quickly notice that the riders with the lower price are requesting rides at a significantly higher rate, and sucking up all the available driver supply in a given area. This leaves the riders with higher prices with not only a higher ride estimate, but longer ETAs when they open up their app, making them even less likely to request. When you look at your experiment results you're not sure if the decreased ride request rate in the higher price group was due to the higher prices they saw or the fact that their ETAs went up- your experiment results are polluted! You've inadvertently introduced *bias* to your results via your experimental design. In this scenario, you could consider running a Switchback test on your marketplace. To do this, you might switch 100% of your riders and drivers in a given metro in and out of the new pricing plan hourly and understand the impact on overall ride request rates during hours at which rider prices were higher vs. lower. # Switchback Testing on Statsig ## Methodology Our Switchback Testing methodology for computing results consists of 3 steps: 1. Attribute events to the corresponding switchback bucket, where each bucket is defined by the time window and grouping attribute. 2. Calculate the variant-level and bucket-level metrics based on the attributed events. 3. Calculate the difference in means between test and control. Use bootstrapping to obtain the confidence intervals. ### Event Attribution Attribution of events to a particular bucket is based on the timestamp and unit\_id of the exposure, the length of the attribution window, and the timestamp of subsequent events for that unit\_id. For example: User 123 is exposed to bucket A at 9:15 am. The test has an attribution window of 90 minutes. This means all events triggered by user 123 between 9:15 am and 10:45 am will be included in the metric calculations for bucket A. ### Bucket-level Metrics Once we have all the events corresponding to a bucket, we calculate the scorecard metrics derived from these events. Bucket metrics table summarizing switchback exposures For sum and count metrics, we use the mean value per unit exposed to that bucket. Metric detail view showing per-unit averages within a bucket ### Variant-level Metrics Similarly, we calculate the overall metric means for test and control by aggregating the values across all the buckets in that variant. So if there are **M** buckets in the test group, the mean value of a ratio metric is given by: Formula showing ratio metric mean calculation across switchback buckets The mean of a sum or count metric would be: Formula for sum or count metric averages in switchback tests ### Deltas and Confidence Intervals The treatment effect is calculated as: Equation for treatment effect delta between test and control The bootstrapped confidence intervals are obtained as follows: 1. Collect a bootstrap sample with replacement from the set of test buckets and separately from the set of control buckets. 2. Calculate the difference in means between test and control samples. 3. Repeat steps one and two 10 thousand times. This gives us a distribution of the metric deltas 4. The 95% confidence interval is the range from the 2.5% quantile to the 97.5% quantile from the distribution of deltas in step three. In general, the confidence interval with significance level $\alpha$ is given by Bootstrap confidence interval formula for switchback tests ## Setup To set up a Switchback test on Statsig, when you create an experiment tap **Advanced Settings** → **Experiment Type** and select "Switchback Test". Experiment type menu selecting switchback test There are a few new aspects of experiment configuration when setting up a Switchback test, namely- 1. **Targeting**- The defined population(s) you will be running your experiment on. 2. **Schedule**- The switching frequency and starting treatments for different pre-defined populations. There are a few different ways to define targeting, namely- * **Targeting Gate**- Specify a targeting gate to define your target experiment population, similar to any other experiment on Statsig. * **Bucketing Method**- Bucket users based on either pre-defined buckets or randomized across an ID type. Switchback targeting configuration showing gate and bucketing options **Buckets** enable you to specify pre-defined buckets, such as *Country*, *Locale*, or a *Custom Field* you log. This is useful when you have a few, pre-defined populations you want to switch in and out of Test/ Control over the course of the experiment. Buckets configuration table listing predefined regions **ID Type** lets you specify an ID type to randomize across, e.g. choosing a custom ID such as CityID will automatically randomize different CityIDs across Treatment/ Control over the course of the different switchback windows. This is useful if you have a very large or dynamic number of experiment units you want to randomize across over the course of the experiment. Randomized bucketing is an advanced feature, please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. Randomized ID bucketing interface selecting custom ID Depending on which bucketing method you've chosen, the **Schedule** section of experiment setup enables you to configure- * Start time * Duration (in days) * Assignment window size (in minutes) * Burn-in/ burn-out periods (in minutes) * *(Pre-defined bucketing only)* Starting phase (treatment group) for each bucket Switchback schedule editor specifying assignment windows and bucket starting phases Burn-in/ burn-out periods enable you to define periods at both the beginning and end of your switchback windows to discard exposures from analysis. This is typically leveraged when there are risks of "bleed over effect" from the previous treatment while a population is switching between test and control. ## Reading Results Both Diagnostics and Pulse metric lifts results for Switchback tests will look and feel like Statsig's traditional A/B tests, with a few modifications- * **No hourly Pulse-** At the beginning of a traditional A/B/n experiment on Statsig, you can start to see hourly Pulse results flow through within \~10-15 minutes of experiment start. Given in a Switchback you will only see either *all* Test or *all* Control exposures right at experiment start, we have disabled Hourly Pulse until you have a meaningful amount of data. However, in lieu of Hourly Pulse you can still leverage the more real-time **Diagnostics** tab to verify checks are coming in and bucketing as expected. * **No time-series**- The Daily and Days Since First Exposure time-series are not available for Switchback tests. This is due to the bootstrapping methodology used to obtain the statistics, which relies on pooling all the available days together in order to have enough statistical power. * **No dimension breakdown**- Breaking down a metric by user property or event property is not available for Switchback tests. * **Advanced statistical techniques-** CUPED and Sequential Testing are not yet available on Switchback tests. Switchback experiment pulse results showing bucket-level metrics # Switchback V2 Source: https://docs.statsig.com/experiments/types/switchback-v2 Run switchback experiments in Statsig to alternate treatment exposure across time windows for marketplace, ranking, and other globally shared systems. ## What is a Switchback Experiment? A **switchback experiment** tests two versions of a system by **alternating them over time**. This methodology is ideal when it’s not possible to isolate user experiences between treatment and control. Sometimes you can’t run a traditional AB test because you can’t cleanly isolate experiences between treatment and control. For example, on a rideshare platform, offering lower prices to a treatment group might increase demand for cars and indirectly affect the experience for control riders. A switchback experiment solves this by measuring impact over time while alternating between experiences. ## Overview At a high level Switchback Experiments work by 1. Define a cluster and it’s schedule: A cluster represents a grouping of users who will switch between experiences on the same cadence. *Example:* All users in New York and Chicago follow the schedule * 9AM - 10AM: Control * 10AM - 11AM: Treatment * 11AM - 12PM: Control * 12PM - 1PM: Treatment 2. Aggregate data by each time bucket: The bucket is each individual window of time where a user’s experience is kept consistent. Each bucket will be used as a data point for analysis *Example:* The switchback experiment from above has 4 buckets, 2 for control and 2 in treatment * Control * Bucket 1: 9AM -10AM * Bucket 3: 11AM - 12PM * Treatment * Bucket 2: 10AM -11AM * Bucket 4: 12PM - 1PM 3. Run regression: Account for the impact of things like time-of-day, day-of-week, or cluster attributes on the measured difference between treatment and control 4. Compare results: The final output resembles a traditional AB test with things like estimated lift and confidence intervals At a high level, switchback experiments work as follows: 1. **Define clusters and their schedule:** A **cluster** is a group of users that switch between experiences on the same cadence. *Example:* All users in **New York** and **Chicago** follow this schedule: * **9:00–10:00 AM:** Control * **10:00–11:00 AM:** Treatment * **11:00 AM–12:00 PM:** Control * **12:00–1:00 PM:** Treatment 2. **Aggregate data by time buckets:** A bucket represents a single window of time during which the user experience remains constant. Each bucket is treated as one data point in the analysis. *Example:* In the schedule above, the experiment produces **four buckets**—two for control and two for treatment. * **Control** * Bucket 1: 9:00–10:00 AM * Bucket 3: 11:00 AM–12:00 PM * **Treatment** * Bucket 2: 10:00–11:00 AM * Bucket 4: 12:00–1:00 PM 3. **Run regression analysis:** Statistical models account for factors such as time of day, day of week, or cluster attributes when estimating the difference between treatment and control. 4. **Compare results:** The final output resembles a traditional A/B test, including metrics such as estimated lift and confidence intervals. # Setting up an Switchback experiment Defining the hypothesis, metrics, groups, targeting, and parameters follows the same general workflow as a traditional A/B test. What makes switchback experiments different are three additional configurations: **clusters**, **scheduling**, and **analysis configuration**. ## Defining Cluster(s) Clusters are groups of users who follow the same experience cadence. In traditional A/B tests, the selected ID Type acts as both the randomization unit and the unit at which metrics are calculated. In a switchback experiment, however, the ID Type defines the unit for metric calculation, while clusters determine which experience a user receives over time. In Statsig, there are three ways to define a cluster. | Method | Description | Inputs | | :----- | :------------------------------------------------------------------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Single | Single Cluster where all users eligible for the experiment follows the same cadence | **Start With:** defines which experience (control or treatment) starts the switchback | | Auto | Provides a two-cluster configuration where users are automatically assigned to each cluster based on the specified inputs. | **Cluster ID Type:** Select a custom ID from the Exposure User Object that Statsig will use to split users into clusters. For example, if server\_id is present on the user object, Statsig will randomly assign server\_id values to each cluster, and users will be clustered based on their server\_id. | | Manual | Two-cluster configuration where users are manually assigned to each cluster. | **Cluster Field:** Select a field from the Exposure User Object that will be used to assign users to clusters. For example, if Country is selected as the Cluster Field, you can assign specific countries to either Cluster 1 or Cluster 2. Users will then be placed into clusters based on the value of that field (e.g., their country). | Fields used as Cluster ID Type (Auto) and Cluster Field (Manual) can be used as covariates in the regression analysis and will be available to break down metric data in the results section. ## Defining Scheduling Once the clusters are configured, you can define the schedule for experiences within each cluster. Screenshot2026 02 25at9 45 50AM ### Inputs **Window Size / Unit:** The length of each window during which a user’s experience remains constant. **Experiment Start Date / Time / Timezone:** The date and time when the switchback experiment begins. All clusters start simultaneously. The experiment cannot be started if the selected start date or time is in the past when the experiment is started. **Target Duration:** The intended length of time the experiment should run. By default, the experiment will **not automatically stop** when this duration is reached—users will continue to receive the experiment’s switched experiences according to the configured schedule. If **“Stop Experiment at target duration”** is enabled, the experiment will automatically stop at the end of the specified duration. At that point, users will be served the default experiment value configured in code. ## Define Analysis Configuration In Statsig, you can configure how exposures and metrics are handled during the transition periods between switchback windows. For example, if a rideshare marketplace switches from Control to Treatment at 9 AM, the system may still experience lingering effects from the Control period—such as drivers already on active trips or riders remaining in the queue from earlier periods. In these cases, you may want to exclude exposures and metric data recorded shortly after the switch, since they may still reflect the previous experience. In Statsig, this can be done by configuring **Burn-in** and **Burn-out** periods in the **Analysis Configuration** section. Screenshot2026 02 25at2 26 44PM ### Inputs **Burn-in Period:** The amount of time at the beginning of each window that is excluded from analysis. **Burn-out Period:** The amount of time at the end of each window that is excluded from analysis. **Metric Calculation:** Determines how metric events are attributed to an exposure. The following options define how metrics are aggregated within each switchback window. Screenshot2026 02 25at2 44 24PM * **Period from first exposure:** Aggregates metric data for a specified period of time after the user’s first exposure. * **Entire window:** Aggregates metric data across the full switchback window. * **Period between burn-in and burn-out:** Aggregates metric data only within the portion of the window between the burn-in and burn-out periods. **Exposure Calculation:** Defines how exposures logged during switchback windows are handled in the analysis. The following options are available: Screenshot2026 02 25at2 50 36PM * **Include exposures in burn periods:** Considers all exposures recorded during the switchback window, including those that occur within the burn-in and burn-out periods. * **Exclude exposures in burn periods:** Considers only exposures recorded between the burn-in and burn-out periods. **\[Coming soon] Specify Pre-computed User Dimensions:** Configure user dimensions that can be used to break down experiment results in the results section. These fields are selected from the exposure user properties or entity properties. # FAQ Source: https://docs.statsig.com/faq Answers to common questions about Statsig SDKs, experiments, feature gates, dynamic configs, billing, data residency, and day-to-day platform usage. ## SDKs and APIs ### How does bucketing in the Statsig SDKs work? See [How Evaluation Works](/sdks/how-evaluation-works). *** ### Can I add a layer to a running experiment? No. Layers are fixed once an experiment starts to preserve the integrity of results. We may support editing layers in the future. *** ### Can I rename an existing experiment or feature gate? Yes, you can rename existing experiments and feature gates after they are created. Note that when you are renaming an entity (e.g., feature gate, experiment, layer), you are only renaming its display name. The underlying ID of the entity (which are referenced your code) remains unchanged as they are immutable to prevent breaking existing implementations. *** ### Why define parameters instead of just reading the experiment group? Parameters let you iterate quickly without code changes and support richer experiment setups. For example: ```js theme={null} // Group-based approach — requires code changes for each variant if (otherEngine.getExperiment('button_color_test').getGroup() === 'Control') { color = 'BLACK'; } else if (otherEngine.getExperiment('button_color_test').getGroup() === 'Blue') { color = 'BLUE'; } // Statsig parameter approach — variants can be changed from the console const color = statsig.getExperiment('button_color_test').getString('button_color', 'BLACK'); ``` *** ### Why aren't exposures or custom events showing up? In short-lived environments (scripts, edge workers), the process may exit before events flush. Call `statsig.flush()` before shutdown. Details live in the [Node.js server SDK docs](/server/nodejsServerSDK#flushing-events). *** ### My SDK language isn't listed. Can I still use Statsig? Likely yes. Let us know in the Statsig Slack community and we'll discuss options. *** ### How do I retrieve all exposures for a user? The [Users tab](https://console.statsig.com/users) shows historical exposures. For hypothetical assignments (e.g., to bootstrap clients) you can call `getClientInitializeResponse` on the server. Pass `{ hash: 'none' }` if you need readable keys: ```js theme={null} const assignments = statsig.getClientInitializeResponse(user, 'client-key', { hash: 'none' }); ``` *** ### What happens if I check a config that doesn't exist? The SDK returns defaults—`false` for gates and the supplied fallback for experiments/layers. You'll see the evaluation reason `Unrecognized`; learn more in [SDK debugging](/sdks/debugging#evaluation-reason). This applies to deleted, archived, or unseen configs (e.g., filtered by [target apps](/sdks/target-apps)). *** ## Feature Gates ### If I change the rollout percentage, do existing users keep their result? Yes. Increasing the pass percentage (e.g., 10% → 20%) keeps the original 10% and adds new traffic until you reach the new percentage. Decreasing it removes the newest slice first. To reshuffle everyone, you must resalt the gate. (Experiments behave differently—use targeting gates for deterministic control.) *** ## Statistics ### What statistical tests does Statsig use? We use a two-sample Z-test for most experiments and [Welch's t-test](/experiments/statistical-methods/p-value#welchs-t-test) when sample sizes are small or variances differ. *** ### How does Statsig handle low sample sizes? We fall back to [Welch's t-test](/experiments/statistical-methods/p-value#welchs-t-test) and offer CUPED/winsorization to boost power. *** ### When should I use one-sided vs. two-sided tests? Use one-sided when you only care about movement in a single direction; it increases power but hides movement in the opposite direction. *** ## Experimentation ### How do I get started with an A/B test? If the feature isn't live yet, wrap it in a [feature gate](/guides/first-feature) and roll out. If it's already in production, create an [experiment](/guides/abn-tests). Results appear in the Pulse view. *** ### Can I target experiments to specific users (e.g., iOS only)? Yes. Use a feature gate with targeting rules as a pre-filter for your experiment. Targeting iOS users in an experiment setup *** ## Billing ### What counts as a [billable event](https://statsig.com/pricing#faq)? Any gate/experiment check or event logged via the SDK or APIs. Pre-computed metrics and custom metrics based on existing data also count. *** ### How do I monitor and manage billable volume? 1. Export usage from the **Usage and Billing** tab. 2. Pivot by event to identify heavy hitters. 3. Admins receive alerts at 50/75/100% of contract. Usage dashboard showing event volume *** ### How many projects can I create with a Pro subscription? Each Statsig Pro plan unlocks one project with pro features and 5M events. Additional projects require their own upgrade. Enterprise plans can cover multiple projects—[contact us](https://statsig.com/contact/demo) to discuss. *** ## Platform Usability ### When should I create a new project? Projects have distinct boundaries. If you're using the same userIDs and metrics across surfaces, apps or environments, put them in the same project. Create a new project when you're managing a separate product with unique user IDs and metrics. For example, if you have a marketing website (anonymous users) and a product (signed-in users), you may want to separate them. However, if you want to track success across both you should manage them in the same project. (e.g. from user signup on the marketing website to user engagement within the product) Some reasons to NOT create a new project * to segregate by environment. Statsig has rich support for environments - you can even customize these. You can turn features or experiments on and off by environment. * to segregate by platform. If you have an iOS app and Web app - it's helpful to have both collect data in the same project and capture metadata on platform. This lets you look at data by platform, but also understand if you've increased the overall metric - or just cannibalized users (pushed the same users from platform to the other platform). *** ### How can I monitor the health of Statsig and get support? You can check the live operational status of Statsig at status.statsig.com. To report an issue or receive help, reach out directly in our Slack Community. *** # Best practices for Feature Gates Source: https://docs.statsig.com/feature-flags/best-practices Best practices for implementing Statsig feature gates, including naming, code structure, team collaboration, governance, and managing flag lifecycles at scale. Statsig classifies the best practices for using feature gates into four categories: implementation, development, collaboration, and governance. ## Implementation 1. **Manage for ease and maintainability** – Use simple if/else gate checks for short lived gates that you can quickly clean up after the release. Use configuration parameters for longer lived gates to avoid nesting multiple gates and growing complexity in your code over time. 2. **Select the gating decision point** – Implement client-side feature gates to select users where most context is available and/or when the feature is primarily developed in the presentation layer (e.g. user registration flow). Implement server-side feature gates when most context is available with the application server and/or when the feature is primarily related to backend system behavior (e.g. new cache layer for better performance). Localizing these gating decisions within the service whose behavior is being changed is the best option in these cases. 3. **Focus on one feature** – Using one feature gate to control multiple features at a time can be confusing and can make troubleshooting issues difficult. If there are multiple parts of a feature that must work together, create a master feature gate to control child feature gates for these individual parts. ## Development 1. **Speed up development** – Shipping new functionality behind a feature gate ensures that the code path is not activated until you're ready to integrate with your dependencies. This enables you to ship service components faster without being blocked by any dependencies. 2. **Always be testing** - Use feature gates to ensure that in-development features remain inactive in production while continuing to test new functionality in staging or pre-production environments. 3. **Progressive delivery** – Ship code for in-development features early and often. Shipping code as part of the main branch that can be deployed to production at any time avoids painful merging of long-lived branches at a later point. 4. **Validate functionality with trusted users** – Use features gates to only expose new functionality to trusted and friendly users such as teammates, company employees, and beta customers before launching publicly. Verify that the new functionality is working as expected. 5. **Set up a phased canary release** – Use feature gates to progressively expose new functionality to a small percentage of users, validate user experience, and monitor production system health before launching broadly. In the initial stages, scale up slowly. We recommend the following rollout strategy, which increases by the same multiplier each time, increasing the bucket of new users exposed to the experiment by a larger margin each time: 0% -> 2% -> 10% -> 50% -> 100% 6. **Validate user and system impact** - Compare key user and system metrics against the default behavior. Common user metrics to test include daily and weekly active users, weekly retention, and conversion rates for key user actions. Common system metrics include error response rates, application crash rates, p50/p90/p99 request-response latency, and CPU utilization. You can create a metric for any key user or system behavior by logging the event that best proxies that behavior with Statsig. 7. **Ramp up or roll back** – Identify issues, and negative impact on users early. Use metric-based evidence to decide whether to release the feature more broadly. If you decide to launch, ramp up deployment (e.g. 10% -> 50% -> 100%). 8. **Clean-up after releases** – Once the release is complete, remove unnecessary gates from code. Once they are no longer checked, you are free to turn them off/delete them. ## Collaboration 1. Scoped Access - Invite verified teammates to create, review, and approve feature gates for a specific project. 2. Role-based Access Control (RBAC) - Ensure that only project members with the appropriate privileges can create and edit feature gate configurations with role-based access control. ## Governance 1. **Audit and record** - Set up audit logs for any changes that your team makes to any feature gates. 2. **Monitor and automate** - Set up automated health monitoring and alerts to improve visibility of your feature gates, reduce response times for any issues, and create automated workflows for common response patterns. # Feature Gate rule criteria Source: https://docs.statsig.com/feature-flags/conditions Understand how Statsig evaluates feature gate rules from top to bottom and see the full list of supported targeting conditions and operators. Statsig feature gates contain a list of rules that are evaluated in order from top to bottom. The page describes in more detail how these rules are evaluated and lists all currently supported conditions. ## Rule Evaluation The rules that you create are evaluated in the order they're listed. For each rule, the **criteria** or **conditions** determine which users *qualify* for the Pass/Fail treatments. The Pass percentage further determines the percentage of *qualifying* users that will be exposed to the new feature. The remaining *qualifying* users will see the feature disabled. Suppose you set up your rules as shown below, the following flow chart illustrates how Statsig evaluates these rules. Example Rules Gate Feature gate rules evaluation flowchart Note that as soon as a user qualifies based on the condition in a given rule, Statsig doesn't evaluate subsequent rules for this user. Statsig then picks the qualifying user to be in either the Pass or Fail group of that rule. Also note that in the example, the third rule for **Remaining Folks** captures all users who don't qualify for previous two rules. If we were to remove this third rule, then only a subset of your users (users in pools 1 and 2) would qualify for this feature gate and for further analysis, not your total user base. ### Client vs Server SDKs All of the following conditions work on both client and server SDKs. Client SDKs handle these conditions a bit more automatically for you - if you do not provide a userID, client SDKs rely on an auto-generated "stable identifier" which is persisted to local storage. In addition, if you do not automatically set an IP or User Agent (UA), the client SDK will infer these attributes from the request IP and UA. Similarly, on mobile, the client SDK will automatically pass your app version and locale to the server so conditions using these attributes can be evaluated without having to set them explicitly. ### Stability Evaluations at a given percentage are *stable* with respect to the unitID. For example, if the gate/config/experiment/layer has a unit type of "userID", and userID = 4 passes a condition at a 50% rollout, they will always pass at that 50% rollout. The same applies for `customIDs`, if the unit type of the entity is that `customID`. Want to reset that stability? See "Resalting" below. ### Resalting Gate evaluations are stable for a given gate, percentage rollout, and user ID. This is made possible by the salt associated with a feature gate. If you want to reset a gate, triggering a reshuffle of users, you can "resalt" a gate from the dropdown menu in the top right of the feature gate details page. Resalt UI ### Partial Rollouts While 0% or 100% rollouts for gates are simply "on for users matching this rule"/"off for users matching this rule", each rule allows you to specify a percentage of qualifying users who should pass (see the new feature). If you want to get [Pulse Results](/pulse/read-pulse) (metric movements caused by a feature), simply specifying a number between 0% and 100% will create a random allocation of users in Pass/Fail or "test"/"control" groups for a simple A/B test. You can use this to validate that a new feature does not regress existing metrics as you roll it out to everyone. Statsig suggests a 2% -> 10% -> 50% -> 100% roll out strategy. Each progressive roll out will generate its own Pulse Results as shown below. Metric Lifts ### User Object Fields Evaluation uses the set of properties defined in the [StatsigUser object](/concepts/user). There are a set of reserved top-level fields, but these keywords are reserved and recognized in the `custom` and `privateAttributes` maps as well. For example, if you set `user.country`, `user.custom.country` OR `user.privateAttributes.country`, it will be used to evaluated a country condition in any of those places (and in that order! top level > custom > privateAttributes), case insensitively. So if user.country is not defined, but user.custom.COUNTRY is, that will be used to evaluate a country condition. ## Supported Conditions ### User ID Usage: Simple lists of User IDs to explicitly target or exclude from a gate. Supported Operators: `Any of, none of` Example usage: Add yourself (or a small group like your team) when you just start building a new feature. Or exclude your designer until it's ready for their eyes. User ID condition example ### Email Usage: Target based on the email of the user Supported Operators: `any of, none of, contains any of, contains none of` Example: Show new feature to people in the Statsig company with an authenticated @statsig.com email address Email condition example ### Everyone Usage: Percentage rollout on the remainder of users that reach this condition. Think of it as "everybody else" - there could be a dozen other rules/conditions above it, but for everyone else, what percentage do you want to pass? Supported Operators: `None. Percentage based only.` Example usage: 50/50 rollout to A/B test a new feature. Or 0% to hide the feature for all people not matching a set of rules. Or 100% to show the feature to the remaining users who did not meet a condition above. Everyone 50/50 condition example ### App Version Usage: User's on a particular version of your app/website will pass. Particularly useful for mobile app development, where a feature may not be fully ready (or maybe be broken) in a particular app version. Supported Operators: `>=, >, <, <=, any of, none of` Example: Turn off a feature for all users on app versions 3.0.0 through 3.1.0 as it was broken. Disable broken versions example ### Browser Version Usage: A particular version of a browser, parsed from the user agent. Should likely be combined with the browser name condition. Supported Operators: `>=, >, <, <=, any of, none of` Example: Turn off a feature for old versions of chrome which don't support a certain API Browser Name and Version Example ### Browser Name Usage: A particular browser, parsed from the user agent: ('Chrome', 'Chrome Mobile', 'Edge', 'Edge Mobile', 'IE', 'IE Mobile', 'Opera', 'Opera Mobile', 'Firefox', 'Firefox Mobile', 'Mobile Safari', 'Safari'). Often combined with the Browser Version condition. Supported Operators: `>=, >, <, <=, any of, none of` Example: Turn off a feature for old versions of chrome which don't support a certain API To test: The Browser Name is inferred from the `userAgent`, but if you need to set it explicitly, you can set `browserName` in the user object. Browser Name and Version Example ### OS Version Usage: A particular os version the user is on, parsed from the user agent. Should likely be combined with the Operating System condition. Supported Operators: `>=, >, <, <=, any of, none of` Example: Turn off a feature for versions of macOS which don't support a certain API OS name and version example ### Operating System Usage: A particular operating system, parsed from the user agent: ('Android', 'iOS', 'Linux', 'Mac OS X', 'Windows'). Often combined with the OS Version condition. Supported Operators: `any of, none of` Example: Turn off a feature for versions of macOS which don't support a certain API To test: The OS is inferred from the `userAgent`, but if you need to set it explicitly, you can set `deviceOS` in the user object. OS name and version example ### Device Model Usage: The device model of the mobile device the user is on. Supported Operators: `any of, none of, is null, is not null, contains any of, contains none of, regex` Example: Turn off a feature for older device models that your app does not support. To test: The Device Model is automatically inferred by the SDK, but if you need to set it explicitly, you can set `deviceModel` in the user object. Device Model Example ### Country Usage: A 2 letter country code, either passed as the user's country or determined by the IP address. Supported Operators: `any of, none of` Example: Show a cookie consent banner for users accessing your service from the EU. Country example ### IP address Usage: An IP address string Supported Operators: `any of, none of` Example: Show a different about us page on [www.statsig.com](http://www.statsig.com) to people in a certain IP address range Matching IP Addresses example ### Passes Target gate Usage: The condition passes if the referenced gate passes for the given user Supported Operators: `gate_id` Example: Only show feature X to people who also see feature Y: Only show a toggle to turn off ranking to people who also pass the new ranking algorithm gate. Passes Target Gate ### Fails Target gate Usage: Inverse of passes target gate: the condition passes if the referenced gate returns false for the given user Supported Operators: `gate_id` Example: Only show a new UI element to people who are not using the redesigned UI. Fails Target Gate ### User in Segment Usage: The condition passes if the user passes the rules defining the referenced segment. See the [segments guide](/guides/using-environments) for more information on segments. Supported Operators: `segment_id` Example: Only show a new UI element to people who are not using the redesigned UI. In Segment ### User not in Segment Usage: The condition passes if the user fails the rules defining the referenced segment. See the [segments guide](/guides/using-environments) for more information on segments. Supported Operators: `segment_id` Example: Only show a feature to people who are not in the premium/paid tier. Not In Segment ### Environment Tier Usage: The condition passes if the evaluation is happening in the given environment tier (development/staging/production). See the [environments guide](/guides/using-environments) for more information on environments. Supported Operators: `development/staging/production/(other)` Example: Only show a feature on development builds Development Environment Condition ### Custom Field Usage: Specify the key in the custom object to fetch the value use on the left hand side of a comparison Supported Operators: * string: `any of, none of, contains any of, contains none of` * number: `any of, none of, less than, greater than` * version: `any of, none of, less than, greater than, less than or equal to, greater than or equal to` * date: `before, after` Example: Only show a feature to user's who have turned on dark mode, as marked by the custom object having "darkmode": true. Custom Field ### Unit ID Usage: Select custom ids to explicitly target or exclude from a gate. Supported Operators: `any of, none of, is null, is not null, contains any of, contains none of, regex` Example: Add yourself (or a small group like your team) when you just start building a new feature. Unit ID example ### Time Usage: Evaluate a gate relative to the current time Supported Operators: `after time, before time` Value: time (displayed and input on Console in your browser's local time, but converted to a unix timestamp for evaluation) Example: Show the labor day sale banner on labor day Time condition ### Private Attributes Usage: Any user field conditions, or custom field conditions, can be used via a private attribute. If you are targeting an email, but don't want it to be logged, create an email condition, but put "email": "[xyz@email.com](mailto:xyz@email.com)" in the privateAttributes dictionary. For custom fields, create a custom field condition, but put the key/value pair in privateAttributes instead. Remember that privateAttributes are used for evaluation only, and are not stored or kept on logEvents. Supported Operators: dependent on the condition Example: Only show a feature to 20 somethings, as marked by the privateAttributes object having "age": "20-29". Private attributes ## FAQs Once a user is exposed, they will be included in the analysis going forward. They saw the new feature and were affected. If the feature gate rules are modified or the user's attributes change in a way that the user no longer qualifies, they will stop receiving the new feature. However, they will continue to be counted for analysis. Once you roll out the feature, all users will see the new feature; alternatively, if you turn off the feature gate, all users will see the control (feature disabled). In either case (roll out or turn off), Statsig performs no further analysis. When you add user IDs in the **Pass** or **Fail** lists of your feature gate, these users will see the appropriate treatment but will not be included in the analysis. # Create a Feature Gate Source: https://docs.statsig.com/feature-flags/create Step-by-step guide to creating a feature gate in the Statsig console, adding targeting rules, implementing the gate in code, and reviewing examples. ## In the Statsig console ### Create a new Feature Gate 1. Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com). 2. On the left-hand navigation panel, under **Feature Management**, select **Feature Gates**. 3. Click on the **Create** button. 4. Enter the name and the description of the Feature Gate you want to create. It's best to name your gate based on what you are rolling out, such as "Zippy Home Page" for a new homepage. Feature gate creation form 5. Click **Create** to complete creating your Feature Gate. 6. At this point, you've successfully created a new Feature Gate without any evaluation rules or conditions set up yet. Newly created feature gate without rules ### Add a rule to your Feature Gate By default, a Feature Gate will return `false` when there are no rules configured to target the gate to a set of users. In other words, all users are "gated" by default from seeing the feature until you've set rules for who gets to "pass". Next, we'll walk through steps for adding evaluation rules or conditions for a Feature Gate. 1. In Statsig console, under **Feature Management**, select **Feature Gates**. 2. Select the feature gate where you want to add a targeting rule 3. Click the **+ Add New Rule** button. Add rule button interface 4. Name your rule with something descriptive that other teammates will understand, such as "Mobile Users Only". 5. Configure your Feature Gate with the following options: * **Environment(s)** - The [staging environment(s)](/guides/using-environments) you want your gate to apply to * [**Criteria**](/feature-flags/conditions) - Specific evaluation conditions for the rule. Read more details about criteria and condition evaluation [here](/feature-flags/conditions). * **Split %** - The percentage of users who meet the criteria that you want to pass or fail the gate check. The Fail % is automatically calculated based on the Pass % you set * [**Overrides**](/feature-flags/overrides) - A list of users you want to always bypass your gate (i.e., a "whitelist") 6. Changes to Feature Gates and targeting rules are NOT auto-saved. Make sure to save your changes by clicking the "Save" button at the bottom right when you are ready for your changes to take effect. ## In your code So far in this doc, we've walked through the set up of Feature Gates in the web console. For these gates to *actually* impact the behavior of your application and your users' experiences, you have to update your product code. It's sort of like setting policies; it's not enough to just list the new rules you'd like to enforce. You need to also set up the infrastructure to make sure those new policies actually take effect in realtime, and that's where the Statsig SDK comes in. ### Initialize the Statsig SDK If you haven't already initialized the Statsig SDK, follow the [Installation Steps](/client/React#installation) in the language of your choice. ### Check a Feature Gate Use the `checkGate` function in the Statsig SDK in the language of your choice. You can find a code snippet for any particular gate by clicking on the code snippet button on that gates page in the statsig console and selecting the correct SDK you want to use. Gate code snippet button Here's an example in [React](/client/React#basics-check-gate): ```tsx theme={null} const { client } = useStatsigClient(); return (
Gate is {client.checkGate("example_gate") ? "passing" : "failing"}.
); ``` In the code snippet example above, the Statsig SDK is checking from a client app whether you've set any rules named "Example Gate". It will render the text "Gate is passing" for users who pass your gate based on conditions you set, and "Gate is failing" for users who fail the gate conditions. Statsig offers over 20 client and server-side SDKs. Check out the full list of [SDKs](/sdks/quickstart#all-sdks) to find the one that best fits your needs. ## Common Feature Gate setups ### Kill switches You can set up a simple "kill switch" by first setting an `Everyone` criteria's Pass percentage to 100%. Then, if you need to completely disable the feature for all users at some point after code deployment, you can set the Pass percentage to 0%, effectively killing the feature for all users. ### OS-based flags You can target users based on common application properties such as the operating system that the application is running on. For example, to target iOS users only, you can create a rule with the "Operating System" criteria, the "Any of" operator, and then listing "iOS" in the text field. ### Internal employees only You can target users based on known user attributes. For example, you can use the user's **Email** attribute and the **Contains any of** operator, then enter the email domain of your company to target only internal employees. ### Defined user segments You can also target users in a defined user [Segment](/segments). ### Parent/child flags You can target users based on their eligibility of other Feature Gates, enabling powerful chaining, hierarchy, and dependencies of flags. [Flag lifecycle management](/feature-flags/feature-flags-lifecycle) makes it easy to manage, deprecate, and maintain dependent flags. # Managing Feature Gate lifecycles Source: https://docs.statsig.com/feature-flags/feature-flags-lifecycle Learn how to manage feature gates through different phases of their lifecycle - from testing to full rollout to cleanup and archival. A feature can go through different phases throughout its lifecycle - maybe it's still being tested out by a few users, or only recently fully rolled out to the world, or maybe it's been tried and true and you no longer need the feature behind a toggle. Whatever phase the feature may be in, its gate should clearly reflect that, for a few important reasons - * **Prevent incidents**: prevent a scenario where old code for a deprecated feature is accidentally touched or repurposed, with real business consequences like how [Knight Capital lost half-a-billion dollars](https://www.statsig.com/blog/lose-half-a-billion-dollars-with-bad-feature-flags-knight-capital). * **Maintain healthy codebase**: messy code base with dead references to flags mean that your team has more volume of code to navigate on a daily basis, and it can even slow down new developers onboarding. * **Reduce mental load**: mental tracking of all your features is no longer necessary because you will be able to see easily what next steps you need to take for the product (e.g. launch or kill a feature), as well as not having to worry about old features that are no longer relevant. ## Managing Feature Gate lifecycles Statsig makes it easy for your feature gates to reflect the phase your feature is in by using **status**. A gate can be in one of four statuses: Feature gate status options interface | Status | What it represents | Implication | | ----------- | ----------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | In Progress | this feature is in the process of being rolled out and tested. | N/A; it's the default status when you create a gate | | Launched | this feature has been rolled out to everyone | This gate will always return **default value = TRUE**, and will stop generating billable exposure events; you'll stop incurring costs. The gate reference is likely safe to be cleaned up in the codebase | | Disabled | this feature has been rolled back from everyone | This gate will always return **default value = FALSE**, and will stop generating billable exposure events; you'll stop incurring costs. The gate reference is likely safe to be cleaned up in the codebase | | Archived | this feature is no longer referenced in code or checked; history on the gate is preserved | This gate has been receiving 0 checks for the last 7 days, and no checks will be sent this gate anymore | ## When to update Feature Gate lifecycles There are 3 points throughout the gate's lifecycle when you'd want to take action, either on Console or in your codebase: Feature gate lifecycle workflow diagram **1) The gate has been fully rolled out or rolled back, and you're ready to skip rule evaluation and assign default value (stop incurring costs for your gates)** * Go to the feature gate page and click on "…" menu on the upper right corner to select **Launch or Disable**. It will open up the following window - * **In Progress → Launch**: when the gate has been rolled out to 100% a while ago (we recommend >30 days), and you feel comfortable with the gate always returning TRUE * **In Progress → Disable**: when the gate has been rolled back to 0% a while go (we recommend >30 days), and you feel comfortable with the gate always returning FALSE * To find *all* gates that are good candidates to be **Launched** or **Disabled** (i.e. have been rolled out to 100% or rolled back to 0% more than 30 days ago): * Go to Feature Gates catalog * Click on filter icon: * Status = In Progress * Pass Rate = 100% AND 0% * In the search bar: "Modified: ` Feature gates catalog filter interface **2) You're ready to clean up the gate reference from your codebase** Confirm that the gate has been set to either **Launched** or **Disabled** (i.e. returning a default value) for a while (we recommend >60 days) so you don't unintentionally break any rule evaluation and you've had enough time to ensure no negative impact on your metrics before you clean up the gate reference. Once confirmed, * Go to your codebase and * for **Launched** gates: remove the gate reference (but leave the code related to the feature as a permanent fixture to the codebase) * for **Disabled** gates: remove the gate reference + all code related to the feature Please confirm that these gates are not included in any active Holdouts before removing reference * To find *all* gates that are good candidates to be removed from your codebase (i.e. have been **Launched** or **Disabled** more than 60 days ago) * Go to Feature Gates catalog * Click on filter icon: Status = **Launched** AND **Disabled** * In search bar: "Modified: ` Feature gates catalog with launched and disabled filter **3) After you've cleaned up the gate reference from your codebase** * **Launched or Disabled → Archived:** you'll want to update this status to mark that the gate has been removed from your codebase, so that it will be filtered out from the list of candidate gates to be cleaned up for the future (as part of step #2) * Go to the feature gate page and click on "…" menu on the upper right corner to select "Archive". * To find *all* gates that should be marked as **Archived**: * Go to Feature Gates catalog * Click the filter icon: * Status = **Launched** or **Disabled** * Checks = 0 checks in last 7 days or 30 days, depending on your comfort level Feature gates catalog with zero checks filter ## Feature Gate lifecycles FAQs We recommend having a quarterly "feature gate cleanup party", where the team blocks out a chunk of time to identify all gates that need to be cleaned up (step #2) and remove the references from their codebase. One person can then follow up after 7 days to make sure all those gates are now receiving 0 checks on Statsig and can mark them as "Archived". Overtime, your team should see more **Archived** gates than **Launched/Disabled/In Progress** gates. Consistent with any other changes to a gate, anyone will be able to make a change, but it will require the same review process for the change to be approved. Yes, once the feature gate is **Launched** or **Disabled**, you will see a banner with an option to re-enable rule evaluation. In that case, Statsig will return false. Unarchive the gate and make it permanent if you will need to reference it indefinitely (i.e. from an older version of a mobile app that still has users). We recommend that you use Delete only for mistakes. Deletion removes the gate and its history from Statsig, and having your Feature Gate Catalog retain history of your gates will help you see valuable information like velocity of your team's feature releases, # of launches decisions made, etc. Archival of a gate implies that any reference to the gate has been completely and permanently removed from your code. Therefore, as best practice, we recommend that you clone an archived gate, essentially creating a new gate with the same rules, instead of reusing a previously archived gate. # Measuring multiple rollout stages Source: https://docs.statsig.com/feature-flags/multiple-rollout-stages Learn how Statsig handles continuous analysis and multi-stage feature flag rollouts for comprehensive measurement and experimentation. ## Continuous Analysis For gate rules that rollout with a pass percentage ≤ 50% and without any rollback, data collected at earlier stages and later stages in the rollout will be consolidated into one analysis. This allows for a more complete and thorough analysis. Valid continuous analysis rollouts include: 10% → 20% 10% → 20% → 50% Invalid continuous analysis rollouts include: 10% → 5% (rollback) 10% → 70% (exceeds 50%) When a rollout is no longer valid for continuous analysis, the new rollout step is analyzed separately from previous steps. ## Compatibility with Other Gate Features ### Balanced Gates Gate rules that have multiple rollout stages are also [balanced](/feature-flags/view-exposures#balanced-gates) using downsampling. In cases where pass percentage ≤ 50% we use the same hashing as the pass/fail and take an equal percentage of the other group. When pass percentage > 50% we use an orthogonal random hashing to sample the larger group at a rate of $\frac{1-p}{p}$ where p is the fraction of users in the larger group. In the case of a gradual rollout, this prevents bias based on enrollment time period. ### CUPED/CURE CUPED/CURE is available only when there have been no rollbacks AND when the rollout percentage has not exceeded 50% within a gate rule. Analysis is no longer continuous when there's a rollback or when 50% is exceeded, which means the pre-exposure data of a given unit is actually biased by earlier treatments. # Feature Gate overrides Source: https://docs.statsig.com/feature-flags/overrides During development, it can be useful to explicitly state which users should pass or fail a given feature gate. This is where overrides come in. ## Override results of a feature gate During development, it can be useful to explicitly state which users should pass or fail a given feature gate. This is where overrides come in. Overrides are based of user IDs and can be set to pass or fail for a given user ID. During evaluation of a gate, if the user ID is overridden, the overridden result will be returned immediately before any rules are evaluated. ### Adding an Override * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Feature Gates** * Select the feature gate where you want to add an Override * Click the **Add Override** button Feature gate page with Add Override button highlighted * Select either 'Pass List' or 'Fail List' from the tabs in the dialog Manage overrides dialog showing pass and fail lists * For users you want to pass the gate, add them to the 'Pass List' * For users you want to fail the gate, add them to the 'Fail List' A user can only exist on one of these lists at a time. * Once you have added the user IDs, hit **Save** ### Deleting an Override If you add an override but later decide it is no longer needed. You can remove it so the rules will be evaluated as normal. * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Feature Gates** * Select the feature gate where you want to add an Override * Click the **Edit Overrides** button Edit overrides button within feature gate settings * Hit the trashcan icon next to the user ID you wish to remove from a list. Override list with trash icon to remove user ID * Once you have updated the lists, hit **Save** ### Testing an Override Once your override has been added, you can test it in the "Test Gate" window by simply adding userID as a property of the user object. Test gate console showing user object with override result Users that pass will see "PASS (User ID Override)" Test gate output indicating PASS due to user ID override Users that fail will see "FAIL (User ID Override)" Test gate output indicating FAIL due to user ID override # Feature Flags Source: https://docs.statsig.com/feature-flags/overview Feature Gates, commonly known as feature flags, allow you to toggle the behavior of your product in real time without deploying new code. **Feature Gates**, commonly known as feature flags, allow you to toggle the behavior of your product in real time without deploying new code. Devs often use them to turn on certain features for a small percentage of the total user base. This allows for safer, gradual software releases because you can monitor the impact of system behavior. Feature Gates also enable you to limit application behavior for a specific set of users, like dogfooding [environments](/guides/using-environments). Feature Gates hero image ## When to use #### Use when you need to... * Schedule gradual feature rollouts to safely deploy new code * Set up dev staging environments before code hits production, like dogfooding * Have a just-in-case "kill switch" that lets you immediately turn off a particular code branch for users in production * Modify the user experience based on attributes like username, email, or other identifiers * Change app behavior based on context like device, browser type, version, and other environment attributes #### Not recommended if... * You need to return structured or multi-value data based on targeting rules and conditions—use a [Dynamic Config](/dynamic-config/overview) * You want to test complex hypotheses beyond simple A/B tests and launch impact—set up an [Experiment](/experiments/overview) instead ## How it works 1. First, [create a Feature Gate](/feature-flags/create) with [targeting rules](/feature-flags/conditions) in the Statsig console. 2. For the Feature Gate to actually impact users, you'll need to integrate the [Statsig SDK](/sdks/getting-started) into your product code. The SDK will query the gate value during runtime and return a true/false result based on user attributes, environment data, and other conditions you define. 3. You can [test a Feature Gate](/feature-flags/test-gate) to make sure it's behaving as expected before you actually roll it out. 4. For finer targeting control, you can also set up [Feature Gate overrides](/feature-flags/overrides), which are like "bypass lists" for the gate. 5. Once your Feature Gate is live, you can [view Feature Gate exposures](/feature-flags/view-exposures) in the Statsig console to monitor who is encountering your gate. 6. You can easily set up deprecation rules and clean up old flags by [managing Feature Gate lifecycles](/feature-flags/feature-flags-lifecycle). ## Key capabilities ### Scheduled Rollouts Gradually deploy a feature over time by setting up a Feature Gate as a [Scheduled Rollout](/feature-flags/scheduled-rollouts). ### Overrides and bypass lists Implement Feature Gates with [Overrides](/feature-flags/overrides) to allow a specific list of users to bypass your gate. ### Chained flag dependencies Chain Feature Gates together in parent-child or other dependent relationships so a top-level gate can enable or disable all its dependent flags in one go, perfect for global kill switches guarding sub-features. ### Built-in A/B tests You can run simple A/B tests without additional setup using [Pulse](/feature-flags/view-exposures). In practice, this means treating the users who see the new feature as the "treatment" group, and the users who are gated (and therefore do not see the new feature yet) as the "control". ### Feature Gate testing [Test your Feature Gates](/feature-flags/test-gate) with Statsig's built-in tools to check whether your Feature Gate is configured to target the right people. ## FAQs Dynamic Configs are key-value configs that let you return structured data (not just true/false) based on targeting rules. They're more for customizing behavior, tuning parameters, or supporting complex logic *beyond booleans*. Feature Gates are simpler on/off switches for gating access to a feature. Technically, you can set up a Dynamic Config as a Feature Gate. We have a full guide on [Choosing Feature Flags vs. Experiments](/guides/featureflags-or-experiments). (TL;DR—Use Feature Gates when you just want to measure the general impact of a feature rollout, and use Experiments when you have a more specific hypothesis or "test" in mind.) ## Related tutorials * [Set up dev environments with Feature Gates](/guides/using-environments) * [Customize dev environments with Feature Gates](/guides/testing) * [Choosing Feature Flags vs. Experiments](/guides/featureflags-or-experiments) * [Best Practices for Feature Gates](/feature-flags/best-practices) # Permanent and Stale Gates Source: https://docs.statsig.com/feature-flags/permanent-and-stale-gates Manage the lifecycle of Statsig feature gates with Types to identify temporary flags ready for cleanup and mark gates intended as permanent integrations. It is important for your codebase and team to bring feature gates to a final state (i.e. flags now permanently part of your codebase or completely removed) when they have served their purpose, as described [here](/feature-flags/feature-flags-lifecycle). On Statsig, you can use feature gate **Types** to easily keep track of your flags that might be ready to brought to their final state. ## Types In your feature gates catalog, you'll see different **Types** displayed in the Status column, as well as under the filter option - Feature gates catalog with status types filter * **Permanent Gates** (set by you) * **Permanent feature gates** are expected to live in your codebase for an extended period of time, beyond a feature release, usually for operations or infrastructure control. Common examples include user permissions (e.g. premium features based on subscription level) or circuit breakers/kill switches (e.g. terminating a connection to prevent negative customer impact) or even supporting legacy features in old app versions. * There are two ways to mark a gate as **Permanent**: * When creating the gate: the **Permanent** box in the gate creation flow Permanent checkbox in gate creation flow * After a gate has been created: click on the "..." menu and then "Mark Gate Permanent" Mark Gate Permanent menu option * Implications of marking a gate as **Permanent** * No change in the gate's behavior when called * Easy filtering on feature gates catalog * More caution in its management: while you are able to archive or delete a **permanent** feature gate, there will be a warning shown before proceeding * Statsig will forgo reminding you to clean up these gates and may display them differently in the console * You will be able to change the gate back to **Temporary** at any point. * All newly created feature gates are marked as **Temporary**, unless marked otherwise (i.e. Permanent). Therefore, Statsig will not display the phrase **Temporary** in the feature gates Catalog or within the individual gates page. * **Stale Gates** (set by Statsig) * **Stale feature gates** indicate to your team that these gates could be a good candidate for cleanup. Statsig automatically marks gates as stale based on the following definition (excludes Permanent and archived gates) * Gates created less than 30 days ago, modified in the last 30 days, or referenced by other gates/experiments/dynamic configs are never consider stale * Otherwise, any of the following conditions make a gate stale: * The gate has had 0 checks within last 30 days * The gate is still being checked but its earliest check was at least 30 days ago * Implications of gates being marked as **Stale** * No change in the gate's behavior when called * Easy filtering on Feature Gates catalog * Will include the gate in Statsig's nudges for cleanup (more below) * **Stale Reasons** are the reason why a gate has been set as stale. This information can be queried on the [Console API](/console-api/gates). * **None** No Stale Gates should have their reason set as None, this is exclusively for **Temporary** or **Permanent** gates. * **STALE\_PROBABLY\_DEAD\_CHECK** There have been no checks in the last 30 days. * **STALE\_PROBABLY\_LAUNCHED** The Gate is marked as launched or has an everyone rule passing 100% (rollout rate of 100%). * **STALE\_PROBABLY\_UNLAUNCHED** The Gate is marked as disabled or has an everyone rule passing 0% (rollout rate of 0%). * **STALE\_PROBABLY\_FORGOTTEN** This gate appears to have been only partially launched for some time. You might want to launch/disable it, or make it permanent if you need to keep it around. * **STALE\_NO\_RULES** The Gate has no set rules. * **STALE\_ALL\_TRUE** The Gate has been returning true every time it has been checked for the last 30 days (or number of days configured in project settings). It could probably be removed. * **STALE\_ALL\_FALSE** The Gate has been returning false every time it has been checked for the last 30 days (or number of days configured in project settings). It could probably be removed. * **STALE\_EMPTY\_CHECKS** The Gate has been returning empty (probably indicating an error) every time it has been checked for the last 30 days (or number of days configured in project settings). It could probably be removed or might need to be investigated. ## Nudges to clean up Stale gates Using the **Stale** type discussed above, Statsig provides both in-console and external nudges to remind you to cleanup (or make Permanent) your feature gates. * **In-console:** See the reminder at the top of the individual feature gate page Stale gate cleanup reminder notification * **Email/Slack:** We will proactively reach out to you with an email/ Slack reminder (if you've enabled the Slack integration) to clean up or mark permanent any stale gates you own. We will send this reminder nudge monthly until the gates are either cleaned up or marked as permanent. # Pre-Post Results Source: https://docs.statsig.com/feature-flags/pre-post-results Use Pre-Post Results in Statsig to analyze feature impact by comparing metrics before and after rollout when A/B testing isn't possible or practical. # Pre-Post Results **Cloud Only Feature** This feature is currently available only on Statsig Cloud. As we work on a WHN solution, please reach out to Statsig if you're interested in being an early customer. ## What are Pre-Post Results in Statsig? Pre-Post Results is an analysis mode for Feature Gates in Statsig that allows you to roughly estimate the impact of feature rollouts when a traditional A/B testing isn't possible. By comparing key metrics before and after a feature gate is rolled out from 0% to 100% of users, you can identify the directional impact of your features in production. Pre-Post Results Interface Pre-Post is particularly valuable for: * **Emergency rollouts** - Features that needed to be shipped immediately without time for slow rollout * **Infrastructure changes** - Backend improvements or technical features that affect all units/pods/users and cannot be partially rolled out * **Retroactive analysis** - Understanding the rough impact of features that were already rolled out without experiments * **Regulatory or ethical features** - Changes that can't be withheld from a control group ## Pre-Post Results are not Experiments Pre-Post analysis is an approximate way of measuring the change in a metric around a specific point in time, amongst a specific set of exposed units. However, it does not live up to the stringent requirements of running a proper AB test or feature gate partial rollout. Pre-Post analyses are better thought of as snapshot measurements around the time you launched your feature; however, because so many other things could be happening around the same time, there's no guarantee that your results are due to your feature launch. Remember, correlation does not equal causation. Experiments remain the gold standard for measuring your feature's impact, and Statsig still highly recommends running your launches as an Experiment or partial Feature Gate rollout when accuracy, validity, and extensibility are key. ## When does Statsig calculate Pre-Post Results? Pre-Post Results are available for targeting rules that meet specific rollout conditions: 1. The targeting rule started at 100% pass rate or was rolled out from 0% to 100% in a single step 2. The rollout happened in the last 30 days When you select a qualifying rule in the Metrics Impact tab, Statsig automatically switches to Pre-Post Results mode and displays a banner to indicate you're viewing Pre-Post analysis. ## How does Pre-Post Results work? Pre-Post Results uses a straightforward approach to estimate feature impact: 1. **Identify the participating units** - Find all users who were exposed to the feature after the 100% rollout 2. **Collect pre/post-rollout data** - Gather metric values for these users from the periods before and after the rule change 3. **Bucket metric data into discreet periods** - Statsig automatically groups metric data into buckets of a consistent duration 4. **Calculate the difference** - Compute the mean metric values for both pre and post periods, treating each bucket as a unique observation, then calculate the delta (difference) between them This method ensures we're comparing the same users before and after the feature rollout. ## Supported Metric Types | Metric type | Supported | | ------------------ | --------- | | Event Count | ✅ Yes | | Event Count Custom | ✅ Yes | | Event User | ✅ Yes | | Sum | ✅ Yes | | Mean | ✅ Yes | | Funnel | ❌ No | | Ratio | ❌ No | | Participation Rate | ❌ No | ## Best Practices When using Pre-Post Results, consider these guidelines: * Focus on metrics that are directly related to your feature's intended impact and have sufficient volume. The more directly a metric responds to the feature launch, the easier it will be to see a sudden spike/drop. * Remember that correlation doesn't equal causation. Consider other changes, seasonal effects, or external events that might influence your metrics during the analysis period. * Validate with domain knowledge. Use Pre-Post Results as one data point alongside qualitative feedback, user research, and business context to make informed decisions. * Aim for A/B results when possible. If you have the chance to partially roll out a feature to less than 100% of users, we highly recommended you do so. This way you can measure the metric impact for users seeing the feature vs. not seeing the feature and arrive at true causation. ## Limitations * **30-day window** - Only rollouts from the last 30 days are supported * **No control group** - Results show correlation, not definitive causation * **External factors** - Other changes during the analysis period can influence results * **Metric type restrictions** - Some advanced metric types are not yet supported # Create a Safeguard Source: https://docs.statsig.com/feature-flags/safeguards-create Learn how to create a Safeguard on one or multiple targeting rules within a Feature Gate to monitor regressions and automatically take action when alerts fire. You can create a Safeguard on one or multiple targeting rules within a Feature Gate. To create a Safeguard on a rule, follow the steps below: 1. Go to a Feature Gate's **Setup** tab 2. Pick a targeting rule for which you want to monitor regressions 3. Click the three-dot (...) menu and choose **'Safeguard Rule'** Create Safeguard on a rule 4. Choose the action to take when alerts fire: * **Rollback to 0%** - Assign Default value to all users * **Roll out to 100%** - Assign Pass value to all users * **Pause Rollout** - Stop scheduled rollout progression (only available with active Schedule Rollout policy) Select an action 5. (Optional) Set how long to monitor alerts for safeguarding the rule: | Alert type | Evaluation period | Evaluation start time | | ------------- | ----------------- | ----------------------------------------------- | | Topline alert | Choose your own | Starts when the safeguard is created | | Rollout alert | Fixed (90 days) | Starts whenever targeting rule's pass % changes | Set evaluation period **Recommended:** Ideally you want to monitor topline alerts for crashes, errors, latency, etc. *for a few days* after a Feature Gate rollout to make sure things are stable. We recommend starting with a 14-day evaluation period. 6. Select one or more alerts to monitor: * Rollout alerts - For feature-specific regression detection * Topline alerts - For system-wide health monitoring Add alerts for your Safeguard 7. Click **Save** # Manage a Safeguard Source: https://docs.statsig.com/feature-flags/safeguards-manage View and manage automated safeguards on Statsig feature flags to detect regressions, halt risky rollouts, and protect critical metrics during deployment. ## View a Safeguard To view an existing Safeguard, tap the blue pill on your Feature Gate's targeting rule. Here you can see how your Safeguard is currently defined and make any changes if you want. You can add/remove alerts, change the action, or adjust the evaluation period. View an existing Safeguard ## When a Safeguard triggers When a safeguard is triggered because of an alert: * The configured action executes automatically (rollback/pause/complete) * A banner appears on the targeting rule with action taken, timestamp, and diagnostic link * Further rule modifications are blocked until the alert is resolved * Notifications are sent per your alert configuration Safeguard is triggered on a rule # Safeguards Source: https://docs.statsig.com/feature-flags/safeguards-overview Ship with confidence by automatically monitoring critical metrics and intervening in Feature Gate rollouts when risk thresholds are exceeded Safeguards help you ship with confidence by continuously monitoring critical metrics and automatically intervening in your Feature Gate rollouts when risk thresholds are exceeded. This ensures faster recovery from issues, eliminates the need for constant manual checks, and protects your users from unintended impact. Safeguards is available on our Pro and Enterprise billing tiers. ## When to use Safeguards Use Safeguards when you want to: * Limit the impact of a feature on a critical business or performance metrics * Automate rollout progression to more users based on how your metrics are performing * Maintain system stability by automatically responding to API errors, latency spikes, or infrastructure issues ## How Safeguards work Safeguards work by listening to different types of alerts, such as Rollout Alerts and Topline Alerts, that you have created in your project. When any alert fires due to metric regressions, Safeguards automatically pause your rollout, rolls it back, or finish a rollout based on your settings. Pre-requisite You must create at least one Rollout Alert or Topline Alert before configuring Safeguard on a Feature Gate. See the [Alerts](/product-analytics/alerts-overview) documentation for setup instructions. ## Two types of Safeguards There are two different types of alerts a Safeguard can use to take an action on your Feature Gate: #### Rollout Alert Definition: Monitor the regression of metric delta between users who pass and fail your Feature Gate Use-case: When you want to ensure that your Feature Gate is not causing any negative drift on the users getting the new flag variation. Only works on partially rolled out rules (pass rate between 0% and 100%) #### Topline Alert Definition: Monitor absolute metric values regardless of Feature Gate assignment Use-case: When you want to take an action on your Feature Gate when system metrics breach thresholds, regardless of confirming causation. Works on fully rolled out (0% or 100%) as well as partially rolled out rules ## Getting Started Follow these tutorials to start using Safeguards: * [Create a new Safeguard](/feature-flags/safeguards-create) * [Manage an existing Safeguard](/feature-flags/safeguards-manage) # Scheduled Rollouts Source: https://docs.statsig.com/feature-flags/scheduled-rollouts Configure pre-set, time-based feature rollout schedules in Statsig that automatically increase traffic percentages or flip rules at specific dates and times. Feature Gates are powerful in ensuring a safe, controlled feature rollout. Scheduled Rollouts add a time-based scheduling layer to Feature Gates, enabling you to pre-set any rollout schedule you want, which will execute automatically. This is particularly useful if, for example, you have a feature launch happening in another timezone (and don't want to stay up all night!) or you have a standard, company-wide ramp-up schedule you follow with every feature release. Scheduled rollouts are set at the Feature Gate **rule** level, enabling maximal flexibility. Not all rules in your Gate need to include a scheduled rollout, simply set it up for whichever rules make the most sense. ## Set up a Scheduled Rollout To set up a Scheduled Rollout on a rule in your Feature Gate, simply tap on the "…" in the upper right-hand corner of the rule you want to schedule a rollout for. Feature Gate rule menu options Select **Edit Rule or Rollout**, and then select **Schedule Automated Rollout**. Schedule Automated Rollout selection From here, you can configure each phase of your Scheduled Rollout. You will see in the upper right-hand corner your current pass percentage- this simply reflects the baseline pass percentage you entered for your rule and can be changed via **Edit Rule**. To add phases to your rollout, click **Add Phase** and configure as many phases as you want. Each scheduled rollout phase includes- * Rollout date * Rollout time\* * Pass percentage Rollout times are available in 15 minute increments. Additionally, each configured phase represents a discrete increase to the next rollout percentage, not a gradual rollout amortized over the course of the entire phase. Scheduled rollout configuration interface As you are building your Scheduled Rollout, you will see a preview of the phases below the configuration wizard. This preview is also available for viewers of your Feature Gate on hover over a rule. Scheduled rollout phases preview Rollout phases hover preview ## Execute a Scheduled Rollout Once configured, each phase of a Scheduled Rollout will execute automatically on the schedule specified. The Feature Gate creator, any editors, and anyone Following the Feature Gate will receive Scheduled Rollout notifications. Notifications will be sent via: * Email * Console * (Optional) Slack * To configure Slack notifications on Statsig, go to "Account Settings → Notifications" # Test your Feature Gate Source: https://docs.statsig.com/feature-flags/test-gate Validate a Statsig feature gate using built-in tools, test apps, and live diagnostics in the console to confirm targeting before rolling out to users. There are three ways to test your feature gate and to validate that it's working as expected with the rules you have created: 1. Using the built-in **Test Gate** tool in the Statsig console 2. Using the prototype Javascript **Test App** available in the Statsig console 3. Using the **Diagnostics** tab in the Statsig console ## Option 1: Use the Test Gate tool To validate your feature gate using the built-in Test Gate tool: * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Feature Gates** * Select the feature gate that you want to validate * At the bottom of the page, the **Test Gate** window that lists all properties available in the rules you have created as shown below. Test Gate interface showing property fields * Click in the window and edit the value of the Email property to include the users that you want to target. For example, type [jdoe@example.com](mailto:jdoe@example.com) as shown below. When email domain matches "@example.com", the feature gate check succeeds and the window shows a PASS. Otherwise, it fails and the window shows a FAIL. Test Gate showing PASS result for email validation ## Option 2: Use the Statsig Test App To validate your feature gate using the Test App: * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Feature Gates** * Select the feature gate that you want to validate * At the bottom of the page, click on **Check Gate in Test App** at the top right of the Test Gate window as shown below by the red arrow; this will open a new browser window with a prototype Javascript client that initializes and calls the Statsig `checkGate` API. Check Gate in Test App button location ## Option 3: Use the Diagnostics tab To validate your feature gate using a live log stream: * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Feature Gates** * Select the feature gate that you want to validate * Click on the **Diagnostics** tab (next to the Setup tab) * Scroll down to the **Exposure Stream** panel, where you will see a live stream of gate check events as they happen as shown below Exposure Stream panel showing live gate check events * In the **Event Count by Group panel** as shown below, you can also validate that your application is recording events as expected for users who are exposed to the new feature (or not). Specifically, if you've started to record a new event type to test the impact of a new feature, you can also validate that these events are starting to show as more users are exposed to the new feature. Event Count by Group panel showing feature exposure metrics # Viewing Feature Gate exposures Source: https://docs.statsig.com/feature-flags/view-exposures Monitor feature impact in Statsig by viewing gate exposures, balanced gate analysis, and downstream metric lifts directly in the console. ## Gate Exposures To see the number of users who are being exposed to a feature gate, * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Feature Gates** * Select the feature gate that you want to test * Click on the **Pulse Results** tab * The **Cumulative Exposures** panel shows total exposures of a feature gate, broken down into three groups- 1. Units that passed the feature gate, and were used for analysis 2. Units that did not pass the feature gate, and were used for analysis vs. the "Pass" group 3. Units that did not pass the feature gate, and were *not* used for analysis vs. the "Pass" group Feature gate cumulative exposures view ## Balanced Gates Statsig balances gates and holdouts by default in Cloud, and as an opt-in option for Warehouse Native holdouts (on the Setup page). This reduces false positives, trading that off for losing sample size. Generally, this is considered a best practice and Statsig lands on the side of preferring a balanced analysis. See section 7 of [AB Testing Intuition Busters](https://drive.google.com/file/d/1oK2HpKKXeQLX6gQeQpfEaCGZtNr2kR76/view) for more discussion. This sampling is achieved during analysis by downsampling the larger arm proportionally to match the smaller group - e.g. a `(100 - large_group_pct)/(large_group_pct)` sampling rate is applied to the larger group using an unbiased hashing approach, keeping the user pool consistent. The hash salt is rotated across different gates and holdouts. ## Metric Lifts The **Metrics Lifts** panel shows how your feature is performing based on lifts in any business metrics added to the list of **Monitoring Metrics** for your gate. After working with experimentation experts across the industry, we aligned on an equal variant comparison (i.e. 10% vs 10% in this example) for calculating metric lifts for gate rollouts. You can read more about the advantages of this methodology in ["A/B Testing Intuition Busters: Common Misunderstandings in Online Controlled Experiments"](https://www.researchgate.net/publication/361226478_AB_Testing_Intuition_Busters_Common_Misunderstandings_in_Online_Controlled_Experiments) by Ron Kohavi, Alex Deng, & Lukas Vermeer. In the example below, the rises in *product view count* and *purchase event count* are statistically significant, suggesting this feature positively impacts the number of product views, but may actually be *negatively* impacting conversions to purchases itself. Metric lifts showing feature performance ## Bots & Filtering Information on Bots & Filtering has been moved to its [own page](/experiments/monitoring/bots) # Running an A/A Test using Sidecar Source: https://docs.statsig.com/guides/aa-sidecar Run an A/A test with the Statsig Sidecar Chrome extension to validate your experimentation setup and confirm metrics fire correctly before launching A/B tests. In this guide, we will walk you through how to leverage Statsig’s sidecar to run an A/A test on your product. This guide assumes that you have successfully set up and configured Statsig Sidecar. For a step-by-step guide on how to do this, see our ["setting up Sidecar"](/guides/sidecar-experiments/setup) guide. ## Why run an A/A (aa) test? There are many reasons to run an A/A test, one of the most common being to validate a new experimentation engine you may be integrating with (in this case Statsig). For new users just getting started with Statsig, we often recommend running an A/A test to provide a “low-stakes” first test environment, ensuring that you’ve got your metrics set up correctly and are seeing exposures flowing through as expected before kicking off your first real A/B test. ## How to run an A/A test ### Step 1: Create a new Experiment in Sidecar Navigate to the page on your website that you want to run an A/A Test. Open the Statsig Side car extension and click on 'New Experiment'. Fill in the title of your A/A test. Sidecar A/A test experiment setup interface Then, determine of the URI filter (i.e. All Pages, contains, etc.). After you have configured the URI, it is time to set up the variants. With the variant 'Control', pick an action. In this example we are simply changing the content of an element, specifically the title, 'Getting Started is Simple'. Sidecar A/A test variant configuration screen Repeating the step above, you'll do the exact same action for the variant 'Test'. Your set up should look like the following. Sidecar A/A test final configuration showing identical variants From there, all you'll need to do is click 'Publish', this will push out the experiment to Statsig as a draft. ### Step 2: Configure Experiment Scorecard in Statsig Console Once the experiment is pushed out to end users, you will need to edit the scorecard to your experiment within the console. Navigate to the console, click on the Experiments tab, and go into the experiment you just created. In the Setup tab, you can fill out the scorecard for the experiment Hypothesis, and any primary metrics you are interested in watching. While Statsig will show you experiment results for all your metrics, these key metrics represent your hypothesis for the experiment. Establishing a hypothesis upfront ensures that the experiment serves to improve your understanding of users rather than simply serving data points to bolster the case for shipping or not shipping your experiment. In the Allocation and Targeting section, for an AA test, we recommend to allocate 100% of users to the experiment while targeting everyone. Then, make sure to save and push your experiment. Your test is now set up to start measuring metrics associated with the A/A Test! ### Step 3: Review A/A test results Within 24 hours of starting your experiment, you'll see the cumulative exposures in the **Pulse Results** tab of your experiment. his will break down your logged exposures (as well as the distribution of the logged exposures). If something looks off, check the **Diagnostics** tab for more granular, day-by-day exposure breakdowns at both the Checks and User level. In the **Scorecard** panel, you can see the full picture of how all your tagged metrics are performing. pulse_results_empty What should you expect to see? * **Exposures**- make sure you’re seeing exposures flowing through as expected from your product. If you’re not seeing exposures, use the **Diagnostics** tab and the **Exposure Stream** to debug * **Pulse results**- roughly 5% of your metrics in Pulse should be showing a statistically significant change due to the 95% confidence interval of Statsig’s stats engine We recommend running your A/A long enough to reach most of your weekly active users, or at least a week. # Run your first A/B test Source: https://docs.statsig.com/guides/abn-tests Run A/B/n experiments in Statsig with three or more variants to compare multiple candidate designs against a single control group in one test. In this guide, you will create and implement your first experiment in Statsig from end to end. There are many types of experiments you can set up in Statsig, but this guide will walk through the most common one: an A/B test. By the end of this tutorial, you will have: * Created a new user-level **Experiment** in the Statsig console, with **parameters** set for a Control and Experiment group * **Checked the experiment** in your application code using the **Statsig Client SDK** `getExperiment` function ## Prerequisites 1. You already have a [Statsig account](https://console.statsig.com/sign_up) 2. You already [installed the Statsig Client SDK](/sdks/quickstart) in an existing application ## Step 1: In the Statsig console ### Create an experiment 1. Log in to the Statsig console at [https://console.statsig.com/](https://console.statsig.com/) and navigate to Experiments in the left-hand navigation panel. 2. Click on the **Create** button and enter the name `Quickstart Experiment` and enter a brief hypothesis. For example, "A new homepage banner will improve engagement." 3. Click **Create**. ### Setup scorecards Next, in the Experiment Setup tab, we will fill out scorecards for the metrics you are interested in watching. For the purposes of this SDK tutorial, we'll leave most of the settings to default. 1. In the **Hypothesis** card, enter a hypothesis for what you are testing: "Showing a homepage banner will increase user engagement, measured by daily\_stickiness." 2. In the **Primary Metrics** card, add a new metric called `daily_stickiness`. 3. We will leave the **Secondary Metrics** and **Duration** settings to their default values. For more details on experiment setup, see the product docs on [Creating an experiment](/experiments/create-new). ### Configure groups and parameters Next, we're going to configure the experiment. By default, the experiment is set up to run on 100% of users; 50% of those users are assigned to a Control group and 50% are assigned to an Experiment group. 1. In the **Groups and Parameters** card, create a parameter called `enable_banner` and select the type as `Boolean`. 2. Set `enable_banner` to `false` for the Control group and `true` for the Experiment group. 3. Click **Save**. 4. Hit **Save** in the bottom right to finalize this experiment setup. 5. This experiment is not live yet. To launch it, click **Start** at the top of the page. This will make production traffic eligible for the experiment. Experiment parameters and groups cannot be configured after launching an experiment. If you want to test in a staging environment, check out our docs on [using environments](/guides/using-environments/#configuring-environments). ## Step 2: In your application code ### Check the experiment This tutorial assumes you have already [installed the Statsig Client SDK](/sdks/quickstart) in an existing application. Next, we'll use a Statsig Client SDK to check a user's assigned experiment group and parameters in real-time. This will change the user's experience according to the variant they are assigned to. In this case, we fetch the value of the `enable_banner` parameter that we had created earlier. ```tsx Check Experiment theme={null} const quickstartExperiment = myStatsigClient.getExperiment(user, "quickstart_experiment"); // the second parameter is the default fallback value if the experiment is not found if (quickstartExperiment.get("enable_banner", false)) { showBanner(); } ``` In this snippet, if this user is assigned to the Experiment group, they will be shown the banner. If they are assigned to the Control group, the banner will not be shown. Notice that we never actually have to [hardcode the experiment group name](/experiments/implementation/getting-group) ("Control" or "Experiment") in the code. This is because the Statsig SDK will automatically fetch the experiment configuration from the Statsig server and return the correct value based on the user's variant. ## Step 3: Monitor experiment diagnostics Now, if you run your code, the `showBanner()` function is called for users in the Test group. If you navigate to your experiment in the Statsig console and select the **Diagnostics** tab, you can see a live log stream of checks and events related to this experiment from your application. You can read more about the Diagnostics tab [here](/experiments/implementation/getting-group#rules). Experiment Diagnostics ## Step 4: Read experiment results Within 24 hours of starting your experiment, you'll see the cumulative exposures in the **Results** tab in the Statsig console, and how they have impacted the various scorecard metrics you set up in the **Setup** panel. Here's a sample of what that looks like: Example experiment results ## Next steps In this tutorial, we used `daily_stickiness` as the primary metric. This metric is automatically logged to Statsig for you, so you don't need to log it explicitly. If you want to measure other events or custom metrics that happen downstream after checking the experiment, you'll need to log them to Statsig. Continue to the next tutorial to learn how to log custom events and metrics with the Statsig SDK. ### Related tutorials * [Initialization techniques for Statsig SDKs](/client/concepts/initialize) * [Run a device-level experiment](/guides/first-device-level-experiment) * [Run an experiment on custom unit ID types](/guides/experiment-on-custom-id-types) * [Configure the Statsig SDK for staging environments](/guides/using-environments) # CDN Edge Testing for Cached Resources Source: https://docs.statsig.com/guides/cdn-edge-testing Run feature gate and experiment evaluations at the CDN edge with Statsig integrations for Cloudflare Workers, Fastly Compute, and Akamai EdgeWorkers. ## Background Most customers with heavy web traffic use a CDN to serve resources from cache, in order to minimize hit to their web servers. This has historically made testing challenging because you don’t have the luxury of calling the SDK for all requests — but now with the emergence of Edge compute (offered by *most providers*), customers can now run code at their CDN edge, allowing them to assign users to tests and determine which resources to serve in a convenient and performant way. This pattern is optimal for testing with cached content without sacrificing cache-hit ratio. For scenarios where running the Statsig SDK at the edge is not possible, the sdk must be implemented on the origin server, or you should consider using a client SDK.
architecture
**example only — implementation may vary depending on your provider**
## Best Practices * Use a provider-specific "Config Sync" [integration](https://console.statsig.com/integrations) — Statsig will automatically sync your Statsig configuration data to your provider's Edge Storage. * Use a provider-specific Data Adapter — This will allow the Statsig SDK to initialize using the configuration stored at the edge near your function. Links for each provider provided below. * Use [statsig-node-lite](https://www.npmjs.com/package/statsig-node-lite) — This is a slimmed down version of the Node SDK, which includes only the essentials and dramatically improves initialization performance for cold-start requests. After initialization, evaluate with Node Lite's sync methods, such as `checkGateSync`, `getFeatureGateSync`, `getExperimentSync`, and `getLayerSync`. * Persist a uuid — As always, you'll need some sort of user identifier that can be used to consistently assign a user to your test buckets. This can typically be solved by generating a uuid in your function and setting it to a cookie. * Persist assignments in a cookie — Some customers will set assignments to a cookie for the purpose of a performance optimization, allowing your code to skip sdk calls if the user is already assigned to a test. This is best defined as a session-cookie (a cookie that expires when user closes their browser). * Persist client instance — [This pattern](/guides/serverless#usage) allows the Statsig client instance to persists across requests when the edge function remains warm and will improve performance. * Use [Target Apps](/sdk-keys/target-apps) — Target apps will allow you to sync a specific subset of experiments/gates to your edge function, reducing the footprint of your project config and improving performance. Target Apps are compatible with all of the "config sync" integrations. * Consider peeking at experiment assignments and avoiding automatic exposure logging. You should only log exposure events once a user has been exposed to the treatment, otherwise your test results may become polluted with users that didn't see the treatment. In Node Lite, use the explicit exposure-disabled sync methods, such as `getExperimentWithExposureLoggingDisabledSync`, `checkGateWithExposureLoggingDisabledSync`, `getFeatureGateWithExposureLoggingDisabledSync`, `getLayerWithExposureLoggingDisabledSync`, or `getConfigWithExposureLoggingDisabledSync`. If you use the top-level `Statsig` wrapper, log the exposure later with `manuallyLogExperimentExposure`, `manuallyLogGateExposure`, `manuallyLogLayerParameterExposure`, or `manuallyLogConfigExposure`; if you use a `StatsigServer` instance directly, use `logExperimentExposure`, `logGateExposure`, `logLayerParameterExposure`, or `logConfigExposure`. * Node Lite does not use the Node Core `EvaluationOptions` pattern for disabling exposure logging. See the current [top-level `Statsig` API](https://github.com/statsig-io/node-js-lite-server-sdk/blob/main/src/index.ts) and [`StatsigServer` API](https://github.com/statsig-io/node-js-lite-server-sdk/blob/main/src/StatsigServer.ts) for the full method lists. * You should consider how this can be managed downstream in your application. All SDKs have a means of tracking exposures manually, and you can also consider using the [HTTP API for logging exposures](/http-api#log-exposure-event). * If you choose to log exposures from your edge function, you should consider using the `context.waitUntil` method if supported by your edge provider ([examples](/server/nodejsServerSDK#environment-specific-setup)). ## Cloudflare Implementation Cloudflare allows its users to easily stand up a serverless edge worker that will get invoked prior to cache lookup and also allow you to modify the response to the viewer. The Worker architecture gives you full code control over how you'd like to map test assignments to resources you should serve to the user, allow you to modify both the URL (serving as cache-key) and headers used to fetch your resource from the CDN, passing the modified request through to origin on cache-miss. The Cloudflare Worker runtime allows you to install [Statsig Node Lite SDK](https://github.com/statsig-io/node-js-lite-server-sdk) as a dependency and use it to determine test assignments before fetching a given resource. We offer both an [integration with Cloudflare KV](/integrations/cloudflare), and a pre-built [KV Data Adapter](https://github.com/statsig-io/cloudflare-data-adapter-node/tree/master), ensuring that SDK initialization is performant and doesn't depend on requests over the network back to Statsig. ## Fastly Implementation Fastly Compute platform supports Functions at the Edge, affording customers the ability to handle assignment at the edge. ### KV (fully supported) [Fastly's KV storage solution](https://www.fastly.com/blog/be-among-the-first-to-try-the-greatest-kv-store-ever-made) is touted as being highly-performant and is now their recommended solution over their legacy ConfigStore documented below. Statsig offers both a Config-Sync for Fastly KV as well as a KV DataAdapter which can be found [here](/integrations/fastly). ### ConfigStore (not recommended) We initially built the [integration with Fastly's ConfigStore](/integrations/fastly), which was their only offering at the time. ConfigStore values are limited to 8 kilobytes, which will be met rather quickly as the number of gates and tests in your project increases. This will be paired with our [Fastly Config Store Data Adapter](https://www.npmjs.com/package/statsig-node-fastly), allow the SDK to initialize from ConfigStore rather than making a request over the network back to Statsig. ### Recommended pattern The Fastly pattern is a bit different than some of the other providers — as recommended by [the Fastly AB Testing guide](https://www.fastly.com/documentation/solutions/tutorials/ab-testing-edge-compute/#use-the-allocations-on-your-origin-server), it does not involve modifying the cache-key (resource URL), but instead attaching `Fastly-ABTest-` headers, which will tell your origin server how to render the resource, and the origin response should include a `Vary` response header to tell Fastly CDN what to cache. This approach is documented thoroughly by Fastly, and it's recommended that their guide serves as the source of truth for the design pattern. ### Fastly VCL Implementation Their legacy Varnish-based platform only supports configuration based caching and minimal scripting at the edge using VCL scripting language. Practically speaking, there is no Statsig SDK (or any SDK for that matter) support at the edge for this reason. In this instance, the Statsig SDK must be implemented on the origin server, and cache rules must be configured to force a cache-miss and allow assignments to take place on the origin server. ## AWS Implementation AWS has a variety of serverless solutions, below we will detail the recommended pattern and various limitations associated with each. ### Lambda\@Edge Implementation Lambda\@Edge are Lambda functions that are designed to be triggered by CloudFront events. Lambda\@Edge implementations carry a few unique challenges: * At the time of writing this, AWS does not offer a KV store solution compatible with Lambda\@Edge. Their [KeyValueStore](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/kvs-with-functions.html) offering is only compatible with Cloudfront Functions. This means, you will either have to: * Use the default Statsig SDK initialization, which will incur a hit over the network to Statsig to load configurations * Bundle your project config statically into the function package. You can consider using our [Config-Change Webhook](/integrations/event_webhook#config-change-webhooks) to automate this. You'll need to download the configuration JSON at the following URL in order to bundle it into your Lambda package: [https://api.statsigcdn.com/v1/download\_config\_specs/SERVER\_SDK\_KEY.json](https://api.statsigcdn.com/v1/download_config_specs/SERVER_SDK_KEY.json) * Store your project config in S3 and use Cloudfront as means to serve it optimally when fetching it from your Lambda\@Edge function. * Function size is capped at 1MB in package size (for viewer request and viewer response triggers, which is what is required for this setup). * Lambda model uses [four events triggers](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-cloudfront-trigger-events.html), calling your function at various stages of the request lifecycle. For this use case, you’ll need to use two: `Viewer Request` and `View Response`. AWS provides documentation on how requests can be handled and manipulated [here](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-at-the-edge.html). #### Viewer Request function Viewer Request will be called before cache lookup, and will be used to define a uuid, generate test assignment and modify the request URL via `Request.URI`. This type of function cannot modify response headers, so you must pass any assignment and uuid info through to the Viewer Response function in order to set cookies. Event structure for each event documented here. #### Viewer Response function Viewer Response function is responsible only for setting uuid and assignment cookies as needed. ### CloudFront Functions Implementation SDKs are not supported here. Cloudfront Functions have 1ms max runtime and 10KB max function size. You’ll have to use Lambda\@Edge, or consider incurring Cache-misses to do assignment at origin. ## Akamai Implementation [Akamai has a documented pattern](https://www.akamai.com/blog/developers/better-a-b-testing-with-edgeworkes-edgekv) of using EdgeWorkers and EdgeKV for running AB tests using cached content, providing both code samples, as well as an overview of the [architecture](https://www.akamai.com/solutions/edge/serverless-computing/use-cases). You’ll need to add the Statsig SDK to the worker function as a dependency and call it for assignment (rather than the rudimentary coin flip using `Math.random()` shown in their code examples). You can now use our [Akamai Edge integration](/integrations/akamai), which includes an integration with EdgeKV as well as a Node package that has been tested and validated within Akamai edge runtime. # Create your first feature flag Source: https://docs.statsig.com/guides/check-gate How to call checkGate in Statsig client and server SDKs to evaluate a feature gate, including parameters, return values, and exposure logging. This tutorial walks you through how to check your first Feature Gate in Statsig from end to end. Feature Gates, also known as feature flags, are a way to safely control the rollout of new features to your users without deploying additional code. Common examples for using Feature Gates include shipping new UI elements, API endpoints, or product features. By the end of this tutorial, you will have: * Created a **Feature Gate** in the Statsig console, with **targeting rules** to enable the feature for a segment of Users * Initialized a **Statsig Client SDK** * Checked a single **feature gate** in your code using the `checkGate` function ## Prerequisites 1. A [Statsig account](https://console.statsig.com/sign_up) 2. An existing application you can integrate the Statsig Client SDK into ## Step 1: In the Statsig console ### Create a Feature Gate For the purposes of this tutorial, we will pretend we are adding a Feature Gate to deploy a new UI element to a user with the "statsig.com" email domain. You can follow along with a specific feature if you have your own scenario in mind. 1. Navigate to [Feature Gates](https://console.statsig.com/gates) in the Statsig console. 2. Then, click on **Get Started** if you don't have any Feature Gates set up yet, or **Create** to create a new one. 3. Name your gate "Example Gate". This name will also be used to identify the Feature Gate later using the SDK. 4. Enter a description for your Feature Gate. It's good practice to describe it in a way that other teammates can easily understand. For example: "This Feature Gate is for launching an example feature for Statsig employees only." ### Create a targeting rule In Statsig, when you create a Feature Gate, they are enabled by default. In other words, all users will be excluded from the feature until you add a rule that lets users "pass" the gate. This means that in order to actually turn on this feature, you will need to add rules to target this Feature Gate to a specific set of folks. Let's walk through doing this in the console. 1. In the console, on the page for the Feature Gate you just created, click on **Add New Rule**. 2. Give this rule a **Name**, such as "Statsig Users Only". 3. Select **Email** as our targeting criteria so we can target users based on their email address. 4. In the User section of the dropdown, select the **Any Of (Case Insensitive)** operator, and then add `statsig.com` for our email-based user targeting. 5. Set the **Pass Percentage** to `100%`. Doing so ensures that all users with the `statsig.com` email domain will pass the Feature Gate and see the new feature. 6. Click **Add Rule** to add this rule to your Feature Gate. 7. Next, hit **Save** on the bottom right to commit these changes to the Feature Gate. You can now test this feature gate by configuring the User object in the "Test Gate" section. ### Create a Client API Key Now that you've set up the Feature Gate from the console, it's time to integrate it into your product with the Statsig SDK. We'll first need to create a new Client API key to use in our product. 1. Navigate to [**Keys & Environments**](https://console.statsig.com/api_keys) in the Statsig console. You can also find this by going to **Settings** at the bottom left of the Statsig console. 2. Scroll down to **API Keys**. Click on **Generate New Key**. 3. In the dropdown, select **Client**. 4. Copy the Client API Key you just created to your clipboard. ## Step 2: In your code ### Initialize the Statsig SDK Now that we have our Client API Key, we can go ahead and integrate the Statsig Client SDK into our product. For the purposes of this tutorial, we will use the React SDK, but you can follow along with a different SDK if you prefer. Statsig offers over 20 client and server-side SDKs. Check out the full list of [SDKs](/sdks/quickstart#all-sdks) to find the one that best fits your needs. 1. Install the Statsig React SDK using your preferred package manager. For this tutorial, we will use npm. ```bash npm theme={null} npm install @statsig/react ``` 2. Import the SDK in your `App.js` file: ```tsx Import SDK theme={null} import { StatsigProvider } from "@statsig/react-bindings"; ``` 3. Next, wrap your app's content within the `StatsigProvider` component. In the following code snippet, we're also creating a [User](/concepts/user) object so that we can target our Feature Gate. ```tsx Setup Provider theme={null} function App() { return (
Hello world
); } export default App; ```
4. Make sure to also replace `client-KEY` with the Client API Key you copied in Step 3. ### Check your Feature Gate Finally, you can now evaluate a Feature Gate in your product code by getting the client with the `useStatsigClient` hook, and then calling `checkGate`. If you're not sure yet about where and how you want to place your flag, check out our doc on [Best Practices for feature flags](/feature-flags/best-practices). 1. Add the following code to your `App.js` file. In this snippet, the `example_gate` is the name of the Feature Gate you created in Step 1. ```tsx Check Gate theme={null} const { client } = useStatsigClient(); return (
Gate is {client.checkGate('check_user') ? 'passing' : 'failing'}.
); ```
2. Run your app and see the result! The app should render the text "Gate is passing" since we already [Created a targeting rule](#create-a-targeting-rule) that targets all users with the `statsig.com` email domain, and we are using that same email domain in this client's User object. 3. Once you've set up your gate, you can easily [monitor the impact of your new feature rollout](/feature-flags/view-exposures) or [manage flag lifecycles](/feature-flags/feature-flags-lifecycle). ## Next steps In this tutorial, we configured a simple feature flag. You can monitor basic metric impacts with this, but if you want to do more complex feature rollouts or metric analysis, continue to the next tutorial to run your first A/B test in Statsig. # Guide to General CMS Integrations Source: https://docs.statsig.com/guides/cms-integrations Integrate Statsig with CMS platforms like Contentful, Webflow, and Framer to run content experiments and personalize pages without redeploying code. ### Using Statsig with a CMS One fairly common question we get is around how to use Statsig with an existing CMS. While we also offer a no-code solution - [Sidecar](/guides/sidecar-experiments/introduction), there are clever ways you can set up your code to integrate your CMS and Statsig so you can write code once, and then run experiments on arbitrary combinations of parameters in the future. We recommend using [Layers](/layers) to wire this up, so take some time to read up before continuing. Layers are a unit of mutual exclusion between experiments in Statsig. Every user participates in only one experiment in a layer at any given time. As such, we recommend you set up a layer for each surface you will be experimenting on with the help of your CMS For the remainder of this guide, we will assume you are experimenting on a single surface - but repeat these steps if you plan to experiment on separate surfaces like your landing page, product page, blog, etc. For the sake of example, let's assume we are parameterizing the Statsig landing page to plug in our CMS. Statsig landing page diagram showing CMS-provided sections First, lets create a layer. Navigate to "Experiments" in the left hand column, and then "Layers" in the title bar: Layers list and create button in Statsig experiments UI Here, we'll create one for all content or parameters we want to experiment with on our landing page, so we call it "statsig\_landing\_page" Create layer dialog naming statsig_landing_page Next, lets create some parameters. One for the title, subtitle, and primary CTA. For the value, we will use the actual ID of the content in the CMS Layer parameter creation form for CMS content IDs It should look like this when you are done: Layer parameters table listing title, subtitle, and CTA defaults Note that each layer parameter has a default value. If the user is not in any experiments in that layer, that's the default they will get, which will be backed by the cms. Now, in code, your integration will look something like this: ```js theme={null} const landingPageCmsIds = statsig.getLayer("statsig_landing_page"); const titleID = landingPageCmsIds.get("title", ""); // note that you have a default value in code as well // exact library and function call will map to your cms client library cmsClient.getEntry(titleID); ``` If you repeat this for the subtitle, CTA, and all the other parameters on your landing page, they all become dynamic! When you put a new CMS ID into statsig, your code will pull the updated content for that section. Now, you can create new content in your CMS, and create an experiment in Statsig to try out that new variant. After creating the content, come back to your layer and hit "Create Experiment in Layer": Create experiment in layer button location Fill out the resulting form: Layer experiment setup form with control and test weights And you have created an experiment! Now, we just need to set up the test and control groups for the experiment, and say which content will be used for each of the parameters we have set up. In the Groups and parameters section, select "Add Parameter" and then choose one of your existing parameters, like title: Groups and parameters section showing Add Parameter menu Parameter value editor selecting CMS ID per group Update the value of the parameter to the id of the new title: Updated parameter table showing new CMS content IDs per variation If you want to create multiple experiment groups, or add more parameters, keep on adding until your experiment setup is complete. After you have validated the experience in all the groups is what you expect, start your experiment and wait for results - no code changes are required for all those parameters you already created, statsig will pull the updated ID, and then your code will load the updated content for each of those automatically! The experimentation flow is the same as all other experiments on Statsig at this point, the value just ties to your CMS. If you need more help setting up and running experiments, see [Experiments](/experiments-plus) # Config History Source: https://docs.statsig.com/guides/config-history View and audit configuration history in Statsig for feature gates, experiments, dynamic configs, and metrics, including who changed what and when. ## Entity Change History Change history for entities like Feature Gates, Experiments, Dynamic Configs, and Segments can be accessed by clicking the "History" button on the top right of their respective page. Config History This page shows each change to a given config and the time that change was published. History timeline listing published config changes For Feature Gates and Dynamic Configs, there is also a "Preview" and "Restore" option. These are meant to make it easy to revert to a previous state, particularly if you start rolling something out that is causing issues in production. The "preview" action will show a diff view between the current state of the gate and the previous state you selected, so you can make sure you are reverting to the correct state. Preview and restore dialog comparing config versions For Gates, Dynamic Configs, and Segments, you can also select two different versions, and compare the differences between the states of the entity between the two versions. Compare Configs Note that changes in nested configs are not listed (e.g. if this gate references another gate or segment via a "Passes Target Gate" or "User in Segment" condition, changes to the other gate or segment will not show in the history view). ## Audit Logs Audit logs will show you the change history for all available entities across the entire project. These audit logs are stored indefinitely and you can filter by entity type or name, tag, target app, environment, action, and user who triggered the action. Audit logs can be accessed programmatically or though the Statsig Console UI in **Settings** > [Audit Logs](https://console.statsig.com/audit_logs). # Guide to Contentful Source: https://docs.statsig.com/guides/contentful Integrate Statsig with Contentful to run experiments on CMS-managed content, including connecting accounts, mapping variants, and tracking metrics. The Statsig Contentful integration lets you create A/B/n tests and test different content blocks against each other directly from within Contentful. You can assess impact using business metrics on Statsig Cloud or Warehouse Native. Marketers can optimize content, obtain insights, and iterate continuously right from within Contentful. * Run experiments on CMS content without engineering involvement * Configure content to serve with each variation * No performance penalty or flicker The Statsig Contentful app will add a Statsig container that is connected to an experiment in Statsig. The user can then add Content Blocks to that container to start a test. The Statsig Contentful app lets marketers measure progress towards business objectives by testing content for lift in any core business metrics configured in Statsig. ## Integrating with Contentful Our Contentful Marketplace App is publicly available. You can find it [here](https://www.contentful.com/marketplace/statsig). To use this integration effectively, you will need to do setup around the Contentful marketplace app, your Content types, and your actual codebase. These are one-time setups, then you will be able to seamlessly run A/B/n tests directly inside Contentful. ### Setting up the Statsig Marketplace App * Navigate to the Marketplace in Contentful, and find the Statsig app. Click 'Install'. * Statsig will prompt you to enter a Console API Key. You can find an existing Console API Key in your Statsig project under Settings > Keys & Environments. It's important that this key is **of type 'Console', and has read and write permissions**. Feel free to generate a new key of type 'Console' if a suitable one does not already exist for your project. image.png * Once your API Key is entered, hit 'Install to selected environments'. Your app should now be configured. Returning to this page later will only show the *obfuscated* API Key. ### Setting up Statsig Variant Container Once configured, a new Content model should have been added to your space called 'Statsig variant container'. We can check to make sure this is setup properly: * Navigate to the 'Content model' tab in Contentful, and select the 'Statsig variant container'. image.png * You should see a list of 4 fields: Statsig Experiment Id, Entry Name, Default Variation (control), Treatment Variations. image.png * If your 'Statsig Experiment Id' field shows `Excluded from api response` next to it, we will need to update this field to be fetchable in API calls. We can do this by clicking the three dots on the right of the field, and click 'Include in API response'. Then click 'Save'. image.png Your 'Statsig variant container' is now setup and ready to associate with other Content types. ### Setting up Experiments in Content Types You can configure your existing content types to run Statsig experiments in, automatically serving different variants of this content type to your users. The steps below walk through how to add a 'Statsig experiment' field to your target content type. * Navigate to the 'Content model' tab in Contentful, and select your target content type (in this example, `page - Blog post`). You should see the list of fields for this content type: image.png * Click 'Add field', and choose 'Reference'. Enter `Statsig experiment` for the Name, then click 'Add and Configure'. image.png * Under 'Validation', select 'Accept only specified entry type', and choose 'Statsig variant container' from the dropdown. image.png * Confirm your new field, and save your content type. Your content type is now setup to use Statsig Experiments! Feel free to repeat this process for any other content types you would like to be able to run experiments with. ### Running an Experiment on your Content To run an experiment on your content, you can link a Statsig Experiment to it. Here's how: * Navigate to the 'Content' tab in Contentful, and select your existing entry from the list. At the bottom of the Editor tab, you should now see an editable field for 'Statsig experiment': image.png * Click on 'Add content', and select 'Statsig variant container' from the New content dropdown. You should see a new Statsig variant container layover: image.png * Under the Statsig tab, enter the name of your experiment under the 'Entry Name' field. Add your control and treatment variations. In this example, we will add `component - Rich image` variations to experiment with. Please note that experiment name should exclude special characters. * When your experiment setup is finalized, hit 'Publish' on the new Statsig variant container entry. Ensure your experiment setup is finalized before publishing, as this will create your experiment inside of Statsig. image.png * You should now be prompted to start your newly created experiment inside of Statsig. Follow the 'Go to Statsig Experiment' link to finalize your experiment's setup, add metrics, and start your experiment. * Once your experiment has been started on Statsig, you should see a green banner at the top of your Statsig variant container, and your variation fields will no longer be editable. * Return to your original entry, and hit 'Publish changes'. Your experiment is now live! ### Integrating Statsig Experiments in your Codebase We have provided an [example repository](https://github.com/statsig-io/contentful-blog-webapp-nextjs-example/tree/main) that outlines how you can integrate your Statsig experiments created from Contentful into your codebase. The `README` walks through the setup process, including pulling experiment fields from Contentful, calling a Statsig SDK, and matching assigned users to their respective variant. ### Troubleshoot Common Problems #### I created an experiment and the 'Go to Experiment' button doesn't show up - what happened? This indicates that the Statsig Experiment Id was not saved properly. To fix this error, navigate into 'Editor' and manually add the Statsig Experiment Id. Once you save it, then the button should populate as expected. Contentful Statsig Experiment Id # Email AB Testing with Customer.io Source: https://docs.statsig.com/guides/customer-io-email-abtest Run email A/B tests with Statsig and Customer.io by sending exposure events from email sends to compare open, click, and conversion metrics. Email campaigns are a critical tool for any Marketing team. Finding the best performing Email template is a perfect use-case for an A/B test. Statsig allows you to run simple but powerful A/B tests on different parts of your email content. Since Statsig can integrate seamlessly with product analytics, you can run email experiments and understand deeper business level impact on product metrics easily. This guide assumes you have an existing Statsig account. Please go here to create a new free account if you don't already have one: [https://statsig.com/signup](https://statsig.com/signup) ### Step 1: Create an experiment Start by creating a new Experiment on Statsig console. Put in a name and leave the rest of the fields empty/default. For the purposes of this walkthrough, that should do. Experiment creation interface ### Step 2: Start the experiment Since you can't start an experiment without a parameter, let's go ahead and add a dummy parameter. Experiment parameter configuration Experiment setup completion Save the experiment setup and **Start** it. We're all set with the experiment set up. Experiment start confirmation While you're at it, copy the **Experiment Name**. We'll use this in a bit. Experiment name copy interface ### Step 3: Set up Exposure Webhooks In your Customer.io campaign, create a **Random Cohort Branch** which a flow similar to the following: Customer.io random cohort branch setup In the Webhook actions, put in a post request similar to the following, with your api key and experiment name filled in: Webhook configuration for experiment exposure Pass in any other custom IDs and user attributes inside the post body. For each webhook, make sure to expose the correct group that you'd like to attribute your branch to. In the above webhook, we are exposing the "Control" group. Experiment group exposure configuration In Statsig, you'll now have exposures for each of your experiment groups. ## Holdouts To use Statsig Holdouts with Customer.io, it's recommended to identify users that are part of a holdout via customer.io's identify function: [https://customer.io/docs/sdk/ios/identify/](https://customer.io/docs/sdk/ios/identify/) Where you call Customer.io's identify method, you could check a Statsig holdout gate, and add an attribute to the user to mark that user as being in a holdout. In your campaign, you'll be able to create a True/False branch to check whether a user is in the holdout. Customer.io holdout branch configuration # A/B Test Email Campaigns Source: https://docs.statsig.com/guides/email-campaign-test How to run controlled A/B tests on email campaigns with Statsig, including audience targeting, exposure logging, and metric attribution best practices. A/B Testing an email campaign and getting experiment results on downstream product metrics (in addition to top level email interaction metrics), is a common use case for Statsig customers. Email marketing tools often have native A/B testing capabilities, but are limited to measuring email open rates or link click rates. These standard patterns can be used across almost any email marketing tool (Braze, [Customer.io](/guides/customer-io-email-abtest/), [Sendgrid](/guides/sendgrid-email-abtest), and Salesforce Marketing Cloud). Two common patterns used on Statsig are: #### 1. Do assignments in the marketing automation tool, do analysis in Statsig (requires Statsig WHN) Use Case: The marketing automation tools only measures email open/click through rates; but what is useful to measure is impact on product usage.\ Solution: Use the email automation tool to bucket users into Control/Test. Write these assignments into a table in your warehouse. Statsig Warehouse Native can use this table as the list of assignments for the experiment and can analyze experiment impact using the full catalog of business/product metrics configured on Statsig. Visit [this guide](/statsig-warehouse-native/guides/email-experiments) for more best practices with Email Experimentation using Warehouse Native. #### 2. Do assignments and analysis using Statsig Use Case: Want email campaigns to be coordinated with in-app messaging/promotions (like access to a certain promotion).\ Solution: Call the Statsig SDK to assign each user into Control/Test and use that in the email marketing system (e.g. Salesforce Marketing Cloud lets you create a custom extension to do this). You can either perform a bulk evaluation for a list of email addresses selected for a campaign (using a Python script + Statsig Python SDK) or call the SDK one by one. This works both on Statsig Warehouse Native and Statsig Cloud. If you're using a script to call the Python SDK, this is very performant. Remember to either `flush()` or `shutdown()` the SDK before exiting the script so the exposures get logged to Statsig. You can verify these were flushed on the Diagnostics page of your experiment. # Experiment on custom Unit ID types Source: https://docs.statsig.com/guides/experiment-on-custom-id-types Run Statsig experiments using custom ID types like company ID, account ID, or device ID instead of the default user ID for B2B and device-level tests. In certain cases, you may want to randomize experiment bucketing using a custom Unit ID instead of the default `userID` or Statsig-generated Stable ID. For instance, if you're running a task management tool for companies and want to experiment on company-wide behaviors, you might use `companyID` as the Unit ID. This ensures all users from the same company get the same experience, allowing you to measure overall productivity impacts at a company level. Custom ID types allow for this flexibility in your experiments and feature gates, enabling you to randomize and control rollouts based on any identifier you need. ### When to use custom Unit IDs: * **Organization-wide experiments**: Group users by `companyID` for company-wide consistency. * **Session-based experiments**: Use a session or device ID to control the experiment within a specific session. * **Group-level rollouts**: Target teams, regions, or other specific cohorts using a relevant ID type. Follow these three simple steps to set up an experiment with a custom Unit ID. In the following examples, we will use `companyID`, but you can replace it with any ID relevant to your use case. These steps also apply to feature gates, allowing you to partially roll out features based on custom Unit IDs. *** ### Step 1: Add `companyID` as a Custom Unit ID 1. **Log into the Statsig Console**: Head over to [Statsig Console](https://console.statsig.com/) and navigate to **Project Settings**. 2. **Find Custom Unit IDs**: Under **Manage Account** > **Info**, look for the **Custom Unit IDs** section. Custom Unit ID Settings 3. **Add a New Custom Unit ID**: * Click the **Edit** button. * Enter `companyID` as the new ID type and provide a description. * Save the changes. Add Custom ID Once added, this custom Unit ID can be used across all your Statsig configurations: experiments, gates, layers, dynamic configs, and autotunes. You only need to configure it once for your project! This unified configuration streamlines your workflow, allowing you to consistently use your custom identifier across various Statsig tools and features. Whether you're setting up a new experiment, configuring a feature flag, or creating a dynamic config, you can rely on your custom Unit ID to accurately target and analyze your specific user base or entities. ### Step 2: Select `companyID` as the ID Type in Your Experiment 1. **Create a New Experiment**: * Navigate to **Experiments+** in the Statsig Console. * When setting up a new experiment, find the **ID Type** dropdown and select `companyID`. Select Custom ID Type 2. **Complete Experiment Setup**: * Finish configuring your experiment as you would for any [user-level experiment](/guides/abn-tests). * Define your hypothesis, metrics, and target audience. Once done, click **Save** to finalize the setup. > **Tip**: Choosing the right custom Unit ID ensures that all users with the same `companyID` are placed in the same experiment group, making it easier to measure performance at the group level (e.g., company-wide performance). *** ### Step 3: Provide `companyID` in the Statsig SDK To use the custom Unit ID in your application, ensure it is provided when initializing the Statsig SDK. The `customIDs` field in the Statsig user object allows you to pass the `companyID` (or any other custom ID) along with the usual user information. #### Example (JavaScript): ```javascript theme={null} var user = { userID: "some_user_id", // Standard user identifier customIDs: { companyID: "some_company_id" // Custom ID for grouping }, // Other attributes (optional) email: "user@example.com", appVersion: "1.0.0" }; // Initialize the Statsig Client const client = new StatsigClient(sdkKey, user); await client.initializeAsync(); ``` * **`customIDs` field**: This allows you to pass a dictionary of custom IDs, including `companyID`, to ensure the experiment targets users based on that ID. Once you've provided the necessary IDs in the SDK, you can start logging events and fetching experiment configurations based on the custom Unit ID. *** # When to Use Feature Gates vs. Experiments? Source: https://docs.statsig.com/guides/featureflags-or-experiments Decide whether to ship a change behind a Statsig feature gate or run an experiment, and learn how feature flags and A/B testing work together in practice. In Statsig, feature flags are called feature gates. The terminology is interchangeable throughout this guide. Both feature gates and experiments create control/test groups. Use this guide to pick the right tool for your launch and measurement goals. *** ## Quick Guidance * **Choose a feature gate** when you want to roll out a feature gradually or monitor impact as you ramp. * **Choose an experiment** when you need to compare multiple variants and quantify the lift across metrics. *** ## Key Differences ### Variants * **Feature gate** → Two experiences only: pass vs. fail. * **Experiment** → Any number of variants. When viewing gate exposures you’ll see three buckets: Pass, Fail, and Fail – Not in Analysis. Only the balanced subset of the fail group is used for metric comparisons. Learn more in the [gate exposure methodology](/feature-flags/view-exposures#gate-exposures). ### Return Values * **Feature gate** → Boolean (`true`/`false`) so your application toggles code paths. * **Experiment** → JSON config that describes the variant (colors, copy, thresholds, etc.). ### Ramping knobs * **Feature gate** → Adjust Pass % to send more traffic to the new experience. You can go beyond 50/50 (e.g. 99% vs 1%). * **Experiment** → Adjust Allocation % to enroll more users, but splits cap at 50/50. Once a user is assigned, neither control reshuffles existing users—you can safely ramp without re-bucketing. Pass% versus Allocation% controls *** ## When Experiments Shine Use experiments when you need: 1. **Multiple variants or personalization** – compare more than two options or tailor experiences via contextual bandits/layers. 2. **Stable identifiers and custom IDs** – analyze behavior before signup with stable IDs, or use custom IDs for sessions, workspaces, or geography. 3. **Isolated universes** – run parallel experiments safely by placing them in their own layers. *** ## When Feature Gates Shine Feature gates are great for: * **Safe rollouts** – gradually increase exposure while observing metrics. * **Targeting audiences** – use gates as pre-filters before enrolling users in an experiment. In experiment setups, gates often act as targeting criteria. The flow looks like this: 1. **Targeting gate** picks the eligible audience. 2. **Allocation %** (experiment) decides how much of that audience participates. 3. **Split %** distributes participants across variants. Once you choose a winner, you can lift the targeting gate and let the winning variant reach everyone. *** ## Putting It Together * Start with a **feature gate** if you have a single variant to launch carefully. * Reach for **experiments** when you need quantitative comparisons across variants. * Combine both when you want precise audience control plus rigorous measurement. Need more depth? Check out: * [Feature gate exposures](/feature-flags/view-exposures) * [Experiments overview](/experiments/overview) * [Layers for mutual exclusion](/experiments/layers-overview) # Build your first Device-level Experiment Source: https://docs.statsig.com/guides/first-device-level-experiment Step-by-step guide to running your first device-level experiment in Statsig, where assignment is based on device or stable ID rather than user ID. When you cannot identify a user via their user ID, device-level experiments allow you to randomize experiments based on a consistent identifier for the user's device. While Statsig can automatically generate a stable ID, it's recommended to use your own cookie or logged-out ID when possible. Device-level experiments are ideal in scenarios such as: * **Anonymous or first-time users**: When users haven't signed in yet or are browsing anonymously. * **Cross-device consistency**: Ensuring the same experience on the same device, regardless of user sign-in status. You can implement a device-level experiment almost exactly like a traditional user-level experiment. The key difference is setting the experiment’s ID type. In this example, we set the `stableID` as the ID type, but if you have your own identifier you can substitute that as well. ## Step 1: Create a Device-level Experiment 1. **Log into the Statsig Console**: Visit [Statsig Console](https://console.statsig.com/) and navigate to **Experiments+** on the left-hand sidebar. 2. **Create a New Experiment**: * Click on **Create** and fill out the **name** and **description** of your experiment. * Enable the **Use Stable ID** option during setup. * Click **Create** to proceed. Stable ID Setup 3. **Define Experiment Metrics**: Add a hypothesis, primary metrics, and secondary metrics in the **Scorecard** section, just like you would for a user-level experiment. 4. **Set Groups and Parameters**: * In the **Groups** section, define the parameters for your experiment. For instance, you can experiment with a simple boolean parameter like `"enabled"`. Experiment Groups 5. **Set Allocation**: * By default, the experiment targets 100% of your user base. Adjust the allocation if needed. Starting with a smaller rollout is often recommended until you're confident in the new variant. Allocation Panel 6. **Save and Start**: Once everything is configured, click **Save** to finalize your experiment. When you're ready to launch, click **Start** to roll it out. ## Step 2: Initialize the SDK in Your Application After setting up your experiment in the Statsig console, the next step is to integrate it into your client application using one of Statsig’s SDKs. **Important:** * **`userID`** should **only** be set for authenticated, logged-in users. * For logged-out or anonymous users, use **`stableID`** (Statsig’s auto-generated device ID) or your own custom deviceID to identify the device. See [customID types](/guides/experiment-on-custom-id-types) if you have your own deviceID * Always pass all known IDs to the SDK — Statsig will use the correct one for evaluation depending on the experiment or gate’s ID type. * If you do rely on stableID, it is only generated by Statsig client SDKs (javascript, react, mobile, etc) - server SDKs are unable to generate this ID for you * **User attributes**: You can pass additional attributes like `appVersion`, and `custom` properties for experiment targeting. ### Example (JavaScript): ```javascript theme={null} const user = { userID: userID: isLoggedIn() ? getLoggedInUserID() : undefined, // Optional attributes to help with targeting appVersion: "1.0.0", custom: { promoCode: "New30Off" } }; // Initialize the Statsig Client const client = new StatsigClient(sdkKey, user, { environment: { tier: "production" } }); await client.initializeAsync(); ``` If your app collects other relevant attributes (e.g., device type, region), pass them in the `user` object to improve experiment precision. ## Step 3 (Optional): Update User Info for Logged-in Users When a user signs in or creates an account, call the SDK’s updateUser method to attach userID and other logged-in attributes. This allows user-level experiments and gates to evaluate with the authenticated identifier. ### Example (JavaScript): ```javascript theme={null} const updatedUser = { ...user, // continue to send the deviceID! userID: loggedInID, email: signUpEmail }; // Update the user object await client.updateUserAsync(user); ``` Adding userID after login enables user-level experiments/gates to target and evaluate using userID. *Device-level* experiment evaluations remain based on stableID (or deviceID) and are not changed by adding userID, as long as you continue to pass that identifier as well! This preserves consistency of device-level bucketing. *** # Guided Tutorial Source: https://docs.statsig.com/guides/first-dynamic-config Step-by-step tutorial to create your first Statsig dynamic config, including building a flexible homepage banner that you can update without redeploying. Now that you have created your Statsig account, and perhaps even your [first feature](/guides/first-feature), this guide will help you make your app a bit more flexible with Dynamic Config. Whether it be for server-side rendered UI, rate limiting configurations, ranking systems, algorithms, etc. Dynamic Config can help you update configurations on the fly. This guide focuses on a client-side example, as it is easier to illustrate how Dynamic Configs work and can be used. You can and still should use Dynamic Config on the server side as well. In this guide, we will be building a front page banner - something that may show up on an ecommerce site to advertise an upcoming sale, for instance. Here, I will apply it to a stripped down version of the Statsig homepage. It will look something like this green banner when we are finished: UI for mac users **Codepen Example** This example is implemented in [a codepen](https://codepen.io/tore-statsig/pen/mdWEajo) where you can see the actual code, or fork it to quickly test your own Dynamic Configs. ### Step 1 - Create a new Dynamic Config Start by creating a new Dynamic Config. Dynamic Configs live under the "Feature Management" tab in the Statsig Console.Pick a name related to the set of variables this config will hold. Since we are building a homepage banner, let's call it "Banner Config": Create Dynamic Config Great! We have created an (empty) Dynamic Config. It returns the default value (an empty JSON object) whenever it is evaluated. For our example, we are going to add some options to populate a homepage banner. The end goal is to show a targeted banner to a few different sets of developers: those likely to use a .NET SDK, and those more likely to use a Swift SDK. Let's keep our targeting simple here: we want to promote the new .NET SDK to anyone that is browsing our site on the Windows operating system, and the new Swift SDK to anyone browsing on MacOS. That takes us to steps 2 and 3. ### Step 2 - Create Targeting Rule(s) and Conditions First, let's add the Window's rule. Select "Add Targeting", and then "Add new Rule": Add Rule Though Feature Gates and Dynamic Config share the same powerful user segmentation and targeting tools, they differ in what you return for those different sets of users. Feature Gates are intended to be super lightweight, and only return a boolean indicating the user matched one of the rules. Dynamic Configs, on the other hand, can return a different JSON blob of data for each rule! ### Step 3 - Update the Return Value for Users Matching Each Rule This is the rough schema we want to return: ```js theme={null} { text, backgroundColor, color, fontSize, isCloseable, } ``` Let's edit the return value for the windows rule, and fill it in. Under "Return" on the right hand side, hit "edit". Then paste the following: ```js theme={null} { text: 'New! Introducing the Statsig .NET SDK', backgroundColor: '#194b7d', color: 'white', fontSize: 14, isCloseable: false, } ``` Return Value Remember to hit "Save" in the bottom right. Config with one rule Let's repeat this process for people using Mac OS. Adding the same OS rule, but this time select "Mac OS X". Then, lets update the return value: ```js theme={null} { text: 'New! Introducing the Statsig Swift SDK', backgroundColor: '#197d4b', color: 'white', fontSize: 16, } ``` Once again, don't forget to click "Save" to apply these new rules to your config. Your Dynamic Config should now look something like this: Dynamic Config with two rules ### Step 4 - Call getDynamicConfig In Your App This is a specific guide for creating a Dynamic Config using the Statsig Javascript SDK - for more guides to calling the statsig SDKs in code, use the Statsig SDK [quickstart guides](/sdks/quickstart) where you can select which SDK you'll be using! You can also reference the code snippet you'll want to use for a particular dynamic config by clicking into the code snippet button on the dynamic config page and selecting the desired SDK dynamic config code snippet button Now let's use this Dynamic Config to create a different landing page experience for different sets of developers. Check out this [codepen](https://codepen.io/tore-statsig/pen/mdWEajo) to follow along. If you fork the code and put in the API Key from your Statsig project, you should be able to see the banner for your platform. After adding the SDK to the webpage via the [jsdelivr cdn](https://www.jsdelivr.com/package/npm/@statsig/js-client), we initialize the SDK: ```js theme={null} const client = new window.Statsig.StatsigClient("", {}); ``` Now, let's fetch our config and construct the banner: ```js theme={null} const bannerConfig = client.getDynamicConfig("banner_config"); const text = bannerConfig.get("text", null); const backgroundColor = bannerConfig.get("backgroundColor", "black"); const color = bannerConfig.get("color", "white"); const fontSize = bannerConfig.get("fontSize", 14); if (text == null) { return; } const banner = document.getElementById("homepageBanner"); const bannerText = document.createElement("p"); banner.style.display = "block"; bannerText.innerHTML = text; banner.style.color = color; banner.style.fontSize = fontSize + "px"; banner.style.backgroundColor = backgroundColor; banner.appendChild(bannerText); ``` Note that this js relies on the html page having the homepageBanner div: ```html theme={null}
``` And that's it! With just a handful of javascript, we integrated with the Statsig SDK and started using the Dynamic Config we created. Now, without updating our website, we can add a new rule to the Dynamic Config and return a completely new banner to a different set of people! Here's what it looks like for me, viewing this webpage in chrome on a Mac: UI for mac users We hope this inspires some ideas of what you could do with your app/website/backend service, and we can't wait to see what you build. # Build Your First Feature Source: https://docs.statsig.com/guides/first-feature Walk through creating your first Statsig feature gate, targeting specific audiences, and rolling out a new feature gradually with the JavaScript SDK. Statsig refers to feature flags as feature gates across the console and SDKs. The terms are interchangeable throughout this guide. Once your Statsig account is ready, follow the steps below to create and test-drive a new feature gate. Navigate to the [Feature Gates page](https://console.statsig.com/gates) and click Get Started (or Create if you already have gates). Feature Gates page with Create button Give the gate a clear name and description—for example, Mobile Registration with a note about the new mobile sign-up flow. New gates default to returning false until you add targeting. Click Add New Rule, choose Operating System → Any of, and select Android and iOS. Set the pass percentage to 100% and click Add Rule, then Save. Adding a mobile targeting rule Layer on a second rule for your team—for example Email → Contains any of with your company domain—so employees can exercise the feature regardless of device. Gate with mobile and email rules Head to [Project Settings → API Keys](https://console.statsig.com/api_keys) and copy the Client API key. Keep server secret keys on backends only, and use console API keys for programmatic configuration work. Statsig supports many platforms—see [Client SDK options](/sdks/getting-started) for alternatives. This walkthrough uses the browser SDK so you can experiment directly in DevTools. Paste the snippet below into the browser console on any site to fetch the SDK from jsDelivr: ```js theme={null} const script = document.createElement('script'); script.src = 'https://cdn.jsdelivr.net/npm/@statsig/js-client@3/build/statsig-js-client+session-replay+web-analytics.min.js'; document.head.appendChild(script); ``` Injecting the SDK via DevTools Replace YOUR\_SDK\_KEY with the client key from Step 4 and run: ```js theme={null} const client = new window.Statsig.StatsigClient('YOUR_SDK_KEY', {}); await client.initializeAsync(); ``` Then call: ```js theme={null} client.checkGate('mobile_registration'); ``` You should see false because the current session is not mobile and doesn’t use the employee email domain. Enable the mobile device toolbar in Chrome DevTools. Chrome DevTools mobile device toolbar icon Re-evaluate the user to pick up the new environment and re-check the gate: ```js theme={null} await client.updateUserAsync({}); client.checkGate('mobile_registration'); ``` The gate should now return true for the mobile profile. Gate returning true for mobile rule Switch DevTools back to the desktop view and update the user with a company email: ```js theme={null} await client.updateUserAsync({ email: 'teammate@statsig.com' }); client.checkGate('mobile_registration'); ``` The gate passes again thanks to the email rule. Gate returning true via email rule ```js theme={null} client.flush(); ``` Open the gate’s Diagnostics tab to confirm each exposure, including the failing desktop check, mobile pass, and employee pass. Diagnostics exposure stream showing recent checks ## Use the gate in production Wrap feature logic in a gate check so only targeted users see the experience: ```js theme={null} if (client.checkGate('mobile_registration')) { show(mobileRegistrationPage); } else { show(oldRegistrationPage); } ``` Happy feature gating! # Integrating Statsig with Framer Source: https://docs.statsig.com/guides/framer-analytics Integrate Statsig product analytics with Framer sites to autocapture events, track conversions, and run no-code A/B tests on Framer-built pages. ## Introduction Framer allows you to create interactive prototypes and websites with ease. By integrating Statsig, you can capture user behavior, and log custom events, directly within your Framer projects. This guide will walk you through integrating Statsig into your Framer project using the JavaScript SDK. ## Installation To start tracking user interactions with Statsig in your Framer project, follow these steps: 1. Copy your web snippet from Statsig, replacing "YOUR\_CLIENT\_KEY" with a Client API Key from your Statsig project, which you can find at [console.statsig.com/api\_keys](https://console.statsig.com/api_keys). ```html theme={null} ``` Go to your Framer project settings by clicking the gear icon in the top right corner - note you'll have to be on the "mini" site plan or above above to enable custom code. In the General tab, scroll down to the Custom Code section and paste your Statsig snippet into the end of `` tag section. Save your changes and publish your site, then Statsig will be ready to track basic user interactions. ## Capture Custom Events To track custom events within your Framer project, you can use window\.statsig.logEvent() inside a custom component. Go to the Assets tab in your Framer project. Click the plus icon next to Code and create a new code component. Name the file CaptureButton and select New Component. Replace the default code in the file with the following: ```js theme={null} export default function CaptureButton() { const handleClick = () => { window.statsig.logEvent("click", "button"); }; return ( ); } ``` Save your changes (Cmd/Ctrl + S). Drag your new CaptureButton component from the Code tab onto your Framer page, where you'd like it. After publishing your site and clicking the button, after a couple minutes you should see the event show up in your Statsig metrics. # Log your first custom event Source: https://docs.statsig.com/guides/logging-events How to log custom events to Statsig from client and server SDKs, including event names, values, and metadata used for metrics and analytics. The first step towards building better products is tracking events. As the saying goes, "If you can't measure something, you can't understand it. If you can't understand it, you can't control it. If you can't control it, you can't improve it." Regardless of if you plan to use Statsig for web or product analytics, or for experimentation, defining the key events and metrics for your product is the best place to start. Event data is consumable in [Metrics Explorer](/product-analytics/overview), can be turned into [custom metrics](/metrics/how-metrics-work), visualized in [dashboards](/product-analytics/dashboards), and used in [experiment results](/metrics/pulse) to show the impact of your experiments on the metrics you care about. We try to make this easy in a few ways: 1. Our web SDKs offer [autocaptured web analytics](/webanalytics/overview) to automatically log common events like pageviews, clicks, and more. 2. We offer [integrations](/integrations/introduction) to connect your existing event data in Segment, mParticle, and other sources. 3. All of our client and server SDKs provide a simple `logEvent` API to instrument your own events in your app or webserver. For general guidance on event logging and core concepts, read on or jump to the "Logging events via SDKs" section below. ## Identifying Users and the "StatsigUser" object Many analytics platforms have a concept of "identifying" a user. In Statsig, this is the StatsigUser object that is set a initialization time in client SDKs, or with each event in Server SDKs. The [`StatsigUser`](/concepts/user) is a set of properties that describe the user. It roughly has the same json definition across all SDKs and integrations: ```json StatsigUser Object theme={null} { "userID": "123", "customIDs": {}, "email": "user@example.com", "ip": "192.168.1.1", "userAgent": "Mozilla/5.0...", "country": "US", "locale": "en-US", "appVersion": "1.0.0", "custom": {}, "privateAttributes": [] } ``` The `userID` field is reserved for a unique identifier for the user. This should be the ID of the logged in user. `customIDs` are explained in "Group Analytics" below. The other fields are fairly self explanatory, and power not just targeting and evaluation for feature flags, but can also be used for custom metrics and data queries in metrics explorer. ## Group Analytics Another core concept in analytics is the idea of different "groups". For example, a user could belong to a certain company, organization, page, subreddit, facebook group, etc. In Statsig, groups are represented by the `customIDs` field. This is a dictionary that can contain multiple IDs for a single user. For example, a user could have the following customIDs: ```json Group Example theme={null} { "userID": "123", "customIDs": { "companyID": "456", "projectID": "abc" } } ``` Statsig computes all metrics at the user level, but also for each custom identifier. This means you can run experiments at a company level, where all users in a company will get the same experience, and compare the impact of company level metrics. You can also use metrics explorer to slice and dice metrics at the company level, rather than the user level, if you set up a customID for it. For example, in the Statsig project, I can look at the console page views at a user or company level in metrics explorer: Viewing Console Page Views at a User Level To set up a new "group" or "customID": 1. Go to your [project settings](https://console.statsig.com/settings) 2. Locate the "Custom IDs" section and hit "edit" CustomIDs in project settings 3. Name the new customID and give it a description so other people in your project know what it means 4. Start logging events with that customID in the `customIDs` field of the StatsigUser object You must provide the set of all IDs on each StatsigUser object. ## Best practices ### Set all known IDs on each StatsigUser object In order to generate metrics for each user/group, Statsig needs to know the set of all IDs on each StatsigUser object. Whether this is at client sdk initialization time, or when calling `logEvent` in a server SDK, you need to pass each identifier for the user for it to contribute to those metrics. If you use a logged out identifier for anonymous users, you should continue to pass that identifier even after the user logs in and you populate the logged in userID. This will enable any gates or experiments on the logged out identifier to continue to bucket the user appropriately, and calculate metrics at both the logged out and logged in level. ### Deduplicating custom events At the moment, Statsig does not provide a way to deduplicate custom events that you log. ### Naming conventions The most important aspect is consistency - you should pick a convention and stick with it. Your events and metrics catalogue can quickly become messy and difficult to navigate when there are similar events with different conventions. Technically speaking, Statsig can accept any type of name convention - E.g.; "Page View", "PageView", "Page-view" and "page\_view". Practically speaking, we recommend using "page\_view" — that is, to avoid spacing, and to use all lowercase characters. One thing to note, special characters cannot be used in event names. Statsig drops events that match this regex/contain this character set: `"\\[\]{}<>#=;&$%|\u0000\n\r"`. This will ensure there are no duplicate entries with different casing, and ensure other downstream systems connected via integration can accept and process the event properly given this is the most universally-compatible convention. ### What to measure In general, if you're doing normal experimentation/product analytics, you'll want events that correspond to user actions. How you classify them will depend on what you're trying to do. In general, keep the following in mind: 1. Events should not have high cardinality if you don't have a lot of users. Otherwise, experimental/metrics data will be too sparse. E.g.; an eCommerce website should not log a separate event name for every single product page. But instead, should have a generic "product\_page\_view" event and include contextual page information within the `optional_event_metadata` object. 2. If you have a well-defined user funnel, you'll want each step to be a separate event. 3. If you have a less defined user journey, you'll want to log generic events (eg. "page\_view"), and add more details to either the value field (which should be the primary dimension of interest), or the `optional_event_metadata` object. ## Logging events via SDKs All our SDKs provide a very simple API to log events. They look like this: ```js JavaScript theme={null} statsig.logEvent( event_name, optional_event_value, optional_event_metadata ); ``` Where event name describes a notable event in your product, like `sign_up`, `achievement_unlocked`, `add_to_cart`, `check_out`, etc. These events can optionally take an event value. For instance, you can use this field to specify the type of achievement that was unlocked, or the name of the product that was added to cart, or the price of the item that was purchased. In experiments, the "value" will generate an automatic breakdown of the event if there are fewer than 8 distinct values. And finally, the metadata field allows you to specify even more details about the event. Here's an couple examples of how a `logEvent` call looks like: ```js JavaScript Example theme={null} statsig.logEvent( 'add_to_cart', // Name 19.99, // Price { item_id: 'BC22010', cart_size: '2', user_segment: 'first_time_purchaser', } ); ``` ```csharp C# Example theme={null} StatsigClient.LogEvent( "level_completed", // Event Name 11, // Level number new Dictionary() { { "score", "452" } } ); ``` **Size limits on event payload** There are limits to how large each event field can be. Object fields have an overall limit of 4096 on its stringified length. String fields have a limit of 64 on its length. See the [SDK reference](/sdks/getting-started) for more details on logging events in the language of your choice. # Migrate your analytics data from Amplitude Source: https://docs.statsig.com/guides/migrate-from-amplitude Step-by-step guide to migrating product analytics from Amplitude to Statsig, including event mapping, user identity, and metric parity. Migrating from Amplitude to Statsig is a strategic choice. Statsig is an all-in-one platform that offers analytics, experimentation, and feature flagging under one umbrella. Using all these products in a single tool is much more powerful. Migrating amplitude data into Statsig usually involves two steps: export and ingest. This guide provides the essentials. For anything beyond these basics, please contact us. ## Step 1. Export your data from Amplitude Amplitude offers a few different export methods. Pick the one that matches your data size and setup: **1. S3 Export** For high volume backfills, you can dump your Amplitude data into an S3 bucket. **2. Warehouse Export** If your Amplitude data is already in Snowflake, BigQuery, or Redshift, you can skip file downloads. Statsig can ingest directly from these warehouses (see Step 2.1) **3. Export API** Use Amplitude's Export API to pull gzipped JSON * **Limit**: 4 GB per request so use hourly windows for large ranges * **Example**: ```bash theme={null} curl --location --request GET 'https://amplitude.com/api/2/export?start=&end=' \ -u '{api_key}:{secret_key}' ``` **4. UI Download (CSV/JSON)** Go to Organization Settings → Project → Export Data * Best for small datasets or initial testing ## Step 2. Transform your data Amplitude and Statsig store events in slightly different formats. To make your Amplitude exports work in Statsig, you'll need to map your Amplitude data to Statsig's format. This step is required irrespective of how you choose to import data into Statsig in the next step. | Amplitude field | Statsig field | | ------------------ | ---------------------------- | | `event_type` | `event` | | `event_time` | `timestamp` (ms since epoch) | | `user_id` | `user.userID` | | `device_id` | `user.stableID` | | `event_properties` | `metadata` | | `user_properties` | `user` fields | **Before transform** ```json theme={null} // Amplitude event { "event_type": "purchase", "user_id": "123", "device_id": "device_abc", "event_time": "2023-08-17T00:00:00Z", "event_properties": { "amount": 25, "currency": "USD" }, "user_properties": { "plan": "premium" } } ``` **After transform** ```json theme={null} // Statsig event { "event": "purchase", "user": { "userID": "123", "stableID": "device_abc", "plan": "premium" }, "timestamp": 1692230400000, "metadata": { "amount": 25, "currency": "USD" } } ``` ## Step 3. Import into Statsig Once your data look like Statsig events, you can start to bring them in. There are a few paths to import your data depending on how you exported: | If you exported from Amplitude via... | Import into Statsig using... | Best when... | | ------------------------------------- | ---------------------------- | ------------------------------------------------------- | | S3 export | S3 ingestion | You're backfilling large datasets | | Warehouse (Snowflake/BQ/Redshift) | Warehouse ingestion | Your Amplitude data already lives in a warehouse | | Export API | Event Webhook | You're moving a few days/weeks of data programmatically | | UI download (CSV/JSON) | Event Webhook | You're testing or moving a small slice of data | **S3 ingestion** * Just ensure the files are transformed to Statsig schema, in Parquet/JSON/CSV form, and then follow [Statsig's S3 ingestion](/data-warehouse-ingestion/s3) steps. * Please note that you need to shard your Amplitude raw data into 1 day's data per directory for to be able into Statsig **Warehouse ingestion** * [Connect your warehouse to Statsig](/data-warehouse-ingestion/introduction) * Point Statsig at a query that outputs events in the expected schema * Statsig ingests on a recurring schedule **UI download or Export API** The simple way to get these events into Statsig is to replay them through the [Event Webhook](/http-api#post-event-webhook). Think of it as a direct POST call: you take each row or JSON object, reshape it, and send it to Statsig one at a time or in small batches. This is best for test runs or initial migrations, not for millions of events. ```bash theme={null} curl -X POST https://api.statsig.com/v1/webhooks/event_webhook \ -H "Content-Type: application/json" \ -H "STATSIG-API-KEY: $STATSIG_SERVER_SECRET" \ -d '{ "event": "signup", "user": { "userID": "abc" }, "timestamp": 1692230400000 }' ``` ## Not sure where to start or need help? If you're unsure how to approach Amplitude migration, please reach out to our team. We have worked with ex-Amplitude customers closely in the past to offer them hands on migration support. We're always happy to discuss your team's individual needs or any other question you have - reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack). # LaunchDarkly Migration Guide Source: https://docs.statsig.com/guides/migrate-from-launchdarkly Step-by-step guide to migrating feature flags and rollouts from LaunchDarkly to Statsig, including flag mapping, SDK swap, and rule recreation. ## Overview Migrating from LaunchDarkly to Statsig is a strategic move. It can lead to efficient feature flag management and a stronger experimentation culture. By following this guide, you'll be well equipped to make the transition with confidence. We will cover the following topics in this guide: 1. Conceptual differences between LaunchDarkly and Statsig 2. Deciding what to migrate vs. not 3. Importing flags into Statsig 4. Flipping evaluation from LaunchDarkly to Statsig 5. How to run the migration process ## Conceptual differences between LaunchDarkly and Statsig It is important to understand a few fundamental differences in how LaunchDarkly and Statsig structure their feature management data models: **Environment**: LaunchDarkly treats environments as top level concept where flags and segments must be duplicated and managed separately across environments. In Statsig, we have a centralized model where flags/configs handle environment-specific logic in their targeting rules. **Flag types**: LaunchDarkly uses a mix of boolean, multivariate, and JSON flags. Statsig distinguishes between Feature Gates (boolean) and Dynamic Configs (typed multivariate configs with JSON values). **Targeting**: LaunchDarkly relies on Contexts to evaluate flags. Statsig evaluates based on what we call a StatsigUser object. #### Side by side comparison | LaunchDarkly concept | Can we migrate? | Statsig notes | | ------------------------------------ | --------------- | ---------------------------------------------------------------------------------------- | | Project | ✅ Yes | Convert to Project | | Environment | ✅ Yes | Convert to Environment (mark critical as production in Statsig) | | Boolean Flags | ✅ Yes | Convert to Feature Gates | | String, Number, and JSON Flags | ✅ Yes | Convert to Dynamic Configs | | Segments | ✅ Yes | Convert to Segments (Big ID list segments won't be imported) | | Targeting Rules | ✅ Yes | Convert to Rules | | Context kind | ✅ Yes | Convert to Custom Unit ID in Statsig | | Context attribute | ✅ Yes | Convert to Custom Fields in Statsig | | Flag owner, tags, teams, and history | ❌ No | Statsig does not preserve any metadata or historical versions of a flag during migration | #### User Context mapping example LaunchDarkly supports multi-kind, structured user contexts. Statsig requires a user object to achieve this. In Statsig, User ID or Custom ID is equivalent to LD's key. Known top-level fields in Statsig include userID, email, ip, userAgent, and custom. All other fields go under the custom object. **Example 1: LD User context to Statsig User object conversion** ```javascript theme={null} // 1 - LD User context { "kind": "user", "key": "user-key-123abc", "name": "Anna", "email": "anna@globalhealthexample.com", "organization": "Global Health Services", "jobFunction": "doctor", "device": "iPad" } // 1 - Statsig User object { "userID": "user-key-123abc", "email": "anna@globalhealthexample.com", "custom": { "name": "Anna", "organization": "Global Health Services", "jobFunction": "doctor", "device": "iPad" } } ``` **Example 2: LD Multi context kind to Statsig User object conversion** ```javascript theme={null} // 2 - LD Multi context kind { "kind": "multi", "user": { "key": "user_abc", "name": "Anna", "email": "abc@company.com", "region": "us-east" }, "org": { "key": "org_xyz", "tier": "enterprise" } } // 2 - Statsig user object { "userID": "user_abc", "email": "abc@company.com", "customIDs": { "org_id": "org_xyz" }, "custom": { "user_name": "Anna", "user_region": "us-east", "org_tier": "enterprise" } } ``` In Statsig, email is a top-level reserved field for the user object, so it should be placed directly as email (not user\_email). Statsig expects fields like userID, email, ip, and userAgent at the top level for user targeting and analytics. ## Deciding what to migrate vs. not Before you begin migrating flags from LaunchDarkly to Statsig, take this opportunity to audit your flags in the current system and do a cleanup. Many organizations accumulate flag debt over time - from stale flags, dead experiments, deprecated toggles, and legacy kill switches. Migration is a chance to start fresh with only what's valuable and active. Utilize filters such as 'Lifecycle' and 'Type' in LaunchDarkly to determine which flags are worth importing into Statsig. Below is a decision framework you can use to decide which flags to import into Statsig. Our migration script follows this framework by default but you can alter it if you need. Migration Decision Framework ## Importing flags into Statsig To import feature flags from LaunchDarkly to Statsig, you can use our official import tool which is designed for this specific purpose. The import tool fetches flags from LaunchDarkly, translates them into Statsig's format, and creates corresponding feature gates in Statsig. Additionally, it tracks the migration status and details in a CSV file. There are two ways to invoke this tool: 1. **[Open source script](/guides/open-source-script) (Recommended)** - This is a good option if you want to customize the integration logic. It will also spit out a CSV of all of your LaunchDarkly flags, along with migration status and relevant URLs to the flag in LaunchDarkly and the gate in Statsig. This imports all of your environments. 2. **[Statsig console](/guides/ui-based-tool)** - UI-based wizard to help you import LaunchDarkly feature flags and segments into Statsig. It will tell you which gates and segments were migrated and which weren't. It only imports the "production" environment at the moment. > 👉 If you are migrating from a different system, you will need to recreate flags manually in Statsig. Ideally you have done the cleaning in the previous step, so you will migrate a small number of flags. If you need assistance, please reach out over email or slack. ## Flipping evaluation from LaunchDarkly to Statsig Once your flags have been imported into Statsig, the next step is to flip evaluation logic in your application. Instead of replacing every LaunchDarkly flag evaluation with Statsig calls, we recommend introducing a wrapper with gradual migration capabilities. This allows you to run both systems in parallel, compare outputs, and gradually switch over with minimal risk. The wrapper approach provides several key benefits: * **Parallel execution**: Run both systems simultaneously to validate behavior * **Gradual rollout**: Migrate flags one at a time or by percentage * **Easy rollback**: Quickly revert to LaunchDarkly if issues arise * **Consistent interface**: Maintain existing application code structure #### Implementation Guide **1. Before migration: LaunchDarkly Evaluation** Here's what a typical LaunchDarkly setup might look like: ```javascript theme={null} import { LDClient } from 'launchdarkly-js-client-sdk'; const ldClient = LDClient.initialize('client-key', { key: 'user_abc', custom: { plan: 'pro' } }); ldClient.on('ready', () => { const isNewHomepageEnabled = ldClient.variation('new_homepage_flag', false); const buttonColor = ldClient.variation('button_config', 'gray'); }); ``` **2. Create the Migration Wrapper** Create a comprehensive wrapper (`featureWrapper.js`) that handles both systems. The wrapper should check Statsig first, and if unavailable, fallback to LaunchDarkly: ```javascript theme={null} import { getLDClient } from './launchdarklyService'; import { getStatsigClient } from './statsigService'; export const wrapperFlags = { // For boolean flags wrapperGetFlag(flagKey, defaultValue = false) { const statsigClient = getStatsigClient(); if (statsigClient) { return statsigClient.checkGate(flagKey); } const ldClient = getLDClient(); if (ldClient) { return ldClient.variation(flagKey, defaultValue); } return defaultValue; }, // For dynamic configs (string, number, or json flags) wrapperGetConfig(user, configKey) { const statsigClient = getStatsigClient(); if (statsigClient) { const config = statsigClient.getConfig(user, configKey); return { get: (paramKey, defaultValue) => config.get(paramKey, defaultValue) }; } const ldClient = getLDClient(); if (ldClient) { return { get: (paramKey, defaultValue) => { const ldKey = `${configKey}.${paramKey}`; return ldClient.variation(ldKey, defaultValue); } }; } return { get: (_key, defaultValue) => defaultValue }; } }; ``` **3. Refactor application code to use the Wrapper** Once the wrapper is in place, you'll want to route all flag checks through it. The application logic itself doesn't change, only the mechanism by which flags are retrieved. ```javascript theme={null} import { wrapperFlags } from './featureWrapper'; const user = { userID: 'user_abc', custom: { plan: 'pro' } }; // Boolean gate if (wrapperFlags.wrapperGetFlag('new_homepage_flag', user)) { showNewHomepage(); } // Multivariate config const buttonConfig = wrapperFlags.wrapperGetConfig('button_config', user); const buttonColor = buttonConfig.get('color', 'gray'); ``` **4. Validate and gradually cut over** Once you've validated that Statsig is working as expected and that your migrated flags are returning correct values, you can begin migrating more flags. We recommend that you repeat the above steps for 2-3 engineering teams to instill confidence that different use cases are covered. After you've maintained flags in both systems long enough to ensure things are working as expected, phase out LaunchDarkly. 1. **Remove LaunchDarkly fallback from the wrapper** - Update the wrapper functions to rely solely on Statsig. This simplifies the logic and ensures LaunchDarkly is no longer queried in production. 2. **Delete LD initialization logic and SDK imports** - Any references to `LDClient.initialize` or `ldClient.variation` can now be safely removed. ## How to run the migration process To ensure a safe and manageable transition to Statsig, we recommend a phased rollout that each team can adopt independently. This approach allows for gradual migration, scoped validation, and shared learnings across the org. | Phase | Description | Who | Duration | | ------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | --------------------------- | | 1. Auditing existing flags in LaunchDarkly | Each team reviews their LaunchDarkly flags and identifies which flags are worth migrating | Individual Teams | Ongoing (2-3 days per team) | | 2. Start creating all new flags in Statsig | Starting immediately, all new flags to be created in Statsig to avoid using legacy system out of habit | Org-wide | 1 week | | 3. Pilot migration with one team | Select one team to migrate a small set of LaunchDarkly flags to Statsig using the wrapper. Validate that migrated flags work as expected | Pilot team | 1–2 weeks | | 4. Org-Wide migration and cutover | Repeat migration for more teams in waves. Create training docs and guides for org wide adoption. Once Statsig is stable and adopted org-wide, remove LaunchDarkly fallback from wrapper | All teams + central guidance | 3–4 weeks (rolling) | We hope this guide helped you better understand how to approach LaunchDarkly to Statsig feature flag migration. If you need any further assistance or just want to talk through your specific case, please talk to us and we'll happy to work with you. # Migrate your analytics data from Mixpanel Source: https://docs.statsig.com/guides/migrate-from-mixpanel Step-by-step guide to migrating product analytics from Mixpanel to Statsig, including event mapping, user identity, and dashboard recreation. # Mixpanel Migration Guide Switching from Mixpanel to Statsig is a smart move for teams seeking a unified platform that combines analytics, experimentation, and feature flagging. This all-in-one approach empowers faster, more informed decisions without data silos. Migrating Mixpanel data into Statsig usually involves three steps: export, transform, and ingest. This guide provides the essentials. For anything beyond these basics, please contact us. We will only cover importing raw events into Statsig. We don't support importing user/group profiles, dashboards, reports, etc. Once your raw events are in the Statsig project, you can re-create your critical dashboards here. ## Step 1. Export your data from Mixpanel Mixpanel offers a few different export methods. Pick the one that matches your data size and setup: **1. CSV Export** Export small batches of events as CSV via the Events tab → query events → click "Export" button. **2. Export API** Use Mixpanel's [Raw Event Export API](https://developer.mixpanel.com/reference/raw-event-export) to pull JSONL data: * **Limit**: Export one day's data at a time for optimal performance * **Format**: JSONL where each line is a valid JSON object ```bash theme={null} curl --location --request GET 'https://data.mixpanel.com/api/2.0/export?from_date=2023-01-01&to_date=2023-01-01' \ -u '{project_id}:{service_account_secret}' ``` **3. Data Pipelines (Bulk Export)** For large data volumes, use Mixpanel's [Data Pipelines](https://docs.mixpanel.com/docs/data-pipelines) feature to export to: * Cloud Storage (AWS S3, Google Cloud Storage, Azure Blob Storage) * Data Warehouse (BigQuery, Redshift, Snowflake) ## Step 2. Transform your data Mixpanel and Statsig store events in slightly different formats. Map your Mixpanel data to Statsig's format: | Mixpanel field | Statsig field | | ------------------------------------------------ | ---------------------------- | | `event` | `event` | | `properties.time` | `timestamp` (ms since epoch) | | `properties.distinct_id` or `properties.user_id` | `user.userID` | | `properties.device_id` | `user.stableID` | | `properties.*` (other fields) | `metadata` | **Before transform** ```json theme={null} // Mixpanel event { "event": "Signed up", "properties": { "time": 1618716477, "distinct_id": "user-123", "device_id": "xyz", "Referred_by": "Friend", "URL": "website.com/signup" } } ``` **After transform** ```json theme={null} // Statsig event { "event": "Signed up", "user": { "userID": "user-123", "stableID": "xyz" }, "timestamp": 1618716477000, "metadata": { "Referred_by": "Friend", "URL": "website.com/signup" } } ``` ## Step 3. Import into Statsig Once your data looks like Statsig events, you can start bringing them in: | If you exported from Mixpanel via... | Import into Statsig using... | Best when... | | --------------------------------------- | ------------------------------------------------------------- | ------------------------------------------------------- | | S3 export | [S3 ingestion](/data-warehouse-ingestion/s3) | You're backfilling large datasets | | Warehouse (Snowflake/BigQuery/Redshift) | [Warehouse ingestion](/data-warehouse-ingestion/introduction) | Your Mixpanel data already lives in a warehouse | | Export API | [Event Webhook](/http-api/overview) | You're moving a few days/weeks of data programmatically | | CSV download | [Event Webhook](/http-api/overview) | You're testing or moving a small slice of data | **Event Webhook (for API/CSV exports)** ```bash theme={null} curl -X POST https://api.statsig.com/v1/webhooks/event_webhook \ -H "Content-Type: application/json" \ -H "STATSIG-API-KEY: $STATSIG_SERVER_SECRET" \ -d '{ "event": "Signed up", "user": { "userID": "user-123", "stableID": "xyz" }, "timestamp": 1618716477000, "metadata": { "Referred_by": "Friend", "URL": "website.com/signup" } }' ``` **Important notes:** * **S3 ingestion**: Shard your Mixpanel data into 1 day's data per directory for Statsig * **Scale gradually**: After small tests, backfill in chunks to manage loads * **Future tracking**: After historical import, switch Mixpanel code calls to Statsig SDKs ## Not sure where to start or need help? If you're unsure how to approach Mixpanel migration, please reach out to our team. We have worked with other Mixpanel customers in the past to help them switch over to Statsig. We're always happy to discuss your team's individual needs or any other question you have - reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack). # Migration Overview Source: https://docs.statsig.com/guides/migration-overview Overview of migration paths to Statsig from other feature flagging, experimentation, and product analytics platforms with planning checklists. Statsig combines feature flags, experimentation, and product analytics in one platform. Migrating ensures your data, flag, experiments, and decision workflows live in a single source of truth. This guide outlines the overall migration process. For provider-specific steps (Amplitude, LaunchDarkly, etc.), see our dedicated guides. ## Migration Phases ### 1. Audit & Plan * Identify the datasets, events, and feature flags you want to move * Decide what needs full historical backfill vs. what can start fresh * Document any dashboards or KPIs that need rebuilding in Statsig ### 2. Set Up Live Data * Implement Statsig SDKs to start streaming new events and feature flag evaluations * Validate critical events are firing with the correct schema * Use Statsig for the newly recorded events and flags ### 3. Import Historical Data * Export data from your existing tool (S3 or warehouse is preferred) * Transform the schema to Statsig's event format (`event`, `user`, `timestamp`, `metadata`) * Import via Statsig's Event Webhook, S3 ingestion, or warehouse ingestion ### 4. Validate & Decommission * Compare metrics between your legacy tool and Statsig to ensure parity * Rebuild dashboards and charts in Statsig * Decommission old pipelines once Statsig is your single source of truth ## Best Practices * **Start small**: Run a pilot project or test migration before backfilling all history * **Align IDs early**: Ensure `userID` and `stableID` mapping is consistent. Identity mismatches are the most common failure point * **Shard historical imports**: Break large datasets into daily partitions for stability * **Rebuild insights intentionally**: Don't port all events and flags directly. Use migration as a chance to clean up stale data * **Plan change management**: Teams need time to adjust workflows, queries, and dashboards so migrate for 1-2 teams before championing in the broader org ## Provider-Specific Guides * [Migrate from Amplitude](/guides/migrate-from-amplitude) * [Migrate from Mixpanel](/guides/migrate-from-mixpanel) * [Migrate from LaunchDarkly (Feature Flags)](/guides/migrate-from-launchdarkly) * Additional guides coming soon ## Get Help Our team has helped many customers move off other tools so that everything is in Statsig. For tailored guidance, reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack). # Open Source Script Source: https://docs.statsig.com/guides/open-source-script Use the open-source Statsig migration scripts to move feature flags, experiments, and metrics from third-party platforms into your Statsig project. [This package](https://github.com/statsig-io/migrations) is designed to help automate migration of feature flags from LaunchDarkly to Statsig. It fetches feature flags from LaunchDarkly, translates them into Statsig's format, and creates corresponding feature gates in Statsig. ## Considerations This script should work out of the box. It's recommended you start with a test environment of 5-10 flags. However, before running the script on a large scale, consider the following: * **IMPORTANT**: If you don't need to customize this import script, you can just use [Statsig's in-console tool](/guides/ui-based-tool) * The script uses a tag `Imported from LaunchDarkly` to identify migrated flags in Statsig. Ensure this tag is unique and recognizable. * The script includes a function to delete all Statsig feature gates with a specific tag. Use this with caution to clean up after a test or failed migration. * The script requires API keys for both LaunchDarkly and Statsig, which should be kept secure. ## Installation To run the script, you need Node.js and npm installed on your system. You can execute directly: ```bash theme={null} npx @statsig/migrations --from launchdarkly --launchdarkly-project-id default ``` ## Configuration * Provide your [LaunchDarkly API key](https://docs.launchdarkly.com/home/account/api) and [Statsig Console API key](/console-api/introduction) in the script * Map LaunchDarkly environments to Statsig environments that aren't already the same by using `--environment-name-mapping` to the script. * Map LaunchDarkly context kind to Statsig's custom unit ids and custom fields. You can find the detailed instructions for these steps on the [Github repo](https://github.com/statsig-io/migrations). ## Running the Script To execute the migration script, run the following command in your terminal: ```bash theme={null} node index.js ``` The script will perform the following actions: 1. Fetch all feature flags from LaunchDarkly. 2. Translate each flag into Statsig's format. 3. Create feature gates in Statsig. 4. Write the migration status and details to a CSV file named `flag_migration_tracker.csv` ## Example Translations The following examples show how LaunchDarkly feature flags are translated into Statsig feature gates: **Example 1:** LaunchDarkly flag with email and name targeting conditions Example 1 - LaunchDarkly to Statsig translation **Example 2:** LaunchDarkly flag with country-based targeting (off flag) Example 2 - LaunchDarkly to Statsig translation ## Troubleshooting If you encounter issues during the migration, check the following: * Ensure that the API keys are correct and have the necessary permissions. * Review the error messages in the console for clues on what might have gone wrong. Pull requests and feedback welcome! # Pre-commit Webhooks Source: https://docs.statsig.com/guides/pre-commit-webhooks Configure pre-commit and pre-publish webhooks in Statsig to enforce review checks, run validations, or trigger CI workflows before changes are saved. Pre-commit Webhooks allow you to integrate external validation into your change approval workflow. This enables you to: * Route changes through internal approval systems * Run automated tests before changes go live * Enforce custom business logic or compliance checks * Integrate with CI/CD pipelines ## How It Works When a Statsig user submits a change for review and it's approved by a reviewer, clicking "Commit Changes" triggers your configured webhook. Statsig sends the change details to your endpoint, which then validates the change (e.g., runs tests, checks policies). Your system responds to Statsig with either an approval or rejection, determining if the user can finally commit the change or if it remains blocked. ## Setup Instructions ### Step 1: Configure Your Webhook Endpoint Your webhook endpoint should: * Accept POST requests from Statsig * Process the change payload (see format below) * Call the Change Validation API to approve or reject See the [Change Validation API documentation](/console-api/change-validation/) for implementation details. ### Step 2: Configure Statsig Console Settings Navigate to **Settings** → **Product Configuration** → [**General**](https://console.statsig.com/settings/products): 1. **Webhook URL**: The endpoint where Statsig will send change notifications 2. **Webhook Key**: A secret key sent in the `x-statsig-webhook-key` header for authentication 3. **Initial Message**: A message shown to users while validation is in progress * Example: "Your change is being validated by our CI/CD system. This may take up to 2 minutes..." Webhook URL and initial message configuration Webhook verification key configuration ## Usage Instructions ### Step 3: Send a Webhook payload Every payload will have these fields at a minimum: ```json theme={null} { "review_id": "string", // Required for API response "submitter": "user@example.com", // Who created the review "committer": "user@example.com", // Who is committing the change "config_type": "gate | dynamic_config | segment | experiment", "config_name": "string", // For gates, configs, segments "experiment_name": "string", // For experiments "type": "string" // Change type (see below) } ``` Currently, only changes to gates, dynamic configs, segments, and experiments trigger pre-commit webhooks, and only the following changes: * `rules`: Updating rules (for gates, dynamic configs, segments) * will contain payloads `old_config` and `new_config` with the same format returned by the console API for the entity type (e.g. [/api-reference/gates/read-gate](/api-reference/gates/read-gate) for gates) * `update_target_apps`: Updating target applications * `update_allocation`: Changing the pass percentage of an ongoing experiment * `start_experiment` * `ship_experiment` * `abandon_experiment` * `update_experiment_settings`: Changing settings of an ongoing experiment * will contain payloads `old_experiment` and `new_experiment` with the same format returned by the [Console API](/console-api/introduction) ### Step 4: Respond to Webhooks Your system must call the Change Validation API to approve or reject the change: **Endpoint:** `POST https://statsigapi.net/console/v1/change_validation` When you respond to validation, you can also set the `message` field to provide a message to the user. You can set the message after the validation is done or provide progressive updates while the validation is running. ```json theme={null} { "reviewID": "review_123", "message": "⏳ Running tests... (45/150 complete)" } ``` **What Users See** When a review is pending validation, users see: * **Title**: "Changes to this config have been reviewed and are pending validation" * **Description**: * The company-level `precommit_webhook_message` (if set in console settings) * The review-specific `precommit_message` (if your webhook has set it) ## Bypassing Validation Project Admins can grant the "Bypass Pre-commit Webhook" permission to specific roles. Users with this permission will see a "Commit Changes (Bypass Validation)" option, allowing them to skip the webhook validation process entirely. ## Best Practices 1. **Respond quickly**: You have 2 minutes before the request times out 2. **Include debug links**: Help users troubleshoot failures quickly 3. **Use progressive updates**: Keep users informed during long validations 4. **Set a helpful initial message**: Explain what's happening and expected wait time 5. **Handle errors gracefully**: If your validation system is down, consider auto-approving or notifying users # Using Private Attributes Source: https://docs.statsig.com/guides/private-attributes Use private attributes in Statsig SDKs to evaluate feature gates and experiments without sending sensitive user data to Statsig servers. ## Evaluating feature gates, dynamic configs, segments, and experiments without logging user data to Statsig We take privacy, and the privacy of your user data, very seriously. If you have legal requirements that prevent you from sending PII to third parties, or you are just uncomfortable sending PII to a third party service, it is still possible to use Statsig for feature gating, configs, or experiments. ## How It Works Any field you wish to evaluate on can be made private. Note that if you make the userID field private, your experience in the Statsig console will be broken (the user's tab, metrics charts, event logs, and pulse metrics will be unable to populate accurately). If you wish to hide the UserID, we recommend you use a stable, one-way hash to maintain the ability to analyze user behavior. For example, let's say you want to use an email condition: 1. Create a feature gate with a condition that passes on certain emails or domains 2. Pass `email: email@domain.com` in `privateAttributes` rather than in the top level `email` field 3. Test it out in the `Test Gate` console after saving changes Private attributes When evaluating rules and conditions, Statsig first checks for fields at the top level, then in the custom attributes, and finally, in the private attributes. By simply not passing email at the top level, the evaluator will keep looking in other possible places before evaluating the condition (`custom`, and then `privateAttributes`). So if you want to keep the ip address and user agent private, but still use browser or IP checks in the console, simply put them in the `privateAttributes` dictionary instead of at the top level of the user. The same goes for country, locale, custom fields, and so on. Our statsig-node SDK illustrates how this works (and is exactly how we evaluate gates on our servers as well): [https://github.com/statsig-io/node-js-server-sdk/blob/d1cb9431fb68b40f840254fce70363de1dc51aa5/src/Evaluator.js#L374](https://github.com/statsig-io/node-js-server-sdk/blob/d1cb9431fb68b40f840254fce70363de1dc51aa5/src/Evaluator.js#L374) Do not provide a `privateAttribute` key anywhere else in the `user` object. The entire `privateAttributes` dictionary is dropped, but any duplicate fields at the top level or in the custom object will still be logged (and evaluated against) ## Client vs Server SDKs Client/single user environment SDKs send the user object with the `initialize` call to evaluate the user against every gate in your Statsig project. *privateAttributes will be sent with this call but will not be stored or logged on Statsig servers*. In order to evaluate the conditions and rules for a gate, config, or experiment, Statsig servers need to know the privateAttributes field. It will be stripped from the users for any logging on Statsig servers. Client SDKs will remove privateAttributes before any events are logged - they are only needed for gate evaluation. Example with the `@statsig/js-client` SDK: [https://github.com/statsig-io/js-client-monorepo/blob/17fb70f1bea00d07e156cf6ff03b8024bbc1b197/packages/client-core/src/EventLogger.ts#L325](https://github.com/statsig-io/js-client-monorepo/blob/17fb70f1bea00d07e156cf6ff03b8024bbc1b197/packages/client-core/src/EventLogger.ts#L325) If this does not meet your needs, our Server SDKs are able to make a stronger guarantee - `privateAttributes` will never leave your server. Statsig server SDKs download the definition of each gate/config/experiment and evaluate them locally. `privateAttributes` stripped from event logs with the `statsig-node` SDK: [https://github.com/statsig-io/node-js-server-sdk/blob/d1cb9431fb68b40f840254fce70363de1dc51aa5/src/LogEvent.js#L21](https://github.com/statsig-io/node-js-server-sdk/blob/d1cb9431fb68b40f840254fce70363de1dc51aa5/src/LogEvent.js#L21) Evaluation happening locally to the server on `privateAttributes` in the `statsig-node` SDK: [https://github.com/statsig-io/node-js-server-sdk/blob/d1cb9431fb68b40f840254fce70363de1dc51aa5/src/Evaluator.js#L374](https://github.com/statsig-io/node-js-server-sdk/blob/d1cb9431fb68b40f840254fce70363de1dc51aa5/src/Evaluator.js#L374) Don't just take our word for it - all of our SDKs are open source and [available on github](https://github.com/statsig-io). Feel free to dive in to the implementation of `privateAttributes` in the SDK you are using, or reach out to us on [slack](https://www.statsig.com/slack) and we can point you in the right direction. To ensure that user PII is never transmitted over the network back to Statsig during Client SDK initialization, you should use [Client Boostrapping](/client/concepts/initialize#bootstrapping-overview) and provide the `privateAttributes` as part of the user object on the server to the `getClientInitializeResponse()` call. This will generate all of the assignments locally on your server, and these assignments can then be passed as `initializeValues` to the client SDK, negating the need to send any user attributes from the client device to Statsig. ## Event Logging The privateAttributes field will be stripped from the user object for any `logEvent` calls on client or server SDKs. On server SDKs, you can simply not provide that field for `logEvent` calls if you wish - no evaluation is happening, so they are not necessary. If you are using the same user everywhere, the SDK will handle dropping the `privateAttributes` for you. # Email AB Testing with SendGrid Source: https://docs.statsig.com/guides/sendgrid-email-abtest Run email A/B tests with Statsig and SendGrid by logging exposure events from sends to compare open, click, and conversion metrics across variants. Email campaigns are a critical tool for any Marketing team. Finding the best performing Email template is a perfect use-case for an A/B test. Statsig allows you to run simple but powerful A/B tests on different parts of your email content. Since Statsig can integrate seamlessly with product analytics, you can run email experiments and understand deeper business level impact easily. This guide assumes you have an existing Statsig account. Please go here to create a new free account if you don't already have one: [https://statsig.com/signup](https://statsig.com/signup) ### Step 1: Create an experiment Start by creating a new Experiment on Statsig console. Put in a name and leave the rest of the fields empty/default. For the purposes of this walkthrough, that should do. Experiment creation interface ### Step 2: Start the experiment Since you can't start an experiment without a parameter, let's go ahead and add a dummy parameter. Experiment parameter configuration Experiment setup completion Save the experiment setup and **Start** it. We're all set with the experiment set up. Experiment start confirmation While you're at it, copy the **Experiment Name**. We'll use this in a bit. Experiment name copy interface ### Step 3: Set up Webhook In your SendGrid console, go to **Settings** -> **Mail Settings** -> **Event Webhook**. In the HTTP Post URL, put in: `https://sendgrid-webhook.statsig.workers.dev/?apikey=[YOUR STATSIG API KEY]` You can find your API Key by navigating to Statsig Project Settings -> API Keys, and copying the 'Client API Key'. Statsig API key location It should look like this: `client-abcd123efg...` Make sure all the **Deliverability Data** and **Engagement Data** checkboxes are checked. Next, Enable the **Event Webhook Status** and hit Save. The set up should look like this: SendGrid webhook configuration ### Step 4: Create Single Sends Now on your SendGrid app, create two new **Single Send** and name them using the experiment-name like this. The first one would be the "Control", which is the baseline. That one should be named `[experiment_name]/control`. For example, in our case it will be `drip_campaign_ab_test/control`. Control email template setup The second one would be the "Test", which is the template you are comparing with the baseline. That one should be named `[experiment_name]/test`. For example, in our case it will be `drip_campaign_ab_test/test`. Test email template setup You can customize these templates however you want, and even use different subjects. The most important thing here is you need to split the recipient list evenly between the Control & Test. This will aid in a 'balanced' experiment. In order to avoid introducing any bias, it is best to split the recipient list at random. For instance, you want to ensure recipients within the same company are distributed evenly between the two lists. Now you're good to go. Send those emails and Statsig will automatically track how well each variant in your A/B test is performing across email opens, clicks, etc. ## Monitoring the set up When you've started the sends, you can verify everything is working as expected by navigating to the Diagnostics Tab in your experiment and looking at the 'Exposure Stream' at the bottom of the page. This shows a realtime stream of the page loads along with the variant they were allocated. Experiment exposure stream in diagnostics tab ## Interpreting results By going to the **Pulse Results** tab in the Experiment page, you can add metrics you want to monitor and verify which variant is doing better. To learn more about reading Pulse Results, check this article out: [Reading Pulse Results](/experiments/interpreting-results/read-results). Pulse results tab showing experiment metrics ## Using API instead of Single Send Statsig also supports A/B testing when using API or Automation to send marketing emails. In order to enable this, you would use unique arguments ([https://docs.sendgrid.com/for-developers/sending-email/unique-arguments](https://docs.sendgrid.com/for-developers/sending-email/unique-arguments)) and pass in unique\_args as below: ```json theme={null} { "unique_args": { "statsig_experiment_name": "[Experiment Name]", "statsig_variant_name": "[control or test]" } } ``` So in our example above, you will set up the Control variant like this: ```json theme={null} { "unique_args": { "statsig_experiment_name": "drip_campaign_ab_test", "statsig_variant_name": "control" } } ``` And the Test variant would look like this: ```json theme={null} { "unique_args": { "statsig_experiment_name": "drip_campaign_ab_test", "statsig_variant_name": "control" } } ``` ## More than two variants It's simple to extend this setup to run ABC or ABn tests. You can add more variants in the Experiment Setup tab, like below. Make sure the variant name is correctly applied in either the Single Send name or the unique arguments in the API. ABC test variant configuration interface # Setting up Reviews for Team Workflows Source: https://docs.statsig.com/guides/setting-up-reviews Set up review workflows in Statsig to require approvals for feature gate, experiment, and dynamic config changes before they reach production users. You can enable reviews for all Statsig resources such as feature gates, dynamic configs, segments, and experiments that you'll likely deploy to a production environment. ### Turning on Change Reviews for a Project As a Project Admin, you can configure your project to require reviews for any changes. To enable reviews for your project, navigate to the **Project Settings** page, switch to the Reviews tab and toggle this on. Project settings reviews configuration interface * You can optionally allow different roles to bypass the review requirement and self-approve review requests by customizing the permissions available to user roles: User role permissions configuration screen * Now when you make any configuration changes, say to a feature gate or experiment, you'll be asked to **Submit for Review**; you can add reviewers when you submit the change for review Submit for Review modal prompting for reviewer selection * Reviewers will now see a notification on the Statsig console as shown below. When they click on **View Proposed Changes**, they will see a diff of the *current version* in production and *new version*. Reviewers can now **Approve** or **Reject** the submitted changes. proposed changes example review ### Teams To create a predefined group of reviewers, you can create Teams Teams creation interface You can now use these predefined **Teams** when you submit any changes for review. Team selection for review submission ### Enforcing Team Reviews You can restrict who can make changes to your Project by (a) turning on **Reviews Required** for your Project and (b) adding designated **Teams** or **Reviewers** when you create the Feature Gate or Experiment. For (a), see section **Turning on Change Reviews for a Project** to turn on project-wide reviews. For (b), as an owner of a Feature Gate or Experiment, you can add designated **Teams** or **Reviewers** at any time as shown below. This ensures that only these designated groups or members can review and approve any subsequent changes. When another member now tries to edit these designated review groups/reviewers, this will require approval from currently designated reviewers. Team review configuration settings Teams and reviewers selector for enforcing approvals ### Configuring Review Settings for Different Environments Many teams build, test, and launch new features and experiments across multiple development environments. Statsig makes creating and using environments in feature launches easy via our [Environments support](/guides/using-environments#configuring-environments). You can also configure which environments require reviews via your **Project Settings**. To do so, go to **Project Settings** → [**Keys & Environments**](https://console.statsig.com/BPJcDV1K1g87fTib5ZEMk/api_keys) → tap **Edit** on **Environments**. By default if you have turned on "Reviews Required" for your Project, reviews will be required for Production, but not non-Production (lower) environments. Environments settings showing environments and order Manage environments dialog with environment list and review settings #### Team-based Required Reviews per Environment You can assign specific teams as reviewers for each environment. This ensures that only designated team members can approve changes for that environment. Assigning teams as reviewers for specific environments #### Code Freeze Use Case During code freeze periods, you can prevent feature flags or configs from being deployed to production by assigning a dedicated code freeze team as the production reviewer. This ensures that only members of that team (such as your SRE team or designated code freeze owners) can approve production changes. Once the code freeze period ends, you can remove the team assignment to restore normal review workflows. # A/B Testing on Shopify Source: https://docs.statsig.com/guides/shopify-ab-test Run A/B tests on a Shopify storefront with Statsig, including adding the SDK to your theme, defining variants, and tracking order conversion metrics. ## Use cases & considerations Shopify provides solutions for commerce businesses to build and manage all aspects of their online storefront, including product catalogue, inventory, site content, marketing, and user experience. For experimenting with the more static aspects of the store experience (static landing pages, visual aspects), we recommend using [Statsig Sidecar](/guides/sidecar-experiments/introduction) to both build your test treatments and to assign users to experiments when they land on your site — all without writing any code. Customers looking to experiment on the more dynamic aspects of their online store (ie; your product catalogue, search capabilities) should use [Shopify Headless Commerce](https://www.shopify.com/plus/solutions/headless-commerce) and integrate our [SDKs](/sdks/getting-started) to unlock full control for experimenting within business logic. ## Using Traditional Shopify + Sidecar for No-code testing The traditional Shopify service is a fully-managed platform for businesses that provides both a backend administration tool for managing your product catalogue & site content, and powers the storefront experience for your shoppers. While Statsig does not have an integration in the Shopify App Store today, you can easily integrate Sidecar for running simple UX experiments on the storefront. The below steps will guide you through the process of setting up Sidecar within the traditional Shopify stack. #### Install the Visual Editor Chrome extension [Follow the Visual Editor setup guide](/guides/sidecar-experiments/sidecar-v3#2-installing-the-chrome-extension-for-editing) to install the Statsig Visual Editor Chrome extension. This simple, lightweight Chrome extension will allow non-technical users to build experiments and their treatments. You can easily indicate where the test should run based on URL, and then configure treatments such as content changes, style changes, image swaps, as well as injecting arbitrary JavaScript for more sophisticated use-cases where the visual editor tools cannot accommodate. #### Add the Visual Editor script to your Storefront's page source * Log in to your Shopify dashboard * click on **Online Store**, then Themes * locate and click the more menu \[...], find the **Edit Code** option statsig product overview * Copy the Visual Editor script tag below, replacing `client-xxx` with your Client SDK key from [Settings > Keys & Environments](https://console.statsig.com/api_keys): ```html theme={null} ``` * Navigate to the `theme.liquid` file in your Shopify theme editor * Paste the Visual Editor script tag toward the top of the page `` as shown below. statsig product overview * Save your `theme.liquid` file. * Sidecar is now installed across your entire website 🎉. ### Configure event tracking Shopify's [Custom Pixel framework](https://help.shopify.com/en/manual/promoting-marketing/pixels/custom-pixels) is ideal for tracking customer events to Statsig. The custom pixel framework offers a [wide set of events](https://shopify.dev/docs/api/web-pixels-api/standard-events) you can subscribe to, and namely, the ability to perform tracking during the checkout experience. Note that code deployed outside the scope of a custom pixel will not fire during checkout experience as documented [here](https://help.shopify.com/en/manual/promoting-marketing/pixels/overview#pixels-sandbox-environment). configure shopify pixel Below is boilerplate custom pixel code that provides a function to send events back to Statsig. You should subscribe to the various events and event metadata necessary for your experimentation practices. This [sample GTM pixel](https://help.shopify.com/en/manual/promoting-marketing/pixels/custom-pixels/gtm-tutorial) shows some of the common events and metadata that you can capture and track to Statsig. ```js theme={null} const getStableID = () => { // New gen JS-SDK stores in dynamic keyed localstorage entries for (let i = 0; i < localStorage.length; i++) { const key = localStorage.key(i); if (key && key.includes('statsig.stable_id.')) { const value = localStorage.getItem(key); return value.replace(/"/gi, ''); } } // Old gen JS-SDK stores in specific localStorage key const fallback = localStorage.getItem('STATSIG_LOCAL_STORAGE_STABLE_ID'); if (fallback) { return fallback.replace(/"/gi, ''); } }; /** * Util function for tracking events back to statsig */ const statsigEvent = async (eventKey, eventValue = null, metadata = {}, userObject = {}) => { Object.assign(userObject, { customIDs: {stableID: stableID} // attach stableID automatically }); await fetch('https://events.statsigapi.net/v1/log_event', { method: 'POST', headers: { 'Content-Type': 'application/json', 'statsig-api-key': 'client-STATSIG_CLIENT_KEY' }, body: JSON.stringify({ "events": [{"user": userObject, "eventName": eventKey, "metadata": metadata}] }) }); } analytics.subscribe("checkout_completed", event => { statsigEvent('checkout', null, { orderId: event.data?.checkout?.order?.id, currency: event.data?.checkout?.currencyCode, subtotal: event.data?.checkout?.subtotalPrice?.amount, shipping: event.data?.checkout?.shippingLine?.price?.amount, value: event.data?.checkout?.totalPrice?.amount, tax: event.data?.checkout?.totalTax?.amount, }); }); analytics.subscribe("product_viewed", (event) => { statsigEvent('product_viewed', null, { product_title: event.data?.productVariant?.title, }); }); ``` For tracking behaviors in the main storefront experience, you can also leverage the following: * [Autocapture](/guides/sidecar-experiments/measuring-experiments#using-autocapture) loads by default with Sidecar, and will automatically collect various user behaviors. * [Instrumenting custom event logging](/guides/sidecar-experiments/measuring-experiments#using-the-tracking-api) to track behaviors using code. ## Using Shopify Headless + Statsig SDKs for deeper experimentation Using [Shopify Headless](https://shopify.dev/docs/storefronts/headless) gives you full control over customizing your storefront by decoupling the Shopify admin backend and the storefront application. This means that Shopify effectively serves as a data store, providing APIs to fetch & serve products, content, and manage the entire shopping experience using code. Whether you're using Shopify's [Hydrogen app](https://shopify.dev/docs/storefronts/headless/hydrogen/fundamentals) and its frameworks or a [custom headless stack](https://shopify.dev/docs/storefronts/headless/bring-your-own-stack), you can integrate Statsig's SDK as needed in order to assign users to experiments. Integrating Statsig in this architecture will follow a similar pattern to our recommendation to [integrating with headless CMS platforms](/guides/cms-integrations). #### Integrating data sources for experiment metrics Along with the measuring simple click stream and point-of-sale behavior as [outlined here](/guides/shopify-ab-test/#configure-event-tracking), commerce businesses performing deeper experimentation often want to integrate offline data systems and measure experiments using existing metrics that the broader business uses. Commonly, the Data Warehouse is the source of truth for user purchase data and other categories of offline data. This affords customers the ability to define more [bespoke metrics](/statsig-warehouse-native/configuration/metrics#metric-types) using filtering, aggregations and incorporating other datasets in the warehouse for segmenting experiment results. # Advanced Configurations Source: https://docs.statsig.com/guides/sidecar-experiments/advanced-configurations Configure advanced Sidecar features including Single Page App support, targeting, segmentation, consent management, and cross-domain tracking. This page covers the legacy Sidecar flow. For Visual Editor v3, see [Advanced Configurations](/guides/sidecar-experiments/advanced-configurations-v3). ## Single Page App Support Sidecar now officially supports integration within Single Page Apps. This includes new Sidecar configuration tools and SDK methods that you can use to configure your own customer trigger points, giving you full flexibility to control when an experiment should run. * **Disable Auto Run** - When selected, Sidecar will not automatically attempt to run the test on page-load. * **Custom activation** using `StatsigSidecar.activateExperiment("")` to activate an experiment manually. * **Prerun Script** - This allows you to define a custom script that will run only once per experiment if the url filter passes. You should use this to bind any listeners or evaluate any custom logic you need to control when to trigger the experiment using `activateExperiment`. Single Page App configuration settings in Sidecar ## Advanced Targeting & Segmentation This approach allows you to set User Identity and Attributes for Sidecar, enabling you to perform more advanced targeting and results segmentation. By default, Sidecar & Autocapture use `stableID` (an auto-generated device-level guid that gets stored in the user's localStorage) for tracking purposes. If you wish to enrich autocapture events with known user identities and attributes, you can define the following object *prior* to autocapture/sidecar being loaded. Watch this [video tutorial](https://tinyurl.com/sidecar-targeting) on how to configure more advanced targeting for your Sidecar experiments. ```js theme={null} window.statsigUser = { userID: "", custom: { // optional attributes object isLoggedIn: false } } ``` ## Accessing the Statsig js client For accessing the underlying Statsig js client instance, you can call `StatsigSidecar.getStatsigInstance()`. ## Configuring Runtime Options This allows you to handle Consent Management, GDPR compliance and more. All of the [StatsigOptions](/client/javascript-sdk/#statsig-options) provided by the JavaScript SDK are also fully-supported with Sidecar. These can be passed to Sidecar via: ```js theme={null} window.statsigOptions = { // example of disabling logging for loggingEnabled: 'disabled' } ``` ## Managing Consent Prior to Sidecar script tag, configure these runtime options to disable browser storage and tracking: ```js theme={null} window.statsigOptions = { loggingEnabled: "disabled", disableStorage: true } ``` Later on, after the user gives consent, re-enable storage and tracking: ```js theme={null} __STATSIG__.instance().updateRuntimeOptions({loggingEnabled: "browser-only", disableStorage: false}); ``` ## Persisting stableID across subdomains Statsig uses `localStorage` as the preferred mechanism for storing the user's stableID. Localstorage keys do not persist across any origin boundaries including across subdomains. For example, a user visiting `https://example.com`, `https://show.example.com` and `https://account.example.com` would be issued three distinct stableIDs. If you are assigning a user to a test on one subdomain, and tracking behavior for metrics purposes on a different subdomain, you'll need to have this solution in place to ensure Statsig can properly attribute cross-origin behavior to the Test Group assignment that took place on the initial experiment domain. To install, simply paste the following in your HEAD section. ```html theme={null} ``` # Advanced Configurations Source: https://docs.statsig.com/guides/sidecar-experiments/advanced-configurations-v3 Configure advanced Statsig Sidecar v3 settings, including custom triggers, multi-page tests, and integrations with analytics tools and consent providers. ## Runtime settings for SPAs and manual activation Visual Editor v3 includes runtime settings that let you control when an experiment starts running on the page. These settings are configured on the experiment setup page in the Statsig Console. * **Disable Auto Run** prevents Sidecar from automatically attempting to run the experiment on page load. * **Custom activation** lets you start the experiment manually with `StatsigSidecar.activateExperiment("experiment_name")`. * **Prerun Script** lets you define a custom script that runs once before the experiment starts when the targeting rules pass. Use this to bind listeners or evaluate custom logic before manually activating the experiment. ## Advanced Targeting & Segmentation This approach allows you to set User Identity and Attributes for Sidecar, enabling you to perform more advanced targeting and results segmentation. By default, Sidecar & Autocapture use `stableID` (an auto-generated device-level guid that gets stored in the user's localStorage) for tracking purposes. If you wish to enrich autocapture events with known user identities and attributes, you can define the following object *prior* to autocapture/sidecar being loaded. Watch this [video tutorial](https://tinyurl.com/sidecar-targeting) on how to configure more advanced targeting for your Sidecar experiments. ```js theme={null} window.statsigUser = { userID: "", custom: { // optional attributes object isLoggedIn: false } } ``` ## Available Sidecar methods Sidecar exposes a small runtime API on `window.StatsigSidecar` for manual activation, manual event logging, and access to the underlying Statsig JavaScript client. After Sidecar initializes, `StatsigSidecar.getStatsigInstance()` returns the underlying client. ```js theme={null} // Start a Visual Editor experiment manually when Disable Auto Run is enabled StatsigSidecar.activateExperiment("experiment_name"); // Log a custom event StatsigSidecar.logEvent("event_name", null, { metadata_key: "metadata_value", }); // Access the underlying Statsig JavaScript client after Sidecar initializes window.postExperimentCallback = function () { const client = StatsigSidecar.getStatsigInstance(); const config = client.getDynamicConfig("config_name"); const context = client.getContext(); }; ``` ## Configuring Runtime Options This allows you to handle Consent Management, GDPR compliance and more. All of the [StatsigOptions](/client/javascript-sdk/#statsig-options) provided by the JavaScript SDK are also fully-supported with Sidecar. These can be passed to Sidecar via: ```js theme={null} window.statsigOptions = { // example of disabling logging for loggingEnabled: 'disabled' } ``` ## Managing Consent Prior to Sidecar script tag, configure these runtime options to disable browser storage and tracking: ```js theme={null} window.statsigOptions = { loggingEnabled: "disabled", disableStorage: true } ``` Later on, after the user gives consent, re-enable storage and tracking: ```js theme={null} __STATSIG__.instance().updateRuntimeOptions({loggingEnabled: "browser-only", disableStorage: false}); ``` ## Persisting stableID across subdomains Statsig uses `localStorage` as the preferred mechanism for storing the user's stableID. Localstorage keys do not persist across any origin boundaries including across subdomains. For example, a user visiting `https://example.com`, `https://show.example.com` and `https://account.example.com` would be issued three distinct stableIDs. If you are assigning a user to a test on one subdomain, and tracking behavior for metrics purposes on a different subdomain, you'll need to have this solution in place to ensure Statsig can properly attribute cross-origin behavior to the Test Group assignment that took place on the initial experiment domain. To install, simply paste the following in your HEAD section. ```html theme={null} ``` # Creating Your First Experiment Source: https://docs.statsig.com/guides/sidecar-experiments/creating-experiments Create and configure A/B experiments using the Statsig Sidecar Chrome extension without writing code or deploying changes to your production application. Sidecar allows you to create and run A/B experiments easily without having to write code or push code to production. Here we'll see how you can create one such experiment and get results This guide assumes you have followed the previous steps of installing side-car, creating a statsig account and setting up the API Keys correctly. Check out [Setup](/guides/sidecar-experiments/setup) for those instructions. ### Step 1: Navigate to the web page Navigate to the web page you want to experiment on. Sidecar experiment interface showing the main dashboard ### Step 2: New experiment Hit the *New Experiment* button and fill out the details. This will create a local experiment which hasn't been published yet. This allows you to configure all the details, verify that everything works and then you can publish it. New experiment creation form ### Step 3 (Optional): Add url filter You have the option to select what pages the experiment will run on.
This will be evaluated prior to any targeting rules you configure on the experiment within Statsig console. *You can configure URL targeting using the following methods:* * All Pages - anywhere Sidecar client is installed * Contains - The page URL must contain the value as a substring * Exact Match - The page URL must match the exact value specified here. * Regex - Regular expressions, for example `(http|https):\/\/www.statsig.com\/pricing` matches pages `http://www.statsig.com/pricing` or `https://www.statsig.com/pricing`, and will activate this experiment on those pages. URL targeting configuration interface with filter options ### Step 4: Add actions Click the *Add Action* button and you'll see a list of actions you can perform with this experiment. Let's try one of them here. Go ahead and choose *Change content of an element*. This will set you up to run an A/B test changing the content of an HTML element - like Headlines, descriptions, CTA, etc. Add action selection menu with available experiment actions #### 💡 Use Redirect Action for running Landing Page and Split URL experiments For running Landing Page and Split URL experiments, you can quickly add the "Redirect to another page" for any of your test groups and indicate the destination url as desired. Any query string parameters will be preserved and passed to the destination URL. Redirect action configuration for landing page experiments ### Step 5: Select an element In order to run a content change experiment, you will need two things: 1. the element that you want to test with, 2. the content you want to change. Click on the yellow *Target element path* text-box. This will activate an element selector mode. Now as you move your mouse over your web page you'll see a red selection rectangle. Choose the element you want by clicking on it. In this example, we're choosing the main Headline. Element selection with red highlight showing target element Sidecar will now reflect the path of the element that was selected. Selected element path display in the configuration panel ### Step 6: Update content Now, you can choose the two different text content you want to A/B test. In the *Control content* text box, add your control text ("Build Better Products") and in the *Test content* text box, add your test variant ("Experiment Like a Pro"). You can validate these changes in realtime by clicking on the ▶ button above the text box for each variant. This will immediately change the element's content so you can visually inspect how things look before publishing. Content update interface with preview functionality ### Step 7: Add more actions Feel free to add more actions within the same experiment and play with the capabilities of the tool Adding additional experiment actions to the same test #### Congratulations! You have created your first no-code experiment ## Next up: [Measuring Experiments](/guides/sidecar-experiments/measuring-experiments) # Integrating Sidecar with GTM for tracking Source: https://docs.statsig.com/guides/sidecar-experiments/integrating-gtm Set up the Google Tag Manager integration with Statsig Sidecar to automatically send GTM-tagged events to Statsig for experiment metric tracking. This integration is for tracking purposes only. We strongly advise against loading Sidecar itself via GTM, as this will delay the changes from being applied and result in "flickering". ### Overview This integration with GTM will automatically send any GTM-tagged events to Statsig.
No additional coding or tagging is required after completing these steps. Statsig logstream showing GTM events flowing in *(statsig logstream showing gtm events flowing in)* ### Step 1: Create new tag GTM create new tag interface for Statsig integration ### Step 2: Choose tag type Choose "Custom HTML" for tag type, and paste [this GTM code](#gtm-code) (including script tag) GTM tag configuration screen with custom HTML setup ### Step 3: Adjust fire options Under Advanced Settings under "Tag Firing options", select "Once per page" GTM tag firing options settings for once per page ### Step 4: Set Tag Trigger Below the "Tag Configuration" section, set the Trigger to "Initialization - All Pages" Option. GTM tag trigger configuration for all pages initialization ### Step 5: Save tag and test After saving the tag, and publishing your updated GTM tag, tracking will be done automatically without any additional configuration. To debug the integration, you can set a local storage entry `debug_ss_gtm` with any value on your webpage. Now, you'll console log statements for each tracking call being dispatched to Statsig. You can also inspect your browser's network traffic to see events being tracked. ### GTM Code ```html theme={null} ``` # Statsig Visual Editor (Low-code Experiments) Source: https://docs.statsig.com/guides/sidecar-experiments/introduction Learn about Statsig Sidecar, a low-code tool that simplifies A/B testing, enabling marketers to independently execute experiments with ease. ## Overview A/B testing, or split testing, is a fundamental method in digital marketing for validating website changes. It involves comparing two versions of a webpage to see which performs better with a target audience. Traditionally, implementing A/B tests required substantial technical skills, often necessitating collaboration between marketers and developers - leading to longer implementation times and reduced agility of marketing teams. ## Statsig's visual editor Statsig's Visual Editor simplifies this process, enabling marketers to independently execute A/B tests with limited technical dependency. It provides an intuitive, point-and-shoot interface, making it easy to set up and manage A/B tests on your website. You can experiment on styling, text content, calls to action, and even inject scripts that change page behavior. This rapid testing and feedback process is crucial for fine-tuning your website and ensuring it's optimized for user preferences and behavior patterns. Combined with Statsig's [industry-leading stats engine](/experiments-plus), Sidecar is a powerful tool in a marketer's toolkit. Statsig sidecar This guide assumes you have an existing Statsig account. Please go here to create a new free account if you don't already have one: [https://statsig.com/signup](https://statsig.com/signup) ## A word on performance Sidecar (along with any type of client side tooling you install, be it open-source or via experimentation providers) will introduce some degree of page load latency. We recommend taking Sidecar for a spin and determining if the impact of your performance metrics fall within your acceptable threshold. Customers that are very performance-minded typically use our [JS-SDK](/client/javascript-sdk) for testing on the web. When using the JS/React SDKs with [Client Bootstrapping](/client/javascript-sdk/init-strategies/#2-bootstrap-initialization), the latency introduced is very minimal since there are no network requests required for initialization. # Measuring Experiments Source: https://docs.statsig.com/guides/sidecar-experiments/measuring-experiments Learn how to measure experiment results using autocapture, the manual tracking API, and Sidecar callbacks for analytics integrations. ## Using Autocapture Sidecar automatically tracks various web activities, allowing you to create both simple and complex Metrics within Statsig console without writing a line of code. Create a new metric in the Metrics tab on the Statsig console to get started, and read more about the metrics we automatically log in [Autocapture on the Web](/webanalytics/autocapture). See [Autocapture on the Web](/webanalytics/autocapture) ## Using the tracking API You can track events manually for actions that are not autocaptured by the feature described above. To track events back to Statsig, you can call `StatsigSidecar.logEvent` which takes the same arguments as the Statsig JS SDK as documented [here](/client/javascript-sdk#logging-an-event). This method can be called prior to completion of the init routine. ```js theme={null} // example order event StatsigSidecar.logEvent('Order', null, { total: 54.66, units: 3, unitAvgCost: 18.22 }); ``` ## Per-assignment callback for outbound integrations You can bind a callback that gets invoked each time Sidecar activates an experiment assignment, including experiments activated later by prerun scripts. *This method should be defined anywhere prior to the Sidecar client script.* ```js theme={null} window.statsigSidecarConfig = { onExperimentEvaluation: function (event) { /** * add your own callback routine here * ie; annotating 3rd party analytics tools with assignment info */ window.dataLayer = window.dataLayer || []; window.dataLayer.push({ event: "statsig_experiment_evaluation", experiment_name: event.experimentName, experiment_group_name: event.groupName, }); } } ``` The callback payload includes: * `event.name` - always `"experiment_evaluation"` * `event.experiment` - the full Statsig experiment object * `event.experimentName` - the Sidecar experiment name, or the Statsig experiment name as a fallback * `event.groupName` - the assigned group / variant, or `null` if unavailable ## Post-Experiment Callback for one-time readiness hooks You can still bind `window.postExperimentCallback` if you need a callback after Sidecar finishes its initial run. This callback can fire even when there are no experiments, and it does not cover experiments that are activated later by prerun scripts. ```js theme={null} window.postExperimentCallback = function(statsigClient, experimentIds) { // One-time initialization hook after Sidecar finishes the initial run } ``` #### Disabling All Logging To disable all logging to statsig (both autocapture events and logging who has seen your experiments) append the following query string parameter to the Sidecar script URL: `&autostart=0`. This may be useful if you're dealing with GDPR compliance, and you can later re-enable events with `client.updateRuntimeOptions({disableLogging: false})` # Taking your experiments to production Source: https://docs.statsig.com/guides/sidecar-experiments/publishing-experiments Publish, QA, and launch Statsig Sidecar experiments in production with step-by-step guidance covering preview links, approvals, and traffic ramp-up. With the experiment configuration out of the way, we need to take this to production. Sidecar makes this easy with just a few clicks. This guide assumes you have followed the previous steps of creating an experiment in Sidecar. Check out [Creating Experiments](/guides/sidecar-experiments/creating-experiments) for those instructions. ### Step 1: Publish the experiments Once you are satisfied with the experiment configuration, go ahead and hit the blue *Publish* button. This is essentially a way to store all of your configurations in Statsig. If you want to make sure these changes have been stored successfully, you can on the `...` menu and choose *Go to Experiment Console*. Publishing changes will not start any experiments, it will do the following: * Sync any unsaved changes to Statsig (making them accessible in Console where you can configure Metrics and other targeting conditions if applicable). * Include any configured tests in the Sidecar script installed on your website. * Allow you to QA experiments on your site while they're in an Unstarted state. Statsig experiment console interface The experiment console will look like this, and allows you to configure rich targeting, metrics, and tweak advanced statistical knobs. More on this later. Statsig experiment setup tab with checklist and scorecard fields ### Step 2: Preview & QA the experiment At this point, your experiment is in a pre-started state, meaning your experiment will not be active to your site visitors. You can pass a query string to your test page url by using the `overrideuser` query string parameter. The override method uses the following convention to force a test & test group:
`https://www.DOMAIN.com/?overrideuser=_` The image below depicts where you can find the experiment ID and each variation ID. Based on this example, you can force a preview of the Test Group by visiting the following URL:
`https://www.DOMAIN.com/?overrideuser=name_color_test_1` Sidecar experiment QA and preview interface Note, this works best with the default test/control group names - if you change one of your group names, you'll also have to modify it in the Statsig Console by clicking "Manage Overrides". ### Step 3: Start the experiment Refresh the page on your browser with the script embedded. Sidecar will automatically pick up the experiment you have published and display all the experiment properties. You can now start the experiment by clicking on the `...` menu and clicking on *Start Experiment*. This will automatically start the experiment, serve the right variants for control and test, and start collecting metrics on your behalf. Sidecar experiment start interface ### Congratulations! You have successfully built and shipped an experiment 🎉 # Setting up Sidecar Source: https://docs.statsig.com/guides/sidecar-experiments/setup Install and configure the Statsig Sidecar Chrome extension, connect it to your website, and verify the integration before launching no-code experiments. * This guide assumes you have an existing Statsig account. Please go here to create a new free account if you don't already have one: [https://statsig.com/signup](https://statsig.com/signup) * You will need to use Google Chrome web browser for this exercise. * This page covers the legacy Sidecar extension flow. For Visual Editor v3, use the [Visual Editor setup guide](/guides/sidecar-experiments/sidecar-v3). ## Setup Sidecar Chrome Extension ### Step 1: Install Chrome Extension If you don't already have the Sidecar extension, visit the [Chrome store](https://chromewebstore.google.com/detail/statsig-sidecar/blkgemeefnlkmicphlkodgdkhceibgcb) and click on the "Add to Chrome" button Statsig Sidecar Extension in the Chrome Web Store ### Step 2: Activate the extension Click on the Extensions toolbar button and select "Statsig Sidecar" to activate the Sidecar extension Chrome extensions menu with Statsig Sidecar option You will now see an Experiment Config UI like this: Sidecar Empty Interface ### Step 3: Update settings You will need to update API keys in the Settings Dialog for the extension to work. You can invoke the Settings dialog from the "Settings" link on the top header. Sidecar Settings Dialog You can retrieve these keys from your Statsig project. In order to get this, login to Statsig Console here: [https://console.statsig.com](https://console.statsig.com) and navigate to the Settings page ([https://console.statsig.com/settings](https://console.statsig.com/settings)) Once you're there, select the "Keys & Environments" panel and copy both the Console API Key and Client API Key and paste them in the Settings dialog. Statsig console Keys & Environments panel showing API keys Hit "OK" to commit the API Keys. ## Install Sidecar on your website Add a single script tag within the `` portion of your website, replacing with your own [Client SDK Key](/access-management/api-keys) as shown below. ``` ``` Installing Sidecar JS via a Tag Manager can potentially lead to flickering and other unpredictable behavior. We strongly encourage installing Sidecar as a synchronous script tag. ### Additional Options Add these query string parameters to the Sidecar script URL for additional controls over Sidecar client behavior * `&reduceflicker=0` will disable the brief hiding of the `` tag while the client initializes * `&autocapture=0` will disable event autocapture Most website builders also support the ability to add script tags on your website. Here are some common website builder examples: [Webflow](https://university.webflow.com/lesson/custom-code-in-the-head-and-body-tags?topics=site-settings), [Wordpress](https://wordpress.com/go/website-building/how-to-properly-add-javascript-to-wordpress-3-top-methods/), [Wix](https://support.wix.com/en/article/embedding-custom-code-on-your-site), [Squarespace](https://support.squarespace.com/hc/en-us/articles/205815928-Adding-custom-code-to-your-site), [Weebly](https://weeblytutorials.com/embed-javascript-weebly). You can copy the script code from within the Sidecar Chrome extension Sidecar Script Code #### You are now all set to create your first experiment ## Next up: [Creating Experiments](/guides/sidecar-experiments/creating-experiments) # Visual Editor Setup & Usage Source: https://docs.statsig.com/guides/sidecar-experiments/sidecar-v3 Overview of Statsig Sidecar v3, a no-code Chrome extension for running A/B tests directly on any website without engineering involvement. ## Summary Visual Editor v3 provides a simpler interface for point-and-shoot experiments. Different from previous versions, experiments are created in the Statsig Console, and edited directly on top of the page. Sidecar still relies on a javascript script tag, though the package name has changed. Sidecar v3 is in an open Beta release. Feel free to try the product, and share any feedback you have in our [Slack Community](https://statsig.com/slack). ## Prerequisites: ### 1. Installing the Sidecar Script For experiments to take effect, you'll need to have the visual editor ("sidecar") script running on your website, on any page you'd like to experiment on: ```html theme={null} ``` Replace `client-key` with a client key from your Statsig project, which you can find at [Settings > Keys & Environments](https://console.statsig.com/api_keys). Most website builders also support the ability to add script tags on your website, like: [Webflow](https://university.webflow.com/lesson/custom-code-in-the-head-and-body-tags?topics=site-settings), [Wordpress](https://wordpress.com/go/website-building/how-to-properly-add-javascript-to-wordpress-3-top-methods/), [Wix](https://support.wix.com/en/article/embedding-custom-code-on-your-site), [Squarespace](https://support.squarespace.com/hc/en-us/articles/205815928-Adding-custom-code-to-your-site), [Weebly](https://weeblytutorials.com/embed-javascript-weebly). ### 2. Installing the Chrome Extension (for editing) Additionally, to create edits for experiment variants - you'll need the new Statsig Visual Editor [Chrome Extension](https://chromewebstore.google.com/detail/statsig-sidecar-v3/mmgjfcbidnlghegclgpkgegpdhbopjhn) installed. You also need to be a project admin, or have permissions to access console API keys, to use all Sidecar functionality. ## Creating an Experiment You now create experiments in the console, on the Create Experiment dialogue, by changing the experiment type to "Visual Editor". Create Visual Editor Experiment ## Setting up an Experiment ### Metrics When you add the Statsig Visual Editor script to your website, it will automatically begin tracking events like clicks, page views, Core Web Vitals, and more. We call these [autocapture](/webanalytics/autocapture/) metrics. See [Metrics](/metrics/101) for more info. You can add any of these metrics to your experiment, or customize them to be filtered to certain attributes (e.g. clicking a certain button, visiting a certain page) by creating a new metric in the [Metrics Catalog](https://console.statsig.com/metrics/metrics_catalog) tab. You must add at least one metric to your experiment before continuing. ### URL Filters Often, you'll want an experiment to only be evaluated on some subset of the pages on your website. You can configure which URLs your experiment should run on by: * **All Pages** - anywhere Sidecar client is installed * **Contains** - The page URL must contain the value entered, for example "pricing" * **Exact Match** - The page URL must match the exact value specified here. * **Regex** - Regular expressions, for example `(http|https):\/\/www.statsig.com\/pricing` matches pages `http://www.statsig.com/pricing` or \`[https://www.statsig.com/](https://www.statsig.com/) URL Targeting ### Audiences Statsig will also infer information about each user - like their country, device and browser type, a unique identifier called StableID, and more. You can target based on any of these attributes with the "audiences" targeting section. You can also target on custom attributes added to the Window\.statsigUser object, though this requires a line or two of code. ### Visual Editor Starting URL Before you begin editing experiment variants, you'll also need to add a Visual Editor URL, where the editor will open when you first begin editing. Add a URL, being sure it starts with "https\://", then click save and you're ready to begin editing. To setup an experiment - enter your metrics, any targeting (on URL, or other user attributes per the [StatsigUser object](/concepts/user)), and the URL you'd like to start editing your experiment from (make sure to start it with "https\://"). Click save, then click the "Open in Editor" button, at which point you'll see an editor bar appear along the bottom of the page: Visual Editor Page ### Runtime settings For more advanced experiment flows, you can also configure runtime settings on the experiment setup page before opening the editor. * **Disable Auto Run** prevents the experiment from applying automatically on page load. * **Prerun Script** lets you run custom JavaScript before the experiment starts. This is useful for SPAs or other advanced flows where you need to bind listeners or evaluate custom logic before manually activating the experiment with `StatsigSidecar.activateExperiment("experiment_name")`. See [Advanced Configurations](/guides/sidecar-experiments/advanced-configurations-v3) for more detail on runtime settings, manual activation, and prerun scripts. ## Making your edits By clicking the "Element selector" or pressing command + E, you enable the element editor, which provides a preset set of editable attributes depending on the element you select, for example: * *Text:* Text, Font attributes * *Button:* Text, target link * *Image:* Image source Plus all elements offer the opportunity to rearrange or hide attributes. Visual Editor Editor ## Starting your experiment By clicking save in the bottom right hand side of the visual editor, your changes will be saved to the console. If you return to the page in console and refresh, your changes should be listed. You can preview your experiment by clicking the three dots in the variant table. When you're confident with your changes, you can go ahead and launch the experiment by clicking "Start" in the top right hand corner. After this, your experiment will begin appearing for end users and your metrics will be collected. ### Advanced Script Setup Add these query string parameters to the Sidecar script URL for additional controls over Sidecar client behavior * `&reduceflicker=0` will disable the brief hiding of the `` tag while the client initializes ## Advanced topics If you need more advanced Sidecar controls after the basic setup is working: * See [Advanced Configurations](/guides/sidecar-experiments/advanced-configurations-v3) for runtime settings, manual activation, prerun scripts, script URL tuning, consent, and identity configuration. * See [Measuring Experiments](/guides/sidecar-experiments/measuring-experiments#per-assignment-callback-for-outbound-integrations) for outbound analytics callbacks and other measurement patterns. # Statsig ID Resolver Source: https://docs.statsig.com/guides/statsig-id-resolver Use the Statsig ID resolver to deduplicate and stitch user identities across anonymous and logged-in sessions for consistent experiment exposures. ## What is Statsig ID Resolver? Statsig ID Resolver is an integration set up at the project level that brings your ID names into console. IDs are used everywhere within Console, but unless you are an ID Wizard it is hard to tell at a glance who or what an ID belongs to. Take Feature Gate rules for example: Feature gate rules with raw user IDs Each of the IDs shown represent a superhero with a name and other identifying information. After setting up ID Resolver you will be able to see an ID’s “name” next to each ID. In this example, the ID’s name is the superhero’s name followed by their publisher. You have the power to define “name” as whatever string is most useful for your project. Feature gate rules with resolved ID names Additionally, after setting up ID Resolver Autocomplete you can begin typing in an ID’s name and have it auto-resolve to the correct ID ID autocomplete functionality in action ID Resolver can be used wherever you enter IDs, for example in Feature Gate rules, Overrides, Users tab, and Segment ID lists ID resolver usage across console interfaces ## Step 1 - Create your ID Resolver Webhook You will need to create and host your own webhook for this integration. This webhook should take in an `id` and a possibly null `unit_type` and return `name`. `unit_type` will come in the form of userID, stableID, or one of your custom ID types. The length of `name` should be under 100 characters. ```js theme={null} const inputId = req.body.id as string | null; const unitType = req.body.unit_type as string | null; if (!inputId) { res.status(200).json({ success: true, data: { name: "", }, }); } const result = IDResolverDatabase.find((d) => d.id === inputId); res.status(200).json({ success: true, data: { name: result ? result.name + ", " + result.Publisher : "", }, }); ``` ## Step 2 - Create your ID Resolver Autocomplete Webhook This webhook should take in a `name` (the current partially typed name) and a possibly null `unit_type` and return the array `results` which contains potential matches in the shape of `{name: string, id: string}`. It should return at most 100 results, and the length per item should be under 100 characters. ```js theme={null} const partialName = req.body.name as string | null; const unitType = req.body.unit_type as string | null; if (!partialName) { res.status(200).json({ success: true, data: { results: [], }, }); } const results = IDResolverDatabase.filter((d) => d.name.match(new RegExp(`^${partialName}`)) ).limit(100); res.status(200).json({ success: true, data: { results: results.map((result) => { return { name: result.name + ", (" + result.Publisher + ")", id: result.id, }; }), }, }); ``` ## Step 3 - Integrate your webhooks with Statsig You can find Statsig ID Resolver under the Integrations tab within Project Settings ID resolver integration setup screen ID resolver webhook configuration interface ### Secure Your Webhook With an API Key (Optional, But Recommended) Statsig accepts an optional API key in the integration configuration. If you submit a string here, we will call your webhook with the HTTP Header "Authorization: Bearer \." This can be any random string. Your server should reject any request that does not supply the same string you supplied when setting up the integration. One way to generate this is by running `openssl rand -hex 32`. Make sure to store and read this securely on your server. *And that's it! You're off to the races with easier-to-recognize IDs throughout the Console.* # Testing your Gates/Experiments Source: https://docs.statsig.com/guides/testing Test Statsig integration locally with overrides, local evaluation, and unit tests so you can validate feature gate and experiment behavior before launch. Statsig enhances your engineering velocity by offering tools and features that allow you to test configurations quickly while ensuring reliable outcomes. This page highlights key features across the product to help you test efficiently. *** ## Overrides: Test Features, Experiments, or Holdouts Overrides allow you to manually configure [features](/feature-flags/overrides#adding-an-override), [experiments](/experiments-plus/overrides), or [holdouts](/holdouts) for testing purposes. This method enables safe testing without affecting live production data or skewing experiment results. **Overrides are excluded from Pulse analysis** to maintain unbiased results. * Use **Segments** to target overrides to pre-production environments or specific groups (e.g., employees) for testing. Feature Override Example Experiment Override Example For more details on adding overrides, see: * [Feature Overrides](/feature-flags/overrides#adding-an-override) * [Experiment Overrides](/experiments-plus/overrides) * [Holdout Overrides](/holdouts) *** ## Unit Testing with Statsig Statsig's **Server SDKs** offer a `localMode` feature that disables network access, ensuring that tests run locally and independently of production systems. When `localMode` is active, the SDK returns default values, allowing you to mock features and experiments in a controlled test environment. ### Override APIs for Testing You can use the `overrideGate` and `overrideConfig` APIs to set specific overrides for users or globally during testing. ```js theme={null} function overrideGate( gateName: string, value: boolean, userID?: string, ): void; ``` ```js theme={null} function overrideConfig( configName: string, value: object, userID?: string, ): void; ``` For example, to override a gate for testing: ```js theme={null} statsig.overrideGate("example_gate", true); ``` * For more information on how to mock Statsig for testing, refer to [Node.js Server SDK](/server/nodejsServerSDK#how-can-i-mock-statsig-for-testing) or [JavaScript Client SDK](/client/javascript-sdk#testing). *** ## Environments: Configuring Development, Staging, and Production Statsig enables you to assign environments to feature gates, experiments, and events. By default, checks without a defined environment are assigned to **Production**. You can customize environments (such as **Development**, **Staging**, or **Production**) and use them to target different versions of feature gates or segments. * Non-production events appear in diagnostics and can be used to track cumulative exposures and metric results when testing experiments in lower environments with **Enable for Environments**. Production data is prioritized for final Pulse result analyses. ### Customizing Environments You can map your internal environments to Statsig's built-in environments or create custom mappings. Common setups include: * Assigning Dev One boxes to **Development**. * Creating an **Early Access** slice (e.g., 1% of production users) as part of the **Production** environment for phased rollouts. For more details on environments, refer to this [Statsig blog post](https://blog.statsig.com/environments-on-statsig-6a818805b3c2). *** # UI-Based Tool Source: https://docs.statsig.com/guides/ui-based-tool Build no-code UI changes with Statsig's visual editor to launch lightweight A/B tests and content experiments without redeploying application code. You can follow this guide to use Statsig's built in LaunchDarkly migration tool. Please note that this UI-based tool only imports the "production" environment at the moment. ## What you need Review the full checklist in the [LaunchDarkly Migration Guide](/guides/migrate-from-launchdarkly#what-you-need), then gather: 1. You will need your project key. Projects in LaunchDarkly have a Name (e.g. "My Mobile App") and a Key (e.g.my\_mobile\_app). 2. You'll need a read-only access token for this project. You can create one in LaunchDarkly -> Account Settings -> Authorization and limit scope to be read-only. 3. A Statsig project to use. We recommend trying this in a test project first. ## How it works These screens mirror the [console walkthrough](/guides/migrate-from-launchdarkly#how-it-works): 1. You will be prompted to Import Feature Gates if you don't have any feature gates in your project. Import Feature Gates prompt 2. Select LaunchDarkly as the platform you want to migrate from. Platform selection interface 3. Enter your LaunchDarkly Project Key and API Key/access token. LaunchDarkly credentials input form 4. Preview the migration summary. We'll highlight what gates we can and can't migrate. Gates we don't migrate include gates with segments (coming soon) and gates with non-Boolean flags. Migration summary preview screen 5. Finish migration of the gates. All your migrated gates will be tagged "Migrated" so you can identify them. Migration completion confirmation # Paranoid about uptime? 10 things to do! Source: https://docs.statsig.com/guides/uptime Check Statsig service uptime, view the public status page, and learn about SLAs and historical reliability for SDKs, console, and pipelines. Statsig serves billions of individual user interactions. Along the way, we designed the service for reliability and availability of your apps that use Statsig. Because of this, in the case where your application cannot reach Statsig for any reason, your application will continue to work exactly as you expect with locally cached values. Collected here are a set of best practices that help maximize your uptime - across potential issues including failed client connectivity (to your server), failed server connectivity to Statsig and buggy or deprecated code. You can also read more about how we [design for failure](https://statsig.com/blog/designing-for-failure). 1. Feature Gates and Experiments have **default values** that are used in an evaluation . You can disable Feature Gates and set a default value for it; however if your SDK has no information this will default to false. For experiments you specify default values in code. Validate these work and don't break the experience. 2. **Test** your code with all the possible assignments in Experiments and Feature Gates. Use overrides to easily do this, along with the inline, real time diagnostics. Don't roll out your experiment and then find a variant (group) that crashes. 3. Use Statsig's support for **pre-production environments** (e.g. Dev, Staging) as part of your validation process. Pre-production environments can remove change approval requirements, allowing swifter iteration. 4. **Start small**, validate and then ramp. With feature gates we recommend rolling out to 2% (check for crashes/obvious bugs) before ramping up to 10%, 50% and then 100% while watching metrics you measure. You can also have [Statsig fire Rollout Alerts](/metrics/rollout-alerts) when thresholds are violated. 5. **Clean up** your code and remove features you've finished launching. This ties back to items #1 and #2; you don't want to leave potential code paths you're not testing/monitoring that can be triggered. 6. Use **change management** on Statsig in production. Changes should be approved by a reviewer. For critical areas, you can enforce an Allowed Reviewer group that has enough context to decide. Statsig Feature Gates allow you to easily audit and roll back changes. 7. **Caching on client SDKs**: Initializing Statsig client SDKs requires them to connect to Statsig and download config. Client SDKs can cache and reuse config (for the same user) if they are offline. You can also choose to bootstrap your client from your own server (and remove the round trip to Statsig) by using [client SDK bootstrapping](/client/concepts/initialize#bootstrapping-overview). 8. **Caching on server SDKs**: Initializing Statsig server SDKs requires them to connect to Statsig and download config. If connectivity to Statsig fails, initialization fails (falling back to default values). Two key things can help mitigate this (and related) risks. 1. Deploy the Statsig [Forward Proxy](/server/concepts/forward_proxy/) so your servers connect to this endpoint to download config, instead of from Statsig directly. This also reduces network traffic and improves consistency of configuration across your server fleet. 2. Reduce connectivity related SDK initialization failures by providing config locally using a [dataAdapter](/server/concepts/data_store#dataadapter-or-datastore). 9. **Test for failure conditions** explicitly (e.g. no Statsig client or server connectivity). Run a disaster simulation (e.g. break DNS routing to Statsig within your data center) and test client app behavior when Statsig is unreachable. Also understand and embrace the variety of capabilities we've built to support different kinds of [testing](/guides/testing). 10. **Implement a custom proxy** to prevent adblockers from blocking events or initialization in client SDKs. Adblockers can interfere with Statsig's default endpoints, preventing your application from logging events or initializing properly. By setting up a [custom proxy](/infrastructure/api_proxy/custom_proxy) on your own domain with custom endpoint names, you can ensure that tracking blockers don't intercept your API calls and that you capture all necessary data. # Environment-based Evaluation Source: https://docs.statsig.com/guides/using-environments Use environments in Statsig to separate development, staging, and production data for feature gates, experiments, and metrics across your project. Statsig SDKs allow you to set the environment tier for your app during initialization. This helps you evaluate feature gates, dynamic configs, and experiments differently in non-production environments like development or staging. All you need to do is configure the appropriate environment in your code and adjust feature rules in the Statsig Console. Here's a step-by-step guide on how to configure and use environments effectively. *** ## SDK Usage There are two key ways to set up environments within your app: 1. **Environment-specific SDK keys**: These determine which rule sets are downloaded by the SDK based on the environment. 2. **Environment tier at SDK initialization**: This defines how rules are evaluated for the app. ### 1. Environment-specific SDK Keys Setting up environment-specific SDK keys allows you to control which rules are sent to the SDK. For instance, if an SDK is initialized with a key for the development environment, it will not receive rules set for staging or production environments. For more information, see [Per-Environment API Keys](#per-environment-api-keys) below. ### 2. Environment Tier Parameter SDK keys can correspond to multiple environments. Therefore, it's important to explicitly set the environment tier during SDK initialization to ensure the correct rules are applied. All SDKs accept an `SDK Key` and an optional `StatsigOptions` dictionary. The `StatsigOptions` parameter includes the `environment` key, which has a `tier` field. This tier corresponds to one of your pre-configured environments (e.g., development, staging). If the environment tier is unset, all checks and event logs will default to "production." Here's an example of setting the environment tier in your code for the **development** environment: #### Example (JS Client SDK): ```javascript theme={null} const client = new StatsigClient(, user, { environment: { tier: 'development' } }); ``` #### Example (Node Server SDK): ```javascript theme={null} await statsig.initialize(, { environment: { tier: 'development' } }); ``` Refer to your language-specific SDK documentation for further details. *** ## Using Environments in Feature Gates To configure environment-specific rules for a **Feature Gate**, follow these steps: 1. **Create a new Feature Gate**: In the Statsig Console, create a new Feature Gate. For example, name it "development mode" to target only your development environment. Feature Gate 2. **Specify Environments**: When configuring the rule, check the **Specify Environments** box and select the environments you want to target. By default, rules are enabled for all environments unless specified otherwise. Specify Environment 3. **Save your settings**: After saving, the environment(s) for which the rule is enabled will be displayed below the rule name. Enabled Environments You can also filter rules by environment using the filter in the upper-right corner of the Feature Gate UI. To edit the target environments of a rule, click the "..." next to the rule name and select **Edit Rule**. Edit Rule *** ## Configuring Environments By default, Statsig provides three environments: **Development**, **Staging**, and **Production**. You can add more environments or rename the default ones, but the **Production** environment cannot be deleted or modified. ### Steps to Add or Edit Environments: 1. Navigate to **Project Settings** → [**Environments & Keys**](https://console.statsig.com/api_keys). Project Settings 2. Click **Edit** to add new environments or reorder the existing ones using drag-and-drop. Edit Environments Reordering environments doesn't affect any rule logic, but it helps convey the rollout hierarchy (e.g., development → staging → production) to your teams. *** ## Per-Environment API Keys To enhance security and privacy, Statsig allows you to create per-environment API keys. This ensures that SDKs initialized with specific environment keys will only access the rules relevant to that environment. ### Steps to Generate Environment-Specific API Keys: 1. Go to **Project Settings** → [**Environments & Keys**](https://console.statsig.com/api_keys). 2. Click **Generate New Key**, and specify the environment for which you want to generate the API key. Generate API Key The default environments—Development, Staging, and Production—share the same server and client-side API keys. You can generate new keys for custom environments as needed. *** # A/B Testing with Webflow and Visual Editor Source: https://docs.statsig.com/guides/webflow-sidecar-ab-test Run no-code A/B tests on a Webflow site using Statsig Sidecar, including connecting the Chrome extension and tracking conversion metrics. ## Use cases & considerations Webflow offers a comprehensive platform for businesses to design, build, and manage visually stunning websites and their content without the need for coding. To experiment on a site that uses Webflow, we recommend using [Statsig Visual Editor](/guides/sidecar-experiments/introduction) to build test treatments and assign users to experiments when they land on your site, all without writing code. #### Install the Visual Editor Chrome extension Install the [Statsig Visual Editor Chrome extension](https://chromewebstore.google.com/detail/statsig-sidecar-v3/mmgjfcbidnlghegclgpkgegpdhbopjhn) and follow the [Visual Editor setup guide](/guides/sidecar-experiments/sidecar-v3#2-installing-the-chrome-extension-for-editing). The extension allows non-technical users to build experiments and treatments directly on top of a live page. You can target by URL and configure text changes, style changes, image swaps, and custom JavaScript for advanced use cases. #### Add the Visual Editor script to your Webflow site 1. Log in to your Webflow dashboard and navigate to your project. 2. Access the **Custom Code** section in your project settings. 3. Copy the Visual Editor script tag from the [Visual Editor setup guide](/guides/sidecar-experiments/sidecar-v3#1-installing-the-sidecar-script) and replace `client-xxx` with your Client SDK key from [Settings > Keys & Environments](https://console.statsig.com/api_keys): `` Webflow custom code interface 4. Paste the script tag into the `` section of your Webflow site's custom code area. 5. Save and publish your changes to apply the Visual Editor script across your website. 6. In Statsig Console, create an experiment and choose **Visual Editor** as the experiment type before launching the editor. #### Optional script parameters Add these query parameters to the script URL for additional client behavior controls: * `&reduceflicker=0` disables briefly hiding `` while the client initializes. #### Configure event tracking and metrics Statsig provides several approaches for tracking events when using Visual Editor. You can use built-in [Autocapture](/webanalytics/autocapture/) tracking or set up custom event logging. For detailed guidance, refer to [Visual Editor Setup & Usage](/guides/sidecar-experiments/sidecar-v3#metrics) and the event tracking [documentation](/guides/sidecar-experiments/measuring-experiments). #### Debugging and Troubleshooting If you encounter issues during the integration or experimentation process, here are some tips: * Verify that the Visual Editor script tag is correctly placed in the `` section of your Webflow site. * Confirm you're using the latest package URL: `statsig-sidecar-v2-beta`. * Check the browser's console for any JavaScript errors that may indicate problems with the script. * Ensure your experiment is created as a **Visual Editor** experiment and is active in Statsig Console. * Confirm the Visual Editor Chrome extension is installed. * Review the [Visual Editor setup documentation](/guides/sidecar-experiments/sidecar-v3) for any missed steps. # HTTP API Source: https://docs.statsig.com/http-api/overview Overview of the Statsig HTTP API for retrieving feature gate, experiment, and dynamic config values and logging events directly without an SDK. While this HTTP API is available for direct use, we strongly recommend using one of our official SDKs for your programming language whenever possible. SDKs offer better performance, automatic error handling, and type safety. They also provide a more idiomatic integration with your codebase. Only use this HTTP API directly if there isn't an SDK available for your language or if you have a specific use case that requires direct API access. Before you can start calling the server APIs, you need to take care of the following steps: Create a free account on the [Statsig sign-up page](https://statsig.com). An account will let you use the Statsig Console, where you can manage all of your Feature Gates, Dynamic Configs, and Experiments. Note that you will also be able to invite others to collaborate on your Statsig Projects, so they can interact with your gates and configs. An API key is required in every API request. There are two types of API keys you can use with the HTTP API: * **Server-side secret Key**: Used only from secure servers and should never be exposed in client-side code. * **Client-SDK Key**: Safe to embed in mobile apps and front-end web apps. If you're working with server-side logic or sensitive data, use the Server-side secret Key. If you're in doubt or working with public-facing code, use the Client-SDK Key. Our API is built on top of HTTPS, and you can authenticate via the `statsig-api-key` header. All API requests use the POST method, and parameters are set by passing a JSON object in the request body. **Why POST?** Even for fetching data, we use POST to ensure secure and flexible transmission of user-specific data (e.g., configurations or experiment results). Statsig automatically logs exposure events whenever you call the APIs. These exposure events help attribute downstream events to experiments or feature gates, which are used to calculate metrics like analytics lift. # Events Mode on Logs Explorer Source: https://docs.statsig.com/infra-analytics/events-mode-logs-explorer Use events mode in Statsig Logs Explorer to analyze log records as discrete events for higher-cardinality grouping, filtering, and visualization. Events Mode brings the searching and filtering abilities from [Log Explorer](/infra-analytics/logs-explorer) to your *existing* Statsig events data. No additional instrumentation required! * Debug user activity with more control * Trace the sequence of actions a user performed * Narrow in on specific events from metadata properties Events Mode is available to all **Statsig Cloud customers**. *** ### To Get Started: 1. Open Logs Explorer from the left navigation. 2. Confirm you've switched from **Logs** mode to **Events** mode (The area next to the search bar should say **Events**). 3. Start searching across event names, properties, and time ranges. *** ### How to Construct your Search You can search through events in two ways: 1. **Writing queries:** Use [syntax-based search](/infra-analytics/logs-explorer-queries) to target specific events and properties 2. **Using the query builder:** Point-and-click to construct filters without syntax overhead. Events Explorer Overview *** ### Example Workflows | Description | Query | | -------------------------------------------------------------- | -------------------------------------------------------------------- | | Find all enterprise account signups from web browser devices | `event_name:signup_completed AND #plan:enterprise AND #platform:web` | | Find all checkout events where the item name contained "black" | `@#custom_event:checkout_event,add_to_cart AND @product:"*black*"` | | Find all events from users with a gmail account | `#user_object.email:"*gmail.com"` | # Getting Started with OTEL + Statsig Source: https://docs.statsig.com/infra-analytics/getting-started Set up OpenTelemetry exporters to send logs, metrics, and traces to Statsig Infra Analytics for use in Logs Explorer, Metrics Explorer, and alerts. This guide helps you setup and send OpenTelemetry telemetry to Statsig so you can use Infra Analytics (Logs Explorer, Metrics Explorer, Alerts). There are two common paths: * Kubernetes/OpenTelemetry Collector: scrape logs and metrics from your cluster and export to Statsig. See [Open Telemetry Logs and Metrics](/server/concepts/open_telemetry) for a more complete guide. * Applications: export traces, metrics, and logs to your OpenTelemetry Collector (or traces directly from TypeScript/Node). See the quick starts below. **Endpoint & Auth** * Endpoint: `https://api.statsig.com/otlp` * Auth header: `statsig-api-key: ` Direct trace export to the Statsig OTLP endpoint is only available for TypeScript/Node. For all other languages, send traces to your OpenTelemetry Collector and forward from the Collector to Statsig over OTLP/HTTP. Need a deeper setup guide? See [Open Telemetry Logs and Metrics](/server/concepts/open_telemetry) for collector installation/config, and the [Traces Explorer quick start](/infra-analytics/send-traces) for language-specific trace examples. *** ## Application Telemetry quick starts Install dependencies: ```bash theme={null} npm install --save \ @opentelemetry/sdk-node \ @opentelemetry/auto-instrumentations-node \ @opentelemetry/exporter-trace-otlp-http \ @opentelemetry/exporter-metrics-otlp-http \ @opentelemetry/api-logs \ @opentelemetry/sdk-logs \ @opentelemetry/exporter-logs-otlp-http \ @opentelemetry/resources \ @opentelemetry/semantic-conventions ``` Initialize OpenTelemetry (e.g., `instrumentation.js`): ```js theme={null} // instrumentation.js const { NodeSDK } = require('@opentelemetry/sdk-node'); const { resourceFromAttributes } = require('@opentelemetry/resources'); const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require('@opentelemetry/semantic-conventions'); // import if you want to enable traces const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http'); const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-http'); const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics'); const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node'); // For troubleshooting, set the log level to DiagLogLevel.DEBUG // const { diag, DiagConsoleLogger, DiagLogLevel } = require('@opentelemetry/api'); // diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG); const statsigKey = process.env.STATSIG_SERVER_SDK_SECRET; const headers = { 'statsig-api-key': statsigKey ?? '' }; const sdk = new NodeSDK({ resource: resourceFromAttributes({ [ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'statsig-node-service', [ATTR_SERVICE_VERSION]: process.env.VERSION || '1', env: process.env.NODE_ENV || 'development', }), traceExporter: new OTLPTraceExporter({ url: 'https://api.statsig.com/otlp/v1/traces', // or // url: /v1/traces headers, }), metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ url: 'https://api.statsig.com/otlp/v1/metrics', // or // url: /v1/metrics headers, }), exportIntervalMillis: 60000, }), instrumentations: [getNodeAutoInstrumentations()], }); sdk.start(); ``` To set up application logs with OTel, you can use the pino or winston bridges. The example below using [pino](https://getpino.io/#/) with [pino auto instrumentation](https://www.npmjs.com/package/@opentelemetry/instrumentation-pino). Install the pino instrumentation: ```bash theme={null} npm i pino @opentelemetry/instrumentation-pino ``` ```js theme={null} // instrumentation.js (continued) const { BatchLogRecordProcessor } = require('@opentelemetry/sdk-logs'); const statsigKey = process.env.STATSIG_SERVER_SDK_SECRET; const headers = { 'statsig-api-key': statsigKey ?? '' }; const sdk = new NodeSDK({ // ... other config ... logRecordProcessors: [ new BatchLogRecordProcessor( new OTLPLogExporter({ url: 'https://api.statsig.com/otlp/v1/logs', // or // url: /v1/logs headers }) ), ], instrumentations: [getNodeAutoInstrumentations(), new PinoInstrumentation()], }); // in your application code, e.g., app.js const pino = require('pino'); const logger = pino(); logger.info('OTel logs initialized'); ``` The Statsig SDK also supports forwarding logs to Log Explorer; see the alternative logging example below. ```js theme={null} // Requires: npm i @statsig/statsig-node-core const { Statsig, StatsigUser } = require('@statsig/statsig-node-core'); const s = new Statsig(process.env.STATSIG_SERVER_SDK_SECRET); await s.initialize(); const user = new StatsigUser({ userID: 'a-user', custom: { service: process.env.OTEL_SERVICE_NAME || 'my-node-service' }, }); // levels: trace, debug, info, log, warn, error s.forwardLogLineEvent(user, 'info', 'service started', { version: process.env.npm_package_version }); try { // your app code } catch (err) { s.forwardLogLineEvent(user, 'error', 'unhandled error', { message: String(err?.message || err), stack: err?.stack, }); } ``` Run your service: make sure that you require or import `instrumentation.js` before any other application code to ensure instrumentation is set up correctly. ```bash theme={null} STATSIG_SERVER_SDK_SECRET=YOUR_SECRET \ OTEL_SERVICE_NAME=my-node-service \ node -r ./instrumentation.js app.js ``` Tip: you can configure exporters via env instead of code: * `OTEL_EXPORTER_OTLP_ENDPOINT=https://api.statsig.com/otlp` * `OTEL_EXPORTER_OTLP_HEADERS=statsig-api-key=${STATSIG_SERVER_SDK_SECRET}` * `OTEL_EXPORTER_OTLP_PROTOCOL=http/json` You can view the official Next.js OpenTelemetry instructions for [pages router here](https://nextjs.org/docs/pages/guides/open-telemetry) and for [app router here](https://nextjs.org/docs/app/guides/open-telemetry) Install dependencies: ```bash theme={null} npm install @opentelemetry/sdk-node @opentelemetry/resources @opentelemetry/semantic-conventions @opentelemetry/sdk-trace-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/auto-instrumentations-node ``` Add `instrumentation.ts` at the app root (Next 13+): ```ts theme={null} // instrumentation.ts import { NodeSDK } from '@opentelemetry/sdk-node'; import { resourceFromAttributes } from '@opentelemetry/resources'; import { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } from '@opentelemetry/semantic-conventions'; // For troubleshooting, set the log level to DiagLogLevel.DEBUG // const { diag, DiagConsoleLogger, DiagLogLevel } = require('@opentelemetry/api'); // diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG); export async function register() { const headers = { 'statsig-api-key': process.env.STATSIG_SERVER_SDK_SECRET ?? '' }; const sdk = new NodeSDK({ resource: new Resource({ [ATTR_SERVICE_NAME]: process.env.OTEL_SERVICE_NAME || 'statsig-node-service', [ATTR_SERVICE_VERSION]: process.env.VERSION || '1', env: process.env.NODE_ENV || 'development', }), traceExporter: new OTLPTraceExporter({ url: 'https://api.statsig.com/otlp/v1/traces', // or // url: /v1/traces headers, }), metricReader: new PeriodicExportingMetricReader({ exporter: new OTLPMetricExporter({ url: 'https://api.statsig.com/otlp/v1/metrics', // or // url: /v1/metrics headers, }), exportIntervalMillis: 60000, }), instrumentations: [getNodeAutoInstrumentations()], }); sdk.start(); } ``` To set up application logs with OTel, you can use the pino or winston bridges. The example below using [Pino](https://getpino.io/#/) with [Pino auto instrumentation](https://www.npmjs.com/package/@opentelemetry/instrumentation-pino). Install the pino instrumentation: ```bash theme={null} npm i pino @opentelemetry/instrumentation-pino ``` ```js theme={null} // instrumentation.ts (continued) import { BatchLogRecordProcessor } from '@opentelemetry/sdk-logs'; const statsigKey = process.env.STATSIG_SERVER_SDK_SECRET; const headers = { 'statsig-api-key': statsigKey ?? '' }; const sdk = new NodeSDK({ // ... other config ... logRecordProcessors: [ new BatchLogRecordProcessor( new OTLPLogExporter({ url: 'https://api.statsig.com/otlp/v1/logs', // or // url: /v1/logs headers }) ), ], instrumentations: [getNodeAutoInstrumentations(), new PinoInstrumentation()], }); // in your application code, e.g., app.ts import pino from 'pino'; const logger = pino(); logger.info('OTel logs initialized'); ``` The Statsig SDK also supports forwarding logs to Log Explorer; see the alternative logging example below. ```js theme={null} // Requires: npm i @statsig/statsig-node-core import { Statsig, StatsigUser } from '@statsig/statsig-node-core'; const s = new Statsig(process.env.STATSIG_SERVER_SDK_SECRET); await s.initialize(); const user = new StatsigUser({ userID: 'a-user', custom: { service: process.env.OTEL_SERVICE_NAME || 'my-node-service' }, }); // levels: trace, debug, info, log, warn, error s.forwardLogLineEvent(user, 'info', 'service started', { version: process.env.npm_package_version }); try { // your app code } catch (err) { s.forwardLogLineEvent(user, 'error', 'unhandled error', { message: String(err?.message || err), stack: err?.stack, }); } ``` In Next.js, mark '@statsig/statsig-node-core' as a server external package in `next.config.js` to avoid bundling. ### Using Vercel + Statsig integration If you're deploying to Vercel, you can use the \[Statsig + Vercel integration]\((/integrations/vercel/) to automatically forward logs from Vercel to Statsig. * Keep `STATSIG_SERVER_SDK_SECRET` out of client bundles (do not use `NEXT_PUBLIC_`). * Client/browser tracing requires separate web tracer setup; do not send secrets client-side. Consider routing via a Collector. Sending OTLP data directly to statsig without a collector is only supported for Node.js applications at this time. For other languages and frameworks, you can send OTLP data to a collector and have the collector forward the data to Statsig. See the [Collector quick starts](#collector-quick-starts) below for example configurations. And see the [OpenTelemetry Language APIs & SDKs documentation](https://opentelemetry.io/docs/languages/) for installation and configuration instructions for other languages and frameworks. *** ## Collector quick starts Running a Collector is optional but recommended for production workloads. Use the OpenTelemetry Collector as a gateway to receive OTLP from your applications and forward to Statsig. This is useful if you want to centralize telemetry collection, add advanced sampling methods like tail-based sampling, or scrape logs/metrics from hosts or Kubernetes. Create a minimal `values.yaml` for the OpenTelemetry Collector that forwards all signals (traces, metrics, logs) to Statsig: ```yaml title="values.yaml" theme={null} config: receivers: otlp: protocols: http: grpc: processors: batch: {} exporters: otlphttp: endpoint: https://api.statsig.com/otlp encoding: json headers: statsig-api-key: ${env:STATSIG_SERVER_SDK_SECRET} service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp] metrics: receivers: [otlp] processors: [batch] exporters: [otlphttp] logs: receivers: [otlp] processors: [batch] exporters: [otlphttp] ``` Install the Collector with Helm: ```bash theme={null} helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update helm install otel-gateway open-telemetry/opentelemetry-collector \ -n otel --create-namespace \ -f values.yaml ``` Provide the Statsig key as an environment variable to the Collector pods (for example via a Secret and envFrom). Your applications then send OTLP to the in-cluster Collector endpoint (for example `http://otel-gateway-collector.otel.svc.cluster.local:4318`). For a production setup that also scrapes Kubernetes logs/metrics, see the full guide: [Open Telemetry Logs and Metrics](/server/concepts/open_telemetry). **Version requirement** The `encoding: json` option in the OTLP HTTP exporter requires Collector v0.95.0 or newer. If you pin the image via Helm values, set `image.tag: "0.95.0"` (or newer). Use Docker Compose to run a Collector gateway that accepts OTLP and forwards to Statsig. ```yaml title="docker-compose.yaml" theme={null} services: otel-collector: image: otel/opentelemetry-collector-contrib:latest command: ["--config=/etc/otel-collector-config.yaml"] environment: - STATSIG_SERVER_SDK_SECRET=${STATSIG_SERVER_SDK_SECRET} volumes: - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml ports: - "4317:4317" # OTLP gRPC - "4318:4318" # OTLP HTTP ``` Create the Collector config referenced above: ```yaml title="otel-collector-config.yaml" theme={null} receivers: otlp: protocols: http: grpc: processors: batch: {} exporters: otlphttp: endpoint: https://api.statsig.com/otlp encoding: json headers: statsig-api-key: ${env:STATSIG_SERVER_SDK_SECRET} service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlphttp] metrics: receivers: [otlp] processors: [batch] exporters: [otlphttp] logs: receivers: [otlp] processors: [batch] exporters: [otlphttp] ``` Start the Collector: ```bash theme={null} export STATSIG_SERVER_SDK_SECRET=YOUR_SECRET docker compose up -d ``` Point your applications at the Collector (HTTP): `http://localhost:4318` (or `http://otel-collector:4318` from other compose services). The Collector forwards to Statsig with your key. You can run the Collector in other environments (VMs, bare metal, etc) using the config below. See the [Collector documentation](https://opentelemetry.io/docs/collector/installation/) for other installation and deployment methods. ```yaml title="otel-collector-config.yaml" theme={null} exporters: otlphttp: endpoint: https://api.statsig.com/otlp encoding: json headers: statsig-api-key: ${env:STATSIG_SERVER_SDK_SECRET} ``` *** ## Common Collector Configs (K8s & Docker) The following examples show popular receivers/processors you can enable in your Collector and still export to Statsig via the same `otlphttp` exporter. These components live in the contrib distribution. Use an image that includes them: * Docker: `otel/opentelemetry-collector-contrib` or newer * Helm: set `image.repository: otel/opentelemetry-collector-contrib` (and a compatible `image.tag`) Helm values (contrib image): ```yaml title="values.yaml" theme={null} image: repository: otel/opentelemetry-collector-contrib tag: "latest" pullPolicy: IfNotPresent ``` ### A. File logs (filelog receiver) Reads and parses logs from files on disk. Useful for hosts, containers, or Kubernetes nodes. Minimal example: ```yaml theme={null} receivers: filelog: include: [ /var/log/myservice/*.json ] start_at: beginning operators: - type: json_parser timestamp: parse_from: attributes.time layout: '%Y-%m-%dT%H:%M:%S%z' processors: batch: {} exporters: otlphttp: endpoint: https://api.statsig.com/otlp encoding: json headers: statsig-api-key: ${env:STATSIG_SERVER_SDK_SECRET} service: pipelines: logs: receivers: [filelog] processors: [batch] exporters: [otlphttp] ``` Kubernetes tip: to tail container logs on nodes, mount host paths (e.g., `/var/log/pods` and `/var/lib/docker/containers`) into the Collector DaemonSet and set `include` to those paths. ### B. EC2 resource detection (resourcedetection processor) Automatically adds AWS EC2 metadata (cloud provider, region/zone, instance id) to your telemetry. ```yaml theme={null} processors: resourcedetection/ec2: detectors: [env, ec2] timeout: 2s override: false service: pipelines: traces: receivers: [otlp] processors: [resourcedetection/ec2, batch] exporters: [otlphttp] metrics: receivers: [otlp] processors: [resourcedetection/ec2, batch] exporters: [otlphttp] logs: receivers: [otlp] processors: [resourcedetection/ec2, batch] exporters: [otlphttp] ``` Permissions: the Collector must be able to reach the EC2 metadata service (IMDS). Ensure network access to `169.254.169.254` and IMDSv2 where required. ### C. Docker container metrics (docker\_stats receiver) Emits container CPU, memory, network, and block IO metrics by querying the Docker daemon. ```yaml theme={null} receivers: docker_stats: endpoint: unix:///var/run/docker.sock collection_interval: 15s processors: batch: {} exporters: otlphttp: endpoint: https://api.statsig.com/otlp encoding: json headers: statsig-api-key: ${env:STATSIG_SERVER_SDK_SECRET} service: pipelines: metrics: receivers: [docker_stats] processors: [batch] exporters: [otlphttp] ``` Requirements: * Linux only (not supported on darwin/windows). * Mount the Docker socket into the Collector container: `/var/run/docker.sock`. *** ## Resources * OpenTelemetry Collector: [https://opentelemetry.io/docs/collector/](https://opentelemetry.io/docs/collector/) * Kubernetes Collector components: [https://opentelemetry.io/docs/platforms/kubernetes/collector/components/](https://opentelemetry.io/docs/platforms/kubernetes/collector/components/) * Helm chart: [https://github.com/open-telemetry/opentelemetry-helm-charts](https://github.com/open-telemetry/opentelemetry-helm-charts) * Collector configuration reference: [https://opentelemetry.io/docs/collector/configuration](https://opentelemetry.io/docs/collector/configuration) * OTLP protocol specification: [https://opentelemetry.io/docs/specs/otlp/](https://opentelemetry.io/docs/specs/otlp/) * Filelog receiver (contrib): [https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver) * Resource detection processor (contrib): [https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/resourcedetectionprocessor) * Docker stats receiver (contrib): [https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/dockerstatsreceiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/dockerstatsreceiver) * Collector contrib distribution: [https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib) # Logs Explorer Overview Source: https://docs.statsig.com/infra-analytics/logs-explorer Search and analyze all of your product’s logs in one place. Logs Explorer lets you query logs, traces, and ingested events from a single interface. Use it the same way whether you’re debugging infrastructure issues or investigating product event streams. * **Search**: Slice logs down to only what's relevant (by service, host, status code, etc.) * **Group**: Aggregate logs by dimensions like region, status, or browser. * **Visualize**: Plot log groupings over time to spot spikes, regressions, or anomalies instantly. *** ### Getting Started with Log Explorer To get started with Log Explorer, follow the [OTEL onboarding guide](/infra-analytics/getting-started.mdx) to set up log ingestion. Once that's ready, you can navigate from **Infra Analytics → Log Explorer** from the Statsig left menu. You can also use Logs Explorer in [Events Mode](/infra-analytics/events-mode-logs-explorer) to search and analyze your existing Statsig Events — no additional instrumentation needed. You can switch between Logs and Events mode using the dropdown left of the search bar. Logs Explorer Switch Mode *** ### Searching in Logs Explorer * **Write custom queries**: Check out our [syntax guide](/infra-analytics/logs-explorer-queries) to craft your search * **Using the query builder**: Point-and-click to construct filters without syntax overhead. Logs Explorer Overview # Query Syntax for Logs Explorer Source: https://docs.statsig.com/infra-analytics/logs-explorer-queries Filter, group, and visualize logs in Statsig Logs Explorer with precision using query syntax, attribute filters, time ranges, and saved query templates. This page covers common syntax you can use in day-to-day investigations, plus a few ready-to-edit examples. Plain text searches will only match against the **log message field**, not the entire log body. ## Basics ### Property Prefixes | Prefix | Description | Example | | ----------- | ----------------------- | -------------------------------------- | | `@property` | Event or log properties | `@traceId:"1a3cg5"` | | `#property` | Reserved properties | `#custom_event:"shopping_cart_opened"` | | `$user` | User identifiers | `$stableID:"abcdef"` | | *none* | User properties | `tier:prod` | ### Logical Operators These are query-level connectors. They combine or negate multiple conditions. | Operator | Description | Example | | ---------------------- | -------------------- | --------------------------------------- | | `AND` (case sensitive) | Match all conditions | `level:error AND service:api-gateway` | | `OR` (case sensitive) | Match any condition | `level:error OR level:warn` | | `!=` or `!:` | Exclude a condition | `status!:error OR service!:api-gateway` | ### Other Operators These are field-level conditions. | Operator | Description | Example | | ------------ | --------------------- | --------------------------------------- | | `:` or `=` | equals | `status_code:200` or `status_code=200` | | `!=` or `!:` | not equals | `level!:debug` or `level!=debug` | | `>` `<` | greater/less than | `latency_ms>500` or `latency_ms<1000` | | `>=` `<=` | greater/less or equal | `latency_ms>=500` or `latency_ms<=2000` | ### Wildcard Search You can use the `*` character as a wildcard in queries. A wildcard matches zero or more characters inside a field value. 🐌 Wildcards can impact the performance of your query, so use them sparingly. ### Additional Examples | Query | Description | Matches | | --------------------------------- | --------------------------------------------------- | ---------------------------------------------------------------- | | `@"log.file.path":"/logs/pods/*"` | Path begins with `/logs/pods/` | `/logs/pods/1234/stdout.log` | | `@"log.file.path":"*/logs/pods/"` | Path ends with `/logs/pods/` | `/mnt/storage/logs/pods/` | | `service:"api-*"` | Services with names starting with `api-` | `api-gateway`, `api-auth` | | `@route:"*products/*"` | Page routes containing the string `products/` | `shopify.com/products/tshirts`, `shopify.com/cart/products/view` | | `@message!:""` | Return logs where the message field is **not null** | any log that has a `message` field | # Infra Analytics Overview Source: https://docs.statsig.com/infra-analytics/overview Monitor and debug the health of your services directly inside Statsig with Infra Analytics, including logs, metrics, traces, alerts, and dashboards. **[Infra Analytics](https://statsig.com/infra-analytics)** pulls in logs, metrics, and alerts so you get system-level observability in the same place you analyze product outcomes. * Collect metrics and traces through OpenTelemetry (OTEL) * Search and analyze logs to investigate issues * Create alerts tied to service health * Connect infrastructure signals to product analytics for a unified understanding of impact ## Feature Highlights * **Logs Explorer**: Debug incidents with search, filters, and visualizations * **Topline Alerts**: Catch regressions and anomalies with log/metric-based triggers * **Metrics Explorer**: Inspect infrastructure metrics alongside product metrics *** ## Ready to Get Started? * 👉 [Set up OTEL ingestion](/infra-analytics/getting-started) to start sending logs and metrics * 👀 Traces are in limited preview — ping us in Slack if you'd like access # Traces Explorer Quick Start Source: https://docs.statsig.com/infra-analytics/send-traces Minimal OTLP/HTTP code examples for exporting distributed traces to Statsig's Traces Explorer from Node.js, Python, Go, Java, and other popular languages. Want Logs and Metrics too? See [Getting Started](/infra-analytics/getting-started) for more in-depth OpenTelemetry Collector setup instructions. Use the OTLP/HTTP traces endpoint to forward spans into Traces Explorer. Authenticate with your Server Secret key in the header: * Endpoint: `https://api.statsig.com/otlp/v1/traces` * Header: `statsig-api-key: ` Direct to API is currently supported for TypeScript/Node only. For all other languages, send traces to your OpenTelemetry Collector and configure it to forward to Statsig over OTLP/HTTP. Point non-TypeScript apps to your Collector (for example `http://localhost:4318/v1/traces`) and configure the Collector to forward to Statsig: ```yaml title="collector.yaml" theme={null} receivers: otlp: protocols: http: exporters: otlphttp: endpoint: https://api.statsig.com/otlp encoding: json headers: statsig-api-key: ${env:STATSIG_SERVER_SDK_SECRET} service: pipelines: traces: receivers: [otlp] exporters: [otlphttp] ``` ```bash theme={null} npm install @opentelemetry/sdk-node @opentelemetry/sdk-trace-node @opentelemetry/sdk-trace-base @opentelemetry/exporter-trace-otlp-http @opentelemetry/resources @opentelemetry/semantic-conventions @opentelemetry/api ``` ```js theme={null} // trace.js const { NodeSDK } = require('@opentelemetry/sdk-node'); const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http'); const { BatchSpanProcessor } = require('@opentelemetry/sdk-trace-base'); const { Resource } = require('@opentelemetry/resources'); const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions'); const { trace } = require('@opentelemetry/api'); const sdk = new NodeSDK({ resource: new Resource({ [SemanticResourceAttributes.SERVICE_NAME]: 'trace-sample-node', }), spanProcessor: new BatchSpanProcessor( new OTLPTraceExporter({ url: 'https://api.statsig.com/otlp/v1/traces', headers: { 'statsig-api-key': process.env.STATSIG_SERVER_SDK_SECRET || '' }, }), ), }); sdk.start().then(() => { const tracer = trace.getTracer('example'); const span = tracer.startSpan('do-work'); span.setAttribute('example', true); span.end(); setTimeout(() => sdk.shutdown(), 1000); }); ``` ```bash theme={null} pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http ``` ```python theme={null} # trace.py from opentelemetry import trace from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter trace.set_tracer_provider( TracerProvider(resource=Resource.create({"service.name": "trace-sample-python"})) ) exporter = OTLPSpanExporter( endpoint="http://localhost:4318/v1/traces", # your Collector ) trace.get_tracer_provider().add_span_processor(BatchSpanProcessor(exporter)) tracer = trace.get_tracer(__name__) with tracer.start_as_current_span("do-work") as span: span.set_attribute("example", True) trace.get_tracer_provider().shutdown() ``` ```bash theme={null} go get go.opentelemetry.io/otel/sdk go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp go.opentelemetry.io/otel/semconv/v1.26.0 ``` ```go theme={null} // main.go package main import ( "context" "log" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp" "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" semconv "go.opentelemetry.io/otel/semconv/v1.26.0" ) func main() { ctx := context.Background() exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpointURL("http://localhost:4318/v1/traces"), // your Collector ) if err != nil { log.Fatal(err) } tp := sdktrace.NewTracerProvider( sdktrace.WithBatcher(exporter), sdktrace.WithResource(resource.NewWithAttributes( semconv.SchemaURL, semconv.ServiceNameKey.String("trace-sample-go"), )), ) otel.SetTracerProvider(tp) tracer := otel.Tracer("example") ctx, span := tracer.Start(ctx, "do-work") span.SetAttributes(attribute.Bool("example", true)) span.End() _ = tp.Shutdown(ctx) } ``` Need a deeper setup guide? See [Open Telemetry Logs and Metrics](/server/concepts/open_telemetry) for collector installation/config, and if you want Logs and Metrics too? See [Getting Started](/infra-analytics/getting-started) for more in-depth OpenTelemetry Collector setup instructions. # Topline Alerts with Logs Source: https://docs.statsig.com/infra-analytics/topline-alerts-logs Detect infrastructure regressions by evaluating logs in real time with Statsig topline alerts on log volume, error rates, and pattern-based thresholds. By combining log filters with [Topline Alerts](/product-analytics/topline-alerts), you will know immediately when things start failing without writing custom scripts or dashboards. ### When to Use Log-based Alerts * **Monitoring success rates** - Catch regressions before they impact SLOs. * **Detecting error spikes** - Trigger when 5xx or other errors rise above baseline. * **Isolate by segment** - Identify failures concentrated in by region, client, or device type ## Create a Log-based Topline Alert (Statsig Cloud) In this example, we're going to create a monitor for the success rate of a GET request.
  • Go to Analytics → Topline Alerts in the product menu.
  • Click +Create.
  • Enter a name for your alert.
Alert setup step 1
  • Select statsig::log\_line as your event.
  • Apply filters to define what constitutes success and failure.
If you're unsure which fields to filter on, open the Logs Explorer and inspect the log body. Alert setup step 2
  • Define the formula for calculating your success rate.
  • (Optional) Add a group-by dimension.
  • Alert setup step 3
  • Set thresholds and the evaluation window.
  • In this example, we will trigger a warning when success rate drops below 99.5%. We will trigger an alert when success rate drops below 99.0%.

    Alert setup step 4
    • Add the alert title and description for context.
    • Include diagnostic hints (e.g. "Check version X" or "Android requests timing out").
    • Add subscribers.
    • Set priority.
    Alert setup step 5
  • Click Save.
  • Once your alert is set up, you can visit the Diagnostics tab to see a history of alert triggers.
  • Alert setup step 6
    *** ## Best Practices * Set up [Slack notifications](/integrations/slack) for team visibility * Keep formulas simple (ratios & percentages are easiest to scan) * Add group-by dimensions (like country or app version) to pinpoint where issues occur * Write clear notification text that explains what the alert means # Custom Proxy for Statsig API Source: https://docs.statsig.com/infrastructure/api_proxy/custom_proxy Run a custom Statsig API proxy in your own infrastructure to cache SDK requests, reduce latency, and control egress traffic from production servers. ## Overview Instead of sending API requests directly to Statsig, you can set up your own environment that proxies requests from your custom domain name to Statsig. This makes it less likely for tracking blockers to intercept your APIs, and allows you to capture more data. There are many ways to set up custom proxies. We are showing instructions for a few common service providers here. **Important: Default Endpoints Are Blocked** The default Statsig endpoints (like `/v1/log_event`) are commonly blocked by tracking blockers. To ensure your proxy works effectively: 1. **Use a custom endpoint name** specific to your product (e.g., `/v1/my-product-data` instead of `/v1/log_event`) 2. **Rewrite the URL in your proxy** to map your custom endpoint to the actual Statsig endpoint (`log_event`) 3. **Do not use passthrough** for the endpoint path - you must rewrite it Additionally, your proxy should not try to deserialize the payload body. This will improve robustness by reducing risk of integration issues from Server SDK -> Proxy -> Client SDK, as well as improve efficiency of the proxy. For example, client SDKs may change encoding to compress payloads, which would break if your proxy does not accept the new format (e.g. gzip). ## Approaches ### AWS CloudFront #### Prerequisites * Write access to your DNS settings. * Write access on your AWS CloudFront and Lambda console. * Access to a SSL certificate for your custom domain. #### Setup On your [AWS CloudFront console](https://console.aws.amazon.com/cloudfront/), * Click on Create distribution. * In the Origin section, * Set the Origin Domain to `api.statsig.com`. * Set the Protocol to `HTTPS only`. Origin section * In the Default cache behavior section, * Set Viewer protocol policy to `Redirect HTTP to HTTPS`. * Set Allowed HTTP methods to `GET, HEAD, OPTIONS, PUT, POST, PATCH, DELETE`. * In the Cache Key and origin requests subsection, allow all headers and parameters to be forwarded to the Origin, and allow CORS requests for the Origin. Default cache behavior section * In Function associations section, * Add a Lambda\@Edge function to Origin request to rewrite the `Host` header to `api.statsig.com` and rewrite custom endpoint paths to Statsig endpoints. Please refer to [the AWS tutorial on creating a Lambda@Edge function](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/lambda-edge-how-it-works-tutorial.html). Function associations section You can use the following javascript code snippet in your Lambda\@Edge function. This example rewrites a custom endpoint `/v1/my-product-data` to the Statsig endpoint `/v1/log_event`: ```javascript theme={null} export const handler = async (event, context, callback) => { const request = event.Records[0].cf.request; request.headers.host[0].value = "api.statsig.com"; // Rewrite custom endpoint to Statsig endpoint // Replace 'my-product-data' with your custom endpoint name if (request.uri.includes('/v1/my-product-data')) { request.uri = request.uri.replace('/v1/my-product-data', '/v1/log_event'); } return callback(null, request); }; ``` * In Settings, * Add an Alternate domain name (CNAME) to be your preferred domain name to use for the custom proxy, e.g. `statsig.example.com`. * Add a Custom SSL certificate. You will need to follow the [AWS guide for Alternate domain name](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/CNAMEs.html#alternate-domain-names-requirements) to add a SSL certificate. * Click on Create distribution. Settings * You will get a Distribution domain name (e.g. `d111111abcdef8.cloudfront.net`) once it is provisioned. In your DNS settings (depending on your DNS provider), * Add a CNAME record in your custom DNS record: * Host name: `statsig.example.com` * Type: `CNAME` * Data: `d111111abcdef8.cloudfront.net` (The Distribution domain name from AWS) * Your proxy should now be setup. See [Using Your Proxy](#using-your-proxy) for instructions on how to configure your Statsig SDK. ### Cloudflare Worker #### Prerequisites You will need a Cloudflare account. Visit [https://www.cloudflare.com](https://www.cloudflare.com/) to set one up. #### Setup Once you are logged into Cloudflare. You can follow these steps: 1. Navigate to "Workers & Pages > Overview" in the left rail to create a new worker. 1-cloudflare-create You may see a different experience if you already have workers on your account. 3. Name you new worker whatever you would like and then click "Deploy". 2-cloudflare-deploy 4. Once deployed, click "Edit Code". 3-cloudflare-edit-code 5. Copy and paste the following snippet into the `worker.js` file, then hit "Deploy". This example rewrites a custom endpoint `/v1/my-product-data` to the Statsig endpoint `/v1/log_event`: ```javascript theme={null} export default { async fetch(request, _env, _ctx) { const url = new URL(request.url); // Rewrite custom endpoint to Statsig endpoint // Replace 'my-product-data' with your custom endpoint name let pathname = url.pathname; if (pathname.includes('/v1/my-product-data')) { pathname = pathname.replace('/v1/my-product-data', '/v1/log_event'); } const original = new Request(request); original.headers.delete("cookie"); return fetch( `https://statsigapi.net${pathname}${url.search}`, original ); }, }; ``` 4-cloudflare-paste-snippet 6. Your worker should now be deployed and ready to use. See [Using Your Proxy](#using-your-proxy) for instructions on how to configure your Statsig SDK. ## Using Your Proxy Once you have a proxy setup, you will need to take its URL and apply it to the SDK. For JavaScript client SDKs, use `StatsigOptions.networkConfig.api`. You can visit [Statsig Options](/client/javascript-sdk#statsig-options) to read about the JavaScript-specific options. Other SDKs may expose the proxy base URL differently, but the goal is the same: point the SDK at your proxy instead of Statsig's default domains. The following is what initializing with a proxy looks like in JavaScript: ```typescript theme={null} Statsig.initialize(mySdkKey, myUser, { networkConfig: { api: "https://my-statsig-proxy.com/v1", }, }); ``` In JavaScript, `networkConfig.api` is a base URL. The SDK appends endpoint paths like `/initialize` and `/rgstr` automatically. Use `initializeUrl` only when you want to override initialization independently of the other endpoints. There is no generic `api` fallback option; failover is configured per endpoint with `initializeFallbackUrls` and `logEventFallbackUrls`. Depending on the SDK type, version, and proxy approach you are using, you may not need to append `'/v1'` to the end of your api string, for example `"https://my-statsig-proxy.com/"`. ### JavaScript Failover Example If you want JavaScript clients to use your proxy as the primary endpoint and a second endpoint as failover, configure fallback URLs per endpoint: ```typescript theme={null} Statsig.initialize(mySdkKey, myUser, { networkConfig: { api: "https://my-statsig-proxy.com/v1", initializeFallbackUrls: [ "https://my-proxy-cache.com/v1/initialize", ], logEventFallbackUrls: [ "https://my-proxy-cache.com/v1/rgstr", ], }, }); ``` ### Configuring Custom Endpoints If you've configured your proxy to use custom endpoint names (recommended to avoid tracking blockers), you'll need to configure the SDK to use those custom endpoints. The SDK will append the endpoint path to your base URL. For example, if your proxy rewrites `/v1/my-product-data` to `/v1/log_event`, you would configure: ```typescript theme={null} Statsig.initialize(mySdkKey, myUser, { networkConfig: { api: "https://my-statsig-proxy.com/v1", logEventUrl: "https://my-statsig-proxy.com/v1/my-product-data", }, }); ``` Check your SDK's documentation for the specific configuration options available for customizing endpoint URLs. # API Proxy Source: https://docs.statsig.com/infrastructure/api_proxy/introduction Introduction to Statsig API proxy options for caching SDK requests, reducing latency, and meeting compliance requirements in your infrastructure. This section provides documentation on the various API proxy options supported when using Statsig. ## Why Use an API Proxy? There are several compelling reasons to implement an API proxy for Statsig network requests: 1. **Mitigate Tracking Blocker Impact**: Reduce the effect of tracking blockers on Statsig API usage. Default Statsig endpoints are commonly blocked by tracking blockers, so using a custom proxy with product-specific endpoint names is essential for reliable data collection. 2. **Meet Security Requirements**: Comply with internal network security and topology standards. 3. **Enhance API Availability**: Improve API availability guarantees within your internal network boundary. ## API Proxy Deployment Options API proxies can be deployed in various forms, each with its own advantages and considerations. The choice between a managed service and a self-hosted solution, or deploying outside versus inside your network, depends on your specific needs. We offer documentation on the following options. For both server SDKs and client SDKs: * [Custom Proxy](/custom_proxy): A fully customizable proxy that you own and operate in your environment. * [Managed Proxy](/infrastructure/managed-proxy): A lightweight, Statsig-owned proxy that provides out-of-the-box functionality. For server SDKs only: * [Forward Proxy](/server/concepts/forward_proxy): A Statsig-built proxy designed for deployment within your own environment. Choose the option that best aligns with your infrastructure requirements and operational preferences. If you have any questions/concerns, drop on in to our [slack channel](https://www.statsig.com/slack) and let us know. # Statsig Managed API Proxy Source: https://docs.statsig.com/infrastructure/api_proxy/managed-proxy Use the Statsig managed API proxy to cache SDK config and event requests, reduce client latency, and improve reliability without running your own proxy. An API proxy gives you a unique URL to send and receive data to/from Statsig servers. This makes it less likely to be intercepted by client-side or DNS-side blockers. This way you'll be able to get the right configuration for your applications and more data back from your applications. The Managed Proxy is available only for Pro or Enterprise tiers. A quick-and-easy way to prevent adblocking, we recommend the custom proxy to low volume customers, or before you have the time to setup the [Custom Proxy](/custom_proxy), which is a more robust and customizable solution. ## Why use a proxy A significant number of web-browser instances have some sort of tracking blockers installed. Sometimes these blockers end up blocking feature flags, experiments and even runtime dynamic config data, resulting in the exclusion of those users in the statistical power. Using a proxy that's unique to your application signals these tracking blockers that this is a necessary component of your application that's required for its functioning. ## Setting up a managed proxy If your project is in pro-tier or enterprise-tier, you will see an option to create a unique proxy for your SDK in the Settings -> Project -> Keys & Environments tab as shown below: Proxy creation option in Keys & Environments settings Clicking on 'Create a proxy' will generate a new unique worker on AWS and give you back a URL that you can start using immediately. Generated proxy URL configuration screen Currently the managed proxy that Statsig creates is hosted in `ap-south-1` region. If you want this hosted in a different region, reach out to Statsig support ## Using Your Proxy Once you have a proxy setup, you will need to take its URL and apply it to the SDK. For JavaScript client SDKs, use `StatsigOptions.networkConfig.api`. You can visit [Statsig Options](/client/javascript-sdk#statsig-options) to read about the JavaScript-specific options. Other SDKs may expose the proxy base URL differently, but the managed proxy URL serves the same purpose. The following is what initializing with a proxy looks like in JavaScript: ```typescript theme={null} Statsig.initialize(mySdkKey, myUser, { networkConfig: { api: "https://my-statsig-proxy.com/v1", }, }); ``` In JavaScript, `networkConfig.api` is a base URL, not the same thing as `initializeUrl`. Use `initializeUrl` only when you want to override initialization independently. There is no generic `api` fallback option; endpoint failover is configured with `initializeFallbackUrls` and `logEventFallbackUrls`. # Infrastructure Ops Overview Source: https://docs.statsig.com/infrastructure/introduction An overview of the Statsig infrastructure that powers reliable feature flagging, experimentation, and analytics at enterprise scale across global regions. ## Why Infrastructure Matters for Feature Flagging & Experimentation Feature flagging and experimentation platforms require robust infrastructure to deliver consistent, low-latency responses that don't impact an application's performance. Every feature gate evaluation, experiment assignment, and analytics event must be processed reliably to ensure: * **Consistent User Experiences**: Users should always receive the same feature variant throughout their session * **Real-time Decision Making**: Feature flags need to evaluate in milliseconds to avoid blocking your application * **Accurate Experiment Results**: Every user interaction must be captured and processed to generate reliable statistical insights * **Business Continuity**: Infrastructure downtime can't block feature releases or compromise running experiments ## Enterprise-Grade Scale & Reliability Statsig is built to process massive volumes of data while maintaining enterprise-grade reliability standards: ### Key Metrics * **2+ Trillion** events processed per day * **Over 3 Billion** unique monthly experiment subjects * **99.99%** infrastructure uptime for API & Console ### Scalable Architecture Statsig's infrastructure has been battle-tested by companies including including OpenAI, Atlassian, Microsoft, Notion, Flipkart, and other enterprises operating at massive scale. But all customers get access to enterprise-grade infrastructure from day one. This means never a reason to switch tools or migrate to an "enterprise-grade" solution as you grow. Statsig can scale with you regardless of volume. Our investment in scaled infrastructure also allows us to offer affordable pricing across all products and customer tiers. Infrastructure isn't limited to just serving configs and logging events. Statsig also provides comprehensive infrastructure management including: * Real-time health checks and monitoring * Automated guardrails on feature rollouts and releases * Multi-region deployment for global availability * Built-in redundancy and failover capabilities ## Getting Started The section includes documentation about infrastructure setup and operations guide for your team when using Statsig. In general, Statsig works out of box with no additional setup required. Some instructions here apply to specific use cases due to special requirements. If you have any questions/concerns, drop on in to our [slack channel](https://www.statsig.com/slack) and let us know. # Reliability FAQs Source: https://docs.statsig.com/infrastructure/reliability-faq Frequently asked questions about Statsig reliability, including SDK failover behavior, multi-region architecture, SLAs, and customer-side mitigations. Integrating your product with Statsig means depending on Statsig, and we take reliability seriously. Here are some questions many people have when trying to evaluate the risks, please feel free to reach out on Slack if you have questions that are not listed here. ## What does Statsig do to stay highly-available? * We actively track internal Service Level Objectives (SLOs) for availability to make sure our service has a high uptime. * Here are some examples of measures that we take to ensure our service reliability: * We handle bursts gracefully via autoscalers and over-provisioned resources * We have mechanisms to reduce unintended/malicious spikes and prevent DDoS attack * Our services are deployed in multiple regions. In case of a region goes down, traffic will be routed to other healthy regions. * We adopt the GitOps approach(e.g. code review, validation, CI/CD, etc.) for all of our infra changes to prevent human errors. * We have a 24/7 eng oncall rotation to solve customer-facing alerts and issues. ## Does Statsig use any caching to help with latency? * We use a combination of caching solutions, depending on the problem we are solving. For serving our console and API requests, most caching is done at the region or host level. ## What else does Statsig do to make sure the service is resilient? * Our SDKs are designed to be resilient in case there is an issue in the API requests, to make sure a seamless experience on your side. * Client SDKs: * The SDKs will use the latest values from Statsig server if the user is able to reach Statsig server; * Then it will use cached value from a previous session will be used, if available; * After that, the APIs require you to have set default values in your code, so that will be used. This means the worst case scenario your users will get the default experiences. * The SDKs will automatically retry failed event requests if for some reason Statsig event servers are unreachable. The Client SDKs will even persist failed log requests to local storage and retry in subsequent sessions. * Server SDKs: * They store rules for gates and experiments in memory, so it will be able to continue evaluating them even if Statsig is down. * There is also an option to “bootstrap” your server SDKs with rule values from a previous session if Statsig is down when your server is starting up. This can be done with a [Server Data Store](/server/concepts/data_store/), which lets you plug a storage provider into the Statsig SDK, to store your rule values. * The SDKs will automatically retry failed event requests if for some reason Statsig event servers are unreachable. ## What kind of automated testing does Statsig do? * Unit and integration tests run on every pull request * Continuous CI/CD running our unit/integrations test suites * Synthetic tests for Console and API use cases, mimicking customer requests * Stress tests to detect any performance issues * Continuous SDK tests run on every pull request and on schedule ## What does Statsig do to protect runtime code? We use Github and Dockerhub for code and binary storage. We keep track of the entire CI/CD process from source code to production deployment with traceable versioning and binary verification. # Monitoring the SDK Source: https://docs.statsig.com/infrastructure/sdk-monitoring Monitor Statsig SDK health from the Statsig console, including initialization success rate, evaluation latency, network errors, and version adoption. This latest release of structured logging and metrics, is currently only [available by the Python SDK](/server/pythonSDK/#sdk-monitoring-). Want it in another? Reach out in our [Support Slack](https://statsig.com/slack). ## SDK Metrics Some Statsig SDKs provide built-in metrics to help you monitor its performance and impact on your application. The specific implementation may vary by programming language, refer to the documentation for the language-specific SDK interface. ### Metric Interface Methods The following interface methods are provided by the Statsig SDK to track various metrics: * **Initialization (`init`)**: This method is called on sdk initialization and allows users to initialize their observability client (such as StatsD, OpenTelemetry, etc.), preparing the SDK to send metrics and logs to the chosen observability tool. * **Shutdown (`shutdown`)**: This method is called on sdk shutdown, and allow users to perform any actions to ensure graceful shutdown of the observability client, such as ensuring that any pending metrics or logs are properly handled and sent before the SDK is terminated. * **Counter**: A method that tracks occurrences of specific events. * **Gauge**: A method used to record point-in-time values, such as the number of active connections or other metrics that don’t accumulate over time. * **Distribution**: A method that tracks distributions of numerical data over time, such as latency or response times. * **Should Enable High Cardinality Tags**: This method is called on high cardinality tags and allows users to define if certain high cardinality tags (which can generate large data volumes) should be enabled for detailed tracking. By default, all high cardinality tags are disabled. ### List of Metrics Below is a list of the primary metrics currently available in the SDK: | **Metric Name** | **Type** | **Tags** | **Description** | | ------------------------------------- | ------------ | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `initialization` | distribution | `success`, `init_source`, `init_source_api`, `store_populated` | Tracks SDK initialization duration. | | `config_propagation_diff` | distribution | `source`, `source_api`, `lcut*`, `prev_lcut*` | Measures the time difference between the last config updated time vs the time that sdk received the config. | | `config_no_update` | counter | `source`, `source_api` | Tracks occurrences of no configuration updates. | | `events_successfully_sent_count` | counter | N/A | Tracks number of events sent successfully to the Statsig server. | | `sdk_exceptions_count` | counter | N/A | Tracks occurrences of unexpected exceptions caught. | | `grpc_received_message` | counter | N/A | GRPC Streaming received a new message | | `grpc_reconnected` | counter | N/A | GRPC streaming reconnected | | `grpc_streaming_failed_with_retry_ct` | distribution | N/A | Streaming failed and how many retry we are on (to estimate how long the client has been disconnected) | * All metrics are prefixed by `statsig.sdk.`, for example, the full initialization metric name in your integration will be `statsig.sdk.initialization`. * While `sdk_exceptions_count` metric captures all exceptions, certain errors (e.g., temporary network connectivity issues or timeouts) are expected to occur occasionally and are generally not cause for concern. Use this metric to identify unexpected or persistent issues that may require investigation. * Tags marked with `*` (such as `lcut` and `prev_lcut`) are high cardinality tags. ### Metric Tags High cardinality tags are tags that can generate large data dimensions when enabled. These tags are disabled by default, but can be enabled as through `Should Enable High Cardinality Tags` method on the observability client interface. High cardinality tags include: * `lcut`: The last configuration update timestamp. * `prev_lcut`: The previous configuration update timestamp. Metric Tags: * `source`: The source of the configuration update, such as network/bootstrap/datastore. * `source_api`: The API endpoint used to fetch the configuration update. * `success`: Indicates whether the initialization was successful. * `store_populated`: Indicates whether the configuration store was populated. # Statsig Domains Source: https://docs.statsig.com/infrastructure/statsig_domains Reference list of Statsig domain names used by SDKs and the console so you can allowlist outbound traffic from your network and firewall. Statsig uses the following domain names for its services. If you have a network policy set up inside your systems, you should allowlist all of the domains below or select domains based on the features you use. ## Statsig Console * `console.statsig.com` * `cdn.console.statsig.com` * `console.statsigcdn.com` ## Statsig API Services These domains are used by our SDKs to communicate with our backend for feature gates, dynamic configs and event logging. They are also used for other Statsig APIs, e.g. console APIs, integrations. * `api.statsig.com` * `featuregates.org` * `statsigapi.net` * `events.statsigapi.net` * `api.statsigcdn.com` * `featureassets.org` * `assetsconfigcdn.org` * `prodregistryv2.org` * `cloudflare-dns.com` * `beyondwickedmapping.org` Note that Statsig's SDKs may switch between these domains on-the-fly as part of our DNS resolution logic. **Why such odd names?** We constantly and dynamically update these domains to prevent overzealous blocking from browser ad blockers. These are updated whenever they pick up the existing ones. ### Don't care about ad-blockers? Use the following approach to minimize the number of Statsig domains you need to whitelist. The [networkConfig](/client/javascript-sdk#networkconfig-object) initialization option allows you to specify a single domain for both the initialization server and the log event server. ```js theme={null} new Statsig.StatsigClient('client-YOUR_KEY', {/* CONTEXT */}, { networkConfig: { initializeUrl: 'https://featureassets.org/v1/initialize', logEventUrl: 'https://prodregistryv2.org/v1/rgstr' } }); ``` ### Server-side APIs If you're just looking for a list of apis used by our backend/server SDKs, you need to allow: * `api.statsig.com` * `statsigapi.net` * `api.statsigcdn.com` * `prodregistryv2.org` * `idliststorage.blob.core.windows.net` (see below) ### Statsig User Segment Storage API The domain is used by our server SDKs to download the segment list for your project. If you do not use big id lists, you won't need this one. * `idliststorage.blob.core.windows.net` # Statsig IP Ranges Source: https://docs.statsig.com/infrastructure/statsig_ip_ranges Reference list of Statsig IP ranges used by SDKs and webhooks so you can allowlist inbound and outbound traffic in your firewall and network policies. Statsig reserves the following IP addresses and ranges for use by its services. If you have a network policy set up inside your systems, you should allowlist all of the IPs below or select IPs based on the direction of network requests. ## Outbound (Statsig -> Your Servers) These IPs are used when Statsig sends requests to your servers/systems. For example, Statsig imports data from your data warehouse that has a network policy allowing only certain IPs. * 20.29.232.235 * 20.190.14.199 * 34.138.242.148/30 (4 addresses) * 34.126.186.120/30 (4 addresses) * 34.168.242.172/30 (4 addresses) * 34.38.207.120/30 (4 addresses) Webhook requests can be very high volume and may not be initiated from these IP Ranges. Consider using [Webhook Signatures](/integrations/event_webhook#webhook-signature) to validate webhook requests. ## Inbound (Your Clients/Servers -> Statsig) These IPs back the domains (e.g. `api.statsig.com`, `featuregates.org`, `statsigapi.net`) of Statsig APIs. * 34.120.214.181 * 34.128.128.0/29 (8 addresses) # Agent Skills Repository Source: https://docs.statsig.com/integrations/agent-skills Use the public Statsig agent-skills repository to extend Cursor, Claude Code, and other coding agents with reusable Statsig workflows and capabilities. The [statsig-io/agent-skills](https://github.com/statsig-io/agent-skills) repository contains reusable skill packs for coding agents. Skills are packaged, shareable workflows for repeated tasks, built on top of the Statsig MCP tools and Console API. These skills build on the open [Agent Skills standard](https://agentskills.io/). ## Supported skills today | Name | Description | | ----------------------------- | ------------------------------------------------------------------------------------------------------------- | | `statsig` | Manage Statsig experiments, gates, dynamic configs, segments, layers, and audit logs through the Statsig MCP. | | `statsig-create-cloud-metric` | Draft or execute Statsig Cloud metric creation requests through the Statsig Console API. | | `statsig-dashboard` | Create, read, and update you Statsig dashboards. | ## Install from the repository Install a skill from this repo with the Vercel `skills` CLI: ```bash theme={null} npx skills add statsig-io/agent-skills ``` You can also: * List installable skills: `npx skills add statsig-io/agent-skills --list` * Install a skill globally for your user: `npx skills add -g statsig-io/agent-skills --skill statsig-dashboard` * Install every skill in this repo: `npx skills add statsig-io/agent-skills --all` After installation, compatible agents can discover each skill from its metadata. ## Requirements * `STATSIG_CONSOLE_API_KEY` for Statsig Console API access * Python 3 if you want to run bundled helper scripts directly # AI Development with Statsig Source: https://docs.statsig.com/integrations/ai_development_with_statsig Use the Statsig MCP server with AI coding agents like Claude Code, Cursor, and Codex to add killswitches and log critical events for AI-written code. ## Statsig's Role in AI Development Increasingly, developers are leaning on AI agents to accomplish small-to-midsized tasks. For small feature additions, this is low risk - but can still introduce bugs or low-quality user experiences. Having the agent be aware of Statsig means that it can automatically add logging for potential errors, and add feature gates to new features (or make sure they're behind existing gates) so that you can manage rollouts safely and measure any degredation. This adds up when considering the high velocity individuals can now achieve with help from these agents, and the ability of people to develop within codebases they have less context in with AI assistance. ## Recommended Approach Our recommended approach is to use an `AGENTS.md` (or any analagous file, such as `CLAUDE.md`) to provide instructions that you can tailor to your own workflow. The behaviors that are critical are: * Adding gates to new features or codepaths, and tuning how aggressively to do so * Automatically adding your user ID, and any test IDs your agent may log in as, to those gates for testing in development * Adding critical logs for errors and user behaviors * Currently adding these to tracking is manual, but we have plans to make this automatable via MCP for event count metrics. ### Boilerplate `Agents.md` file Fill in your Statsig ID, and follow the [MCP guide](/integrations/mcp/overview) to set up the MCP server. Consider updating naming conventions and logic around when to ask. We recommend using a limited-access console API key to control risk around a scenario where an agent deletes entities in Statsig. ``` # Statsig Development Guidelines It is important that we selectively killswitch new features or any new, risky codepaths using Statsig Feature Gates so that we can turn them off if they cause issues. ## When To Use Gates - We should consider adding a gate whenever we make significant changes to product features that have any risk of causing issues. - It is possible for this to be spammy, so clarify with users during planning what should be killswitched. - Start by assuming atomic, session-level features will be killswitched - e.g. "A new modal" would be, but not a copy change within a modal, or a bugfix. - Sometimes, adding a gate is just as dangerous as the feature itself. - Identify situations where this could occur - usually when adding the gate adds a large amount of complexity - and flag the risk to the user when asking if they want to create a gate. Always ask the user if they want to gate the feature. Often, users will want to include the feature behind an existing gate, so ask in this way, when appropriate: "This seems like a feature that you might want to be able to turn off remotely. Would you like to gate this feature, add it to an existing gate, or proceed without using gates?" Look for gates already being used for this feature type - often in the same file or nearby in the file tree for client code. Don't spend a bunch of time investigating code before asking, since it leads to a lot of hang time. This should be an early step in the process. Quickly check in with the user on gating before you do a deep codebase scan - though you might want to use a quick context check to identify risk. ## How to Use Gates - Create a new gate, if requested, using the statsig-local MCP server. If this server does not exist, ignore the above. - After creation, use that gate in code, following the local codebase patterns for accessing statsig. - This will normally be Statsig.useGate() or similar, but they may have an internal wrapper or use a general feature flag library which wraps Statsig. - When creating the gate, start it with a default-fail rules. The user, or we, can add rules later to release the gate. - Add a rule called "Development" which passes the user's ID, specified below. Make sure this is the first evaluated rule. ## Override instructions - The user's ID is the string "" - Use a custom ID gate condition, with the userID field set to that ID ## Naming Conventions - For these features, create a feature gate which is a 3-4 word description separated by dashes. - Do not use the word killswitch. Propose the name of the gate before creating it, allowing the user to update it if desired. Do not include your name (e.g. codex, claude, cursor) in the gate name. ## Logging - We would like to measure our gate changes as we roll out features. - If there's key interactions in the code which are not logged to Statsig, consider logging those events to statsig - Include critical metadata (e.g. information about what happened, key numeric values) Propose these logging changes to the user and see what they say, and only if they're very clearly needed. Similar to the gate checks, pattern match to what is in the repo already, e.g. Statsig.logEvent or a wrapper like LoggingContext.log() ## Statsig Implementation Instructions [left empty - fill with specific instructions about your Statsig implementation if it exists, such as specific calling patterns or gotchas] ## Guardrails and Critical Rules - In this context, only use this MCP server to create gates - Never run deletions or other destructive tools without explicit instructions from the user ``` # Akamai Edge KV Source: https://docs.statsig.com/integrations/akamai Run Statsig feature gate and experiment evaluations at the edge with Akamai EdgeWorkers for low-latency rules evaluation in your CDN layer. ## Overview Statsig’s Akamai Edge KV integration pushes Statsig Configs to Edge KV, providing low latency for gate and experiment evaluations directly in Akamai Edge KV. If you have the correct prerequisites - this should take \~60 minutes to get to the point where you are able to check experiments or gates on Akamai. ## Prerequisites You must have these prerequisites: 1. An Akamai account with [EdgeWorkers added to your contract](https://techdocs.akamai.com/edgeworkers/docs/add-edgeworkers-to-contract) and a Statsig account 2. [The Akamai CLI](https://developer.akamai.com/getting-started/cli) ## Setup Akamai EdgeWorker 1. Create an [EdgeWorker ID](https://techdocs.akamai.com/edgeworkers/docs/create-an-edgeworker-id-1) 2. Add the [EdgeWorker Behavior](https://techdocs.akamai.com/edgeworkers/docs/add-the-edgeworker-behavior-1) 3. Install the [Akamai CLI](https://developer.akamai.com/getting-started/cli) 4. Install the [Edgeworkers CLI](https://techdocs.akamai.com/edgeworkers/docs/akamai-cli#edgeworkers-cli) 5. Generate [EdgeGrid credentials](https://techdocs.akamai.com/developer/docs/edgegrid) ## Configure Integration First, enable the Akamai integration in the Statsig Console. Navigate to [Project Settings -> Integrations](https://console.statsig.com/integrations), and then select Akamai ### Authentication You will need to input the following EdgeGrid credentials from the previous section: Authentication ### Configuration Finalize the configuration by selecting a [namespace](https://techdocs.akamai.com/edgekv/docs/manage-access-to-edgekv) and [environment](https://techdocs.akamai.com/edgekv/docs/sandbox-support-edgekv). There is also an option to filter the configs that are synced into your KV namespace by a [Target App](/sdk-keys/target-apps). You may wish to enable this in the future as the size of your config payload grows. For now, you can leave this unchecked. Configuration After filling this out, click **Enable**. Within a minute, the Statsig backend should generate a config payload from your statsig project and push it into your KV namespace. Under your KV namespace, navigate to **KV Pairs** - you should see an entry starting with the prefix `statsig-`. The next part of that is your project ID. Copy that to the clipboard - you'll need it later. ## Add Statsig SDK to Worker 1. Create a Statsig [server secret key](/access-management/api-keys#server-secret-keys) and [configure it on Akamai](https://techdocs.akamai.com/developer/docs/edgegrid#credentials-as-environment-variables) 2. Clone the [Statsig Akamai Edge KV](https://github.com/statsig-io/akamai-statsig-example?tab=readme-ov-file#getting-started) repo (or if you already have an Edge KV package, apply these changes to it) 3. Follow instructions in the repo ## Additional Links [Akamai EdgeWorkers](https://developer.akamai.com/akamai-edgeworkers-overview) [Statsig Akamai Edge KV Repo](https://github.com/statsig-io/akamai-statsig-example?tab=readme-ov-file#getting-started) # Capturing Metrics Source: https://docs.statsig.com/integrations/azureai/capturing-metrics Capture AI metrics from Azure OpenAI calls in Statsig, including latency, cost, token usage, and per-prompt success signals for evaluation. Azure AI SDK automatically captures relevant invocation and usage metrics from each API call and logs them to Statsig. You can see these events streaming in real-time in the console at: [https://console.statsig.com/metrics/events](https://console.statsig.com/metrics/events). Statsig console events streaming interface ### Metrics captured Completion invocations capture the following metrics automatically: completion token length, prompt token length, latency, model name, total token length. For example, a completion API call results in a usage log that looks like this: Azure AI completion API usage log example These could be used to compare multiple deployments with each other, and also to run experiments against different sets of parameters, aiding in optimization of cost, responsiveness, user experience, etc. # Completions Source: https://docs.statsig.com/integrations/azureai/completions Instrument Azure OpenAI completions with Statsig to log prompts, completions, and metadata for AI evaluations and experiment analysis. Chat completions are AI-generated responses used to generate text. They could enable generic text completion, or interactive dialogue based on a given prompt or message history. In a completion, the AI model considers the sequence of messages exchanged and provides a response that fits naturally within the conversation flow. ## Simple completions ```js theme={null} const messages = [ { role: "system", content: "You are a helpful assistant. You will talk like a pirate." }, { role: "user", content: "What is the best way to train a parrot?" }, ]; const result = await modelClient.complete(messages); for (const choice of result.choices) { console.log(choice.message.content); } ``` ```python theme={null} response = modelClient.complete([ SystemMessage(content="You are a helpful assistant. You will talk like a pirate."), UserMessage(content="What is the best way to train a parrot?") ]) self.assertIsNotNone(response, "Expected response to not be None") for item in response.choices: content = item.message.content print(content) ``` ```csharp theme={null} var completion = await modelClient.Complete( "You are a helpful assistant that speaks like a pirate", "How do you train a parrot in 10 easy steps?" ); Console.WriteLine(completion); ``` #### Output ``` Arrr, trainin’ a parrot be quite the adventure, savvy? Here be some tips fer ye: 1. **Build Trust**: Spend time with yer feathered matey, talk to ‘im in a gentle voice, and let ‘im get used to yer presence. 2. **Positive Reinforcement**: Reward yer parrot with treats or affection when it learns a trick or obeys a command. Parrots, like all creatures, respond well to praise! 3. **Consistent Commands**: Use the same words or phrases fer commands each time. Repeatin’ yerself helps the parrot make connections. 4. **Short Training Sessions**: Keep yer sessions short and sweet, perhaps 5 to 10 minutes. Parrots have short attention spans, ye see! 5. **Be Patient**: Not every parrot learns at the same pace. Keep yer cool, and don't lose yer temper; patience be key! 6. **Fun and Play**: Incorporate games into yer training to keep it interestin’. A happy parrot be a learnin’ parrot! Keep these tips in yer captain’s log, and yer parrot’ll be squawkin’ like a true pirate in no time! Arrr! ``` ## Streaming Completions Sometimes you want to start streaming the AI responses back to your client, to reduce latency and increase responsiveness. You can accomplish that by using the Streaming API ```js theme={null} const messages = [ { role: "system", content: "You are a helpful assistant. You will talk like a pirate." }, { role: "user", content: "What is the best way to train a parrot?" }, ]; const stream = await modelClient.streamComplete(messages); for await (const event of stream) { if (event.data === "[DONE]") { return; } for (const choice of JSON.parse(event.data)?.choices) { process.stdout.write(choice.delta?.content ?? ""); } } ``` ```python theme={null} response = modelClient.stream_complete([ SystemMessage(content="You are a helpful assistant. You will talk like a pirate."), UserMessage(content="What is the best way to train a parrot?") ]) for update in response: print(update.choices[0].delta.content or "", end="", flush=True) ``` ```csharp theme={null} var completion = await modelClient.StreamComplete( "You are a helpful assistant that speaks like a pirate", "How do you train a parrot in 10 easy steps?" ); await foreach (var update in completion) { if (!string.IsNullOrEmpty(update.ContentUpdate)) { Console.Write(update.ContentUpdate); } } ``` # Text Embeddings Source: https://docs.statsig.com/integrations/azureai/embeddings Instrument Azure OpenAI embeddings calls with Statsig to log inputs, vectors, and metadata for AI evaluations and downstream experiments. Embeddings are numerical representations of data (like text, images, or audio) that capture their essential features in a compact form, typically as vectors. For text, embeddings map words or sentences to vector spaces, where similar items are closer together, enabling comparisons and efficient searches. Developers use embeddings in tasks like semantic search, recommendation engines, and clustering, as they allow for analyzing and processing unstructured data with machine learning models that recognize and work with these patterns. ## Generate text embeddings ```js theme={null} const result = await modelClient.getEmbeddings([ "Hello, world!", "Goodbye, world!", ]); for (const data of result.data) { console.log(`Embedding: ${data.embedding}`); } ``` ```python theme={null} response = modelClient.get_embeddings(["Hello, world!", "Goodbye, world!"]) for item in response.data: length = len(item.embedding) print( f"data[{item.index}]: length={length}, [{item.embedding[0]}, {item.embedding[1]}, " f"..., {item.embedding[length-2]}, {item.embedding[length-1]}]" ) ``` ```csharp theme={null} var embedding = await client.GetEmbeddings(["Hello, world!", "Goodbye, world!"]); Console.WriteLine(embedding.First().ToArray()); Console.WriteLine(embedding.Last().ToArray()); ``` #### Output ``` Embedding: -0.01918462,-0.025279032,-0.0017195191,0.018848283,-0.033795066,-0.019695852,-0.020947022,0.05158053,-0.03212684,-0.03037789,-0.0021458254,-0.028978731,-0.0024737532,-0.031481072,0.01033225,0.018606123,-0.046145335,0.041463535,0.00044186175,0.041221373,0.053679265,0.001873393,0.004567446,0.01002282,0.047867376,0.0022013208,-0.009834472,0.03847687,0.00089213194,-0.052118666,0.051150016,-0.03255735,-0.0140319485,-0.01263279, ..... ``` # Getting Started Source: https://docs.statsig.com/integrations/azureai/getting-started Get started with the Statsig Azure OpenAI integration to log AI requests, capture metrics, and run experiments on prompts, models, and parameters. ## Step 1: Install the SDK in your server app Start by installing the Statsig Azure AI SDK. Depending on your language/framework you would use the right package manager to install the SDK in your project ```shell theme={null} npm i @statsig/azure-ai ``` ```shell theme={null} pip install azureai-statsig ``` ```shell theme={null} dotnet add package StatsigAzureAI ``` ## Step 2: Create a model deployment in Azure AI Studio Log into your Azure AI Studio console and create a new deployment that you'd like to use. Azure AI Studio model deployment creation interface Once confirmed, you will be taken to the details page of that model. Azure AI model deployment details page Now, copy the **Target URI** (this will be your *endpoint* in code) and the **Key** - you'll need this in the SDK to call the APIs. ## Step 3: Get Statsig server SDK key This guide assumes you have an existing Statsig account. Please go here to create a new free account if you don't already have one: [https://statsig.com/signup](https://statsig.com/signup) Go to your **Project Settings** and choose the **Keys & Environments** tab on the left. Scroll down to **API Keys** section and copy the Server Secret Key. If one doesn't exist, you will have to create one, or ask your project admin to create one for you. Statsig project settings API keys section ## Step 4: Initialize Azure AI Server ```js theme={null} import { AzureAI } from "@statsig/azure-ai"; await AzureAI.initialize(""); ``` ```python theme={null} AzureAI.initialize("") ``` ```csharp theme={null} using Statsig; using Statsig.AzureAI; await Server.Initialize(""); ``` # Azure AI Source: https://docs.statsig.com/integrations/azureai/introduction Introduction to the Statsig Azure OpenAI integration for logging AI calls, capturing metrics, and running experiments on prompts and model parameters. Statsig offers SDKs for integrating Azure AI models into server applications. These SDKs simplify the implementation of features like completions and embeddings in your server application. They provide easy-to-use APIs and automatically track metrics such as latency, token length, and model details, which you can use for optimization and experimentation. Use cases include: * Implement Azure AI Models in your code with a [single lightweight framework](/azureai/model-client) * [Stream completions](/azureai/completions) and [generate embeddings](/azureai/embeddings) * [Capture invocation and usage metrics](/azureai/capturing-metrics/) with no extra work * [Run A/B tests on parameters](/azureai/running-experiments) like model, prompt, temperature and more Azure AI integration architecture diagram ## Currently supported SDKs * Node JS: [https://github.com/statsig-io/azureai-nodejs/](https://github.com/statsig-io/azureai-nodejs/) * Python: [https://github.com/statsig-io/azureai-python/](https://github.com/statsig-io/azureai-python/) * .Net: [https://github.com/statsig-io/azureai-dotnet/](https://github.com/statsig-io/azureai-dotnet/) # AI Model Client Source: https://docs.statsig.com/integrations/azureai/model-client Use the Statsig Azure OpenAI model client to wrap chat, completion, and embedding calls with automatic logging and experiment-aware routing. In order to invoke Azure AI methods, you'll need to instantiate a Model Client. You have two ways of instantiating a Model Client ## Option 1: (Recommended) Using Statsig Dynamic Config Using Statsig's Dynamic Config is a clean way to configure your AI Client, which provides maximum flexibility in being able to adjust the AI invocation parameters and even the endpoint without needing to modify code. In your Statsig console, create a new **Dynamic Config** by going to: [https://console.statsig.com/dynamic\_configs](https://console.statsig.com/dynamic_configs), and clicking on **Create** button. Dynamic Config creation interface Once created, you can fill in the properties of this deployment like this: Dynamic Config properties configuration screen The JSON of this looks like this: ``` { endpoint: "https://FILL_IN_YOUR_ENDPOINT", key: "FILL_IN_YOUR_KEY", completion_defaults: { frequency_penalty: 0, presence_penalty: 0, temperature: 1, top_p: 1, max_tokens: 0, stop: [], seed: 0, }, } ``` Once this is done, you can instantiate your Model Client directly by using the **id** of this Dynamic Config like this: ```js theme={null} const client = AzureAI.getModelClient(""); ``` ```python theme={null} client = AzureAI.get_model_client("") ``` ```csharp theme={null} var client = Server.GetModelClient(""); ``` ## Option 2: Using hard-coded endpoint and key This is the most direct way of instantiating an AI model client. However, this would mean you'll have to embed the endpoint and key in your code, or use another way (like an environment variable) to get the model endpoint and key into the process. ```js theme={null} const modelClient = AzureAI.getModelClientFromEndpoint( "", "" ); ``` ```python theme={null} modelClient = AzureAI.get_model_client_from_endpoint("", "") ``` ```csharp theme={null} var modelClient = Server.GetModelClientFromEndpoint( "", "" ); ``` # Running A/B Tests Source: https://docs.statsig.com/integrations/azureai/running-experiments Run experiments on Azure OpenAI prompts, models, and parameters with Statsig, including variant configuration, exposure logging, and result analysis. Azure AI SDK helps you easily and quickly run A/B tests to measure the effectiveness of different models and related parameters. By leveraging Statsig's powerful stats engine, you can gain real-time insights into model performance, optimizing for metrics like cost, accuracy, and latency. This integration enables you to experiment with various configurations, such as model type, prompt settings, or response parameters, and make data-driven decisions to enhance your application's efficiency and user experience. ## Example: Test GPT4o vs. GPT4o-mini ### Step 1: Create configs Create two dynamic configs, one named `gpt-4o` and another named `gpt-4o-mini`. In the **Value** section add the endpoint, key and other default parameters like this: Dynamic config setup interface These will serve as the base deployment configs for our tests, and also allow you to modify it on the fly as you launch ### Step 2: Create some metrics to track Let's take the example of a metric like **latency** and see how to create it in Statsig. Navigate to the **Metrics Catalog** page ([https://console.statsig.com/metrics/metrics\_catalog](https://console.statsig.com/metrics/metrics_catalog)) and click on **Create** button. Metrics catalog creation interface Now, in the **Metric Definition** section, choose: | Property | Value | | ------------------ | ------------------------------- | | Metric Type: | **Aggregation** | | ID Type: | **User ID** | | Aggregation Using: | **Events** | | Aggregation Type: | **Average** | | Rollup Mode: | **Total Experiment** | | Event: | **usage** | | Average Using: | **Metadata** => **latency\_ms** | This will create a metric that averages **latency** across all **usage** events coming from chat completions. Latency metric configuration screen ### Step 3: Create an experiment Create a new experiment in the Statsig console from [https://console.statsig.com/experiments](https://console.statsig.com/experiments) Experiment creation interface In the **Setup** page, add the metrics you created in Step #2 in the **Primary Metrics** field. Primary metrics configuration screen ### Step 4: Set up the variations You can now create the control and test variants for the experiment you want to run. In our case, let's split them evenly 50/50. In the **Groups and Parameters** section, click on **Add Parameter** button and name the parameter *model\_name*, with *String* type Parameter setup interface Now add the two configs we created in Step #1, one each to Control and Test parameters like this: Experiment variant configuration screen ### Step 5: Save and start the experiment Now, hit the **Save** button at the bottom of the page. You will now see a **Start** button appear at the top of the experiment page. Go ahead and click it - this will start the allocation process for the experiment. ### Step 6: Let's write some code The code below: 1. Fetches the experiment configuration from server for a given user. You can pass down the **userID** from your client application or use one from your database. The code below generates a random one for testing purposes. 2. Gets the **config name** from the experiment variant - either from control or test 3. Create a model client using the config that we just fetched 4. Uses that model client to complete text. ```js theme={null} async function testExperiments() { await AzureAI.initialize(statsigServerKey); const experiment = Statsig.getExperimentSync( { userID: Math.random().toString() }, // use a valid userID here "model_experiment_gpt4o_vs_gpt4o-mini", ); const configName = experiment.get("model_name", "gpt-4o"); console.log(`Using model: ${configName}`); const modelClient = AzureAI.getModelClient(configName); const result = await modelClient.complete([{ role: "user", content: "Recite the first 10 digits of pi." }]); result.choices.forEach((choice, i) => { console.log(choice.message.content); }); await AzureAI.shutdown(); } ``` ### Step 7: Run the experiment and verify results Run this experiment for several days, and you will now be able to measure latency profiles of **gpt-4o** compared with **gpt-4o-mini** in Statsig console. You can choose whichever one suits your needs. The above is just a simple experiment to test models against each other. You could also tweak other parameters like *temperature*, *frequency\_penalty*, *max\_tokens*, etc. by modifying the config. This could all be done without needing to update code. # Cloudflare KV Source: https://docs.statsig.com/integrations/cloudflare Run Statsig feature gate and experiment evaluations at the edge with Cloudflare Workers for low-latency rules evaluation in your CDN layer. Statsig offers a set of integrations that make usage with Cloudflare easy: * Automatically pushing changes to Cloudflare's KV store, for low-latency SDK startup * A helper pattern that handles Statsig SDK overhead, so you can focus on worker logic Navigate to [Project Settings -> Integrations](https://console.statsig.com/integrations), in the Statsig Console, then select Cloudflare, and input: * **Cloudflare Account ID**: Can be found in Cloudflare portal on the Compute (Workers) page, under Account Details * **KV Namespace ID**: Create a new namespace, then go to Account Home -> Storage and Databases -> Workers KV, and copy the ID from the table view. * **Cloudflare API Key**: In Cloudflare portal under Account Home -> Profile -> API Tokens. Use a token with Account.Workers KV Storage Edit Permissions. You can also filter the configs that are synced to your KV namespace by Target App. This becomes important as you add more configs to your project, but for now, you can leave this unchecked. Click **Enable**, then the Statsig backend will push a config to your KV namespace (\<60 seconds). In your KV namespace, navigate to **KV Pairs** - you'll see entry starting with `statsig-`. This is the `key` associated with your KV storage. Note this key down for later. Now, we'll set up a worker to read those experiments and gates from your KV namespace. If you've never created a worker before, you can follow the instructions [here](https://developers.cloudflare.com/workers/). After creating your worker, you will need to connect your KV store to your worker through a binding. Navigate to **Compute (Workers)** -> **Select Your Worker** -> **Bindings** -> **Add binding** -> **KV namespace**. Name your binding under **Variable name**. Under **KV namespace**, select your KV store name. For more information on connecting your worker to your KV store, you can follow the instructions [here](https://developers.cloudflare.com/pages/functions/bindings/). Install the Statsig serverless SDK: ```bash theme={null} npm install @statsig/serverless-client ``` The helper method takes two arguments, a handler function, and a ParamsObject. **Put *all* of your worker logic in the handler function**, along with your Statsig usage. ```javascript highlight={5} theme={null} import { handleWithStatsig } from '@statsig/serverless-client/cloudflare'; export default handleWithStatsig( async (request, env, ctx, client) => { // Your business, and Statsig logic here }, { kvKey: 'kv_key', envStatsigKey: 'statsig_key', envKvBindingName: 'STATSIG_KV' } ); ``` The required ParamsObject params (kvKey, envStatsigKey, envKvBindingName) must be stored as env variables, either in your wrangler.toml or as Cloudflare secrets. Environment variable name containing your KV pair key Environment variable name containing your Statsig client key Your KV binding name See StatsigOptions [here](/client/javascript-sdk#statsig-options) **Best practice:** * store `envStatsigKey` as a Cloudflare secret. You can set this in the Cloudflare dashboard under, **Worker → Settings → Variables and Secrets** * store `kvKey` and `envKvBindingName` in your wrangler.toml ### Example Usage This is an example of an end-to-end worker function that uses the Statsig SDK and returns a flag value. This is all you need - this will compile as the index.js file in your worker. ```javascript index.js expandable theme={null} import { handleWithStatsig } from '@statsig/serverless-client/cloudflare'; export default handleWithStatsig( async (request, env, ctx, client) => { const randomUserId = Math.floor(Math.random() * 100).toString(); const gate = client.getFeatureGate("test_cloudflare_sync", { userID: randomUserId }); const value = gate.value; client.logEvent('new_event', { userID: randomUserId }); return new Response(`Gate check result: ${value}`); }, { kvKey: 'kv_key', envStatsigKey: 'statsig_key', envKvBindingName: 'STATSIG_KV' } ); ``` ```wrangler wrangler.toml theme={null} name = "test" main = "src/index.js" compatibility_date = "2025-09-10" [vars] kv_key = "statsig-1gh32fg61hds9876" [[kv_namespaces]] binding = "STATSIG_KV" id = "b76664aa8259481e834e7c549443c6541" [observability] enabled = true ``` **That's it!** The helper automatically: * Initializes the Statsig Client with config specs from your KV store * Executes your handler code (Your business logic + Statsig usage) * Flushes all events after your handler completes execution * Cleans up resources **Use the advanced/manual setup if:** * You need fine-grained control over initialization timing * You need fine-grained control over event flushing timing * You need to customize error handling behavior ## Prerequisites 1. Completed the [Statsig Cloudflare KV integration setup](#configure-integration) 2. [Created and bound a KV namespace to your worker](#add-the-statsig-sdk-to-your-worker) ## Installation First, you'll need to install the Statsig serverless sdk. ```bash theme={null} npm install @statsig/serverless-client ``` ## Import Next, import the Cloudflare client. ```bash theme={null} import { StatsigCloudflareClient } from '@statsig/serverless-client/cloudflare'; ``` Then, you need to hook it all up. This involves: 1. Creating a `StatsigCloudflareClient` instance. 2. Initializing the Statsig client 3. Checking a Gate 4. Logging an event 5. Flushing events to Statsig If you've used a Statsig sdk in the past, these steps should be familiar. The usage will be the same, the only difference is the sdk will initialize from the KV store instead of the statsig backend. In our example, we are checking a gate called "test\_cloudflare\_sync" that is set to a 50% pass rate. We create a random userID on every request, and we should see it evaluate to true 50% of the time. ### 1. Creating a `StatsigCloudflareClient` instance ```bash theme={null} const client = new StatsigCloudflareClient(""); ``` The client instantiation takes two arguments: * `sdkKey : string` This is your Statsig client API key. It is available from the [Project Settings](https://console.statsig.com/api_keys) page in the Statsig Console. This is used to authenticate your requests. * `options : StatsigOptions` See here, for more [options](/client/javascript-sdk#statsig-options). For best practice: * store `sdkKey` as a Cloudflare secret. You can set this in the Cloudflare dashboard under, **Worker → Settings → Variables and Secrets** ### 2. Client initialization The following line initializes the client by loading feature gate and experiment configurations directly from your Cloudflare KV store. ```bash theme={null} const initResult = await client.initializeFromKV(env., ); ``` The client initialization takes two arguments: * `KvBinding` This is the binding you named earlier. Remember to provide this argument as `env.YOUR_KV_NAMESPACE_BINDING` * `KvKey : string` This is the KV pair key that was generated through the Statsig integration. It can be found under **Workers KV** -> **Your KV namespace** -> **KV Pairs** For best practice: * store `kvBinding` and `kvKey` in your wrangler.toml ### 3. Checking a Gate ```bash theme={null} const value = client.checkGate("test_cloudflare_sync", { userID: randomUserId }); ``` This is a gate check in code. The `checkGate` method takes two arguments: * `name : string` The name of the Statsig gate that you are checking. * `user : StatsigUser` The Statsig user object for whom the gate is being checked. For more information on the user object, see [here](/sdks/user#introduction-to-the-statsiguser-object). Refer to the [Javascript on device evaluation sdk documentation](/client/jsOnDeviceEvaluationSDK) for how to check other entities like experiments and dynamic configs. ### 4. Logging an event ```bash theme={null} client.logEvent('gate_check', { userID: randomUserId }); ``` This is an event log in code. The `logEvent` method takes two parameters: * `eventOrName : string | StatsigEvent` This is the name and details of the event you are logging. * `user : StatsigUser` The Statsig user object for whom the event is being logged. For more information on event logging, see [here](/client/jsOnDeviceEvaluationSDK#logging-an-event). ### 5. Flushing Events ```bash theme={null} ctx.waitUntil(statsig.flush()); ``` This flushes all events from the sdk to Statsig. **Without this, you won't be able to get diagnostic information in the Statsig Console, nor any event data you logged**. ### Putting it all together ```Javascript theme={null} import { StatsigCloudflareClient } from '@statsig/serverless-client/cloudflare'; export default { async fetch(request, env, ctx) { try { const client = new StatsigCloudflareClient(env.statsig_key); const initResult = await client.initializeFromKV(env.STATSIG_KV, env.kv_key); const randomUserId = Math.floor(Math.random() * 100).toString(); //generates a random user id const value = client.checkGate("test_cloudflare_sync", { userID: randomUserId }); client.logEvent('gate_check', { userID: randomUserId }); ctx.waitUntil(client.flush()); return new Response(`Value: ${value}, userID: ${randomUserId}); } catch (error) { return new Response(`Error: ${error.message}`, { status: 500 }); } } }; ``` ```wrangler theme={null} name = "test" main = "src/index.js" compatibility_date = "2025-09-10" [vars] kv_key = "statsig-1gh32fg61hds9876" [[kv_namespaces]] binding = "STATSIG_KV" id = "b76664aa8259481e834e7c549443c6541" [observability] enabled = true ``` If you want to check on the evaluations you are getting, you can go to the gate you created for this example and look at the evaluations in the Diagnostics tab. Diagnostics Stream If you want to check the events you logged, in the **Statsig Console**, go to **Data** -> **Events** Statsig Events log stream showing gate_check events with timestamps and user IDs And there you have it - a working Cloudflare KV integration for Statsig. ## Other Considerations ### Polling for updates The SDK cannot poll for updates across requests since [**Cloudflare does not allow for timers**](https://developers.cloudflare.com/workers/reference/security-model/#step-1-disallow-timers-and-multi-threading). To optimize for edge use cases, we do not provide an api to recognize updates to your config specs. However, when a change is made to your project definition on the Statsig console, the changes will be propagates to the KV store and will be reflected the next time you initialize the Cloudflare client. ### Flushing events The SDK enqueues logged events and flushes them in batches. In order to ensure events are properly flushed, we recommend calling flush using context.waitUntil. This will keep the request handler alive until events are flushed without blocking the response. ```bash theme={null} context.waitUntil(client.flush()); ``` ### Size Limits Cloudflare KV has maximum size limits that may prevent Statsig from pushing configs into your KV. See [here](https://developers.cloudflare.com/workers/platform/limits/#kv-limits) for the latest Cloudflare KV limits. If your payload continues to grow, you will need to set the option to filter the payload by a Target App in the integration settings. ### Unsupported Features Statsig ID Lists are not currently synced into Cloudflare KVs. If you rely on large (>1000) ID lists, you will not be able to check them in your Cloudflare Worker. # Amplitude Source: https://docs.statsig.com/integrations/data-connectors/amplitude Connect Amplitude with Statsig to import events and metrics for experiment analysis or export Statsig data into Amplitude dashboards and notebooks. ## Overview Statsig supports both incoming and outgoing events for Amplitude. As well as adding Amplitude Cohorts to Statsig ID Lists. ## Incoming - Receiving Events From Amplitude Ingestion with this integration is available only for Statsig Cloud. For Warehouse Native, create a metrics source that references this data in your warehouse. The following steps outline how to forward events from Amplitude into Statsig. 1. Get a Statsig "Server Secret Key" from the API keys page in [Project Settings](https://console.statsig.com/api_keys). 2. Go to Amplitude and navigate to the Data Destinations page. Click the "Add Destination" button in the top right. Amplitude Data Destinations page showing Add Destination button 3. From the Destinations Catalog, search for and select the Statsig Event Streaming destination. Destinations catalog highlighting Statsig event streaming option 4. Give this destination a name and click "Create Sync". Create Sync form for Statsig destination 5. Enter the "Server Secret Key" you copied in Step 1 into the provided field. Select the events you wish to send to Statsig. Ensure that the Status is set to "Enabled" and then click "Save". Statsig destination settings entering server secret key and event selection 6. *Enable the integration* - On the Integrations page for your Statsig project, enable the Amplitude Incoming integration. Statsig integration panel confirming Amplitude connection ## Outgoing - Sending Statsig Events to Amplitude 1. Navigate to Amplitude and click on the Settings button in the bottom-left corner. Amplitude settings menu accessed from bottom-left 2. Click on the Projects tab and choose the Project you wish to send data to. Amplitude Projects tab listing available workspaces 3. Copy the API Key and paste it in the Statsig integration panel. Amplitude project API key display Statsig integration panel fields for Amplitude API key 4. Hit Enable on the integration panel and any data logged to Statsig will show up in your Amplitude Project account. Amplitude event stream showing Statsig exposure events ## First Exposures [First exposures](/pulse/export#first-exposures-file-description) are an enterprise-tier feature that simplifies your project insights. This is available for Enterprise contracts. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. ### What is it? Our Amplitude Integration offers the flexibility to forward first exposures instead of every exposure, reducing the overall number of events being forwarded. First exposures are calculated daily and forwarded to integrations at around 7pm UTC. ### How to enable First ensure that the "first exposure" feature has been enabled for your company by reaching out to support team, your sales contact, or via our [Slack community](https://statsig.com/slack). Once this is done you will be able to go into the event filtering tab of the integration and enabled "First Exposure" setting. ### Example Events In Amplitude Example of a get\_experiment First Exposure in Amplitude. Example first exposure event in Amplitude log ### Accessing Raw Data For a comprehensive view, you can obtain the raw first exposure data in CSV format. Simply make a request to the [console/v1/reports](/console-api/daily-reports#get-/reports) endpoint to receive a download link. ## Cohort Sync - Syncing Amplitude Cohorts to Statsig Segments For up to date configuration information on syncing Cohorts (aka Segments) from Amplitude to Statsig, please take a look at Amplitude's documentation [here](https://www.docs.developers.amplitude.com/data/destinations/statsig-cohort/). Ensure you create and use a console API key from your [Statsig project settings](https://console.statsig.com/api_keys) ## Filtering Events You can customize which events should be sent and received via Amplitude using [Event Filtering](/integrations/event_filtering) # Braze Source: https://docs.statsig.com/integrations/data-connectors/braze Connect Braze with Statsig to send messaging events to Statsig for experiment analysis and to use Statsig audiences in Braze campaigns. ## Overview Enabling the Braze integration allows you to export Statsig exposure events to your configured Braze app with information on the status of each user's feature gate and experimentation groups. These exposures will be forwarded to Braze as a [Custom Attribute](https://www.braze.com/docs/user_guide/data/custom_data/custom_attributes) object on the user. There will be one Custom Attribute per gate/experiment the user has been exposed to. The Custom Attribute in Braze will be named `statsig_exposure::{gate/experiment name}` and be of the form: ``` { group_name: String, timestamp: Time } ``` You can then filter exposed users into a Segment in Braze. Custom Attributes will be forwarded to Braze users by having the unit ID from the gate/experiment as the `external_id` in Braze by default. You can choose to provide a custom Unit ID Type from your Statsig project to be forwarded as the `external_id` for all gate/experiment exposures. This can be provided in the ID Type Mapping section of the Setup dialog for this integration. The integration will attempt to use this custom ID Type if it is provided in the SDK call at the time of exposure, and will fall back to the experiment's Unit ID Type if not. ## Setup in Statsig This is available for Enterprise contracts. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. After it has been enabled, you will be able to find 'Braze' as an option in your Statsig project's [list of integrations](https://console.statsig.com/integrations) from within Statsig console. 1. Open your [Braze dashboard](https://dashboard.braze.com/). Navigate to Settings > APIs and Identifiers, then open the API Keys tab. 2. Create or select an existing API key that has the 'users.track' permission. Enter the API Key Identifier in the Braze Integration Setup dialog in your Statsig project. 3. Find your Instance's REST Endpoint from the [Braze API docs](https://www.braze.com/docs/api/basics/#api-definitions/). Enter it in the Integration Setup dialog. ## Segment Filtering in Braze Once your integration is set up in Statsig, exposures can start firing into your Braze app. When exposures arrive in Braze from a new gate/experiment, you can create a filter on these users. 1. Open your [Braze dashboard](https://dashboard.braze.com/). Navigate to Data Settings > Custom Attributes. You should see your new Custom Attribute from Statsig like below: image.png 2. Click 'Generate Schema'. It will automatically detect the schema like below: image.png 3. Now you can create a Segment from these users. Navigate to Audience > Segments, and click 'Create Segment'. 4. Under the 'Segment Builder' section, add a new filter. Click 'Custom Attributes', then 'Nested Custom Attributes'. image.png 5. Now you can filter to a specific group\_name (true/false for gates, group name for experiments), or timestamp for your set of users. An example Segment filter for all users that have passed a specific gate is like below: image.png ## First Exposures [First exposures](/pulse/export#first-exposures-file-description) are an enterprise-tier feature that simplifies your project insights. This is available for Enterprise contracts. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. ### What is it? Our Braze Integration offers the flexibility to forward first exposures instead of every exposure, reducing the overall number of events being forwarded. First exposures are calculated daily and forwarded to integrations at around 7pm UTC. ### How to enable First ensure that the "first exposure" feature has been enabled for your company by reaching out to support team, your sales contact, or via our [Slack community](https://statsig.com/slack). Once this is done you will be able to go into the event filtering tab of the integration and enabled "First Exposure" setting. # Census Source: https://docs.statsig.com/integrations/data-connectors/census Connect Census reverse ETL with Statsig to sync audience segments and metric data between your data warehouse and Statsig for activation. ## Overview Enabling the [Census](https://getcensus.com/) integration for Statsig allows Statsig to receive events from Census. This enables you to ingest data into Statsig from any sources that Census supports. You can find all events that Statsig receives from Census in the [Metrics](/metrics) tab in the Statsig console. Statsig will automatically include these events in [Pulse](/pulse/read-pulse) and [Experiment](/experiments-plus/monitor) results for your feature gates and experiments respectively. ## Configuring Incoming Events 1. From the [API Keys](https://console.statsig.com/api_keys) tab in the Statsig console, copy the Statsig "Server Secret Key”. 2. From census, create a new [destination](https://docs.getcensus.com/destinations/overview) and select Statsig from the list of options. 3. Paste the Statsig secret into the field and click save. Census destination setup form with Statsig secret key input 4. Create a Sync to the new Statsig destination (see [Sync Configuration](#sync-configuration) section below) 5. On the Statsig [Integrations](https://console.statsig.com/integrations) page, enable the Census integration. ### Sync Configuration A sync key is required to uniquely identify each event. Census sync configuration showing sync key selection The following fields are required when mapping to Statsig events. * `User ID` -> `userID` * `Event Name` -> `eventName` * `Timestamp` -> `timestamp` * `Value` -> `value` Field mapping table aligning Census columns to Statsig event fields All other fields will be included in the `metadata` section of the mapped Statsig event. ### Custom ID Mapping The Census integration allows the mapping of arbitrary fields to Statsig Custom IDs. To do this, visit the Census panel on the Statsig [Integrations](https://console.statsig.com/integrations) page and look for the "Map Identifier" section. Here you can choose fields you would like mapped to a Custom ID. The input Event Field must match the exact spelling as in the original Census event. Statsig integration panel for Census custom ID mapping # Fivetran Source: https://docs.statsig.com/integrations/data-connectors/fivetran Connect Fivetran with Statsig to ingest events from third-party sources via your warehouse for use in Statsig metrics and experiment analysis. ## Overview Enabling the Fivetran integration for Statsig will allow Statsig to push events to your Fivetran account through a webhook. This allows you to forward Statsig data to any connectors available from Fivetran. ## Configuring Outbound Events 1. Follow the steps in the [Fivetran Webhook Setup Guide](https://fivetran.com/docs/events/webhooks/setup-guide) to create a new Webhook URL. 2. On the Statsig [Integrations](https://console.statsig.com/integrations) page, enable the Fivetran integration by pasting in the Fivetran Webhook URL and click **Confirm**. ### Event Format Events will be sent in batches in a JSON format. The structure of a Statsig Event sent will look like the following: | Field | Type | Description | | --------------- | ------ | ---------------------------------------------------------------- | | eventName | String | Name of the event provided | | user | JSON | [Statsig User Object](/concepts/user) | | userID | String | User ID provided | | timestamp | Number | Timestamp in MS of the event | | value | String | Value of the event provided | | metadata | JSON | Custom Metadata provided | | statsigMetadata | JSON | Metadata related to the logging of this event added by Statsig | | timeUUID | String | UUID for the event | | unitID | String | Unit ID of the exposure (e.g. userID, stableID, or the customID) | #### Custom Event Formatting - logEvent > ```json theme={null} { "eventName": "my_custom_event", "user": { "userID": "a_user", "email": "a.user@email.com" }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "a_custom_value", "metadata": { "key_a": "value_a", "key_b": "123" }, "timeUUID": "abd2a983-ec0f-11ec-917a-fb8cdaeda578" } ``` #### Feature Gate Exposure Formatting - checkGate > ```json theme={null} { "eventName": "statsig::gate_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "", "metadata": { "gate": "a_gate", "gateValue": "false", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "timeUUID": "8d7c1040-ec11-11ec-g123-abe2c32fcf46", "unitID": "userID" } ``` #### Dynamic Config Exposure Formatting - getConfig > ```json theme={null} { "eventName": "statsig::config_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "", "metadata": { "config": "a_config", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "timeUUID": "af379f60-ec11-22ad-8e0a-05c3ee70bd0c", "unitID": "userID" } ``` #### Experiment Exposure Formatting - getExperiment > ```json theme={null} { "eventName": "statsig::experiment_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655232119734", "statsigMetadata": { ... }, "value": "", "metadata": { "config": "an_experiment", "ruleID": "4SauZJcM1T7zNvh1igBjwE", "reason": "Network", "time": "1655231249644", "experimentGroupName": "Control" }, "timeUUID": "af379f61-ab22-11ec-8e0a-05c3ee70bd0c", "unitID": "userID" } ``` #### Example Batch > ```json theme={null} [ { "eventName": "page_view", "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566967, "value": "example_value", "metadata": {"page": "home_page"}, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120002" }, { "eventName": "statsig::gate_exposure", "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566968, "value": "", "metadata": {"gate": "test_gate", "gateValue": "true", "ruleID": "default"}, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120003", "unitID": "userID" }, { "eventName": "statsig::experiment_exposure" "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566969, "value": "", "metadata": { "config": "an_experiment", "ruleID": "4SauZJcM1T7zNvh1igBjwE", "reason": "Network", "time": "1655231249644", "experimentGroupName": "Control" }, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120004", "unitID": "userID" } ] ``` ## Filtering Events Once you've enabled outbound events to Fivetran, you can select which categories of Statsig events you want to export by click on the **Event Filtering** button and checking the appropriate boxes as shown below. Event filtering configuration interface Event category selection checkboxes # Google Analytics Source: https://docs.statsig.com/integrations/data-connectors/google-analytics Connect Google Analytics with Statsig to send GA events to Statsig for experiment analysis and metric tracking alongside your other product data. Enabling the Google Analytics 4 integration allows Statsig to send logged events and exposures to GA4. This enhances your existing Google Analytics tracking with additional data collected by Statsig's logging SDKs. Once enabled, Statsig will forward exposures and logged events to a configured Data Stream. These events can be filtered with [event filtering.](/integrations/data-connectors/google-analytics#filtering-events) ### Benefits of using the Google Analytics 4 integration Using the GA4 Integration allows you to log additional events without having the orchestrate two libraries, thus simplifying your code. Furthermore by implementing this integration, you'll be able to join data about experiments you create in Statsig to existing analytics events you care about in Google Analytics, giving you greater insights into different experiments' impact on your user interactions. ## Configuring outbound events to Google Analytics 4 To send events collected by Statsig's SDKs to GA4, you must configure a Data Stream and provide a few pieces of information. 1. Navigate to the GA4 admin settings. Under your app's property click Data Streams and select the stream you'd like to use. If you don't have a stream you'll need to create one. 2. Statsig requires an API secret to send the data to your stream, navigate to your stream and create a new secret: GA4 Data Stream details with Measurement Protocol API secrets section Create API secret dialog in GA4 List of API secrets showing newly generated key Measurement ID highlighted on GA4 stream setup 3. Once created, copy the API secret and the measurement ID (optional). Navigate to your Statsig Project -> Project Settings -> Integrations -> Google Analytics (click enable) -> Google Analytics 4. Provide your API Secret and measurement ID from the previous step and click *confirm*: Statsig integration configuration form for GA4 API secret and measurement ID 4. Verify that you are receiving events now by checking the Realtime overview report for the event with name `statsig`. Account for a couple days of delay for events to be available in other reports. 5. You can also add the following custom event dimensions. Other custom IDs and custom user attributes are available as user dimensions
    • `config` - Name of the experiment/gate/dynamic config
    • `group` - Name of the exposed group (e.g. Control)
    • `value` - Value for custom events
    • `statsig_session_id` - Session ID
    • `category` - Type of exposure or name of the custom event (e.g. `statsig_gate_exposure`)
    • `unit_id` - Value of the unit ID (e.g. '123')
    • `unit_id_type` - Type of the unit ID (e.g. 'stableID')
    ## Filtering Events Once the outgoing integration has been enabled, you can optionally configure event filtering to control whch events are populating the GA4 Data Stream: Event filtering UI specifying which Statsig events flow to GA4 # Heap Source: https://docs.statsig.com/integrations/data-connectors/heap Connect Heap with Statsig to send autocaptured events to Statsig for experiment analysis and metric tracking across your product surfaces. ## Overview Enabling the Heap integration allows you to export Statsig events to your configured Heap app with information on the status of each user's feature gate and experimentation groups. Statsig will send events to Heap when a client SDK is initialized and will also forward events as they are received. ## Client SDK Initialize Events Statsig sends the following events to your Heap app every time you call the `initialize` API from a Statsig client SDK. | Event Name | Properties | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Statsig Feature Gates | For each [Statsig Feature Gate](/feature-flags/overview), this field contains a property that maps the name of the feature gate to `true` or `false`, stating whether the user passes or does not pass the feature gate. | | Statsig Experiments | For each [Statsig Experiment](/experiments-plus), this field contains a property that maps the name of the experiment to the variant that the user is assigned to. | ## Exposures and Custom Event Forwarding Statsig can forward events as they are received from SDKs, Integrations or the HTTP API. Events include exposures (Gate, Experiment, Config) and custom events. ### Event Format Events will be sent in batches in a JSON format. The structure of a Statsig Event sent will look like the following: | Field | Type | Description | | --------------- | ------ | ---------------------------------------------------------------- | | eventName | String | Name of the event provided | | user | JSON | [Statsig User Object](/concepts/user) | | userID | String | User ID provided | | timestamp | Number | Timestamp in MS of the event | | value | String | Value of the event provided | | metadata | JSON | Custom Metadata provided | | statsigMetadata | JSON | Metadata related to the logging of this event added by Statsig | | timeUUID | String | UUID for the event | | unitID | String | Unit ID of the exposure (e.g. userID, stableID, or the customID) | #### Custom Event Formatting - logEvent > ```json theme={null} { "eventName": "my_custom_event", "user": { "userID": "a_user", "email": "a.user@email.com" }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "a_custom_value", "metadata": { "key_a": "value_a", "key_b": "123" }, "timeUUID": "abd2a983-ec0f-11ec-917a-fb8cdaeda578" } ``` #### Feature Gate Exposure Formatting - checkGate > ```json theme={null} { "eventName": "statsig::gate_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "", "metadata": { "gate": "a_gate", "gateValue": "false", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "timeUUID": "8d7c1040-ec11-11ec-g123-abe2c32fcf46", "unitID": "userID" } ``` #### Dynamic Config Exposure Formatting - getConfig > ```json theme={null} { "eventName": "statsig::config_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "", "metadata": { "config": "a_config", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "timeUUID": "af379f60-ec11-22ad-8e0a-05c3ee70bd0c", "unitID": "userID" } ``` #### Experiment Exposure Formatting - getExperiment > ```json theme={null} { "eventName": "statsig::experiment_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655232119734", "statsigMetadata": { ... }, "value": "", "metadata": { "config": "an_experiment", "ruleID": "4SauZJcM1T7zNvh1igBjwE", "reason": "Network", "time": "1655231249644", "experimentGroupName": "Control" }, "timeUUID": "af379f61-ab22-11ec-8e0a-05c3ee70bd0c", "unitID": "userID" } ``` #### Example Batch > ```json theme={null} [ { "eventName": "page_view", "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566967, "value": "example_value", "metadata": {"page": "home_page"}, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120002" }, { "eventName": "statsig::gate_exposure", "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566968, "value": "", "metadata": {"gate": "test_gate", "gateValue": "true", "ruleID": "default"}, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120003", "unitID": "userID" }, { "eventName": "statsig::experiment_exposure" "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566969, "value": "", "metadata": { "config": "an_experiment", "ruleID": "4SauZJcM1T7zNvh1igBjwE", "reason": "Network", "time": "1655231249644", "experimentGroupName": "Control" }, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120004", "unitID": "userID" } ] ``` ## Configuring Outbound Events 1. Navigate to your [Heap Projects](https://heapanalytics.com/app/manage/projects) page to find and copy the App ID for your project. Heap projects page showing App ID value to copy 2. Paste the App ID into the App ID input field for the Heap configuration in the Statsig [Integrations](https://console.statsig.com/integrations) page and save your changes. ## First Exposures [First exposures](/pulse/export#first-exposures-file-description) are an enterprise-tier feature that simplifies your project insights. This is available for Enterprise contracts. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. ### What is it? Our Heap Integration offers the flexibility to forward first exposures instead of every exposure, reducing the overall number of events being forwarded. First exposures are calculated daily and forwarded to integrations at around 7pm UTC. ### How to enable First ensure that the "first exposure" feature has been enabled for your company by reaching out to support team, your sales contact, or via our [Slack community](https://statsig.com/slack). Once this is done you will be able to go into the event filtering tab of the integration and enabled "First Exposure" setting. # Hightouch Source: https://docs.statsig.com/integrations/data-connectors/hightouch Connect Hightouch reverse ETL with Statsig to sync warehouse-defined audiences and metrics into Statsig for targeting and experiment analysis. ## **Overview** Enabling the **[Hightouch](https://hightouch.com/)** integration allows you to send events and keep segments up-to-date in Statsig via Hightouch. You can ingest data into Statsig from [any source](https://hightouch.com/integrations) that Hightouch supports. You can find all events that Statsig receives from Hightouch in the [Metrics](/metrics) tab in the Statsig console, if you’re on a [Pro plan](https://www.statsig.com/pricing). Statsig automatically includes these events in [Pulse](/pulse/read-pulse) and [Experiment](/experiments-plus/monitor) results for your feature gates and experiments respectively. You can view your updated segment lists on the **[Segments](https://console.statsig.com/segments)** page. ## **Event configuration** 1. From the **[API Keys](https://console.statsig.com/api_keys)** tab in the Statsig console, copy the Statsig **Client-SDK API key**. 2. Go to the Hightouch [**Destinations** overview page](https://app.hightouch.com/destinations) and click the **Add destination** button. Select **Statsig** and click **Continue**. Enter the **Client-SDK API key** and click **Continue**. 3. Give your destination a descriptive name, for example, "Statsig prod." 4. Once you've set up your Statsig destination and have a [model](https://hightouch.com/docs/getting-started/concepts#models) to pull data from, you can set up your sync configuration to begin syncing data. Go to the [**Syncs** overview page](https://app.hightouch.com/syncs) and click the **Add sync** button to begin. Then, select the relevant model and the Statsig destination you previously set-up. 5. Select **Events** as the sync type. 6. Enter either a static value or select a column that contains **Event names**. 7. Optionally, select a column that contains the event timestamp. If empty, Statsig uses the time the event arrives at the server. 8. Choose source columns to sync as event metadata and user attributes, such as the **User ID**. 9. Finally, select whether you want the first sync to backfill event data or not. For more information about configuration options see [Hightouch’s Statsig docs](https://hightouch.com/docs/destinations/statsig#events). 10. Run the sync and check that events appear in your Statsig **[Metrics](/metrics)** tab. ## Segment configuration The Hightouch integration lets you keep your segments up-to-date by adding or removing members based on changes in your source dataset. 1. From the **[API Keys](https://console.statsig.com/api_keys)** tab in the Statsig console, copy the Statsig **Console API key.** 2. Go to the Hightouch [**Destinations** overview page](https://app.hightouch.com/destinations) and click the **Add destination** button. Select **Statsig** and click **Continue**. Enter the **Client-SDK API key** and click **Continue**. 3. Give your destination a descriptive name, for example, "Statsig prod." 4. Once you've set up your Statsig destination and have a [model](https://hightouch.com/docs/getting-started/concepts#models) to pull data from, you can set up your sync configuration to begin syncing data. Go to the [**Syncs** overview page](https://app.hightouch.com/syncs) and click the **Add sync** button to begin. Then, select the relevant model and the Statsig destination you previously set-up. 5. Select **Segment** as the sync type. 6. Select whether you would like to create a new audience or use an existing audience to sync data to. If creating a new audience, you can give it a name. If you leave this input blank, Hightouch uses the name of your model for the audience. 7. Select the audience's ID type: either **userID** or **stableID**. 8. Select the source column and Statsig field to match records on. For more information see Hightouch's docs on [record matching](https://hightouch.com/docs/syncs/record-matching). 9. [Schedule your sync](https://hightouch.com/docs/syncs/schedule-sync-ui) to run as frequently as you need. You can view your updated segment lists on the **[Segments](https://console.statsig.com/segments)** page. ## Troubleshooting If you have any questions, don’t hesitate to contact [Hightouch support](https://www.notion.so/Hightouch-page-in-Statsig-docs-42b88b32b82b491d9baf1694049955ab) for assistance. # Mixpanel Source: https://docs.statsig.com/integrations/data-connectors/mixpanel Connect Mixpanel with Statsig to send Mixpanel events to Statsig for experiment analysis and to keep product analytics consistent across both tools. ## Overview The [Mixpanel](https://mixpanel.com/) integration has two functions. * Incoming: Statsig can sync your Mixpanel user cohorts with a Statsig ID list segment. * Outgoing: Statsig can forward Statsig events to Mixpanel. ## Cohort Syncing Ingestion with this integration is available only for Statsig Cloud. For Warehouse Native, create a metrics source that references this data in your warehouse. Statsig can ingest user information via a [Mixpanel Cohort Syncing](https://developer.mixpanel.com/docs/cohort-webhooks) 1. On Statsig, navigate to [Segments](https://console.statsig.com/segments) on the left navigation menu and create a segment. * **Name**: Must match the name of your Mixpanel cohort. * **Type of segment**: Should be ID List. statsig-segment-config 2. On Mixpanel, click on the Data Management navbar item and choose Integrations from the dropdown. mixpanel-integration-menu 3. In the list of integrations, scroll until you find Custom Webhook and then select it. mixpanel-custom-webhook 4. In the dialog that appears, paste the url below, substituting the SERVER\_SECRET\_KEY with a "Server Secret Key" found in [Project Settings](https://console.statsig.com/api_keys), then click Continue. ``` https://api.statsig.com/v1/webhooks/mixpanel?statsig-api-key=SERVER_SECRET_KEY ``` custom-webhook-dialog 5. Click "Enable" (or "Confirm" if you are updating the integration). 6. You can now kick off a cohort sync job on Mixpanel from the Cohorts page. ## Configuring Outbound Events To export your Statsig events to Mixpanel: 1. Get a copy of your "Project Token" from Mixpanel by following this [guide](https://help.mixpanel.com/hc/en-us/articles/115004502806-Find-Project-Token-). 2. Paste your project token into the Outgoing configuration on the Statsig integration panel. 3. Select your **Data Residency Region** based on your Mixpanel project's data residency configuration: * **US** (default): Events are sent to the global `api.mixpanel.com` endpoint * **EU**: For projects using [EU data residency](https://docs.mixpanel.com/docs/privacy/eu-residency) * **India**: For projects using [India data residency](https://docs.mixpanel.com/docs/privacy/in-residency) 4. Hit "Enable" (or "Confirm" if you are updating the integration). mixpanel-outgoing-configuration 5. Verify your events are being forwarded by visiting the Events tab on Mixpanel. mixpanel-event-tab ### Filtering Events You can customize which events should be sent to Mixpanel using [Event Filtering](/integrations/event_filtering#outgoing-event-filtering) # mParticle Source: https://docs.statsig.com/integrations/data-connectors/mparticle Connect mParticle with Statsig to forward customer data platform events to Statsig for experiment analysis, metrics, and audience targeting. ## Overview Enabling the [mParticle](https://www.mparticle.com/) integration for Statsig allows Statsig to receive events from mParticle. You can find all events that Statsig receives from mParticle in the [Metrics](/metrics) tab in the Statsig console. Statsig will automatically include these events in [Pulse](/pulse/read-pulse) and [Experiment](/experiments-plus/monitor) results for your feature gates and experiments respectively. ## Configuring Incoming Events Ingestion with this integration is available only for Statsig Cloud. For Warehouse Native, create a metrics source that references this data in your warehouse. 1. From the [API Keys](https://console.statsig.com/api_keys) tab in the Statsig console, copy the Statsig `Server Secret Key`. 2. Use your Statsig `Server Secret Key` to configure a Statsig Event Integration via mParticle's integrations directory. ### Mapping User IDs In order to associate your mParticle events with your Statsig Feature Gates or Experiments, you must use mParticle's [IDSync framework](https://docs.mparticle.com/guides/idsync/introduction/) to ensure your mParticle events pass along the same user IDs used with the Statsig SDK. ## Configuring Outbound Events To export your Statsig events to mParticle: 1. Go to your mParticle account and choose `Setup` then `Inputs` on the left-hand column to start configuring your integration. mParticle navigation drawer highlighting the Setup to Inputs path 2. Click on the `Feeds` tab within the page that loads, click on `Add Feed Input` button, and then search for `Statsig` and click on the option. Add Feed Input dropdown with Statsig option selected 3. Provide a name for your `Statsig Feed` and click `Save`. Statsig feed configuration form showing name field and active toggle 4. Copy the `Server to Server Key` and `Server to Server Secret` for the next step. 5. Log into the Statsig console and navigate to the [**Integrations**](https://console.statsig.com/integrations) page. 6. Click on the `mParticle` card and switch to the `Outbound` tab. Paste the `Server to Server Key` and `Server to Server Secret` in their respective boxes and **Enable** the integration. # RevenueCat Source: https://docs.statsig.com/integrations/data-connectors/revenuecat Connect RevenueCat with Statsig to send mobile subscription events to Statsig for revenue experiment analysis and lifecycle metric tracking. Ingestion with this integration is available only for Statsig Cloud. For Warehouse Native, create a metrics source that references this data in your warehouse. ## Overview Enabling the RevenueCat integration allows Statsig to pull billing, subscription, and revenue metrics into your Statsig projects. This provides easy mechanisms to optimize purchases and revenue by using Statsig's feature gates or experimentation tools without any additional logging. Statsig integrates with RevenueCat through a Webhook and receives data as mentioned [in the RevenueCat documentation](https://docs.revenuecat.com/docs/webhooks) ## Configuring Incoming Metrics 1. Copy your **Statsig Server Secret Key** from the [API Keys](https://console.statsig.com/api_keys) tab in the Statsig console. 2. Navigate to your app in the RevenueCat dashboard and choose **Statsig** from the Integrations menu. 3. Enter your **Statsig Server Secret** and click **Save**. RevenueCat integration settings entering Statsig secret 4. On the Statsig [Integrations](https://console.statsig.com/integrations) page, enable the RevenueCat integration. If you're running an experiment with the user as your unit type, you must set the RevenueCat `appUserID` to match the `userID` that you log with the Statsig SDK, for example, when you expose the user to a Statsig feature gate or experiment. Check out how to set the `appUserID` on RevenueCat [here](https://docs.revenuecat.com/docs/user-ids#provided-app-user-id). By default, Statsig will not ingest [Sandbox events](https://docs.revenuecat.com/docs/webhooks#testing) from RevenueCat to reduce noise from test events. However, you can explicitly enable ingestion of Sandbox events into Statsig using the Statsig [Integrations](https://console.statsig.com/integrations) page while debugging. # RudderStack Source: https://docs.statsig.com/integrations/data-connectors/rudderstack Connect RudderStack with Statsig to forward customer data platform events to Statsig for experiment analysis, metrics, and audience targeting. ## Overview Enabling the RudderStack integration for Statsig will allow Statsig to pull in your RudderStack events. This allows you to run your experiment analysis on Statsig with all of your existing events from RudderStack without requiring any additional logging. When Statsig receives events from RudderStack, these will be visible and aggregated in the [Metrics](/metrics) tab in the Statsig console. These events will automatically be included in your [Pulse](/pulse/read-pulse) results for A/B tests with Statsig's [feature flags](/feature-flags/overview) as well as all your [Experiment](/experiments-plus/monitor) results. ## Configuring Incoming Events Ingestion with this integration is available only for Statsig Cloud. For Warehouse Native, create a metrics source that references this data in your warehouse. To ingest your events from RudderStack, 1. On [app.rudderstack.com](https://app.rudderstack.com/), navigate to "Connections" and click **Add Destination** . 2. Search for “Statsig” in the Destinations Catalog, and select the “Statsig” destination. 3. Give your connection a name and choose which Source should send data to the “Statsig” destination. 4. From the [Statsig dashboard](https://console.statsig.com/api_keys), copy the Statsig "Server Secret Key”. 5. Enter the Statsig “Server Secret Key” in the “Statsig” destination settings in RudderStack. 6. On the Statsig [Integration page](https://console.statsig.com/integrations) enable the RudderStack integration. 7. As your RudderStack events flow into Statsig, you'll see a live **Log Stream** in the [Metrics](/metrics) tab in the Statsig console. You can click one of these events to see the details that are logged as part of the event. Statsig metrics log stream interface #### User IDs and Custom IDs Statsig automatically detects the `event` and `userID` fields that you log through your RudderStack events. If you're running an experiment with the user as your unit type, this userID should match the user identifier that you log with the Statsig SDK. If you're using a [custom ID](/guides/experiment-on-custom-id-types) as the unit type for your experiment, you can provide this identifier using the key `statsigCustomIDs` as part of the RudderStack `properties` field as shown below. ```bash title="JSON Body" theme={null} { ... properties: { "statsigCustomIDs": [ "companyID", "", "stableID", "",] } } ``` The `statsigCustomIDs` field in properties should be an array, where the even index is the name of the user ID type and the odd index is the value of the previous element in the array. Assuming you've created this custom ID type on Statsig (under **ID Type Settings** in your [Project Settings](https://console.statsig.com/settings)), Statsig will automatically recognize these custom identifiers to compute your experiment results appropriately. #### Environments By default, all events are treated as "production" events, but you can also differentiate your event traffic by specifying the environment that the events are coming from. This allows you to avoid non-production data making it into your production metrics. If you would like to include the environment tier, you can add it to the properties object of your event. The required format is below: ```json theme={null} { ... "properties": { "statsigEnvironment": { "tier": "staging" } } } ``` To learn more about environments see [Using Environment](/guides/using-environments). ## Configuring Outbound Events To export your Statsig events to RudderStack, 1. Log into the Statsig console and navigate to the [**Integrations**](https://console.statsig.com/integrations) page. 2. Click on the **RudderStack** card and switch to the **Outbound** tab. 3. Follow the steps outlined in [RudderStack's Webhook Source](https://www.rudderstack.com/docs/stream-sources/webhook-source/) to get the required "Write Key" and "Data Plane URL". 4. Once filled out, hit enable to save your changes. ## Filtering Events You can customize which events should be sent and received via RudderStack using [Event Filtering](/integrations/event_filtering) # Segment Source: https://docs.statsig.com/integrations/data-connectors/segment Connect Segment with Statsig to forward customer data platform events to Statsig for experiment analysis, metrics, and audience targeting. Enabling the Segment integration for Statsig will allow Statsig to pull in your Segment events. This allows you to run your experiment analysis on Statsig with all of your existing events from Segment without requiring any additional logging. When Statsig receives events from Segment, these will be visible and aggregated in the [Metrics](/metrics) tab in the Statsig console. These events will automatically be included in your [Pulse](/pulse/read-pulse) results for A/B tests with Statsig's feature gates as well as all your [Experiment](/experiments-plus/monitor) results. ### Supported Segment Event Types * [Track](https://segment.com/docs/connections/spec/track/) * [Page](https://segment.com/docs/connections/spec/page/) * [Group](https://segment.com/docs/connections/spec/group/) * [Screen](https://segment.com/docs/connections/spec/screen/) * [Identify\*](https://segment.com/docs/connections/spec/identify/) Identify calls are only supported for syncing Segment Engage Audiences with Statsig Segments ### Benefits of using the Segment integration Using the Segment integration has several benefits over other methods of event ingestion: * Customers who are ingesting customer data with Segment will be able to quickly populate Statsig with metrics and can typically get up and running within a day * Customers will only have to use Statsig's assignment SDKs (gate/experiment allocation), simplifying your code and engineer workflows * Additional logging can be done via the [event logging SDKs](/guides/logging-events#logging-events-via-sdks) but will and additional code orchestration and a collection window * With [event filtering](/integrations/event_filtering) you can control which events are ingested and make billing more predictable * If you have [Segment Replay](https://segment.com/docs/guides/what-is-replay/), you can forward up to 7 days of historical events to Statsig for analysis ## Configuring Inbound Events into Statsig Ingestion with this integration is available only for Statsig Cloud. For Warehouse Native, create a metrics source that references this data in your warehouse. To send events collected from Segment into Statsig, you must configure the Statsig Destination within the your Segment account: ### Using OAuth The easiest way to connect Statsig to Segment is via OAuth. 1. To get started, within the Statsig Console, go to **project settings → integrations → Segment → Enable** 2. Click “Configure Segment OAuth” Statsig integration settings showing Segment OAuth button 3. Select the workspace and source that will send data to Statsig. Click “Allow”: Segment OAuth consent screen selecting workspace and source 4. After the integration has been enabled, you can configure event filtering and map additional identifiers. This is useful for mapping device level identifiers as well as `anonymousIds` generated from Segment. We recommend creating a custom ID called `segmentAnonymousId` and mapping the `anonymousId` from Segment to it. Statsig Segment integration panel with identifier mapping options ### Manual Configuration If you are unable to connect to Segment via OAuth, you can still manually connect Statsig to Segment by configuring 1. Within the [Segment App](https://app.segment.com), navigate to your Destinations, and select "**Add Destination**" 2. Search for “**Statsig**” and select the destination Segment destination catalog listing Statsig 3. Select "Statsig" from the list of available integrations, and then select **sources** that will send data to Statsig. Segment destination setup selecting sources 4. You must provide a Statsig Server SDK key. You can copy an existing server key or create a new one from the [Statsig console settings](https://console.statsig.com/api_keys). * Create or copy a server SDK key Statsig API keys page showing server key Segment destination configuration entering Statsig server key * Put your Server Secret Key in the “API Key” field in the Statsig Destination Segment destination API key input 5. In order to set up mappings you must enable the Segment Integration on the Statsig . Go to the Statsig Console > Project Settings > Integrations Tab > Segment > Click **Enable**: Statsig Segment integration enable toggle * After the integration has been enabled, you can configure event filtering and map additional identifiers. This is useful for mapping device level identifiers as well as `anonymousIds` generated from Segment. Statsig identifier mapping form 6. Now we can verify that events are being properly sent to Statsig using the [Segment Event Tester](https://segment.com/docs/connections/test-connections/). * In the event tester, send a test event that can verify any filtering and/or mapping you have set up: Segment integration panel showing outgoing toggle * In the your [Statsig Console Events Explorer](https://console.statsig.com/metrics/events), click on the test event in the log stream and verify the event: Segment destination configuration for outgoing data * Note: In this example the `anonymousId` is mapped to Statsig’s `segmentAnonymousId` based on the above mapping. The User ID is inferred from the Segment `userID` field Statsig integration for Segment Engage audience mapping 7. As your Segment events flow into Statsig, you'll see a live **Log Stream** in the [Metrics](/metrics) tab in the Statsig console. You can click one of these events to see the details that are logged as part of the event. Log details panel showing Segment event payload fields and metadata ## Working with Users Statsig will join incoming user identifiers to whichever [unit of randomization](/experiments-plus#choosing-the-right-randomization-unit) you choose. This allows you to be flexible with your experimentation and enables testing on known (userID) and unknown (anonymousID) traffic as well as any custom identifiers your team may have (deviceID, companyID, vehicleID, etc). ### User IDs and Custom IDs Statsig automatically detects the `event` and `userId` fields that are logged through your Segment events (see [`track`](https://segment.com/docs/connections/spec/track/) for an example). If you're running an experiment with the userId as your unit type, this `userID` should match the user identifier that you log with the Statsig SDK. If you're using a [custom ID](/guides/experiment-on-custom-id-types) as the unit type for your experiment, you can provide this identifier using the key `statsigCustomIDs` as part of the Segment `properties` field as shown below. ```bash title="JSON Body" theme={null} { ... properties: { "statsigCustomIDs": [ "companyID", ""] } } ``` The `statsigCustomIDs` field in properties should be an array, where the even index is the name of the user ID type and the odd index is the value of the previous element in the array. Assuming you've created this custom ID type on Statsig (under **ID Type Settings** in your [Project Settings](https://console.statsig.com/settings)), Statsig will automatically recognize these custom identifiers to compute your experiment results appropriately. ### Anonymous Users #### Mapping anonymous users The Segment integration also allows the mapping of top level fields to custom IDs you define in Statsig. To do this, visit the Segment panel on the Statsig Integrations page and look for the "Map Identifier" section. Here you can choose fields you would like mapped to a [Custom ID](/guides/experiment-on-custom-id-types). This is particularly useful for working with the [Segment anonymous ID](https://segment.com/docs/connections/sources/catalog/libraries/website/javascript/identity/#anonymous-ids), which is passed as an event field `anonymousId`. By defining a custom ID called `segmentAnonymousId`, Statsig can easily map your anonymous Segment traffic to your metric data. Values passed in `properties.statsigCustomIDs` will take precedence over mapped identifiers below. Segment integration card with Map Identifiers form mapping anonymousId to custom ID #### Experimenting on anonymous traffic For example, if you're running experiments on anonymous users, you can use Segment's `anonymousId` as the unit of randomization. First, you will want to [add a new customer identifier to Statsig](/guides/experiment-on-custom-id-types#step-1---add-companyid-as-a-new-id-type-in-your-project-settings). In the above example, we call our new custom ID `segmentAnonymousId`. Then, when [initializing](/client/javascript-sdk) the Statsig SDK, if you have access to the Segment `anonymousId` you will want to pass it to Statsig as a custom ID. For example, your Statsig initialization may look like this: ```jsx theme={null} import { StatsigClient } from '@statsig/js-client'; await Statsig.initialize( "client-sdk-key", ); const client = new StatsigClient(sdkKey, { userID: "some_user_id", customIDs: { segmentAnonymousId: analytics.user().anonymousId() } }, { environment: { tier: "production" } } ); ``` You can access Segment's `anonymousId` using `analytics.user().anonymousId()` as [outlined in the Segment docs here](https://segment.com/docs/connections/sources/catalog/libraries/website/javascript/identity/). It's important to note, that the Segment SDK may initialize after Statsig SDK, and the `anonymousId` may not be available in that scenario. It's important to check this in your specific implementation. If you're using vanilla javascript, you may want to wait for the `anonymousId` from the Segment SDK after initialization and call the `updateUser()` method on the Statsig SDK to update this ID. #### Example Mapping Flow Refer to the following diagram to help orient you to mapping `anonymousIds` in Segment to a custom ID representing anonymous users in Statsig: Identifier mapping flowchart 1. Initialize the Statsig SDK with your [Statsig User](/concepts/user) which will contain an optional `userID` value and a `customID` that you've created in the Statsig UI - `segmentAnonymousId` in this example. 2. As you orchestrate features/experiments, Statsig will associate this user to a variant using the unit of randomization chosen. For anonymous users, we'll use `segmentAnonymousId`. 3. Your existing Segment implementation tracks user traffic and associates anonymous users to the top-level field `anonymousId`. 4. This `anonymousId` is mapped in Statsig (to `segmentAnonymousId`), properly associating the identifier used in experiment exposures to the same identifier used to track user actions. ### Syncing Statsig Segment ID Lists with Segment Engage Audiences By using [Segment Engage Audiences](https://segment.com/docs/engage/audiences/) you are able to maintain a list of users that can be used for targeting using [Statsig Feature Gates](/feature-flags/overview). To configure this: 1. Create a [Statsig ID List Segment](/segments/create-new) on the Statsig Console. 2. Follow the [Segment guide for Audiences](https://segment.com/docs/engage/audiences/) to create a new Audience and choose `Statsig` as a Destination. The `audience_key` must match the ID of the `Statsig ID List Segment` created. Once these steps have been completed, your Segment Audience will be synced, and you will be able to target those users for features you develop or experiments you run. ### Custom Properties Passing [custom properties to a Statsig User](/concepts/user#user-attributes) (see `custom` field) enables targeting on specific cohorts of your users in feature gates and experimentation. Providing custom user properties also allows you to drill down your results to specific populations (ex: android/iOS, isVIP, etc) when [reading pulse results](/pulse/custom-queries#running-a-custom-query). If you're using custom fields to [target users](/feature-flags/conditions#custom) in your feature gates, you can provide these properties through Segment using the key `statsigCustom` as part of the Segment `properties` field, as an array of key value pairs: `[key1, value1, key2, value2, ...]`. An example is shown below: ```bash title="JSON Body" theme={null} { ... properties: { "statsigCustom": [ "isVIP", "true", "marketing_campaign", "abx343", ...] } } ``` ## Configuring Outbound Events to Segment To export your Statsig events to Segment, 1. In your Segment app, add a new source → search for Statsig → click **Next** Segment outbound configuration selecting Statsig destination 2. Name your source → Create Source → click **Done**: Segment workspace auth prompt for outbound integration 3. Locate your **Write Key** and copy it. 4. Log into the Statsig console and navigate to the [**Integrations**](https://console.statsig.com/integrations) page. 5. Click on the **Segment** card and switch to the **Outbound** tab, paste the **Write Key** into the **API Key** text box shown below, and click **Enable**. Segment outbound integration configuration interface ### Outbound event schema Statsig exports log events and exposure events to segment as `track` events: ``` { type: 'track', userId: event.userID, timestamp: Number(event.timestamp), event: event.eventName, context: { user: event.user, value: event.value, metadata: event.metadata, library: { name: 'statsig', version: '1.0', }, stableID: event.statsigMetadata.stableID // stableID, if you are relying on that for anonymous users }, properties: { value: event.value, metadata: event.metadata, }, }; } ``` Config Change events follow this schema: ``` { userId: statsigUserID, timestamp: Date.now(), event: 'statsig::config_change', context: { library: { name: 'statsig', version: '1.0', }, }, properties: { author: author, configName: configName, description: changeDescription, environment: environment, }, }, ``` ## Environments #### Environments By default, all events are treated as "production" events, but you can also differentiate your event traffic by specifying the environment that the events are coming from. This allows you to avoid non-production data making it into your production metrics. If you would like to include the environment tier, you can add it to the properties object of your event. The required format is below: ```json theme={null} { ... "properties": { "statsigEnvironment": { "tier": "staging" } } } ``` To learn more about environments see [Using Environment](/guides/using-environments). ## Working with Segment Metrics in the Statsig UI Segment events are piped into Statsig and are accessible in the metrics console like any other event. Furthermore, these metrics will be accessible to use as monitoring metrics in your feature gates and experiments so you can utilize your existing metric collection via Segment with Statsig's experimentation platform. Segment events are prepended with `segment::` so they can be easily distinguished from other sources Segment destination configuration for first exposures These metrics will be reported in pulse results among other monitoring metrics: Segment HTTP endpoint settings ## Filtering Events You can customize which events should be sent and received via Segment using [Event Filtering](/integrations/event_filtering) # Stitch Source: https://docs.statsig.com/integrations/data-connectors/stitch Connect Stitch with Statsig to ingest events and tables from third-party sources via your warehouse for use in Statsig metrics and experiments. ## Overview Enabling the Stitch integration for Statsig will allow Statsig to push events to your Stitch account through a webhook. This allows you to forward Statsig data to any connectors available from Stitch. ## Configuring Outbound Events 1. Follow the steps in the [Stitch Webhook Setup Guide](https://www.stitchdata.com/docs/integrations/webhooks/stitch-incoming-webhooks#setup) to create a new Webhook URL. 2. On the Statsig [Integrations](https://console.statsig.com/integrations) page, enable the Stitch integration by pasting in the Stitch Webhook URL and click **Confirm**. ### Event Format Events will be sent in batches in a JSON format. The structure of a Statsig Event sent will look like the following: | Field | Type | Description | | --------- | ------ | ------------------------------------------------------------------------------------------------ | | event | String | Name of the event provided | | user | JSON | [Statsig User Object](/concepts/user) | | userId | String | User ID provided | | stableId | String | Stable ID | | timestamp | Number | Timestamp in MS of the event | | value | String | Value of the event provided | | metadata | JSON | Both custom metadata provided and metadata related to the logging of this event added by Statsig | #### Custom Event Formatting - logEvent > ```json theme={null} { "userId": "a_user", "stableId": "123", "timestamp": 1655231253265, "event": "my_custom_event", "context": { "user": { "userID": "a_user", "email": "a.user@email.com" }, "value": "a_custom_value", "metadata": { }, "library": { "name": "statsig", "version": "1.0" } } } ``` #### Feature Gate Exposure Formatting - checkGate > ```json theme={null} { "userId": "a_user", "stableId": "123", "timestamp": 1655231253265, "event": "statsig::gate_exposure", "context": { "user": { "userID": "a_user", "email": "a.user@email.com" }, "value": "", "metadata": { "gate": "a_gate", "gateValue": "false", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "library": { "name": "statsig", "version": "1.0" } } } ``` #### Dynamic Config Exposure Formatting - getConfig > ```json theme={null} { "userId": "a_user", "stableId": "123", "timestamp": 1655231253265, "event": "statsig::config_exposure", "context": { "user": { "userID": "a_user", "email": "a.user@email.com" }, "value": "", "metadata": { "config": "a_config", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "library": { "name": "statsig", "version": "1.0" } } } ``` #### Experiment Exposure Formatting - getExperiment > ```json theme={null} { "userId": "a_user", "stableId": "123", "timestamp": 1655231253265, "event": "statsig::experiment_exposure", "context": { "user": { "userID": "a_user", "email": "a.user@email.com" }, "value": "", "metadata": { "config": "an_experiment", "ruleID": "4SauZJcM1T7zNvh1igBjwE", "reason": "Network", "time": "1655231249644", "experimentGroupName": "Control" }, "library": { "name": "statsig", "version": "1.0" } } } ``` ## Filtering Events Once you've enabled outbound events to Stitch, you can select which categories of Statsig events you want to export by click on the **Event Filtering** button and checking the appropriate boxes as shown below. Initial Setup Dialog Event Filtering # Data Warehouse Exports Source: https://docs.statsig.com/integrations/data-exports/data_warehouse_exports Export Statsig events, exposures, and metric data into your cloud data warehouse on a schedule for downstream BI, modeling, and custom analysis. ## Introduction You can export your data from Statsig to your data warehouse with a data connection. This lets you send exposures and events directly to your warehouse for further analysis. We currently support connections to Snowflake, Redshift, S3, BigQuery, and Databricks. Data Warehouse Exports are an Enterprise-only feature. If you are on the Developer or Pro tiers and wish to upgrade to Enterprise, feel free to reach out to our team [here](https://www.statsig.com/contact/demo) or via support. ## How to Begin 1. Go to Statsig Console 2. Navigate to the Help and Tools Section on the side navigation bar 3. Go to “Exports List” Statsig Help and Tools menu showing Exports List option On the Exports page, you will find an Export History tab and a Schedule Export tab. The Export History tab shows you a list of all previous experimentation exports triggered from the Experiment Results page. To understand how to export one-off experimentation results, check out the documentation [here](/integrations/data-exports/experiment_result_exports). Under the Scheduled Export tab, you will be prompted to set up a data warehouse connection. You will be required to set up connections with the necessary credentials and grant Statsig Read, Write, and Delete permissions. The warehouse-specific tabs below show the fields Statsig asks for and example SQL you can use to create each export table yourself. Scheduled export setup form for Snowflake connection with credential fields ## Table Schema Statsig can export both Exposures and Events to your destination. The exported columns are logically the same across SQL warehouses, but the physical types vary by warehouse. S3 is a file destination rather than a SQL table destination. **Setup summary** Configure: * BigQuery Project ID * BigQuery Dataset ID * one destination table name per export type Permissions: * grant the Statsig service account `BigQuery User` at the project level * grant the same service account `BigQuery Data Editor` on the target dataset Statsig uses this access to create the table if needed and validate the connection by inserting and deleting a test row. If you prefer to create the export tables yourself, you can use SQL like this: ```sql BigQuery Exposures theme={null} CREATE TABLE IF NOT EXISTS `PROJECT_ID.DATASET_ID.EXPOSURES_TABLE` ( company_id STRING, unit_id STRING, unit_type STRING, exposure_type STRING, name STRING, rule STRING, experiment_group STRING, first_exposure_utc TIMESTAMP, first_exposure_pst_date DATE, as_of_pst_date DATE, percent FLOAT64, rollout BIGINT, user_dimensions STRING, inserted_at TIMESTAMP, rule_name STRING, group_id STRING, non_analytics BOOLEAN ); ``` ```sql BigQuery Events theme={null} CREATE TABLE IF NOT EXISTS `PROJECT_ID.DATASET_ID.EVENTS_TABLE` ( user_id STRING, stable_id STRING, custom_ids JSON, timestamp TIMESTAMP, event_name STRING, event_value STRING, user_object JSON, statsig_metadata JSON, company_metadata JSON ); ``` **Exposures** Please note that we do not export exposures from rules that are 0%/100%. | Field name | Type | Mode | | -------------------------- | --------- | -------- | | company\_id | STRING | NULLABLE | | unit\_id | STRING | NULLABLE | | unit\_type | STRING | NULLABLE | | exposure\_type | STRING | NULLABLE | | name | STRING | NULLABLE | | rule | STRING | NULLABLE | | experiment\_group | STRING | NULLABLE | | first\_exposure\_utc | TIMESTAMP | NULLABLE | | first\_exposure\_pst\_date | DATE | NULLABLE | | as\_of\_pst\_date | DATE | NULLABLE | | percent | FLOAT64 | NULLABLE | | rollout | BIGINT | NULLABLE | | user\_dimensions | STRING | NULLABLE | | inserted\_at | TIMESTAMP | NULLABLE | | rule\_name | STRING | NULLABLE | | group\_id | STRING | NULLABLE | | non\_analytics | BOOLEAN | NULLABLE | **Events** | Field name | Type | Mode | | ----------------- | --------- | -------- | | user\_id | STRING | NULLABLE | | stable\_id | STRING | NULLABLE | | custom\_ids | JSON | NULLABLE | | timestamp | TIMESTAMP | NULLABLE | | event\_name | STRING | NULLABLE | | event\_value | STRING | NULLABLE | | user\_object | JSON | NULLABLE | | statsig\_metadata | JSON | NULLABLE | | company\_metadata | JSON | NULLABLE | **Setup summary** Configure: * Account Name * Database Name * Schema Name * either Username and Password, or Private Key authentication * one destination table name per export type Permissions: * the configured user must be able to create tables and insert/delete rows in the target database and schema If you prefer to create the export tables yourself, you can use SQL like this: ```sql Snowflake Exposures theme={null} CREATE TABLE IF NOT EXISTS DATABASE_NAME.SCHEMA_NAME.EXPOSURES_TABLE ( company_id STRING, unit_id STRING, unit_type STRING, exposure_type STRING, name STRING, rule STRING, experiment_group STRING, first_exposure_utc TIMESTAMP, first_exposure_pst_date DATE, as_of_pst_date DATE, percent DOUBLE, rollout BIGINT, user_dimensions STRING, inserted_at TIMESTAMP, rule_name STRING, group_id STRING, non_analytics BOOLEAN ); ``` ```sql Snowflake Events theme={null} CREATE TABLE IF NOT EXISTS DATABASE_NAME.SCHEMA_NAME.EVENTS_TABLE ( user_id STRING, stable_id STRING, custom_ids STRING, timestamp TIMESTAMP, event_name STRING, event_value STRING, user_object STRING, statsig_metadata STRING, company_metadata STRING ); ``` **Exposures** | Field name | Type | Mode | | -------------------------- | --------- | -------- | | company\_id | STRING | NULLABLE | | unit\_id | STRING | NULLABLE | | unit\_type | STRING | NULLABLE | | exposure\_type | STRING | NULLABLE | | name | STRING | NULLABLE | | rule | STRING | NULLABLE | | experiment\_group | STRING | NULLABLE | | first\_exposure\_utc | TIMESTAMP | NULLABLE | | first\_exposure\_pst\_date | DATE | NULLABLE | | as\_of\_pst\_date | DATE | NULLABLE | | percent | DOUBLE | NULLABLE | | rollout | BIGINT | NULLABLE | | user\_dimensions | STRING | NULLABLE | | inserted\_at | TIMESTAMP | NULLABLE | | rule\_name | STRING | NULLABLE | | group\_id | STRING | NULLABLE | | non\_analytics | BOOLEAN | NULLABLE | **Events** | Field name | Type | Mode | | ----------------- | --------- | -------- | | user\_id | STRING | NULLABLE | | stable\_id | STRING | NULLABLE | | custom\_ids | STRING | NULLABLE | | timestamp | TIMESTAMP | NULLABLE | | event\_name | STRING | NULLABLE | | event\_value | STRING | NULLABLE | | user\_object | STRING | NULLABLE | | statsig\_metadata | STRING | NULLABLE | | company\_metadata | STRING | NULLABLE | **Setup summary** Configure: * Cluster Endpoint * Username and Password * Staging Schema * one destination table name per export type Permissions: * the configured user must be able to create tables and insert/delete rows in the target database and schema * if your cluster is not directly reachable, you can also configure SSH tunneling If you prefer to create the export tables yourself, you can use SQL like this: ```sql Redshift Exposures theme={null} CREATE TABLE IF NOT EXISTS "DATABASE_NAME"."STAGING_SCHEMA"."EXPOSURES_TABLE" ( "company_id" VARCHAR, "unit_id" VARCHAR, "unit_type" VARCHAR, "exposure_type" VARCHAR, "name" VARCHAR, "rule" VARCHAR, "experiment_group" VARCHAR, "first_exposure_utc" TIMESTAMP, "first_exposure_pst_date" DATE, "as_of_pst_date" DATE, "percent" DOUBLE PRECISION, "rollout" BIGINT, "user_dimensions" VARCHAR, "inserted_at" TIMESTAMP, "rule_name" VARCHAR, "group_id" VARCHAR, "non_analytics" BOOLEAN ); ``` ```sql Redshift Events theme={null} CREATE TABLE IF NOT EXISTS "DATABASE_NAME"."STAGING_SCHEMA"."EVENTS_TABLE" ( "user_id" VARCHAR, "stable_id" VARCHAR, "custom_ids" VARCHAR, "timestamp" TIMESTAMP, "event_name" VARCHAR, "event_value" VARCHAR, "user_object" VARCHAR, "statsig_metadata" VARCHAR, "company_metadata" VARCHAR ); ``` Note: the default size for VARCHAR is 256 and we truncate all columns to 256 characters for Redshift. If you anticipate certain columns will exceed these limits, please adjust accordingly and let us know to remove these limits. Common longer columns are `user_dimensions` in exposures and `user_object`, `custom_ids`, and `company_metadata` in events. **Exposures** | Field name | Type | Mode | | -------------------------- | ---------------- | -------- | | company\_id | VARCHAR | NULLABLE | | unit\_id | VARCHAR | NULLABLE | | unit\_type | VARCHAR | NULLABLE | | exposure\_type | VARCHAR | NULLABLE | | name | VARCHAR | NULLABLE | | rule | VARCHAR | NULLABLE | | experiment\_group | VARCHAR | NULLABLE | | first\_exposure\_utc | TIMESTAMP | NULLABLE | | first\_exposure\_pst\_date | DATE | NULLABLE | | as\_of\_pst\_date | DATE | NULLABLE | | percent | DOUBLE PRECISION | NULLABLE | | rollout | BIGINT | NULLABLE | | user\_dimensions | VARCHAR | NULLABLE | | inserted\_at | TIMESTAMP | NULLABLE | | rule\_name | VARCHAR | NULLABLE | | group\_id | VARCHAR | NULLABLE | | non\_analytics | BOOLEAN | NULLABLE | **Events** | Field name | Type | Mode | | ----------------- | --------- | -------- | | user\_id | VARCHAR | NULLABLE | | stable\_id | VARCHAR | NULLABLE | | custom\_ids | VARCHAR | NULLABLE | | timestamp | TIMESTAMP | NULLABLE | | event\_name | VARCHAR | NULLABLE | | event\_value | VARCHAR | NULLABLE | | user\_object | VARCHAR | NULLABLE | | statsig\_metadata | VARCHAR | NULLABLE | | company\_metadata | VARCHAR | NULLABLE | **Setup summary** Configure: * API Key * Server Hostname * HTTP Path * Database * one destination table name per export type Permissions: * the configured user or token must be able to create tables and insert/delete rows in the target database If you prefer to create the export tables yourself, you can use SQL like this: ```sql Databricks Exposures theme={null} CREATE TABLE IF NOT EXISTS DATABASE_NAME.EXPOSURES_TABLE ( company_id STRING, unit_id STRING, unit_type STRING, exposure_type STRING, name STRING, rule STRING, experiment_group STRING, first_exposure_utc TIMESTAMP, first_exposure_pst_date DATE, as_of_pst_date DATE, percent DOUBLE, rollout BIGINT, user_dimensions STRING, inserted_at TIMESTAMP, rule_name STRING, group_id STRING, non_analytics BOOLEAN ); ``` ```sql Databricks Events theme={null} CREATE TABLE IF NOT EXISTS DATABASE_NAME.EVENTS_TABLE ( user_id STRING, stable_id STRING, custom_ids STRING, timestamp TIMESTAMP, event_name STRING, event_value STRING, user_object STRING, statsig_metadata STRING, company_metadata STRING ); ``` **Exposures** | Field name | Type | Mode | | -------------------------- | --------- | -------- | | company\_id | STRING | NULLABLE | | unit\_id | STRING | NULLABLE | | unit\_type | STRING | NULLABLE | | exposure\_type | STRING | NULLABLE | | name | STRING | NULLABLE | | rule | STRING | NULLABLE | | experiment\_group | STRING | NULLABLE | | first\_exposure\_utc | TIMESTAMP | NULLABLE | | first\_exposure\_pst\_date | DATE | NULLABLE | | as\_of\_pst\_date | DATE | NULLABLE | | percent | DOUBLE | NULLABLE | | rollout | BIGINT | NULLABLE | | user\_dimensions | STRING | NULLABLE | | inserted\_at | TIMESTAMP | NULLABLE | | rule\_name | STRING | NULLABLE | | group\_id | STRING | NULLABLE | | non\_analytics | BOOLEAN | NULLABLE | **Events** | Field name | Type | Mode | | ----------------- | --------- | -------- | | user\_id | STRING | NULLABLE | | stable\_id | STRING | NULLABLE | | custom\_ids | STRING | NULLABLE | | timestamp | TIMESTAMP | NULLABLE | | event\_name | STRING | NULLABLE | | event\_value | STRING | NULLABLE | | user\_object | STRING | NULLABLE | | statsig\_metadata | STRING | NULLABLE | | company\_metadata | STRING | NULLABLE | **Setup summary** Configure: * Region * Bucket * one folder name per export type Permissions: * grant the Statsig IAM user bucket-level access to list the bucket and retrieve its location * grant object-level read, write, and delete permissions, including multipart upload management S3 exports are file-based. Statsig writes export files to the configured bucket and folder for each export type. In the export UI, the per-export destination is configured as a folder rather than a table. **Export outputs** | Export type | Destination shape | | ----------- | ----------------------------------------------------- | | Exposures | Files written to the configured folder in your bucket | | Events | Files written to the configured folder in your bucket | ## Troubleshooting Exports If you set up an Export flow in Statsig, Statsig can notify you if your data connection is failing. To enable notifications, go to Settings → My Account → Email Notifications → Edit and click Alerts to subscribe to these notifications. If your Exports are failing, make sure your warehouse is connected with up-to-date credentials and the necessary Read, Write, and Delete permissions. # Experiment Result Exports Source: https://docs.statsig.com/integrations/data-exports/experiment_result_exports Export Statsig experiment results into your data warehouse or BI tool for custom reporting, executive dashboards, and longitudinal experiment analysis. ## Overview Your data is your data. Statsig makes it easy to export both the reports and the raw data your feature rollouts and experiments generate. ## How to 1. [Download experiment results](/pulse/export#how-to-export-pulse-data) from the Console as a CSV file - including a summary view, exposures and the raw data. This is meant for one-off downloads/analysis. Pulse results export interface 2. [Programmatically export the data underlying Pulse](/console-api/daily-reports). 3. For ongoing data exports we support [data integrations](/integrations/introduction) via customer data platforms like Segment, RudderStack and mParticle. There is also a [generic webhook](/integrations/event_webhook) if you want to build your own integration. If you want to set up a daily export into your data warehouse, see [Data Warehouse Exports](/integrations/data-exports/data_warehouse_exports). Grid of event export integrations such as Amplitude, Braze, and Webhook ## Validating data Many teams audit and compare their data in Statsig with what they have in other systems. There are no black box algorithms. We use well-recognized statistical methods and industry best practices and you should be able to reproduce results yourself. Some tips when doing so - 1. Start small: Use a day's worth of data to reduce the variables in play. When comparing experiments, start with a full day (not days the experiment started/stopped where there's partial data). 2. Third party ads/tracking blockers can block events sent to 3rd party services. Using a server side integration or [proxying requests via your domain](/custom_proxy) will remove this. 3. Watch for time zone conversion issues to make sure a consistent definition of day is being used. 4. Statsig applies [winsorization](/stats-engine/variance_reduction#winsorization) on metrics to remove outliers. 5. Statsig applies CUPED to get faster experimental results (reduce variance on metrics using pre-experimental data). Turn this off when looking at results in the console and comparing them. # Azure Metrics Upload (Deprecated) Source: https://docs.statsig.com/integrations/data-imports/azure_upload Import event and metric data into Statsig from Azure Blob Storage on a schedule, including file format options and column-to-event mappings. This solution is still functional, but can be manual and time consuming to set up with minimal error handling. We encourage you to check out the [Data Warehouse Ingestion](/data-warehouse-ingestion/introduction) solution instead. ## Overview Statsig allows you to upload your pre-computed metrics data to a secure Azure blob owned by Statsig. We'll ingest all of your uploaded metrics for a day once you've signalled that a given day is finished uploading. ## Getting Started Reach out in slack or to your primary Statsig point of contact. We'll set up an Azure blob storage container and provide you with credentials to connect. ## Filesystem Format To allow for daily uploads, please set up your blob storage container with the following folders: * `events/` for events data * `metrics/` for metrics data * `signals/` for signal flags when you've finished uploading data for a day. You can omit this folder and instead use the [`mark_data_ready` API](/metrics/ingest) instead, but you must use one or the other We recommend writing folders by date partitions for ease of debugging, i.e. storing day's data in folders with ISO-formatted names (`YYYY-MM-DD`). ### Data Format Please make sure your data conforms to the following schemas. Events ``` | Column | Description | Rules | | -------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | timestamp | UNIX timestamp of the event | UTC timestamp | | event_name | The name of the event | String under 128 characters, using `_` for spaces | | event_value | A string representing the value of a current event. Can represent a 'dimension' or a 'value' | Read as string format; numeric values will be converted into value | | event_metadata | A dictionary in the form of a JSON string, containing named metadata for the event | String format. Not null. Length < 128 characters | | user | A JSON object representing the user this event was logged for; see below | Escaped JSON string including the keys 'custom' and 'customIDs'. A userID or customID must be provided. | | timeuuid | A unique UUID or timeUUID used for deduping. If omitted, will be generated but will not be effective for deduping | UUID format | ``` Please refer to docs for the [Statsig User Object](/concepts/user#user-attributes) for available fields. An example would look like: ``` { userID: "12345", customIDs: { stableID: "", ... } email: "12345@gmail.com", userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.40 Safari/537.36", ip: "192.168.1.101", country: "US", locale: "en_US", appVersion: "1.0.1", systemName: "Android", systemVersion: "15.4", browserName: "Chrome", browserVersion: "45.0", custom: { new_user: "false", age: "22" ... }, } ``` Metrics Make sure to include all of metric\_value, numerator, and denominator, writing `cast(null as double)` for numerator and denominator if you are omitting them (or conversely for metric\_value if sending numerator/denominator). | Column | Description | Rules | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | unit\_id | The unique user identifier this metric is for. This might not necessarily be a user\_id - it could be a custom\_id of some kind | String format. Make sure this is in the same format as your logged unit\_ids | | id\_type | The id\_type the unit\_id represents. | String format. Must be a valid id\_type. The default Statsig types are user\_id/stable\_id, but you may have generated custom id\_types. Make sure this matches (case sensitive) a customID in your project, or you won't get experiment results | | date | Date of the daily metric | Read as string format; can be written as ISO date. Statsig's dates are calculated in PST - we'll load custom metrics to whatever date you use here | | metric\_name | The name of the metric | String format. Not null. Length \< 128 characters | | metric\_value | A numeric value for the metric | Double format. Metric value, or both of numerator/denominator need to be provided for Statsig to process the metric. See details below | | numerator | Numerator for metric calculation | Double format. Required for ratio metrics. If present along with a denominator in any record, the metric will be treated as ratio and only calculated for users with non-null denominators | | denominator | Denominator for metric calculation | Double format. See above | ### Scheduling Because you may be streaming events to your tables or have multiple ETLs pointing to your metrics table, Statsig relies on you signalling that your metric/events for a given day are done. To do this, write a dataset with the single column `finished_date`, which contains all dates of data which have been written to Statsig. For example, once you have written data for `2022-06-22` you would insert a record with `finished_date` of `2022-06-22` to trigger ingestion of data from up to and including `2022-06-22`. Unlike some other integrations like Snowflake, for S3 Statsig will skip dates; if your latest finished date is `2022-06-22` and you insert `2022-07-01`, we will ingest all data as of `2022-07-01` and infer that data for dates between (e.g. `2022-06-25`) is loaded. Alternatively, you can use the `mark_data_ready` API and send a timestamp for which the data previous to that timestamp has finished loading into your container. Note that, for events, Statsig processes days in PST. When you mark data ready for '2022-06-20', statsig will process events from `2022-06-20T00:00` PST to `2022-06-20T23:59....` PST. Keep this in mind when scheduling your signals! #### Checklist These are common errors we've run into - please go through and make sure your setup is looking good! * Field names are set incorrectly * The `id_type` is set correctly * Default types are `user_id` or `stable_id`. If you have custom ids, make sure that the capitalization and spelling matches as these are case sensitive (you can find your custom ID types by going to your Project Settings in the Statsig console). * Your ids match the format of ids logged from SDKs * In some cases, your data warehouse may transform IDs. This may mean we can't join your experiment or feature gate data to your metrics to calculate pulse or other reports. You can go to the Metrics page of your project and view the log stream to check the format of the ids being sent (either `User ID`, or a custom ID in `User Properties`) to confirm they match # BigQuery (Deprecated) Source: https://docs.statsig.com/integrations/data-imports/bigquery Import event and metric data into Statsig from Google BigQuery on a schedule, including authentication, queries, and column-to-event mappings. This solution is still functional, but can be manual and time consuming to set up with minimal error handling. We encourage you to check out the [Data Warehouse Ingestion](/data-warehouse-ingestion/introduction) solution instead. ## Overview The BigQuery integration allows you to export events and/or metrics from your BigQuery instance to Statsig. Here are the steps to take to enable BigQuery integration with Statsig: 1. Set up tables in your BigQuery instance. 2. Give Statsig's service account corresponding permissions on the tables. 3. Enable the BigQuery integration in the Statsig console. 4. Insert data into tables and mark data to be ready for import. ## Set Up Tables in your BigQuery instance 1. In your project, create a new dataset where tables for Statsig should live. You can use an existing dataset, but you will be giving the Statsig server user some permissions on this dataset later. 2. Create a table for pre-computed metrics, and another for signalling when data has landed with the statement below: ``` -- Replace statsig with your dataset name, if not using statsig CREATE TABLE IF NOT EXISTS statsig.statsig_user_metrics( unit_id STRING NOT NULL, id_type STRING NOT NULL, -- stable_id, user_id, etc. date DATE NOT NULL, -- YYYY-MM-DD. Statsig calculates dates according to PST timeuuid STRING, --Generated unique UUID; we will generate if not provided metric_name STRING NOT NULL, metric_value NUMERIC, numerator NUMERIC, denominator NUMERIC ); -- Replace statsig with your dataset name, if not using statsig CREATE TABLE IF NOT EXISTS statsig.statsig_user_metrics_signal( finished_date DATE ); ``` ## Give permissions to Statsig's service account 1. In your Statsig console, navigate to Project Settings -> Integrations -> BigQuery. Here you will see the Statsig service account. Copy it to be used later. 2. In your BigQuery's [IAM & Admin settings](https://console.cloud.google.com/iam-admin/), add the Statsig service account you copied in step 1 as a new principal for your project, and give it "BigQuery Read Session User" role. BigQuery IAM permissions configuration 3. Navigate to your [BigQuery SQL Workspace](https://console.cloud.google.com/bigquery), choose the dataset, click on "+ Sharing" -> "Permissions" -> "Add Principal" to give the same Statsig service account "BigQuery Data Viewer" role on the dataset. BigQuery dataset permissions setup 4. Back to your Statsig console's BigQuery integration dialog, and enter your BigQuery project and dataset name. Then click "Enable". bq_permission_step_3 Now the service account should have the required permissions to export data from this dataset. ## Insert data for import, and signal when it is ready To load data into statsig, you will load data into `statsig_user_metrics` and then mark a day as completed in `statsig_user_metrics_signal` once all of the data for that day is loaded. Your data should conform to these definitions and rules to avoid errors or delays: | Column | Description | Rules | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | unit\_id | The unique user identifier this metric is for. This might not necessarily be a user\_id - it could be a custom\_id of some kind | | | id\_type | The id\_type the unit\_id represents. | Must be a valid id\_type. The default Statsig types are user\_id/stable\_id, but you may have generated custom id\_types | | date | Date of the daily metric | | | timeuuid | A unique timeuuid for the event | This should be a timeuuid, but using a unique id will suffice. If not provided, the table defaults to generating a UUID. | | metric\_name | The name of the metric | Not null. Length \< 128 characters | | metric\_value | A numeric value for the metric | Metric value, or both of numerator/denominator need to be provided for Statsig to process the metric. See details below | | numerator | Numerator for metric calculation | See above, and details below | | denominator | Denominator for metric calculation | See above, and details below | Metric ingestion is for user-day metric pairs. This is useful for measuring experimental results on complex business logic (e.g. LTV estimates) that you generate in your data warehouse. ##### Note on metric values If you provide **both** a numerator and denominator value for any record of a metric, we'll assume that this metric is a ratio metric; we'll filter out users who do not have a denominator value from analysis, and recalculate the metric value ourselves via the numerator and denominator fields. If you only provide a metric\_value, we'll use the metric\_value for analysis. In this case, we'll impute 0 for users without a metric value in experiments. #### Scheduling Because you may be streaming events to your tables or have multiple ETLs pointing to your metrics table, Statsig relies on you signalling that your metric/events for a given day are done. When a day is fully loaded, insert that date as a row in the appropriate signal\_table - `statsig_user_metrics_signal` for metrics or `statsig_events_signal` for events. For example, once all of your metrics data is loaded into `statsig_user_metrics` for `2022-04-01`, you would insert `2022-04-01` into `statsig_user_metrics_signal`. Statsig expects you to load data in order. For example, if you have loaded up to `2022-04-01` and signal that `2022-04-03` has landed, we will wait for you to signal that `2022-04-02` has landed, and load that data before we ingest data from `2022-04-03` This ingestion pipeline is in beta, and does not currently support automatic backfills or updates to data once it's been ingested. Only signal these tables are loaded after you've run data quality checks! #### Checklist These are common errors we've run into - please go through and make sure your setup is looking good! * The `id_type` is set correctly * Default types are `user_id` or `stable_id`. If you have custom ids, make sure that the capitalization and spelling matches as these are case sensitive (you can find your custom ID types by going to your Project Settings in the Statsig console). * Your ids match the format of ids logged from SDKs * In some cases, your data warehouse may transform IDs. This may mean we can't join your experiment or feature gate data to your metrics to calculate pulse or other reports. You can go to the Metrics page of your project and view the log stream to check the format of the ids being sent (either `User ID`, or a custom ID in `User Properties`) to confirm they match If your data is not showing up in the Statsig console * Monitoring is limited today, but you should be able to check your snowflake query history for the Statsig user to understand which data is being pulled, and if queries are not executing (no history) or are failing. * You should see polling queries within a few hours of setting up your integration. * If you have a signal date in the last 28d, you should see a select statement for data from the earliest signal date in that window * If that query fails, try running it yourself to understand if there is a schema issue * If data is loading, it's likely we're just processing. For new metrics, give it a day to catch. If data isn't loaded after a day or two, please check in with us. The most common reason for metrics catalog failures is due to id\_type mismatches. ## Next Steps Now that everything is set up, your data should start flowing into Statsig's metrics tab and experiment results, and should appear on your console by around noon the next day (PST). This may take longer if you choose an early first date, as we'll need to sequentially load your historical data. Please shoot us a message on [Slack](https://www.statsig.com/slack) if you have any questions. We can also help making sure everything is set up correctly for you. # Imports Overview (Deprecated) Source: https://docs.statsig.com/integrations/data-imports/overview Overview of data import options for Statsig, including ingestion from data warehouses, object storage, and customer data platforms via connectors. This solution is still functional, but can be manual and time consuming to set up with minimal error handling. We encourage you to check out the [Data Warehouse Ingestion](/data-warehouse-ingestion/introduction) solution instead. ## Overview Statsig has a number of ways by which you can send your own events or pre-computed metrics. We support some native integrations to read directly from your data warehouse, or you can choose to write your data to Azure storage owned by Statsig. ## How Metric Imports Work Some specifics may vary based on your approach, but in general you will write your metrics data at a user-day-metric granularity to a table (or API endpoint) with a special schema. You'll write dates as rows to a special signal table (or API endpoint) when you've finished adding metrics for that day. Statsig will poll this signal table hourly. Once you've marked your first day of data as ready, or marked the next day of your data as ready, Statsig will pull data from your table and then process it into metrics and experiment results. Some notes: * New imported metrics may not show up in your catalog until the following morning * The earliest signal date we'll consider for your first day of imported metrics is 4 weeks ago * Statsig doesn't show metrics that haven't had a value for 2 weeks in the Metrics Catalog today. This means that you should start your imports on a recent date for them to show up. * Statsig currently imports dates ordinally without gaps. This means if you update the signal table for `2022-06-13` and `2022-06-15`, we won't load the data for `2022-06-15` until you mark `2022-06-14` and we finish ingesting that data. ## How Event Imports Work Event imports behave similarly to Metric imports, with a schematized data source you own and triggers for ingestion. One key distinction is that Statsig will events pull data continuously to avoid large batch loads timing out. ## Support Please reach out in slack if you're having any issues with imports or have feature requests. # Redshift (Deprecated) Source: https://docs.statsig.com/integrations/data-imports/redshift Import event and metric data into Statsig from Amazon Redshift on a schedule, including authentication, queries, and column-to-event mappings. This solution is still functional, but can be manual and time consuming to set up with minimal error handling. We encourage you to check out the [Data Warehouse Ingestion](/data-warehouse-ingestion/introduction) solution instead. ## Overview There are 2 ways to integrate with Redshift: using a data connector, or ingesting events and metrics to Statsig through S3. ## Using a Data Connector To **ingest** events from Redshift, you can use our [Census integration](/integrations/data-connectors/census). To **export** events to Redshift, you can use our [Fivetran integration](/integrations/data-connectors/fivetran). ## Direct Ingestion S3 imports are currently a custom setup flow. You'll need to reach out to Statsig through Slack or through your support PoC in order to set up this integration. The documentation below describes the steps to set up this integration. There are 3 main steps: 1. Create a pipeline to write your metric, event, and (optionally) signal data to an S3 bucket in parquet format 2. Create an IAM user with read and list access on that bucket and send that user's Key/Secret to Statsig. We will securely store these in a keystore service 3. Schedule ingestion through a `signals` dataset or through the `mark_data_ready` API ### Set up a data pipeline to S3 #### Filesystem Format We will expect data in your S3 bucket to be saved in parquet format. To allow for daily uploads, please set up your bucket with the following folders: * `events/` for events data * `metrics/` for metrics data * `signals/` for signal flags when you've finished uploading data for a day. You can omit this folder and instead use the [`mark_data_ready` API](/metrics/ingest) instead, but you must use one or the other We recommend writing folders by date partitions for ease of debugging, i.e. storing day's data in folders with ISO-formatted names (`YYYY-MM-DD`). #### Data Format Please make sure your data conforms to the following schemas. Events ``` | Column | Description | Rules | | -------------- | ----------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- | | timestamp | UNIX timestamp of the event | UTC timestamp | | event_name | The name of the event | String under 128 characters, using `_` for spaces | | event_value | A string representing the value of a current event. Can represent a 'dimension' or a 'value' | Read as string format; numeric values will be converted into value | | event_metadata | A dictionary in the form of a JSON string, containing named metadata for the event | String format | | user | A JSON object representing the user this event was logged for; see below | Escaped JSON string including the keys 'custom' and 'customIDs'. A userID or customID must be provided. | | timeuuid | A unique UUID or timeUUID used for deduping. If omitted, will be generated but will not be effective for deduping | UUID format | ``` Please refer to docs for the [Statsig User Object](/concepts/user#user-attributes) for available fields. An example would look like: ``` { userID: "12345", customIDs: { stableID: "", ... } email: "12345@gmail.com", userAgent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.40 Safari/537.36", ip: "192.168.1.101", country: "US", locale: "en_US", appVersion: "1.0.1", systemName: "Android", systemVersion: "15.4", browserName: "Chrome", browserVersion: "45.0", custom: { new_user: "false", age: "22" ... }, } ``` Metrics Make sure to include all of metric\_value, numerator, and denominator, writing `cast(null as double)` for numerator and denominator if you are omitting them (or conversely for metric\_value if sending numerator/denominator). | Column | Description | Rules | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | unit\_id | The unique user identifier this metric is for. This might not necessarily be a user\_id - it could be a custom\_id of some kind | String format. Make sure this is in the same format as your logged unit\_ids | | id\_type | The id\_type the unit\_id represents. | String format. Must be a valid id\_type. The default Statsig types are user\_id/stable\_id, but you may have generated custom id\_types. Make sure this matches (case sensitive) a customID in your project, or you won't get experiment results | | date | Date of the daily metric | Read as string format; can be written as ISO date. Statsig's dates are calculated in PST - we'll load custom metrics to whatever date you use here | | metric\_name | The name of the metric | String format. Not null. Length \< 128 characters | | metric\_value | A numeric value for the metric | Double format. Metric value, or both of numerator/denominator need to be provided for Statsig to process the metric. See details below | | numerator | Numerator for metric calculation | Double format. Required for ratio metrics. If present along with a denominator in any record, the metric will be treated as ratio and only calculated for users with non-null denominators | | denominator | Denominator for metric calculation | Double format. See above | ### Set up and Provide Credentials * Navigate to your IAM console on AWS * Go to Users->Add User * Select the `Access key - Programmatic access` credential type * Attach an appropriate policy which gives Read and List access to the appropriate bucket. Make sure this is scoped appropriately so the user only has access to the data intended! Example policy: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": "" } ] } ``` Next, modify your bucket access policy (under `permissions` on your S3 bucket's page) to allows this user to access objects. Example policy: ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "statement1", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "s3:ListBucket", "Resource": "" }, { "Sid": "statement2", "Effect": "Allow", "Principal": { "AWS": "" }, "Action": "s3:GetObject", "Resource": "/*" } ] } ``` You can confirm your credentials are sufficient by adding any data to your metrics folder and running the following code in PySpark with the IAM user credentials: ``` sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", '') sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", '') spark.read.parquet("s3:///metrics/*",inferSchema=True).show() ``` ### Scheduling Because you may be streaming events to your tables or have multiple ETLs pointing to your metrics table, Statsig relies on you signalling that your metric/events for a given day are done. To do this, write a dataset with the single column `finished_date`, which contains all dates of data which have been written to Statsig. For example, once you have written data for `2022-06-22` you would insert a record with `finished_date` of `2022-06-22` to trigger ingestion of data from up to and including `2022-06-22`. Unlike some other integrations like Snowflake, for S3 Statsig will skip dates; if your latest finished date is `2022-06-22` and you insert `2022-07-01`, we will ingest all data as of `2022-07-01` and infer that data for dates between (e.g. `2022-06-25`) is loaded. Alternatively, you can use the `mark_data_ready` API and send a timestamp for which the data previous to that timestamp has finished loading into S3. Note that, for events, Statsig processes days according to PST. When you mark data ready for '2022-06-20', statsig will process events from `2022-06-20T00:00` PST to `2022-06-20T23:59....` PST. Keep this in mind when scheduling your signals! # Snowflake (Deprecated) Source: https://docs.statsig.com/integrations/data-imports/snowflake Import event and metric data into Statsig from Snowflake on a schedule, including authentication, queries, and column-to-event mappings. This solution is still functional, but can be manual and time consuming to set up with minimal error handling. We encourage you to check out the [Data Warehouse Ingestion](/data-warehouse-ingestion/introduction) solution instead. ## Overview There are 2 ways to integrate with Snowflake: using a data connector, or through direct ingestion from Snowflake. ## Using a Data Connector To **ingest** events from Snowflake, you can use our [Census integration](/integrations/data-connectors/census). To **export** events to Snowflake, you can use our [Fivetran integration](/integrations/data-connectors/fivetran). ## Direct ingestion from Snowflake We also support direct data ingestion from Snowflake, through which Statsig will automatically pull data from Snowflake into your events. You will need to do the following steps. Please follow the checklist below to avoid delays! ### 1. Set up your Snowflake data warehouse and user for Statsig integration Insert `USER` and `PASSWORD` values to the below and run it in a Snowflake worksheet on an account which has sysadmin and securityadmin roles. This will create the table schemas and setup that Statsig's ingestion will use. Statsig will use the user you create to access tables in the new Statsig schema. Make sure to use a unique and secure username and password and replace the placeholder values in the first 2 statements. ```sql theme={null} BEGIN; -- set up variable values to be used in statements later -- make sure to configure user_name and user_password with your own values SET user_name = ''; -- REPLACE WITH YOUR OWN VALUE SET user_password = ''; -- REPLACE WITH YOUR OWN VALUE SET role_name = 'STATSIG_ROLE'; -- change role to sysadmin for warehouse / database steps USE ROLE sysadmin; -- create a warehouse, database, schema and tables for Statsig CREATE OR REPLACE WAREHOUSE statsig WITH warehouse_size='XSMALL'; CREATE DATABASE IF NOT EXISTS statsig; CREATE SCHEMA IF NOT EXISTS statsig.statsig; -- a table for ingestion of raw events CREATE TABLE IF NOT EXISTS statsig.statsig.statsig_events( time BIGINT NOT NULL, -- unix time timeuuid STRING NOT NULL DEFAULT UUID_STRING(), --generated unique timeuuid user STRING NOT NULL, --json user object event_name STRING NOT NULL, event_value STRING, event_metadata STRING NOT NULL, --json metadata object event_version BIGINT, record_number NUMBER AUTOINCREMENT START 1 INCREMENT 1 ); -- a table for ingestion of metrics/user outcomes CREATE TABLE IF NOT EXISTS statsig.statsig.statsig_user_metrics( unit_id STRING NOT NULL, id_type STRING NOT NULL, -- stable_id, user_id, etc. date DATE NOT NULL, -- YYYY-MM-DD. Statsig calculates dates according to PST timeuuid STRING NOT NULL DEFAULT UUID_STRING(), metric_name STRING NOT NULL, metric_value NUMBER, numerator NUMBER, denominator NUMBER ); CREATE TABLE IF NOT EXISTS statsig.statsig.statsig_events_signal( finished_date DATE ); CREATE TABLE IF NOT EXISTS statsig.statsig.statsig_user_metrics_signal( finished_date DATE ); -- change current role to securityadmin to create role and user for Statsig's access USE ROLE securityadmin; -- create role for Statsig CREATE ROLE IF NOT EXISTS identifier($role_name); GRANT ROLE identifier($role_name) TO ROLE SYSADMIN; -- create a user for Statsig CREATE USER IF NOT EXISTS identifier($user_name) password = $user_password default_role = $role_name default_warehouse = statsig; GRANT ROLE identifier($role_name) TO USER identifier($user_name); -- grant Statsig role access GRANT USAGE ON WAREHOUSE statsig TO ROLE identifier($role_name); GRANT USAGE ON DATABASE statsig TO ROLE identifier($role_name); GRANT USAGE ON SCHEMA statsig.statsig TO ROLE identifier($role_name); GRANT SELECT ON statsig.statsig.statsig_events TO ROLE identifier($role_name); GRANT SELECT ON statsig.statsig.statsig_user_metrics TO ROLE identifier($role_name); GRANT SELECT ON statsig.statsig.statsig_events_signal TO ROLE identifier($role_name); GRANT SELECT ON statsig.statsig.statsig_user_metrics_signal TO ROLE identifier($role_name); COMMIT; ``` Make sure all the statements ran successfully. This will create the schema and user that Statsig's ingestion expects. ### 2. Provide the credentials to Statsig * Go to [console.statsig.com](https://console.statsig.com/) and log in * Go to the settings page and navigate to the integrations tab. * Find Snowflake in the integrations list and provide the requested credentials for the user you just created in step 1. * You can use the "Test Connection" button to make sure we can establish a connection to the table using the credentials provided. Snowflake integration configuration interface ### 3. Load data into the new Statsig tables In step 1 we created 2 data tables and 2 signal tables. To load data into statsig, you will load data into the data tables, and mark a day as completed in the corresponding signal table once all of the data for that day is loaded. The `statsig_events` table is for sending Statsig raw events which were not logged to Statsig through the API. Once loaded, these events will be processed as though they were logged directly. The `statsig_user_metrics` table is for sending pre-computed metrics from your data warehouse. These metrics will be surfaced in the statsig console and in your test results. You may only need one of these use cases - that's fine! Just follow the steps for the relevant table and ignore the other one. Your data should conform to these definitions and rules to avoid errors or delays: #### Events (statsig\_events) | Column | Description | Rules | | --------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | time | The unix time your event was logged at | Not null | | timeuuid | A unique timeuuid for the event | This should be a timeuuid, but using a unique id will suffice. If not provided, the table defaults to generating a UUID. | | user | A user json object. | See below | | event\_name | The name of the event | Not null. Length \< 128 characters | | event\_value | The value of the event | Length \< 128 characters | | event\_metadata | Metadata about the event | Not null. Length \< 16384 characters. Json-formatted - leave empty if none | | event\_version | The version of this event | | The user object is a stringified JSON representation. An example might look like: ```json theme={null} { "os": "Mac OS X", "os_version": "10.15.7", "browser_name": "Electron", "browser_version": "11.5.0", "ip": "1.1.1.1", "country": "KR", "locale": "en-US", "userID": "bbbbb-bbbbb-bbbbb-bbbbb", "custom": { "locale": "en-US", "clientVersion": "23.10.23.10", "desktopVersion": "11.5.0" }, "customIDs": { "deviceId": "ddddd-ddddd-ddddd-ddddd", "stableID": "sssss-sssss-sssss-sssss" } } ``` Key components of the user object are the `userID`, `custom` fields, and the `customIDs` object (notably `stableID`) if you are using any custom identifiers. Make sure these fields are provided where they exist, and that the names of the fields capitalized correctly. Not providing a unit identifier will limit the utility of your events, as we won't be able to use them to build metrics like daily event users. #### Metrics (statsig\_user\_metrics) | Column | Description | Rules | | ------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | | unit\_id | The unique user identifier this metric is for. This might not necessarily be a user\_id - it could be a custom\_id of some kind | | | id\_type | The id\_type the unit\_id represents. | Must be a valid id\_type. The default Statsig types are user\_id/stable\_id, but you may have generated custom id\_types | | date | Date of the daily metric | | | timeuuid | A unique timeuuid for the event | This should be a timeuuid, but using a unique id will suffice. If not provided, the table defaults to generating a UUID. | | metric\_name | The name of the metric | Not null. Length \< 128 characters | | metric\_value | A numeric value for the metric | Metric value, or both of numerator/denominator need to be provided for Statsig to process the metric. See details below | | numerator | Numerator for metric calculation | See above, and details below | | denominator | Denominator for metric calculation | See above, and details below | Metric ingestion is for user-day metric pairs. This is useful for measuring experimental results on complex business logic (e.g. LTV estimates) that you generate in your data warehouse. ##### Note on metric values If you provide **both** a numerator and denominator value for any record of a metric, we'll assume that this metric is a ratio metric; we'll filter out users who do not have a denominator value from analysis, and recalculate the metric value ourselves via the numerator and denominator fields. If you only provide a metric\_value, we'll use the metric\_value for analysis. In this case, we'll impute 0 for users without a metric value in experiments. #### Scheduling Because you may be streaming events to your tables or have multiple ETLs pointing to your metrics table, Statsig relies on you signalling that your metric/events for a given day are done. When a day is fully loaded, insert that date as a row in the appropriate signal\_table - `statsig_user_metrics_signal` for metrics or `statsig_events_signal` for events. For example, once all of your metrics data is loaded into `statsig_user_metrics` for `2022-04-01`, you would insert `2022-04-01` into `statsig_user_metrics_signal`. Statsig expects you to load data in order. For example, if you have loaded up to `2022-04-01` and signal that `2022-04-03` has landed, we will wait for you to signal that `2022-04-02` has landed, and load that data before we ingest data from `2022-04-03` This ingestion pipeline is in beta, and does not currently support automatic backfills or updates to data once it's been ingested. Only signal these tables are loaded after you've run data quality checks! #### Checklist These are common errors we've run into - please go through and make sure your setup is looking good! * Field names are set incorrectly * Some of the field names here may intersect with protected key words. You should be able to generate these by wrapping them in quotes if needed. The above SQL works in the snowflake console. * Run `SELECT *` on your tables when done to confirm that there aren't any special characters in the column names! This will cause our ingestion to fail. * The `id_type` is set correctly * Default types are `user_id` or `stable_id`. If you have custom ids, make sure that the capitalization and spelling matches as these are case sensitive (you can find your custom ID types by going to your Project Settings in the Statsig console). * Your ids match the format of ids logged from SDKs * In some cases, your data warehouse may transform IDs. This may mean we can't join your experiment or feature gate data to your metrics to calculate pulse or other reports. You can go to the Metrics page of your project and view the log stream to check the format of the ids being sent (either `User ID`, or a custom ID in `User Properties`) to confirm they match If your data is not showing up in the Statsig console * Monitoring is limited today, but you should be able to check your snowflake query history for the Statsig user to understand which data is being pulled, and if queries are not executing (no history) or are failing. * You should see polling queries within a few hours of setting up your integration. * If you have a signal date in the last 28d, you should see a select statement for data from the earliest signal date in that window * If that query fails, try running it yourself to understand if there is a schema issue * If data is loading, it's likely we're just processing. For new metrics, give it a day to catch. If data isn't loaded after a day or two, please check in with us. The most common reason for metrics catalog failures is due to id\_type mismatches. # Datadog Source: https://docs.statsig.com/integrations/datadog Connect Datadog with Statsig to forward Datadog events and metrics to Statsig for experiment analysis and to push Statsig events back into Datadog. ### Overview There are four key use-cases to the Datadog integration: 1. [Config Changes](#config-changes) - Streaming changes made in Statsig into Datadog, so you can see what feature was turned on that may have caused a CPU usage spike or some other degradation of performance (most widely-used integration). 2. [Event Forwarding](#events) - Statsig will forward event-count totals to DataDog, purely for the purpose of monitoring your Statsig usage volumes. 3. [Datadog RUM integration](#datadog-rum-integration) - This allows you to enrich DataDog RUM data with flag/experiment assignment info, allowing customer to correlate product feature changes with their impact on system/performance metrics. 4. [DataDog triggers](/integrations/triggers/datadog) - When an alarm goes off in DataDog, it kills a Statsig feature gate. ### Connecting to Datadog 1. To create a Datadog API key, navigate to **Organization Settings** > **API Keys**. If you have the permission to create API keys, click **New Key**. Datadog API key creation interface 2. Paste the API key in the text box at the top of the integration dialog, then hit "Confirm". If the above is out of date, refer to the [Datadog documentation](https://docs.datadoghq.com/account_management/api-app-keys/#add-an-api-key-or-client-token) on how to setup API Keys ### Config Changes This integration will send Datadog Events of your choice when your project's settings change. For instance, we will send an Event when someone edits a Feature Gate. These events can be found in the Datadog Events Explorer. Datadog Events Explorer interface ### Event Totals Forwarding This integration will forward the number of Statsig SDK Events reported to Datadog. This is meant for monitoring your project's Statsig usage. This integration also has the option to allow non-production events to be forwarded to Datadog. Statsig events are mapped to Datadog metrics with listed tags as follows: * statsig::gate\_exposure -> statsig.check\_gate.count * environment * name * value * statsig::config\_exposure -> statsig.get\_config.count * environment * statsig::experiment\_exposure -> statsig.get\_experiment.count * environment * group * name * statsig::layer\_exposure -> statsig.get\_layer.count * environment * name * statsig::holdout\_exposure -> statsig.get\_holdout.count * environment * name * value #### Example of check\_gate metric Datadog check_gate metric visualization ### Datadog RUM integration This integration requires a client-side setup as outlined [here in DataDog documentation](https://docs.datadoghq.com/integrations/statsig-rum/). # Event Filtering Source: https://docs.statsig.com/integrations/event_filtering Configure event filtering in Statsig to control which events are ingested, dropped, or transformed before they reach metrics and experiments. Once you've enabled an integration, you can select specific events that you want to send and/or receive by clicking on the **Event Filtering** button. The steps outline below are for the Segment integration, but the steps to enable **Event Filtering** are the same across integrations. ## Incoming Event Filtering If your integration includes "Incoming" events, you can filter which ones Statsig should care about. All others will not be logged. * Visit the [integrations](https://console.statsig.com/integrations) page on console.statsig.com. * Select the integration you wish to filter events for. * In the dialog that appears, select "Event Filtering" Event filtering dialog interface * You can now search for specific events and select or deselect the events that you want Statsig to ingest. Event selection interface for incoming events ## Outgoing Event Filtering If your integration includes "Outgoing" events, you can select which ones being sent to Statsig should then be forwarded to the integration. * Visit the [integrations](https://console.statsig.com/integrations) page on console.statsig.com. * Select the integration you wish to filter events for. * In the dialog that appears, select "Event Filtering" Outgoing event filtering configuration # Event Webhook Source: https://docs.statsig.com/integrations/event_webhook Configure event webhooks in Statsig to forward exposure, gate, and experiment events to your own HTTP endpoints in near real time. ## Incoming The Statsig Event Webhook allows you to log event data to Statsig from third party apps or other external sources to provide additional insights to your Statsig experiments and metrics. Before using the Webhook, you will need to obtain your Projects' server secret key. An example call to the Statsig Event Webhook should look like the following: ```bash title="HTTP" theme={null} POST https://api.statsig.com/v1/webhooks/event_webhook ``` ```bash title="Headers" theme={null} Content-Type: application/json Accept: */* STATSIG-API-KEY: {STATSIG_SERVER_SECRET} ``` ```bash title="JSON Body" theme={null} { "user": { "userID": {USER_ID}, ... }, "event": {EVENT_NAME}, "value": {VALUE}, "metadata": { "example_field_1": {EXAMPLE_VALUE_1}, "example_field_2": {EXAMPLE_VALUE_2}, ... }, timestamp: {TIMESTAMP} } ```
    *** ## Outgoing
    Statsig Webhook integration card with Enable button
    If you are using a service that we don't have an official integration for, you can always use our Generic Webhook integration. This integration just sends raw events to the provided webhook url. ### Setup In your Project Settings, under the Integrations tab. Enable the Generic Webhook integration. In the dialog that appears, enter the url of your destination webhook and then hit Enable to save the url and enable this integration. integration-dialog You can then set which events you want forwarded to your webhook using the [Event Filtering](#filtering-events) dialog. ### Runtime event webhooks These webhooks are triggered at runtime as users are being assigned to gates, experiments and are triggering events. ### Event Format Events will be sent in batches in a JSON format. The structure of a Statsig Event sent will look like the following: | Field | Type | Description | | --------------- | ------ | ---------------------------------------------------------------- | | eventName | String | Name of the event provided | | user | JSON | [Statsig User Object](/concepts/user) | | userID | String | User ID provided | | timestamp | Number | Timestamp in MS of the event | | value | String | Value of the event provided | | metadata | JSON | Custom Metadata provided | | statsigMetadata | JSON | Metadata related to the logging of this event added by Statsig | | timeUUID | String | UUID for the event | | unitID | String | Unit ID of the exposure (e.g. userID, stableID, or the customID) | #### Custom Event Formatting - logEvent > ```json theme={null} { "eventName": "my_custom_event", "user": { "userID": "a_user", "email": "a.user@email.com" }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "a_custom_value", "metadata": { "key_a": "value_a", "key_b": "123" }, "timeUUID": "abd2a983-ec0f-11ec-917a-fb8cdaeda578" } ``` #### Feature Gate Exposure Formatting - checkGate > ```json theme={null} { "eventName": "statsig::gate_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "", "metadata": { "gate": "a_gate", "gateValue": "false", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "timeUUID": "8d7c1040-ec11-11ec-g123-abe2c32fcf46", "unitID": "userID" } ``` #### Dynamic Config Exposure Formatting - getConfig > ```json theme={null} { "eventName": "statsig::config_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655231253265", "statsigMetadata": { ... }, "value": "", "metadata": { "config": "a_config", "ruleID": "default", "reason": "Network", "time": "1655231249644" }, "timeUUID": "af379f60-ec11-22ad-8e0a-05c3ee70bd0c", "unitID": "userID" } ``` #### Experiment Exposure Formatting - getExperiment > ```json theme={null} { "eventName": "statsig::experiment_exposure", "user": { ... }, "userID": "a_user", "timestamp": "1655232119734", "statsigMetadata": { ... }, "value": "", "metadata": { "config": "an_experiment", "ruleID": "4SauZJcM1T7zNvh1igBjwE", "reason": "Network", "time": "1655231249644", "experimentGroupName": "Control" }, "timeUUID": "af379f61-ab22-11ec-8e0a-05c3ee70bd0c", "unitID": "userID" } ``` #### Example Batch > ```json theme={null} [ { "eventName": "page_view", "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566967, "value": "example_value", "metadata": {"page": "home_page"}, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120002" }, { "eventName": "statsig::gate_exposure", "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566968, "value": "", "metadata": {"gate": "test_gate", "gateValue": "true", "ruleID": "default"}, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120003", "unitID": "userID" }, { "eventName": "statsig::experiment_exposure" "user": {"userID": "user_1", "country": "US"}, "userID": "user_1", "timestamp": 1644520566969, "value": "", "metadata": { "config": "an_experiment", "ruleID": "4SauZJcM1T7zNvh1igBjwE", "reason": "Network", "time": "1655231249644", "experimentGroupName": "Control" }, "statsigMetadata": {}, "timeUUID": "f4c414a0-8ab5-11ec-a8a3-0242ac120004", "unitID": "userID" } ] ```
    ### Config Change webhooks These webhooks are triggered by configuration changes taking place within Console. Each webhook request body will contain a batch of change events in the following format: `{"data": [, ]}`. Batches may contain 1 or more config change events. Below are a few examples of some of the config change payloads. To best capture the exhaustive types of config webhooks and their payloads, it's recommended to run a service such as ngrok or some other service that will help you log incoming webhooks. #### Gate Change ```json theme={null} { "user": { "name": "Test User", "email": "testuser@email.com" }, "timestamp": 1709660061095, "eventName": "statsig::config_change", "metadata": { "type": "Gate", "name": "layout_v2", "description": "Description: Change default page layout", "action": "created" } } ``` #### Experiment Change ```json theme={null} { "user": { "name": "Test User", "email": "testuser@email.com" }, "timestamp": 1709658507446, "eventName": "statsig::config_change", "metadata": { "type": "Experiment", "name": "heading_test", "description": "- Updated experiment settings\n - Groups updated: control, test\n - Parameters added: heading\n - Parameters updated: directives", "action": "updated" } } ``` ### Filtering Events Once you've enabled outbound events to your webhook, you can select which categories of Statsig events you want to export by clicking on the **Event Filtering** button and checking the appropriate boxes as shown below. There are 2 main types of events: *Exposures* (e.g. events logged via the SDK) and *Config Changes* (changelogs for Statsig Console) event-filter-button event-filter-dialog ### Webhook Signature You can verify that a webhook request is coming from Statsig via our Webhook Signature. Follow the following steps to verify the signature: 1. Grab your webhook signing secret from your Webhook integration card Webhook integration card showing signing secret 2. Extract the request time header 'X-Statsig-Request-Timestamp' from the webhook request. 3. Concatenate the version number ("v0"), the timestamp, and the request body together, using a colon (:) as a delimiter to create a signature basestring. Here's an example of a possible base string: ``` v0:1671672194836:{"data":[{"user":{"name":"Joe Zeng","email":"joe@statsig.com"},"timestamp":1671672134833,"eventName":"statsig::config_change","metadata":{"type":"Gate","name":"test","description":"- Updated Rule test rollout from 100.00% to 10.00%","environment":"production"}}]} ``` 4. Hash the signature basestring, using the signing secret as a key, and take the hex digest of the hash. Create the full signature by prefixing the hex digest with the version number ("v0") and an equals sign. See sample pseudo code below. ``` statsig_signature = 'v0=' + hmac.compute_hash_sha256( webhook_signing_secret, signature_basestring ).hexdigest() >>> 'v0=05c50d1513d49f884df8b0469befbbd432bd30364e81f16a606dec69f29e8f18' ``` 5. Compare the resulting signature to the 'X-Statsig-Signature' header on the request ### Developing and Testing Webhooks The actual event payload may look different than the examples above.
    To test webhook configuration and see payloads before starting development, you can use a local HTTP tunnel (e.g. ngrok).
    You can also use the **Debug** tool to 1. See requests made to the webhook, including diagnostic information such as number of events forwarded/filtered, request header & body, and more. 2. Send example requests to the webhook using any recently logged event or exposure Debug Tool # Fastly Source: https://docs.statsig.com/integrations/fastly Run Statsig feature gate and experiment evaluations at the edge with Fastly Compute@Edge for low-latency rules evaluation in your CDN layer. Statsig offers a suite of integration tools that make usage with Fastly easy: * Statsig automatically pushes project changes to Fastly KV/Config Store, providing low-latency SDK startup * Statsig offers a Fastly helper that handles client initialization and event flushing, so you can focus on your business logic. #### Configure Integration First, enable the Fastly integration in the Statsig Console. Navigate to [Project Settings -> Integrations](https://console.statsig.com/integrations), and then select Fastly You will need to input the following: * **Fastly API Key** - Can be found in Fastly portal under **Account** -> **API Tokens**. * Create an **Automation Token** with: * **global** and **global:read** scope * **Store Type** - Select **"Config Store"** or **"KV Store"** depending on your storage type. * **Config Store ID** OR **KV Store ID**- Your Store ID There is also an option to filter the configs that are synced into your KV namespace by a Target App. You may wish to enable this in the future as the size of your config payload grows. For now, you can leave this unchecked. After filling this out, click **Enable**. Within a minute, the Statsig backend will generate a config payload from your statsig project and push it into your store. Under your Config or KV Store, look for a key starting with the prefix `statsig-`. This is the `key` associated with your Statsig config specs in your Store. Note this key down as it will be required later. #### Install the Statsig SDK Install the Statsig serverless SDK: ```bash theme={null} npm install @statsig/serverless-client ``` #### Import the statsig SDK Import the Statsig Helper: ```javascript theme={null} import { handleWithStatsig} from "@statsig/serverless-client/fastly"; ``` #### Use the SDK ```javascript theme={null} handleWithStatsig(handler, params) ``` The helper method takes two arguments: * `handler` This is your Fastly Compute code. * `params : StatsigFastlyHandlerParams` | Parameter | Optional | Type | Description | | ----------------- | -------- | ---------------- | --------------------------------------------------------------------- | | `statsigSdkKey` | No | `string` | Your Statsig client API key | | `fastlyStoreType` | No | `string` | Either `kv` or `config`, signifying your Fastly store type | | `storeId` | No | `string` | Your KV or Config store id | | `keyId` | No | `string` | They key storing your Statsig config specs in your Fastly store | | `apiToken` | No | `string` | Your Fastly API token used to authenticate requests to the Fastly API | | `statsigOptions` | Yes | `StatsigOptions` | See StatsigOptions [here](/client/javascript-sdk#statsig-options) | For best practice: store `statsigSdkKey` and `apiToken` in a Fastly [Secret Store](https://www.fastly.com/documentation/guides/compute/edge-data-storage/working-with-secret-stores/) ### Example Usage ```javascript index.js theme={null} import { handleWithStatsig} from "@statsig/serverless-client/fastly"; async function myHandler(event, client) { const user = { userID: Math.random().toString().substring(2, 5) }; const res = client.checkGate("pass_gate", user); client.logEvent("pass_gate", user); return new Response( JSON.stringify({ res, user }), { status: 200, headers: { 'Content-Type': 'application/json' } } ); } const handleRequest = handleWithStatsig(myHandler,{ statsigSdkKey: "client-LhxVWHSeZt2uor***********", fastlyStoreType: "kv", storeId:"7b12fn*********", keyId:"statsig-3htllY8XxFsJ*****", apiToken:"7NaRxS6R**********" }) addEventListener('fetch', (event) => event.respondWith(handleRequest(event))); ``` **That's it!** The helper automatically: * Initializes the Statsig Client with config specs from your KV or Config store * Executes your handler code (Your business logic + Statsig usage) * Flushes all events after your handler completes execution * Cleans up resources ### Advanced Usage **Use the advanced/manual setup if:** * You need fine-grained control over initialization timing * You need fine-grained control over event flushing timing * You need to customize error handling behavior ### Prerequisites 1. Completed the [Statsig Fastly integration setup](#configure-integration) #### Install the Statsig SDK Install the Statsig serverless SDK: ```bash theme={null} npm install @statsig/serverless-client ``` #### Import the statsig SDK Import the Fastly client: ```javascript theme={null} import { StatsigFastlyClient} from "@statsig/serverless-client/fastly"; ``` #### Creating a `StatsigFastlyClient` instance ```javascript theme={null} const client = new StatsigFastlyClient(""); ``` The client instantiation takes two arguments: * `sdkKey : string` This is your Statsig client API key. It is available from the [Project Settings](https://console.statsig.com/api_keys) page in the Statsig Console. This is used to authenticate your requests. * `options : StatsigOptions` See here, for more [options](/client/javascript-sdk#statsig-options). For best practice: store `sdkKey` in a Fastly [Secret Store](https://www.fastly.com/documentation/guides/compute/edge-data-storage/working-with-secret-stores/) ### Client initialization The following line initializes the client by loading feature gate and experiment configurations directly from your Fastly KV or Config store. ```javascript theme={null} const initResult = await client.initializeFromFastly(, , , ); ``` The client initialization takes four arguments: * `fastlyStoreType : string` This is the Fastly store type you are using represented by `kv` or `config` * `storeId : string` The id of your Fastly store * `keyId : string` This is they key storing the Statsig config specs in your store * `apiToken : string` Your Fastly API Token For best practice: store `apiToken` in a Fastly [Secret Store](https://www.fastly.com/documentation/guides/compute/edge-data-storage/working-with-secret-stores/) #### Checking a Gate ```javascript theme={null} const value = client.checkGate("pass_gate", user); ``` This is a gate check in code. The `checkGate` method takes two arguments: * `name : string` The name of the Statsig gate that you are checking. * `user : StatsigUser` The Statsig user object for whom the gate is being checked. For more information on the user object, see [here](/sdks/user#introduction-to-the-statsiguser-object). Refer to the [Javascript on device evaluation sdk documentation](/client/jsOnDeviceEvaluationSDK) for how to check other entities like experiments and dynamic configs. #### Logging an event ```javascript theme={null} client.logEvent('fastly_gate_check', user, value.toString()); ``` This is an event log in code. The `logEvent` method takes two parameters: * `eventOrName : string | StatsigEvent` This is the name and details of the event you are logging. * `user : StatsigUser` The Statsig user object for whom the event is being logged. * `value : string` A value you would like to associate with this event For more information on event logging, see [here](/client/jsOnDeviceEvaluationSDK#logging-an-event). #### Flushing Events ```javascript theme={null} event.waitUntil(client.flush()); ``` This flushes all events from the sdk to Statsig. **Without this, you won't be able to get diagnostic information in the Statsig Console, nor any event data you logged**. #### Putting it all together ```javascript theme={null} import { StatsigFastlyClient } from "@statsig/serverless-client/fastly"; addEventListener("fetch", (event) => event.respondWith(handleRequest(event))); async function handleRequest(event) { const client = new StatsigFastlyClient("client-LhxVWHSeZt2uor********"); const initResult = await client.initialzeFromFastly( "kv", "7b12fnfm7po7*********", "statsig-3htllY8X**********", "7NaRxS6RMGE-DTp*******" ); const user = { userID: Math.random().toString().substring(2, 5) }; const value = client.checkGate("pass_gate", user); client.logEvent("fastly_gate_check", user, value.toString()); event.waitUntil(client.flush()); return new Response(JSON.stringify({ kv, user }), { status: 200, headers: { "Content-Type": "application/json" }, }); } ``` ## Other Considerations ### Polling for updates v5.13.0+ The SDK cannot poll for updates across requests since Fastly does not allow for timers.To optimize for edge use cases, we do not provide an api to recognize updates to your config specs. However, when a change is made to your project definition on the Statsig console, the changes will be propagates to the KV/Config store and will be reflected the next time you initialize the StatsigFastly client. ### Flushing events v4.16.0+ The SDK enqueues logged events and flushes them in batches. In order to ensure events are properly flushed, we recommend calling flush using `event.waitUntil()`. This will keep the request handler alive until events are flushed without blocking the response. ``` event.waitUntil(statsig.flush()); ``` If you want to check on the evaluations you are getting, you can go to the gate you created for this example and look at the evaluations in the Diagnostics tab. If you want to check the events you logged, in the **Statsig Console**, go to **Data** -> **Events** And there you have it - a working Fastly integration for Statsig. ### Size Limits Fastly Config Store has maximum size limits that may prevent Statsig from pushing configs into Fastly. See [here](https://docs.fastly.com/products/edge-data-storage) for the latest Config Store limits. If your payload continues to grow, you will need to set the option to filter the payload by a Target App in the integration settings. ### Unsupported Features Statsig ID Lists are not currently synced into Fastly KVs or Config Stores. If you rely on large (>1000) ID lists, you will not be able to check them in your Fastly compute services. # Github AI Integration Source: https://docs.statsig.com/integrations/github-ai-integration Integrate Statsig with GitHub AI tools to keep feature gates and experiments visible to AI code reviewers and automate flag cleanup PRs. ## Overview This feature is in open beta and might have rough edges currently. If you have feedback, please share with us! The Statsig GitHub integration links your organization's code repository directly to your Statsig project. This allows Statsig to analyze your code base and create a private Knowledge Graph that maps relationships between your codebase and Statsig services such as Feature Gates, Experiments, and Metrics. This integration connects your software development with measurable business impact, delivering contextual insights within Statsig. Knowledge Graph diagram showing relationships between codebase and Statsig services ### Key Features Integrating your code repository unlocks the following capabilities: * Locate specific lines of code where a Feature Gate or Experiment is implemented * Automatically generate PRs (Pull Requests) to remove stale flags from your code An example of Remove from Code in a stale gate banner * Understand the intent behind metrics and experiments based on code implementation An example of AI description in the right rail of a config * Write natural language queries to create charts to answer business questions (Coming Soon) ## Why Connect Your Codebase In modern product development, there is often a disconnect between the code you write (hosted on GitHub) and its impact and business outcomes (data in Statsig). The Knowledge Graph bridges this divide by analyzing your connected repositories to provide the context behind every feature. * Context: It knows *when* an event is logged or a feature is exposed to a user * Intent: It understands what the *purpose and meaning* of that event and feature * Freshness: It ensures insights in Statsig reflect the current state of your main branch ## Installation Guide Before you begin, ensure you have Admin permissions in both your Statsig Project and the target Github Organization. ### Pre-requisites If your organization uses IP restrictions to limit access to your GitHub resources, you must allowlist the IP addresses that Statsig services use to connect to GitHub. Please perform the steps outlined in this [Github doc](https://docs.github.com/en/enterprise-cloud@latest/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/managing-allowed-ip-addresses-for-your-organization) to allowlist IP addresses for your Organization. Configuring IP addresses in your Github Org If you choose to manually add IP addresses, you can find Statsig's IP ranges [here](/infrastructure/statsig_ip_ranges#statsig-ip-ranges). ### First-time Installation If you are connecting GitHub to Statsig for the first time: 1. Navigate to **Settings** → **Integrations** in the Statsig console 2. Locate and select the **Statsig GitHub App** card 3. Click **Enable Connection** 4. You will be redirected to GitHub. Authenticate and select the target Organization and Repositories. 5. Once authorized, you will be redirected back to Statsig to confirm the connection You can choose to install the app on specific repositories or your entire organization. For the best results with Knowledge Graph, we recommend selecting all relevant repositories containing feature code. ### Connecting Additional Projects GitHub allows a specific GitHub App to be installed only *once* per GitHub Organization. If you have multiple Statsig Projects (e.g., "Project 1" and "Project 2") that need access to the same GitHub Org, they will share the underlying connection. To link a new project to an existing GitHub connection: 1. Navigate to **Settings** → **Integrations** in the Statsig console 2. Locate and select the **Statsig GitHub App** card. We will indicate if an existing connection exists for any other project. 3. Select the Statsig Project you wish to inherit the connection from 4. Confirm the link ### Migrating from Legacy (PAT) Integration If you are using the older Personal Access Token (PAT) integration, we recommend upgrading to the GitHub App for improved security and functionality. 1. Navigate to **Settings** → **Integrations** in the Statsig console 2. Locate and select the **Statsig GitHub App** card 3. You will see a banner or indicator on the GitHub integration card prompting you to upgrade from PAT based connection 4. Follow the flow to install the GitHub App on your repositories 5. Once the App is installed, the system will automatically deprecate the old PAT connection ## Statsig GitHub Integration FAQs 1. **Will my data be used to train LLM models?**\ Statsig does not use your data to train LLM models nor does its sub-processors. Your data is secured to your organization and used to improve the services provided to you. 2. **Does Statsig store my source code?**\ Statsig does not permanently store your source code. Code is temporarily processed to extract metadata and embeddings in order to create the Knowledge Graph, and discarded when no longer necessary. 3. **How long does it take for the code insights to appear in Statsig?**\ Setting up the integration can take up to a few hours depending on the size of the repository. After the initial setup, we will index new data every 7 days. 4. **What security controls apply?**\ Information on Statsig's security practices is available [here](https://www.statsig.com/trust/security). Information on Statsig's AI governance is available [here](/compliance/ai_governance_security_privacy). # Github Code References Source: https://docs.statsig.com/integrations/github_code_references Connect Statsig with GitHub to surface code references for feature gates and dynamic configs, so you can see exactly where each flag is used. ### Deprecated: PAT-based Integration We have deprecated the Personal Access Token (PAT) based integration in favor of our new GitHub App-based integration. The new integration provides improved security and functionality, including AI-powered features like Knowledge Graph. Please migrate to the [GitHub AI Integration](/integrations/github-ai-integration) for the best experience. ## Overview The Statsig Github Integration allows you to find [Feature Gate](/feature-flags/overview) and [Dynamic Config](/dynamic-config) references within your codebase on the Statsig Console. The integration leverages the Github API to access only references to the Feature Gate or Dynamic Config without storing any sensitive information. ## Configuration Create a new Github developer token, either Classic or Fine Grained, with at least read access to the organization or repositories that you use Statsig. This can be found in Github under **Settings** > **Developer Settings** > **Personal Access Token**. Then, login to the Statsig Console and navigate to the Github Code References integration on the Integrations page. This can be found in **Project Settings** > **Integrations Tab** > **Github Code References**. The Integration should provide additional instructions on how to enable Github Code References. GitHub integration configuration interface After inputting your token and organization name the integration will verify it can access repositories and notify you if there are any problems. Finally, navigate to the Feature Gates or Dynamic Configs pages on the Statsig Console and select a gate or dynamic config. Under the Diagnostics tab click on the View Code References link to see your code References! Feature gate diagnostics tab showing code references link Code References will appear based on the feature gate or dynamic config page you are on. Code References can be filtered by repository and file extension. Code references panel listing repositories and files ### Github Code References Action We also have a [Github Action](https://github.com/statsig-io/github-code-references) that can scan your repositories for gates and dynamic configs, then create a PR replacing [Stale Gates](/feature-flags/permanent-and-stale-gates). # GitLab Code References Source: https://docs.statsig.com/integrations/gitlab_code_references Connect Statsig with GitLab to surface code references for feature gates and dynamic configs, so you can see exactly where each flag is used. ## Overview The Statsig GitLab Integration allows you to find [Feature Gate](/feature-flags/overview) and [Dynamic Config](/dynamic-config) references within your codebase on the Statsig Console. The integration leverages the GitLab API to access only references to the Feature Gate or Dynamic Config without storing any sensitive information. ## Configuration Create a new GitLab access token, either [project](https://docs.gitlab.com/user/project/settings/project_access_tokens/) or [personal](https://docs.gitlab.com/user/profile/personal_access_tokens/), with the `read_api` scope. Then, login to the Statsig Console and navigate to the Github Code References integration on the Integrations page. This can be found in **Project Settings** > **Integrations Tab** > **GitLab Code References**. The Integration should provide additional instructions on how to enable GitLab Code References. GitLab integration configuration interface After inputting your token and organization name the integration will verify it can access repositories and notify you if there are any problems. Finally, navigate to the Feature Gates or Dynamic Configs pages on the Statsig Console and select a gate or dynamic config. Under the Diagnostics tab click on the View Code References link to see your code References! Feature gate diagnostics page with GitLab code references link Code References will appear based on the feature gate or dynamic config page you are on. Code References can be filtered by repository and file extension. GitLab code references table listing files and repositories # Google Tag Manager (GTM) Source: https://docs.statsig.com/integrations/gtm Integrate Statsig with Google Tag Manager to forward GTM-tagged events to Statsig for experiment analysis and metric tracking without code changes. ## Inbound Integration (Events flow from GTM dataLayer to Statsig) This integration will allow customers using Statsig on the web to leverage their existing Google Tag manager configuration to track events to Statsig. This saves customers from having to retag their web properties with calls specific to Statsig's SDK. This integration uses a global listener to consume all GTM triggers and dispatch a corresponding event back to Statsig. Statsig log stream showing GTM events *(statsig log stream showing GTM events flowing in)* ## Setup ### Step 1: Broadcast Statsig client readiness to GTM The tracking code within GTM will need to know when the Statsig client is ready to use for tracking. To do so, you'll need to broadcast a window-level event and pass the statsig instance for the GTM tag code to use. In your initialize call, implement the `initCompletionCallback` callback as follows: #### Using @statsig/js-client ```js theme={null} window.statsig = new Statsig.StatsigClient('', {/* USER */}, {/* OPTIONS */}); statsig.on('values_updated', function(evt) { // bind before init is called if(evt.status && evt.status === 'Ready') { window.dispatchEvent(new CustomEvent("statsig:ready", { detail: { statsig: statsig } })); } }); await statsig.initializeAsync(); ``` #### Using statsig-js ```js theme={null} await statsig.initialize('', '', { initCompletionCallback: function (duration, success, message) { window.dispatchEvent(new CustomEvent("statsig:ready", { detail: { statsig: statsig } })); } }); ``` ### Step 2: Create new tag GTM create new tag interface ### Step 3: Choose tag type Choose "Custom HTML" for tag type, and paste [this GTM code](#gtm-code) (including script tag) GTM tag configuration screen ### Step 4: Adjust fire options Under Advanced Settings under "Tag Firing options", select "Once per page" GTM tag firing options settings ### Step 5: Set Tag Trigger Below the "Tag Configuration" section, set the Trigger to "Initialization - All Pages" Option. GTM tag trigger configuration ### Step 6: Save tag and test After saving the tag, and publishing your updated GTM tag, tracking will be done automatically without any additional configuration. To debug the integration, you can set a local storage entry `debug_ss_gtm` with any value on your webpage. Now, you'll console log statements for each tracking call being dispatched to Statsig. You can also inspect your browser's network traffic to see events being tracked. ### GTM Code The code below assumes that the statsig client lives at `window.statsig` ```html theme={null} ``` ## Outbound Integration (GTM Data is enriched with Statsig test assignments) This pattern will allow you to enrich your GTM `dataLayer` with experiment assignment information. gtm datalayer outbound Prior to your Statsig `initialize` call, you should bind an EventEmitter listener that captures the assigned experiment name and test group, and push it into the `dataLayer` as follows. Note that the argument passed to the callback contains rich context about the assignment. Please modify the GTM `dataLayer` properties to your liking. ```js theme={null} // use event emitter to listen for experiment assignments statsigClient.on("experiment_evaluation", function(evt) { window.dataLayer.push({ 'event': 'experiment_viewed', 'experiment_name': evt.experiment.name, 'variant_name': evt.experiment.groupName }) }); await statsigClient.initializeAsync(); ``` # Integrations Overview Source: https://docs.statsig.com/integrations/introduction Overview of Statsig integrations with data warehouses, CDPs, messaging tools, CDNs, and developer tools to fit Statsig into your existing stack. *For Warehouse Integrations, go to this [page](/data-warehouse-ingestion/introduction).* The following data connectors are available for use now, and we're adding more every week: > **Warning:** We only support a single data connection at a time. ### Events Forward any events logged via Statsig APIs or SDKs to the following providers: * [Segment](/integrations/data-connectors/segment) * [Snowflake](/data-warehouse-ingestion/snowflake) * [Amplitude](/integrations/data-connectors/amplitude) * Bugsnag * [Fivetran](/integrations/data-connectors/fivetran) * [Google Analytics](/integrations/data-connectors/google-analytics) * [Heap](/integrations/data-connectors/heap) * [Mixpanel](/integrations/data-connectors/mixpanel) * [RevenueCat](/integrations/data-connectors/revenuecat) * [mParticle](/integrations/data-connectors/mparticle) * [RudderStack](/integrations/data-connectors/rudderstack) * [Webhook](/integrations/event_webhook) * [Google Tag Manager (GTM)](/integrations/gtm) * [Braze](/integrations/data-connectors/braze) ### Changelog and alerts Notify and update the following places when Feature Gates/Experiments/Dynamic Configs change: * [Datadog](/integrations/datadog) * Discord * LogDNA * Microsoft Teams * [Slack](/integrations/slack) * [Jira](/integrations/jira): track Statsig gate rollout status and A/B test results from Jira issues # Jira Source: https://docs.statsig.com/integrations/jira Integrate Statsig with Jira to link feature gates and experiments to tickets, surface rollout status in issues, and automate status updates. ## Overview The Statsig for Jira app allows you to bring insights from your Statsig [Feature Gates](/feature-flags/overview) into your Jira project. Statsig feature flags can be associated with your Jira issues to track rollout status and A/B test results directly in Jira. Jira issue view showing Statsig feature gate rollout widget ## Configuration Get a [Server Secret Key from the statsig console](https://console.statsig.com/api_keys) and then go to the [Atlassian Marketplace](https://marketplace.atlassian.com/apps/1225708/statsig-for-jira?hosting=cloud\&tab=overview) to install the Statsig app for Jira. To configure the app, go to Apps > Statsig > Allow Access. Then, enter your api key: Jira app configuration interface # Statsig ChatGPT App Source: https://docs.statsig.com/integrations/mcp/chatgpt-connector Set up the Statsig ChatGPT connector to query experiments, feature gates, and metric data directly from OpenAI ChatGPT using natural language prompts. ## Overview You can set up a ChatGPT App (built on top of the Statsig MCP server), allowing you to query experiments, manage feature flags, explore analytics -- all directly within OpenAI's ChatGPT conversational interface. ## Installation 1. Navigate to the Statsig App in the ChatGPT Apps Directory [here](https://chatgpt.com/apps/statsig/asdk_app_6967f065ac9481918969c660ff7686e9). 2. Click "Connect" and "Continue" to complete OAuth for your Statsig project. ChatGPT App OAuth only supports Personal Console API Keys. Ensure your Statsig org owner has enabled Personal Console API Keys creation for your role [here](https://console.statsig.com/settings?tab=organization) Statsig listing in the ChatGPT Apps Directory ## Using Statsig MCP within ChatGPT With Statsig MCP configured in ChatGPT, you can: * **Explore Experiments**: "List all my active experiments" * **Manage Gates**: "What gates are currently stale?" * **Configure Dynamic Configs**: "Show me the configuration for the dynamic config 'dynamic-config'" * **Get Insights**: "Show me details about the experiment called 'new-checkout-flow'" ## Next Steps After installation, you can: * List experiments, gates, and dynamic configs * Create and update experiments, gates, and configs * Access your Statsig data directly from Codex For more information about available MCP capabilities, see the [MCP capabilities](/integrations/mcp#current-mcp-capabilities) section. # Statsig MCP with Claude Code Source: https://docs.statsig.com/integrations/mcp/claude-code Set up the Statsig MCP server in Claude Code so the agent can query experiments, feature gates, metrics, and project data directly from your terminal. ## Installation and Authentication via OAuth MCP OAuth only supports Personal Console API Keys. Ensure your Statsig org owner has enabled Personal Console API Keys creation for your role [here](https://console.statsig.com/settings?tab=organization) On Claude Code, we recommend using OAuth and the HTTP transport directly. Run this command on the command line: ```bash theme={null} claude mcp add --transport http statsig https://api.statsig.com/v1/mcp ``` This command will: * Add the Statsig MCP server to your Claude Code configuration * Configure it to use HTTP transport with OAuth authentication * Set up the connection to Statsig's MCP endpoint To authenticate via OAuth, run `/mcp` in Claude Code and follow the setup instructions: 1. Claude Code will open a browser window 2. Sign in to your Statsig account 3. Authorize the MCP server to access your Statsig project 4. The authentication will be saved for future sessions While we recommend setting up Statsig MCP via OAuth, you can also authenticate with your Console API key. Run this command on the command line: ```bash theme={null} claude mcp add --transport http statsig-local https://api.statsig.com/v1/mcp \ --header "statsig-api-key: console-YOUR-CONSOLE-API-KEY" ``` Replace `console-YOUR-CONSOLE-API-KEY` with your actual Statsig Console API key, which you can retrieve [here](https://console.statsig.com/api_keys). Ensure your API key has the right permissions — read-only keys can view data, while write keys can make changes to your project! ## Verification After installation, you can verify the connection by: 1. Opening Claude Code 2. Asking Claude to list your Statsig experiments or gates 3. If configured correctly, Claude will be able to access your Statsig data ## Using Statsig MCP with Claude Code Once configured, you can interact with your Statsig data through Claude Code: * **Query Experiments**: "What experiments are currently running?" * **Manage Gates**: "List all my feature flags" * **Get Details**: "Show me the configuration for gate 'new-feature'" * **Create Entities**: "Create a new experiment called 'checkout-test'" ## Troubleshooting If you encounter issues: * Make sure you have the latest version of Claude Code * Verify your Statsig account has the necessary API permissions * Check that the MCP server URL is correct: `https://api.statsig.com/v1/mcp` * Try re-running the installation command ## Next Steps * Learn about [MCP capabilities](/integrations/mcp#current-mcp-capabilities) * Explore [use cases](/integrations/mcp#use-cases) for Statsig MCP * Set up Statsig MCP in other tools: [Cursor](/integrations/mcp/cursor), [Codex CLI](/integrations/mcp/codex) # Statsig MCP with Codex Source: https://docs.statsig.com/integrations/mcp/codex Configure the Statsig MCP server in Codex CLI, the Codex IDE extension, and the Desktop App so you can query Statsig data and manage features from Codex. ## Overview You can set up the Statsig MCP server using an API key on Codex CLI, IDE Extension, and App today. ## Installation and Authentication via OAuth MCP OAuth only supports Personal Console API Keys. Ensure your Statsig org owner has enabled Personal Console API Keys creation for your role [here](https://console.statsig.com/settings?tab=organization) On Codex, we recommend using OAuth and the HTTP transport directly. Authentication will be saved for future sessions. In Codex Desktop, navigate to Settings > MCP servers and add a new custom server. Set up via Streamable HTTP, with URL set to `https://api.statsig.com/v1/mcp`: Set up Statsig MCP in the Codex Desktop App Upon saving, you should see statsig turned on under custom servers. Statsig MCP in the Codex Desktop App MCP Servers List If you are working in Codex CLI or IDE extension, run the below command to add the Statsig MCP server into your `~/.codex/config.toml` file and restart Codex. ```bash theme={null} codex mcp add statsig --url https://api.statsig.com/v1/mcp ``` This command should open a browser window for you to sign in to your Statsig account and authorize access to your Statsig project. Once you've signed in, restart Codex and run `/mcp` in CLI to validate the Statsig MCP is listed as an available MCP server. Make sure status is set to enabled: Statsig MCP in the Codex CLI MCP Servers List ```toml theme={null} [mcp_servers.statsig] url = "https://api.statsig.com/v1/mcp" command = "npx" args = ["--yes", "mcp-remote", "https://api.statsig.com/v1/mcp", "--header", "statsig-api-key: console-YOUR-CONSOLE-API-KEY"] trust_level = "trusted" enabled = true ``` Replace `console-YOUR-CONSOLE-API-KEY` with your actual Statsig Console API key, which you can retrieve [here](https://console.statsig.com/api_keys). Ensure your API key has the right permissions — read-only keys can view data, while write keys can make changes to your project! ## Using Statsig MCP with Codex With Statsig MCP configured in any Codex environment, you can: * **Explore Experiments**: "List all my active experiments" * **Manage Gates**: "What gates are currently stale?" * **Configure Dynamic Configs**: "Show me the configuration for the dynamic config 'dynamic-config'" * **Get Insights**: "Show me details about the experiment called 'new-checkout-flow'" ## Next Steps After installation, you can: * List experiments, gates, and dynamic configs * Create and update experiments, gates, and configs * Access your Statsig data directly from Codex For more information about available MCP capabilities, see the [MCP capabilities](/integrations/mcp#current-mcp-capabilities) section. # Statsig MCP with Cursor Source: https://docs.statsig.com/integrations/mcp/cursor Set up the Statsig MCP server in Cursor IDE so the agent can read experiments, feature gates, and metric data while writing and reviewing code. ## Installation and Authentication via OAuth MCP OAuth only supports Personal Console API Keys. Ensure your Statsig org owner has enabled Personal Console API Keys creation for your role [here](https://console.statsig.com/settings?tab=organization) You can add the Statsig MCP with OAuth to Cursor in two ways: ### Option 1: Quick Install (Recommended) [Click here](cursor://anysphere.cursor-deeplink/mcp/install?name=statsig\&config=eyJ1cmwiOiJodHRwczovL2FwaS5zdGF0c2lnLmNvbS92MS9tY3AifQ%3D%3D) to automatically add the Statsig MCP to Cursor. ### Option 2: Manual Configuration 1. Open Cursor settings 2. Navigate to **Settings → Cursor Settings → Tools & Integrations** 3. Find the MCP servers section 4. Add the following configuration to `~/.cursor/mcp.json`: ```json theme={null} { "mcpServers": { "statsig": { "url": "https://api.statsig.com/v1/mcp" } } } ``` Cursor will automatically handle OAuth authentication when you first use the Statsig MCP. You'll be prompted to: 1. Sign in to your Statsig account 2. Authorize the MCP server to access your Statsig project 3. Restart Cursor to apply the changes 4. Verify the connection by navigating to **Settings → Cursor Settings → Tools & Integrations**, where Statsig MCP server should be listed and active The authentication will be saved for future sessions. While we recommend setting up Statsig MCP via OAuth, you can also authenticate with your Console API key. [Click here](cursor://anysphere.cursor-deeplink/mcp/install?name=statsig\&config=eyJjb21tYW5kIjoibnB4IG1jcC1yZW1vdGUgaHR0cHM6Ly9hcGkuc3RhdHNpZy5jb20vdjEvbWNwIC0taGVhZGVyIHN0YXRzaWctYXBpLWtleToke0FVVEhfVE9LRU59IiwiZW52Ijp7IkFVVEhfVE9LRU4iOiJpbnNlcnQteW91ci1hcGkta2V5LWhlcmUifX0%3D) to quick install, or manually configure: 1. Open Cursor settings 2. Navigate to **Settings → Cursor Settings → Tools & Integrations** 3. Find the MCP servers sections 4. Add the below configuration to `~/.cursor/mcp.json` ```json theme={null} { "mcpServers": { "statsig": { "command": "npx mcp-remote https://api.statsig.com/v1/mcp --header statsig-api-key:${AUTH_TOKEN}", "env": { "AUTH_TOKEN": "console-YOUR-CONSOLE-API-KEY" } } } } ``` Replace `console-YOUR-CONSOLE-API-KEY` with your actual Statsig Console API key, which you can retrieve [here](https://console.statsig.com/api_keys). Ensure your API key has the right permissions — read-only keys can view data, while write keys can make changes to your project! ## Using Statsig MCP with Cursor Once configured, you can use Statsig MCP commands in Cursor's chat interface: * **Query Experiments**: "What experiments are currently running?" * **Manage Gates**: "List all my feature flags" * **Get Details**: "Show me the configuration for gate 'new-feature'" * **Create Entities**: "Create a new experiment called 'checkout-test'" ## Troubleshooting If the MCP server doesn't appear: * Make sure you've restarted Cursor after adding the configuration * Check that the `mcp.json` file is in the correct location: `~/.cursor/mcp.json` * Verify your Statsig account has the necessary permissions ## Next Steps * Learn about [MCP capabilities](/integrations/mcp#current-mcp-capabilities) * Explore [use cases](/integrations/mcp#use-cases) for Statsig MCP * See examples of [stale gate cleanup](/integrations/mcp#example-prompt-for-stale-gate-cleanup) # Statsig MCP with Other MCP-Compatible Clients Source: https://docs.statsig.com/integrations/mcp/manual-setup Manually configure the Statsig MCP server for any MCP-compatible tool, including authentication, scopes, transport options, and recommended client settings. ## Overview If your tool doesn't have a specific setup guide, you can manually configure the Statsig MCP server. This guide covers the general configuration steps that work with any MCP-compatible client. ## Prerequisites * A Statsig account (sign up at [console.statsig.com](https://console.statsig.com)) * An MCP-compatible tool or client * Access to your tool's configuration files ## Configuration Add the following configuration to your MCP client's configuration file: ```json theme={null} { "mcpServers": { "statsig": { "url": "https://api.statsig.com/v1/mcp" } } } ``` This installs the Statsig MCP Server with OAuth. When you first use the MCP server: 1. Your tool will prompt you to authenticate 2. You'll be redirected to Statsig's OAuth page 3. Sign in and authorize the MCP server 4. The authentication token will be stored automatically MCP OAuth only supports Personal Console API Keys. Ensure your Statsig org owner has enabled Personal Console API Keys creation for your role [here](https://console.statsig.com/settings?tab=organization) While we recommend setting up Statsig MCP via OAuth, you can also authenticate with your Console API key. Add the below configuration to your MCP client's configuration file: ```json theme={null} { "mcpServers": { "statsig": { "command": "npx mcp-remote https://api.statsig.com/v1/mcp --header statsig-api-key:${AUTH_TOKEN}", "env": { "AUTH_TOKEN": "console-YOUR-CONSOLE-API-KEY" } } } } ``` Replace `console-YOUR-CONSOLE-API-KEY` with your actual Statsig Console API key, which you can retrieve [here](https://console.statsig.com/api_keys). Ensure your API key has the right permissions — read-only keys can view data, while write keys can make changes to your project! ## Verification After adding the configuration: 1. Restart your tool to apply the changes 2. Check your tool's MCP server list to verify Statsig appears 3. Try using a Statsig MCP command to test the connection ## Testing the Connection You can test the connection by asking your tool to: * **List experiments**: "Get list of experiments" * **List gates**: "Get list of gates" * **Get details**: "Get details for experiment \[experiment-id]" ## Next Steps * Learn about [MCP capabilities](/integrations/mcp#current-mcp-capabilities) * Explore [use cases](/integrations/mcp#use-cases) for Statsig MCP * Check tool-specific guides: [Cursor](/integrations/mcp/cursor), [Claude Code](/integrations/mcp/claude-code), [Codex CLI](/integrations/mcp/codex) # Overview Source: https://docs.statsig.com/integrations/mcp/overview Connect the Statsig MCP server to Codex, Cursor, and Claude Code so AI assistants can explore experiments, feature gates, and metrics in your project. ## MCP configuration guides Set up Statsig MCP using Codex Desktop App, CLI, or IDE extension. Talk to your Statsig projects from within ChatGPT. Configure Statsig MCP in Cursor IDE. Set up Statsig MCP in Claude Code. Manual configuration for any MCP-compatible tool. ## Current MCP capabilities
    Tool Description
    Get\_Audit\_Logs List audit logs in the project. Filter by:
    • id
    • sortKey
    • sortOrder
    • tags
    • startDate
    • endDate
    Tool Description
    Create\_Dynamic\_Config Create new config
    Get\_Dynamic\_Config\_Details\_by\_ID Retrieve detailed config information
    Get\_List\_of\_Dynamic\_Configs List all dynamic config objects in the project. Filter by:
    • creatorName
    • tags
    Update\_Dynamic\_Config\_Entirely Replace entire dynamic config with new targeting and values
    Tool Description
    Create\_Experiment Create new experiment
    Get\_Experiment\_Details\_by\_ID Get experiment details
    Get\_Experiment\_Overall\_Results Retrieve experiment results/pulse data for a specific experiment. Analyze by:
    • date
    • cuped: whether or not to apply CUPED
    • confidence: Confidence threshold used for result calculations
    Get\_Experiment\_Metric\_
    Retrieve metric results for one experiment metric with dimensional breakdowns. Analyze by:
    • date
    • cuped: whether or not to apply CUPED
    • confidence: Confidence threshold used for result calculations
    Get\_List\_of\_Experiments List all experiments in the project. Filter by:
    • status: Supported values include active, setup,decision\_made , abandoned,archived, experiment\_stopped,assignment\_stopped
    • creatorName
    • tags
    • stale
    Update\_Experiment\_Entirely Replace entire experiment configuration (any excluded data will be removed)
    Tool Description
    Create\_Gate Create new gate/flag
    Get\_Gate\_Details\_by\_ID Get complete gate configuration details
    Get\_Gate\_Results Retrieve pulse/results for a specific gate rule.
    • cuped: Whether or not to apply CUPED. Defaults true
    • confidence: CI percentage, defaults 95
    Get\_List\_of\_Gates List all gates/flags. Filter by:
    • type (e.g., temporary, permanent, stale, tempmlate)
    • creatorName
    • tags
    Update\_Gate\_Entirely Replace entire gate setup with new rules and settings (any excluded data will be removed)
    Tool Description
    Create\_Layer Create a new layer
    Get\_Layer\_Details\_by\_ID Retrieve layer details, including parameters and metadata
    Get\_List\_of\_Layers List all layers in the project
    Update\_Layer\_Entirely Replace the full layer configuration
    Tool Description
    Get\_List\_of\_Metric\_Sources List all metric sources in the project
    Get\_List\_of\_Metrics List all metrics in the project. Filter by:
    • showHiddenMetrics
    • tags
    • filters (supports `any of` and `all of`)
    Get\_Metric\_Definition\_by\_ID Get the full definition for a metric, including its type, source, and configuration details
    Tool Description
    Create\_Segment Create a new segment. Supports id\_list, rule\_based, analysis\_list, and user\_store\_id\_list segment types
    Get\_List\_of\_Segments List all segments in the project
    Get\_Segment\_Details\_by\_ID Retrieve segment details
    Update\_Segment Update an existing segment. Specifically:
    • Update rules for conditional segment types
    • Add IDs to user stores and ID lists
    Need other functions? We're happy to consider additions by request, reach out in Slack. ## Use Cases The Statsig MCP server now supports both `GET` and `POST` requests. This means tools can not only read data (like stale gates) but also make updates, if your API key has write permissions. We've found the Statsig MCP server especially useful for: * Repetitive tasks like cleaning up stale gates * Summarizing console information in your IDE workflows * Bulk creating or deleting gates, and making the necessary changes in your code ### Example prompt for stale gate cleanup ``` You are an expert, diligent Software engineer with the sole goal of reducing the amount of tech debt in the code base. This code base, making use of best practices, leverages feature gates liberally using Statsig. As gates complete their lifecycle in Statsig, they may end up "stale" which means that they're enabled, but no longer checked. Your job is to find these gates, and refactor the codebase to no longer check the gate (instead, changing the check to a constant value). You should follow coding best practices: - You should not simply replace gate calls with "True" or "False" but instead carefully trace the logic through to where it is used and change the behavior that way - adjusting the code in minor ways to make the default behavior what the value is that the gate was returning - You should always strive to write minimal code - readable but terse, never longer than it needs to be - You should never write comments or debug statements. You should use the statsig-local MCP to list feature gates, then look for gates that are marked as stale. You should then grep the codebase for that feature flag name, and do a minimal rewrite of the code to no longer use Statsig, removing the checkGate call or similar. When you use the MCP use the get /console/v1/gates endpoint and parameters type="STALE" and limit =10. You should select only one gate to do this with, before stopping. If you cannot find the gate after a grep, try the next one you found using the MCP. Once you successfully remove a gate, return. ``` # OpenAI Source: https://docs.statsig.com/integrations/openai Integrate Statsig with OpenAI to log AI requests, capture metrics, and run experiments on prompts, models, and parameters across your applications. ## Context When using a pre-trained large language model, several inputs influence user experience, including the prompts used, the "inference parameters" given to the model, often including parameters like temperature, length penalties and repetition penalties, and even the exact model selected. Tools like Statsig can streamline the process of assigning users to various experiments, such as modifying these inputs, and can automatically identify when changes have a statistically significant impact on user experience metrics. This enables efficient iteration and optimization of user experience by tweaking model choices, prompts, and inference parameters. In this case we show how you can log both implicit indicators of user feedback (like response time) and more explicit ones, like self-reported satisfaction. ## Introduction The included Python code offers an example of the simple but powerful interaction between OpenAI's GPT and Statsig to experiment with model inputs and log user events. This example uses OpenAI's ChatCompletion feature to answer questions, plus a Statsig integration to experiment with model versions and log user feedback to the changes. This example assumes you have a funded OpenAI account, plus a Statsig experiment that varies the model selected between "gpt-3.5-turbo" and "gpt-4". For more info on setting up a Statsig experiment, see the [experiments](/experiments-plus) page. ## Code Breakdown ### Initial Configuration Of course, you'll need to install both the Statsig and OpenAI Python packages before starting: ```bash theme={null} pip3 install openai, statsig ``` After that, we can begin coding in a python file: ```python theme={null} import openai from statsig import statsig, StatsigEvent, StatsigUser import time openai.api_key = "your_openai_key" # Replace with your own key statsig.initialize("your_statsig_secret") # Replace with your Statsig secret user = StatsigUser("user-id") #This is a placeholder ID - in a normal experiment Statsig recommends using a user's actual unique ID for consistency in targeting. See https://docs.statsig.com/concepts/user ``` ### The ask\_question Function The following code all occurs in one function titled ask\_question (see the [final code](#final-code)) 1. Get User Input ```python theme={null} #ask the user for a question to query GPT with question = input("\nWhat is your question? ") ``` First, we prompt the user with the question they'd like to ask ChatGPT 2. Query OpenAI's GPT ```python theme={null} #track the start time so we can check response time later start_time = time.time() completion = openai.ChatCompletion.create( model=statsig.get_experiment(user, "statsig_openai_integration").get("model", 'gpt-4'), messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": question} ] ) ``` Next, we request a completion that queries the GPT model specified by the Statsig experiment (either gpt-3.5-turbo or gpt-4). We also start a timer so we can track the response time in our events later on. 3. Display Response & Log Implicit Feedback ```python theme={null} #log "implicit" indicators to Statsig c = completion.choices[0] stats = completion.usage #completion object has the number of tokens used - which is that what GPT usage is charged on. statsig.log_event(StatsigEvent(user, "chat_completion", value=c.finish_reason, metadata={"response_time": time.time() - start_time, "completion_tokens": stats["completion_tokens"], "prompt_tokens": stats["prompt_tokens"], "total_tokens": stats["total_tokens"]})) #print the message back to the user print(f"\nAnswer: {c.message['content']}") ``` With the response from OpenAI we have our first set of useful information to log to Statsig - like the response time and tokens used. We log this information with Statsig's SDK, storing them in the metadata of the request. 4. Collect User Feedback & Log to Statsig ```python theme={null} #track explicit feedback: if the user is satisfied with the answer satisfaction = input("\nDid this answer your question? (y/n): ") # Log "explicit" indicators to Statsig if satisfaction == 'y': statsig.log_event(StatsigEvent(user, "satisfaction")) elif satisfaction == 'n': statsig.log_event(StatsigEvent(user, "dissatisfaction")) ``` Next, we log a more explicit indicator of feedback, the user's self-reported satisfaction or dissatisfaction. The satisfaction metric can provide a strong indicator of the model's overall power. And we're done - we can run this Python program with the following code, now outside of our ask\_question function. ```python theme={null} if __name__ == "__main__": while input("Would you like to ask a question? (y/n): ").lower() == 'y': ask_question() ``` ### Tips for Using Statsig with AI 1. Experimentation: You can also test other model parameters like temperature, top\_p, or initial prompts. 2. Log Useful Data: Consider logging other interesting user interactions or feedback. 3. Analyze and Iterate: After collecting enough data, analyze the results on the Statsig dashboard. 4. User Identification: Consider integrating a mechanism to uniquely identify each user/session. Always ensure you're in compliance with user privacy regulations and that you have user consent where necessary. ### Useful Links * [OpenAI Documentation](https://platform.openai.com/docs/api-reference/chat/create) ## Final code ```python theme={null} import openai from statsig import statsig, StatsigEvent, StatsigUser import time openai.api_key = "your_openai_key" statsig.initialize("your_statsig_secret") user = StatsigUser("user-id") #This is a placeholder ID - in a normal experiment Statsig recommends using a user's actual unique ID for consistency in targeting. See https://docs.statsig.com/concepts/user def ask_question(): #ask the user for a question to query GPT with question = input("\nWhat is your question? ") #track the start time so we can check response time later start_time = time.time() #query GPT with the OpenAI Python library for chat completions completion = openai.ChatCompletion.create( model=statsig.get_experiment(user, "statsig_openai_collab").get("model", 'gpt-4'), #experiment is setup to return either "gpt-3.5-turbo" or "gpt-4". #other than varying the model selected, other attributes could be varied like "temperature", "top_p", "presence_penalty" and more. #See the "Create chat completions" section of the OpenAI documentation for more: https://platform.openai.com/docs/api-reference/chat/create messages=[ {"role": "system", "content": "You are a helpful assistant."}, #Initial prompts are another candidate for experimentation {"role": "user", "content": question} ] ) #log "implicit" indicators to Statsig c = completion.choices[0] #we've only requested one choice, so selecting the first stats = completion.usage #completion object has the number of tokens used - which is that what GPT usage is charged on. statsig.log_event(StatsigEvent(user, "chat_completion", value=c.finish_reason, metadata={"response_time": time.time() - start_time, "completion_tokens": stats["completion_tokens"], "prompt_tokens": stats["prompt_tokens"], "total_tokens": stats["total_tokens"]})) #print the message back to the user print(f"\nAnswer: {c.message['content']}") #track explicit feedback: if the user is satisfied with the answer satisfaction = input("\nDid this answer your question? (y/n): ") #log "explicit" indicators to Statsig if satisfaction == 'y': statsig.log_event(StatsigEvent(user, "satisfaction")) elif satisfaction == 'n': statsig.log_event(StatsigEvent(user, "dissatisfaction")) if __name__ == "__main__": #Let the user abandon the question-asking process when they are done while input("Would you like to ask a question? (y/n): ").lower() == 'y': ask_question() ``` # Pulumi Source: https://docs.statsig.com/integrations/pulumi Manage Statsig feature gates, experiments, and dynamic configs as code with Pulumi, including provider setup and resource definition examples. The [Statsig Pulumi Provider](https://www.pulumi.com/registry/packages/statsig/) allows you to configure your gates and experiments using Pulumi Infrastructure as Code. The provider synchronizes with Statsig via the Console API. If there is something you need to perform that isn't supported by the Pulumi Provider, checkout the [Console API](/console-api/introduction). ## Installation The Statsig provider is available as a package in the following Pulumi languages: * **JavaScript/TypeScript**: [`@statsig/pulumi-statsig`](https://www.npmjs.com/package/@statsig/pulumi-statsig) * **Python**: [`pulumi-statsig`](https://pypi.org/project/pulumi-statsig/) * **Go**: [`github.com/statsig-io/pulumi-statsig/sdk/go/statsig`](https://github.com/statsig-io/pulumi-statsig) * **.NET**: [`Statsig.Pulumi`](https://www.nuget.org/packages/Statsig.Pulumi) ## Configuration The provider needs to be configured with the proper credentials before it can be used. Configure your `Pulumi.yaml` file with your Console API key: ```yaml theme={null} # Pulumi.yaml provider configuration file name: configuration-example runtime: nodejs # or python, go, dotnet config: statsig:consoleApiKey: value: 'YOUR_CONSOLE_API_KEY' ``` ### Configuration Reference * `consoleApiKey` (String) - The Statsig Console API key retrieved from Statsig console. ## Example Usage ### TypeScript ```typescript theme={null} import * as pulumi from "@pulumi/pulumi"; import * as statsig from "@statsig/pulumi-statsig"; // Create a Feature Gate const gate = new statsig.Gate("my-gate", {}); ``` ### Python ```python theme={null} import pulumi import pulumi_statsig as statsig # Create a Feature Gate gate = statsig.Gate("my-gate") ``` ### Go ```go theme={null} package main import ( "github.com/pulumi/pulumi/sdk/v3/go/pulumi" "github.com/statsig-io/pulumi-statsig/sdk/go/statsig" ) func main() { pulumi.Run(func(ctx *pulumi.Context) error { // Create a Feature Gate _, err := statsig.NewGate(ctx, "my-gate", nil) if err != nil { return err } return nil }) } ``` ### C\# ```csharp theme={null} using Pulumi; using Statsig = Statsig.Pulumi; return await Deployment.RunAsync(() => { // Create a Feature Gate var gate = new Statsig.Gate("my-gate"); }); ``` ## Supported Features We currently support the following Statsig configurations: * Gates * Experiments Coming Soon: * Dynamic Configs * Segments If you need more from our Pulumi Provider, please feel free to ask in the [Statsig Slack channel](https://statsig.com/slack). # Serverless Source: https://docs.statsig.com/integrations/serverless Use Statsig integrations with serverless platforms like AWS Lambda, Vercel, and Cloudflare Workers to evaluate flags and log events at the edge. The serverless SDK is a lightweight JavaScript SDK optimized for serverless environments while being compatible for any other platform/environment. This SDK harnesses the functionalities that all Statsig server SDK's offer. ```bash theme={null} npm install @statsig/serverless-client ``` ```bash theme={null} import {StatsigServerlessClient} from '@statsig/serverless-client' ``` ```javascript theme={null} const client = new StatsigServerlessClient(process.env.STATSIG_KEY) ``` The client instantiation takes two arguments: * `sdkKey : string` This is your Statsig client API key. It is available from the [Project Settings](https://console.statsig.com/api_keys) page in the Statsig Console. This is used to authenticate your requests. * `options : StatsigOptions` See [here for more options](/client/javascript-sdk#statsig-options). The following line initializes the client by loading feature gate and experiment configurations over network ```javascript theme={null} const init = await client.initializeAsync(); ``` ```javascript theme={null} const GateResult = client.checkGate('pass_gate', user); ``` This is a gate check in code. The `checkGate` method takes two arguments: * `name : string` The name of the Statsig gate that you are checking. * `user : StatsigUser` The Statsig user object for whom the gate is being checked. For more information on the user object, see [here](/sdks/user#introduction-to-the-statsiguser-object). Refer to the [Javascript on-device evaluation SDK documentation](/client/jsOnDeviceEvaluationSDK) for how to check other entities like experiments and dynamic configs. ```javascript theme={null} client.logEvent('gate_check', { userID: randomUserId }); ``` This is an event log in code. The `logEvent` method takes two parameters: * `eventOrName : string | StatsigEvent` This is the name and details of the event you are logging. * `user : StatsigUser` The Statsig user object for whom the event is being logged. For more information on event logging, see [here](/client/jsOnDeviceEvaluationSDK#logging-an-event). ```javascript theme={null} client.flush(); ``` This flushes all events from the SDK to Statsig. **Without this, you will not be able to get diagnostic information in the Statsig Console, nor any event data you logged**. ### Putting it all together ```javascript index.js theme={null} import { StatsigServerlessClient } from '@statsig/serverless-client'; export default async function handler(request) { const client = new StatsigServerlessClient(process.env.STATSIG_KEY); let init = await client.initializeAsync(); const user = { userID: Math.random().toString().substring(2, 5) }; const passed = client.checkGate('pass_gate', user); client.logEvent('serverless_event', user, passed.toString()); client.flush(); return new Response( JSON.stringify({ passed, user }), ); } ``` ### Bootstrapping You can use the serverless SDK to bootstrap your client SDK. ```javascript theme={null} // Get client initialize response for a user const values = statsig.getClientInitializeResponse(user, options); // Pass values to a client SDK to initialize without a network request ``` For more information on bootstrapping including options and full code example, see [here](/server-core/node-core#client-sdk-bootstrapping-|-ssr) ### Bring your own CDN The Statsig Serverless SDK also supports the bring-your-own-CDN model. You can now store your Statsig config specs anywhere, and initialize your client directly from that source. This use case has been tested and optimized for usage with AWS Lambda/Lambda\@Edge and S3 but is compatible with any other data source. Ensure the URL you provide returns the Statsig config specs for your project. ```javascript theme={null} let init = await client.initializeViaURL("https://my-statsig-config.s3.us-east-1.amazonaws.com/my_dcs.json") ``` After initializing, use the SDK as usual. # Slack Notifications Source: https://docs.statsig.com/integrations/slack Configure the Statsig Slack integration to deliver project, team, and personal notifications for experiments, feature gates, and metric alerts. Depending on your organization’s Slack settings, you may need help from a Slack Admin to complete setup. *** Statsig's Slack integration seamlessly connects your experimentation platform with your team's communication hub. By enabling this integration, you can receive real-time updates and alerts directly in your Slack workspace. Stay informed about critical changes to your feature gates, experiments, and dynamic configs without constantly monitoring the Statsig console or the Statsig operational health [status page](https://status.statsig.com). ## Setting Up Slack **Step 1:** From your Statsig Console, go to **Statsig Settings** -> **[Integrations](https://console.statsig.com/integrations)**. **Step 2:** Select Slack and click **+ Add Connection**. **Step 3:** Allow the Statsig app to connect to your Slack workspace and Slack channel where you would like to receive Statsig's notifications. *** ## Product Notifications **Step 4:** Filter to the relevant Team, Tags, or Target Apps. **Step 5:** Choose the notifications about product-level activities and status changes you want to subscribe to. Slack Product Notification Events *** ## General Notifications **Step 6:** Choose the notifications about Statsig system status or administrative updates you want to subscribe to. Slack General Notifications - Notifications about account health and administrative updates *** ## Personal Notifications **Step 7:** Go to **Settings** -> **[My Account](https://console.statsig.com/account_settings)** and navigate to Notifications section. In this page, you can also manage your email preferences for receiving notifications on your email. Personal notification preferences interface # Statsig Lite Source: https://docs.statsig.com/integrations/statsiglite Use Statsig Lite, a lightweight integration option for embedding Statsig into low-resource environments and lightweight client surfaces. ## What it is Statsig Lite is a free experiment calculator, powered by our stats engine. It lets you visualize experiment results for data from experiments you've already run. You bring anonymized experiment exposure and metrics/events in CSVs, and get to preview it in the Statsig Console without connecting your production applications or data warehouses. ## FAQ ### Should I anonymize my data? Yes. Please use anonymized data with Statsig Lite - there's no reason to do otherwise. ### Who can see my data? We will email you a link to your results - with a secret embedded. You can choose to share this link with anyone you want to have access to this view. ### When would I do this? The most common case for this is teams that use tools like Optimizely for experiment assignment, but can't use the same tool for experiment analysis. This often happens because the data that needs to be analyzed sits in a warehouse or a system Optimizely can't access and teams fall back to manual analysis for experiments. Statsig can now automate that analysis for you. It is also a way to quickly get a sense for what your experimental data will look like on Statsig in a few minutes. ### I don't have real data to upload. Can I get some sample data? Yes. The Statsig Lite website has some sample data you can start with. Tools like ChatGPT also make it easy to create a sample dataset using English language prompts like [this](https://chatgpt.com/share/67bf3105-b984-800c-99b4-02935deb5f5b). ### Is Statsig Lite missing features? Statsig Lite is meant to be a quick preview. Sign up for a Statsig account to get access to features including automated health checks (balanced exposures, pre-experimental bias, outlier detection), advanced out-of-the-box statistical methodologies (stratified sampling, differential impact detection, sequential testing) and the ability to quickly dive deeper into results with custom queries. # Statsig Terraform Provider Source: https://docs.statsig.com/integrations/terraform/introduction Manage Statsig feature gates, experiments, and dynamic configs as code with the Terraform provider, including authentication and resource examples. The Statsig Terraform Provider allows you to configure your gates and experiments with Terraform. The provider synchronizes with Statsig via the Console API. If there is something you need to perform that isn't supported by the Terraform Provider, checkout the [Console API](/console-api/introduction). It is hosted on the Terraform registry at [https://registry.terraform.io/providers/statsig-io/statsig](https://registry.terraform.io/providers/statsig-io/statsig) ## Supported Features We currently only support the following Statsig configurations: * [Gates](/integrations/terraform/terraform_gate) * [Experiments](/integrations/terraform/terraform_experiment) Coming Soon: * Dynamic Configs * Segments If you need more from our Terraform Provider, please feel free to ask in the [Statsig Slack channel](https://statsig.com/slack). # Managing Experiments With Terraform Source: https://docs.statsig.com/integrations/terraform/terraform_experiment Define Statsig experiments as Terraform resources, including variants, allocation, targeting, and scorecard metrics, for fully reproducible setups. You can create a .tf file (Terraform File) to configure your Statsig experiments. All features of [console/v1/experiments](/console-api/experiments) are supported. The layout is very similar to the JSON body of a /experiments request. Requiring the Statsig provider. (You will need to change the version). ```go theme={null} terraform { required_providers { statsig = { version = "x.x.x" source = "statsig-io/statsig" } } } ``` ## Basic Example Creating a basic experiment resource ```go theme={null} resource "statsig_experiment" "my_experiment" { name = "my_experiment" description = "A short description of what we are experimenting on." id_type = "userID" allocation = 10 status = "setup" groups { name = "Test Group" size = 50 parameter_values_json = jsonencode({ "a_string" : "test_string", "a_bool" : true }) } groups { name = "Control Group" size = 50 parameter_values_json = jsonencode({ "a_string" : "control_string", "a_bool" : false }) } } ``` ## Changing Experiment Status You can update the `status` field to four possible values, **setup**, **active**, **decision\_made** and **abandoned**. If you would like to see code examples of how the **Setup -> Run -> Ship** flow works for an experiment, check out our [Terraform Acceptance Tests](#) for experiments. #### Status: setup When an experiment has this status, you are stating that your experiment is not ready, and no values will be served via a Statsig SDK or the HttpAPI. #### Status: active When an experiment has this status, you are marking the experiment as running, and it will start returning values to your users and collecting analytics data. #### Status: decision\_made When an experiment has this status, you are stating that your experiment is complete and you have picked an experiment group that you would like to ship. Changing to this state will require the `launched_group_id` field to be set will the GroupID found on [console.statsig.com](https://console.statsig.com). #### Status: abandoned Experiments with this status will not serve any values and will not collect any analytics information. * You may only create an experiment with the status "setup" or "active". * You can only transition to "decision\_made" from "active". A full experiment example is included in the open source Github repo [https://github.com/statsig-io/terraform-provider-statsig/blob/main/examples/resources/statsig\_experiment/resource.tf](https://github.com/statsig-io/terraform-provider-statsig/blob/main/examples/resources/statsig_experiment/resource.tf). # Managing Gates With Terraform Source: https://docs.statsig.com/integrations/terraform/terraform_gate Define Statsig feature gates as Terraform resources, including rules, conditions, and rollout percentages, for fully reproducible feature flag setups. You can create a .tf file (Terraform File) to configure your Statsig feature gates. All features of [console/v1/gates](/console-api/gates) are supported. The layout is very similar to the JSON body of a /gates request. Requiring the Statsig provider. (You will need to change the version). ```go theme={null} terraform { required_providers { statsig = { version = "x.x.x" source = "statsig-io/statsig" } } } ``` ## Basic Example Creating a basic gate resource ```go theme={null} resource "statsig_gate" "my_gate" { name = "my_gate" description = "A short description of what this Gate is used for." is_enabled = true id_type = "userID" rules { name = "Public" pass_percentage = 100 conditions { type = "public" } } } ``` ## Conditions All Console API conditions are supported but the syntax needs a little tweaking. * **type** | string | The [type](/console-api/rules#all-conditions) of condition it is. * **operator** | string | What form of evaluation should be run against the **target\_value**. * **target\_value** | \[string] | The value or values you wish to evaluate. Note: This must be an array, and elements should be of string type. (You can put quotes on numbers. 31 -> "31") * **field** | string | Only for custom\_field condition type. The name of the field you wish to pull for evaluation from the "custom" object on a user. ```go theme={null} conditions { type = "custom_field" target_value = ["31"] operator = "gt" field = "age" } ``` See the full list of conditions [here](/console-api/rules#all-conditions). A full gate example is included in the open source Github repo [https://github.com/statsig-io/terraform-provider-statsig/blob/main/examples/resources/statsig\_gate/resource.tf](https://github.com/statsig-io/terraform-provider-statsig/blob/main/examples/resources/statsig_gate/resource.tf) # Datadog Triggers Source: https://docs.statsig.com/integrations/triggers/datadog Configure Datadog triggers in Statsig to react to Datadog monitor events with automated rollback, killswitch, or notification actions on feature flags. ### Overview Triggers are a way to make changes to your Statsig project from a 3rd party source. You can create a trigger with a specific action like "Disable a feature gate". This will generate a URL that will perform that action when hit. Triggers can be used with Datadog to toggle a gate on or off depending on the performance of a metric. An example use case would be to create a trigger that disables a gate and hook it up to a Datadog monitor so that if metric regression is detected, you can automatically turn off the feature. ### Trigger Types Currently, the only supported type of triggers are gate triggers. ### Creating a Trigger 1. On Statsig console, navigate to the [integrations](https://console.statsig.com/integrations) tab. 2. Find and open **Datadog** -> **Triggers**. 3. Specify the target gate and action, then click **Create** Statsig trigger configuration interface ### Connecting to Datadog 1. Copy the trigger URL generated from the previous step. 2. In Datadog, create a new [Webhook](https://app.datadoghq.com/integrations/webhooks) using that URL. (You do not need to make any changes to the payload) Datadog webhook configuration screen 3. Now within your Datadog monitor settings, you can add this webhook as a notification target. Datadog monitor notification settings To learn more about how to configure a Datadog monitor, see [Datadog Notifications](https://docs.datadoghq.com/monitors/notify/). # Vercel Source: https://docs.statsig.com/integrations/vercel Integrate Statsig with Vercel to evaluate feature gates and experiments in edge middleware, server components, and serverless functions on Vercel. Statsig offers a suite of integration tools that makes usage with Vercel simple: * Statsig automatically pushes project changes to Vercel's Edge Config, providing low latency SDK startup. * Statsig offers a Vercel helper that handles client initialization and event flushing, so you can focus on your business logic. #### Configure Integration 1. Head over to the [Vercel Marketplace](https://vercel.com/integrations/statsig) and install the Statsig integration. 2. You will be prompted to create a new Statsig account or link an existing Statsig account. * Click **Accept and Create** * Enable Edge Config Syncing to have statsig automatically push configs into Vercel's Edge Config Vercel marketplace installation interface Once created, you can select **Open in Statsig** in the top right to set up gates and experiments for your newly created Statsig project. Vercel marketplace project view interface Your Statsig Integration for Vercel is completed! * Select your project and edge config * Click **Save** Your Statsig Integration is now complete! You can go to the **storage** tab to confirm your Statsig config specs have propagated to your Edge Config. #### Install the Statsig SDK ```bash theme={null} npm install @statsig/vercel-edge ``` #### import the Vercel helper ```bash theme={null} import {handleWithStatsig} from '@statsig/vercel-edge ``` #### Use the SDK ```Javascript theme={null} export default handleWithStatsig(handler, params) ``` The helper method takes two arguments: * `handler` This is your Vercel function code * `params` | Parameter | Optional | Type | Description | | ---------------- | -------- | ---------------- | ----------------------------------------------------------------- | | `configKey` | No | `string` | The Key associated with your Statsig specs in your Edge Config | | `envStatsigKey` | No | `string` | Your Statsig Client API Key | | `statsigOptions` | Yes | `StatsigOptions` | See StatsigOptions [here](/client/javascript-sdk#statsig-options) | For best practice: * Store `configKey` and `envStatsigKey` as environment variables in your Vercel project settings ### Example Usage ```javascript api/index.js theme={null} import { handleWithStatsig } from "@statsig/vercel-edge"; export const config = {runtime: 'edge'}; async function myHandler(request, client){ const user = { userID: Math.random().toString().substring(2, 5) }; //Generates a random user id const passed = client.checkGate('test_vercel_edgeconfig', user); client.logEvent('vercel_wrapper', user, passed.toString()); return new Response( JSON.stringify({ passed, user }) ); export default handleWithStatsig(myHandler,{ configKey : process.env.EDGE_CONFIG_KEY, statsigSdkKey: process.env.STATSIG_KEY }) ``` The `handler` parameter is **your Vercel function code** .This is the same code you would normally export directly in your API route (for example, `myHandler` in the snippet above).\ Instead of exporting it, you pass it into `handleWithStatsig`, which takes care of the Statsig setup and cleanup for you. **Here’s what happens behind the scenes:** 1. The Statsig SDK initializes a client using the config specs stored in your Edge Config. 2. Your function code (the handler) runs as usual. 3. Any events you log are automatically flushed back to Statsig when execution finishes. **Your Vercel function behaves exactly the same, but you no longer need to manually handle Statsig initialization or event flushing.** **That's it!** The helper automatically: * Initializes the Statsig Client with config specs from your Edge Config * Executes your Vercel function code (Your business logic + Statsig usage) * Flushes all events after your handler completes execution * Cleans up resources ## Advanced Implementation **Use the advanced/manual setup if:** * You need fine-grained control over initialization timing * You need fine-grained control over event flushing timing * You need to customize error handling behavior ### Prerequisites * Completed the [Statsig Vercel integration setup](#configure-integration) ```bash theme={null} npm install @statsig/vercel-edge ``` ```bash theme={null} import {StatsigVercelClient} from '@statsig/vercel-edge' ``` ```javascript theme={null} const client = new StatsigVercelClient(process.env.STATSIG_KEY) ``` The client instantiation takes two arguments: * `sdkKey : string` This is your Statsig client API key. It is available from the [Project Settings](https://console.statsig.com/api_keys) page in the Statsig Console. This is used to authenticate your requests. * `options : StatsigOptions` See here, for more [options](/client/javascript-sdk#statsig-options). For best practice: * Store `sdkKey` as an environment variable in your Vercel project settings The following line initializes the client by loading feature gate and experiment configurations directly from your Vercel Edge Config. ```javascript theme={null} const init = await client.initializeFromEdgeConfig(); ``` The client initialization takes one argument: * `ConfigKey : string` The Key associated with your Statsig specs in your Edge Config ```javascript theme={null} const GateResult = client.checkGate('pass_gate', user); ``` This is a gate check in code. The `checkGate` method takes two arguments: * `name : string` The name of the Statsig gate that you are checking. * `user : StatsigUser` The Statsig user object for whom the gate is being checked. For more information on the user object, see [here](/sdks/user#introduction-to-the-statsiguser-object). Refer to the [Javascript on device evaluation sdk documentation](/client/jsOnDeviceEvaluationSDK) for how to check other entities like experiments and dynamic configs. ```javascript theme={null} client.logEvent('gate_check', { userID: randomUserId }); ``` This is an event log in code. The `logEvent` method takes two parameters: * `eventOrName : string | StatsigEvent` This is the name and details of the event you are logging. * `user : StatsigUser` The Statsig user object for whom the event is being logged. For more information on event logging, see [here](/client/jsOnDeviceEvaluationSDK#logging-an-event). ```javascript theme={null} waitUntil(statsig.flush()); ``` This flushes all events from the SDK to Statsig. **Without this, you will not be able to get diagnostic information in the Statsig Console, nor any event data you logged**. ### Putting it all together ```javascript api/index.js theme={null} import { StatsigVercelClient } from '@statsig/vercel-edge'; import { waitUntil } from '@vercel/functions'; export const config = { runtime: 'edge', }; export default async function handler(request) { const client = new StatsigVercelClient(process.env.STATSIG_KEY); let init = await client.initializeFromEdgeConfig(process.env.EDGE_CONFIG_KEY); const user = { userID: Math.random().toString().substring(2, 5) }; const passed = client.checkGate('pass_gate', user); client.logEvent('vercel_event', user, passed.toString()); waitUntil(client.flush()) return new Response( JSON.stringify({ passed, user }), ); } ``` ## Other Considerations ### Polling for updates v5.13.0+ The SDK cannot poll for updates across requests since Vercel Edge Functions do not allow for timers outside of the request handler. To optimize for edge use cases, we do not provide an api to recognize updates to your config specs. However, when a change is made to your project definition on the Statsig console, the changes will be propagated to your Edge Config and will be reflected the next time you initialize the Statsig client. ### Flushing events v4.16.0+ The SDK enqueues logged events and flushes them in batches. In order to ensure events are properly flushed, we recommend calling flush using `waitUntil()` from `@vercel/functions`. This will keep the request handler alive until events are flushed without blocking the response. ``` waitUntil(client.flush()); ``` ### Size Limits Vercel Edge Config has maximum size limits that may prevent Statsig from pushing configs into your Edge Config. See [here](https://vercel.com/docs/concepts/edge-network/edge-config/edge-config-limits) for the latest Vercel Edge Config limits. ### Unsupported Features Statsig ID Lists are not currently synced into Vercel Edge Config. If you rely on large (>1000) ID lists, you will not be able to check them in your Vercel edge functions. This is why we set `initStrategyForIDLists: 'none'` in the SDK initialization. If you want to check on the evaluations you are getting, you can go to the gate you created for this example and look at the evaluations in the Diagnostics tab. Diagnostics Stream If you want to check the events you logged, in the **Statsig Console**, go to **Data** -> **Events** vercel_event example in Statsig showing lineage chart and log stream table And there you have it - a working Vercel Edge Config integration for Statsig. ## Using Flags SDK in NextJS If you're using NextJS in your Vercel project, you can use Statsig through Flags SDK and take advantage of built in precompute patterns for improved performance. See the [Statsig Adapter for Flags SDK docs](https://flags-sdk.dev/docs/api-reference/adapters/statsig) for steps on how to get started. Note that the marketplace app sets all required environment variables for the Flags SDK by default for a quick setup process. ## Sending logs to Statsig You can connect your Vercel logs to Statsig with a Log Drain to start exploring them in Logs Explorer. 1. From the [Vercel dashboard](https://vercel.com/), go to **Settings -> Drains** and click **Add Drain -> Integration**. 2. Select **Statsig**, follow the configuration steps provided, and choose a project to connect with the service. 3. Navigate to [Statsig's Logs Explorer](https://console.statsig.com/logs) to see your logs flow through. # Cloudflare Workers AI Source: https://docs.statsig.com/integrations/workersai Integrate Statsig with Cloudflare Workers AI to run experiments on AI workloads at the edge with low-latency feature gate and experiment evaluation. ## Statsig Cloudflare Workers AI Integration By integrating Statsig with Cloudflare Workers AI, you can easily conduct experiments on different prompts and models, and gather real-time analytics on model performance and usage. Statsig provides the tools to dynamically control variations, measure success metrics, and gain insights into your AI deployments at the edge. For generic setup of Statsig with Cloudflare Workers (including KV namespace configuration and SDK installation), please refer to our [Cloudflare Workers Integration documentation](/integrations/cloudflare). For setting up Workers AI itself, please refer to the [Cloudflare Workers AI documentation](https://developers.cloudflare.com/workers-ai/). ### Vision When you deploy a Cloudflare Worker running AI code, Statsig can automatically inject lightweight instrumentation to capture inference requests and responses. Statsig can track the key metadata for each request (models, latency, token usage), but you can include any others you find valuable (success rates, user interactions, etc). This integration empowers developers to: * **Experimentation:** Easily set up experiments (e.g., prompt “A” vs. prompt “B”, llama vs deepseek models) and define success metrics (conversion, quality rating, user retention). Statsig dynamically determines which variation each request should use, ensuring statistically valid traffic splits. * **Real-time Analytics:** The integrated Statsig SDK sends anonymized event data (model outputs, user interactions, metrics) back to Statsig’s servers in real time. Data is gathered at the edge with minimal overhead, then streamed to Statsig for fast analysis. ### Use Case 1: Prompt and/or Model Experiments This use case demonstrates how to use Statsig experiments to test different prompts and AI models within your Cloudflare Worker. For the sake of this example, we have 4 groups in our experiment. A control, with our default prompt and llama model, and then each possible variant switching to a different prompt and/or model (deepseek, in this case). #### Sample Experiment Setup in Statsig Console prompt and model experiment See the sample code below for the experiment implementation in a Cloudflare Worker with AI. ### Use Case 2: Model Analytics Beyond experiments, the logging mechanism illustrated below provides valuable insights into your AI model's performance and usage patterns. You could keep the default parameters for models and prompts and still get insights from the metadata you log to Statsig. #### What to track for Model Analytics: * **Latency (`ai_inference_ms`):** Crucial for understanding user experience. You can monitor average, P90, P99 latencies in Statsig. * **Model Usage (e.g., `prompt_tokens`, `completion_tokens`):** If your AI provider returns token counts, logging these allows you to track cost and efficiency. * **Error Rates:** Log events when the AI model returns an error or an unexpected response. * **Output Quality (via custom events):** * **User Feedback:** If your application allows users to rate the AI's response (e.g., thumbs up/down), log these as Statsig events. * **Downstream Metrics:** Track how the AI's output influences key business metrics (e.g., conversion rates if the AI is generating product descriptions, or user engagement if it's a chatbot). #### How to view Model Analytics in Statsig By consistently logging these metrics, you can create custom dashboards in Statsig to monitor the health and effectiveness of your AI models in real-time. This allows you to identify performance bottlenecks, cost inefficiencies, and areas for improvement. For instance, within minutes of adding the logging from the example below to your function, you can start to see the breakdown of latency per model with a query like this: metrics explorer #### Example Worker Code for Prompt/Model Experimentation and Analytics ```typescript theme={null} import { CloudflareKVDataAdapter } from 'statsig-node-cloudflare-kv'; import Statsig from 'statsig-node'; import { StatsigUser } from 'statsig-node'; export default { async fetch(request: Request, env: Env, ctx: ExecutionContext): Promise { await initStatsig(env); // ideally, use a logged in userid. In this example, I have the RayID from cloudflare const rayID = request.headers.get('cf-ray') || ''; const user = { userID: rayID, }; const promptExp = Statsig.getExperimentSync( user, "workers_ai_experiment", // Name of your experiment in Statsig Console ); // fetch the prompt and model to use for this ray ID // providing default values in case of failure to initialize statsig from the kv store const prompt = promptExp.get("prompt", "What is the origin of the phrase Hello, World"); const model = promptExp.get("model", "@cf/meta/llama-3.1-8b-instruct"); const start = performance.now(); const response = await env.AI.run(model, { prompt, }); const end = performance.now(); const aiInferenceMs = end - start; logUsageToStatsig(user, model, response, aiInferenceMs); ctx.waitUntil(Statsig.flush(1000)); return new Response(JSON.stringify(response.response)); }, } satisfies ExportedHandler; /** * Logs AI model usage and performance metrics to Statsig. * @param user The StatsigUser object. * @param model The name of the AI model used. * @param response The response object from the AI model (expected to contain a 'usage' field). * @param aiInferenceMs The time taken for AI inference in milliseconds. */ function logUsageToStatsig(user: StatsigUser, model: string, response: any, aiInferenceMs?: number) { const metadata = { ...(response?.usage || {}), ai_inference_ms: aiInferenceMs, }; Statsig.logEvent(user, "cloudflare_ai", model, metadata); } /** * Initializes the Statsig SDK. * Make sure you have the right bindings configured for the KV, and a secret for the Statsig API key * Refer to https://docs.statsig.com/integrations/cloudflare for more details on integrating Statsig with Cloudflare workers * @param env The Workers environment variables. */ async function initStatsig(env: Env) { const dataAdapter = new CloudflareKVDataAdapter(env.STATSIG_KV, 'statsig-YOUR_STATSIG_PROJECT_ID'); // Replace with your actual project ID await Statsig.initialize( env.STATSIG_SERVER_API_KEY, // Your Statsig secret key { dataAdapter: dataAdapter, postLogsRetryLimit: 0, initStrategyForIDLists: 'none', initStrategyForIP3Country: 'none', disableIdListsSync: true, disableRulesetsSync: true, // Optimizations for fast initialization in Cloudflare Workers }, ); } ``` **Explanation:** 1. **`initStatsig(env)`**: This function initializes the Statsig SDK using the `CloudflareKVDataAdapter` to fetch configurations from Cloudflare KV, ensuring low-latency access to your experiment setups. Make sure to replace `'statsig-YOUR_STATSIG_PROJECT_ID'` with your actual Statsig project ID and configure `STATSIG_SERVER_API_KEY` and `STATSIG_KV` as environment variables in your Worker. 2. **`Statsig.getExperimentSync(...)`**: This is the core of the experimentation. It retrieves the assigned experiment variant for the current user (based on `rayID`) for the `workers_ai_experiment` experiment. The `get()` method then safely retrieves the `prompt` and `model` parameters defined in your Statsig experiment, falling back to default values if the experiment or parameter is not found. 3. **`env.AI.run(model, { prompt })`**: This executes the AI model provided by Cloudflare Workers AI with the dynamically chosen `model` and `prompt`. 4. **Latency Measurement**: `performance.now()` is used to capture the start and end times of the AI inference, allowing you to track the `ai_inference_ms` metric. 5. **`logUsageToStatsig(...)`**: This function logs a custom event (`cloudflare_ai`) to Statsig. It includes the `model` used as the event value and attaches metadata such as `ai_inference_ms` and any `usage` information (e.g., token counts) returned by the AI model. This data is crucial for analyzing model performance and cost. 6. **`ctx.waitUntil(Statsig.flush(1000))`**: This ensures that all logged events are asynchronously sent to Statsig before the Worker's execution context is terminated, without blocking the response to the user. ### Other Use Cases enabled by this Integration * **Prompt Tuning:** An e-commerce app running on Workers AI tries two different prompt styles for product descriptions. Statsig tracks cart conversion and time on site, revealing which prompt yields higher sales. * **Model Selection:** A developer tests GPT-3.5 vs. GPT-4 within Cloudflare Workers AI. Statsig shows which model, combined with specific temperature or frequency penalty values, generates more accurate or user-satisfying results. * **Response Latency vs. Quality:** By varying max token length and frequency penalties within an experiment, Statsig helps optimize for speed without sacrificing accuracy, crucial for user-facing chat applications. * **Cost Optimization:** Monitor `prompt_tokens` and `completion_tokens` by model and prompt variant to identify the most cost-effective AI configurations. # Metrics Overview Source: https://docs.statsig.com/metrics/101 Learn the fundamentals of setting up essential product metrics in Statsig, including events, user counts, retention, and how metrics power experiments. **Warehouse Native users**: You're viewing the Cloud docs for this page. Metrics and experiments behave differently in Warehouse Native. Read more in [Data & Semantic Layer in Warehouse Native](/statsig-warehouse-native/configuration/data-and-semantic-layer). This 101-level user guide steps through the basic concepts to help you set up essential product metrics in your Statsig Project. 1. [How Metrics Work on Statsig](/metrics/how-metrics-work) 2. [Raw Events](/metrics/raw-events) * [Types of Raw Events](/metrics/raw-events#types-of-raw-events) * [Unit Identifiers](/metrics/raw-events#unit-identifiers) required for raw events * [ID Mapping Considerations](/experiments-plus/create-new#id-mapping-capabilities) for cross-ID analysis * [Ingesting Raw Events](/metrics/raw-events#ingesting-raw-events) * [Seeing Raw Events in the Statsig Console](/metrics/raw-events#raw-events-in-console) 3. [Auto-generated Events](/metrics/raw-event-metrics) * [Event Count](/metrics/raw-event-metrics#event-count-metric) and [Event DAU](/metrics/raw-event-metrics#event-dau-metric) metrics * [User Accounting](/metrics/raw-event-metrics#user-accounting-metrics) 4. [Custom Metrics](/metrics/custom-metrics) 5. [Importing Precomputed Metrics](/metrics/precomputed-metrics) 6. [Pulse Metrics](/metrics/pulse) **Ask for Help**: Hop on to the [Statsig Slack channel](https://statsig.com/slack) if you have any questions or want to validate the best path to import your metrics. # Metrics 201 - Creating Custom Metrics Source: https://docs.statsig.com/metrics/201 Create custom metrics, organize them with tags, and customize DAU definitions in Statsig as your product analytics project grows beyond the basics. In this 201-level user guide, you'll learn to create your own metrics and organize them as your project grows. You'll also be able to customize the definition of your daily active users (DAU). 1. [Creating Custom Metrics](/metrics/create) for your product in the Statsig console 2. [Creating Metric Tags](/metrics/create-metric-tags) to organize the metrics in your Statsig Project 3. [Customizing the DAU Definition](/metrics/user) that Statsig uses to compute [User Accounting](/metrics/raw-event-metrics#user-accounting-metrics) metrics. # Metrics 301 - Real-time Analytics Source: https://docs.statsig.com/metrics/301 Advanced guide to Statsig product analytics covering real-time event analysis, multi-step user funnels, user flow visualization, and behavioral cohorts. This 301-level user guide shows you: 1. How to use [Events Explorer](/product-analytics/overview) to analyze sampled real-time events emitted by your application 2. How to create [Create User Funnels](/metrics/create-user-funnels) 3. How to create [Create User Flows](/metrics/create-user-flows) Watch this space for more real-time and out-of-box product analytics! **What's on the roadmap?** Join us on the [Statsig Slack channel](https://statsig.com/slack) to ask for previews of Statsig's upcoming features or to submit a request. Most of our roadmap is built based on requests from customers! # Archiving and Deleting Metrics Source: https://docs.statsig.com/metrics/archiving-metrics Manage the end-of-life for Statsig metrics through archiving and deletion options to keep your metrics catalog clean without breaking historical references. Statsig offers two ways to manage the end-of-life for your metrics. * **Archiving a metric**: the metric will no longer be computed, but its history will be retained for your record. Use this for when a metric is no longer relevant, but you still wish to maintain the history of it. Use case example: you can archive older versions of a metric that continues to evolve so you have a record of how the metric has evolved over time. * **Deleting a metric**: the metric will be removed from Statsig completely, including its history. Use this for when you've made a mistake, logged or imported an irrelevant metric, or created a more accurate version of a metric. Use case examples include incorrect definition, incorrect name, duplicate metric that you don't want to confuse others with. ## Archiving Metrics ### Archiving a Metric There are two ways to archive a metric: 1. In your Metric Catalog, select the metric(s) you want to archive to see a toolbar of options appear to **Archive**, **Compare**, or **Tag**. Select the **Archive** icon. Bulk Archive 2. In the Metrics Detail View page, select the "..." in the upper right-hand corner, and select **Archive**. Archive Once you select Archive, Statsig will check if this metric is used in any feature gates, experiments, or other metrics. Archive dependencies While feature gate or experiment dependencies will be shown as soft warnings (no action necessary), metric dependencies will require you to remove the dependency first, before proceeding with the archival. This is because archival of a metric stops its computation, and we don't want other dependent metric values impacted by this archival. Once you have no metric dependencies, you will start a 24-hour grace period during which you'll be able to undo the archival. In the Metrics Detail View page, you will see a new banner appear at the top of the page, indicating the start of the grace period. After the 24-hour grace period, Statsig will stop computing this metric and the Metric Detail View page will update with a new banner indicating that the metric has been archived, and Metric Value will change to a disabled state to indicate that this metric is archived. ### Implications of Archiving a Metric *As soon as the **Archive** button is clicked,* * 24-hour grace period will start * Owners of Experiments and Gates using this metric will receive an email notification to be notified of potential impact upstream *After the 24-hour grace period has ended,* * Archived metrics will no longer be computed (when the 24-hour grace period ends). * Archived metrics will not show up in your Metric Catalog search. To access all archived metrics, go to the last page(s) of your Metrics Catalog. * Archived metrics will be removed from Pulse, including any time the archived metric has been added to the Scorecard of an experiment or the Monitoring Metrics section of a Feature Gate ### Unarchiving a Metric If you mistakenly archived a metric you can undo your Archival. * *During* the 24-hour grace period: Click "undo" on the archival banner at the top of the Metrics Detail View page. Since you **Unarchived** before the grace period ended (when the metric is no longer computed), this will restore the metric to both your Metrics Catalog as well as any experiment results that include the metric. Archive Grace Period * *After* the 24-hour grace period: Either a) go to the last few pages of your Metrics Catalog, select the archived metric(s) you want to **Unarchive** to see a toolbar of options appear, and select the **Unarchive** icon OR b) in the Metrics Detail View page of an archived metric, select "Unarchive" in the banner indicating metric's archival Archive after grace period Since the grace period has ended and the metric has stopped being computed already, its calculation will restart from scratch and history will not be restored. ### Auto-Archival To combat metric clutter, Statsig offers a default auto-archival feature that cleans up metrics that have not been in-use for at least 60 days. Metric creators and admins will get a warning about a week before archival happens, at which time they can either choose to extend the metric for another 60 days or mark it as permanent. The entire process is outlined below: State graph #### How do we measure activity? Statsig counts the number of times the custom metric is used in one or more of the following components: 1. *Scorecard*: Used in experiments, pulse reports, holdouts, etc. 2. *Dashboards*: Used to build dashboards and other analytical assets 3. *Other Metrics*: Used to calculate other composite metrics If a metric is in use, it will be considered as active. You can see a summary of a metric’s usage on the metric’s main page: Metric References At the same time, we are also detecting (1) any edits to the metrics, including changing any fields in the setup or restoring a previous version, (2) adding tags to the metrics, or (3) creating or modifying an alert on the metric. Any such interaction would restart the 60 day clock. #### How to Pause / Stop Auto-Archiving Any tracked action above (adding it to a scorecard, etc) will also take the metric out of the archival queue. Outside of that, if you want to pause the archival process, you may simply extend the metric for another 60 days. We also give you an option to mark it as permanent, which takes it out of the auto-archival process entirely. We recommend this only for the most important and widely reused metrics. Banner Another way to mark it as permanent by clicking into the setup dropdown from the metrics page and selecting “Mark as Permanent” Dropdown If you’d like to turn off auto-archiving entirely for your project, you may do so in the Project Settings page Project Settings # Deleting Metrics ### Deleting a Metric To delete a metric, go the Metrics Detail View page of a metric you wish to delete, select the "..." in the upper right-hand corner, and select **Delete**. Delete metric Once you select Delete, Statsig will check if this metric is used in any feature gates, experiments, or other metrics. While feature gate or experiment dependencies will be shown as soft warnings (no action necessary), metric dependencies will require you to remove the dependency first, before proceeding with the deletion, so that other dependent metric values are not impacted by this deletion. Dependencies Once you have no metric dependencies, you will start a 24-hour grace period during which you'll be able to undo the deletion. In the Metrics Detail View page, you will see a new banner appear at the top of the page, indicating the start of the grace period. Undo delete ***Metric Deletion cannot be undone after the grace period.*** ### Implications of Deleting a Metric *As soon as **Delete** button is clicked* * 24-hour grace period will start * Owners of Experiments and Gates using this metric will receive an email notification to be notified of potential impact upstream *After the 24-hour grace period has ended,* * Deleted metrics and their history will be removed from Statsig, and cannot be restored. * Deleted metrics will be removed from Pulse, including any time the deleted metric has been added to the Scorecard of an experiment or the Monitoring Metrics section of a Feature Gate # Metrics Dashboard Source: https://docs.statsig.com/metrics/console Explore your metrics and events in the Statsig console with real-time visualizations, search, filtering, and organization tools for product analytics. Metrics are available for all unit types enabled in the project. User ID and Stable ID are provided by default and others can be added following [these steps](/guides/experiment-on-custom-id-types#step-1---add-companyid-as-a-new-id-type-in-your-project-settings). Make a selection from the drop down to view event DAU and user accounting metrics calculated based on the desired unit type. Metrics unit type selection ## Events The Metrics console allows you to visualize all the events that you have logged in Statsig. The **Events** tab shows all the events, including a real-time stream of events as they come in. Events tab view You can toggle between a list view or chart view of your events to view the trend line over time. Events chart view From here you can drill into each event and see a detailed view of the logs, broken down by each unique value that was logged. Event detailed view ## Metrics Catalog The **Metrics Catalog** tab allows you to search and tag your metrics, as well as [create custom metrics](/metrics/create). Tags enable you organize your metrics and create collections of metrics that are associated in some way. For example, you could tag a set of metrics focused on a product area, business function, business objective, and so on. You can also create a loose collection of guardrail metrics that teams check in every experiment to ensure there are causing no unexpected effects in other parts of the business. Once you create a tagged collection of metrics, you can easily pull up this set of metrics when viewing your experiment results and zoom into the context that you want to focus on. Metrics catalog Similar to the **Events** tab, you can toggle between a list view or chart view of your metrics to view the trend line over time. Metrics chart view # Count Distinct Metrics Source: https://docs.statsig.com/metrics/count-distinct Count distinct metrics in Statsig compute approximate unique value counts at the unit level using HyperLogLog++ sketches for efficient scalable estimation. ## Purpose of count\_distinct The `count_distinct` metric reports the number of unique values at unit-level. At group-level, the mean is calculated as the sum of the unit-level count distinct divided by the count of unique units exposed to the experiment. Because this metric relies on probabilistic HyperLogLog++ (HLL++) sketches, it carries a small amount of estimation error (around 0.2% - 0.3%), which could increase as the experiment window grows. By default, we set precision p = 12 for `count_distinct`. This provides good accuracy for most use cases while keeping results stable and resource usage efficient. **Beta Feature**: Sketch-based count distinct metrics are in beta. Please reach out to learn more and join our beta testing! ## For context, what are sketches? Sketches are a probabilistic summary of a large dataset that lets us answer certain queries "approximately" but very quickly and using little memory. It does this all while using very little memory (often a fixed-size or sublinear footprint, like O(log n) or even constant space regardless of dataset size). ## Core Principles of HLL++ 1. **Hashing and Leading Zeros** * Each input is hashed uniformly into a 64-bit integer. * The position of the first 1-bit in the hash (the "rank") provides an estimate of how many distinct items precede that hash in sorted order. 2. **Register Array** * The algorithm allocates *m* registers, where *m* = 2^p and *p* is the precision parameter. * Each hashed value maps to one register. The register stores the maximum rank seen so far for values mapped to it. 3. **Sparse and Dense Modes** * **Sparse mode** is used when the number of distinct items is below 2^(p+5). It keeps an exact list of non-zero registers in a compact form. This yields near-zero error for small cardinalities. * **Dense mode** is used when the count exceeds the sparse limit. The sketch is represented as a fixed array of *m* registers, each one byte in size. Error increases slightly but remains tightly bounded. 4. **Estimation and Bias Correction** * The raw estimate is computed as the harmonic mean of 2^(–register\_value) across all registers, multiplied by a constant factor (alpha\_m). * BigQuery applies empirical bias corrections and the HIP (historic inverse probability) estimator to reduce variance and worst-case error. ## Precision, Memory, and Error Relationship The precision parameter *p* controls the number of registers *m* and thus the sketch's size and accuracy: | Precision p | Registers m = 2^p | Memory per Sketch | Approximate Relative Standard Error (1 sigma) | | ----------- | ----------------- | ----------------- | --------------------------------------------- | | 10 | 1,024 | \~1 KB | 0.83 / sqrt(1,024) ≈ 2.6 % | | 12 | 4,096 | \~4 KB | 0.83 / sqrt(4,096) ≈ 1.3 % | | 13 | 8,192 | \~8 KB | 0.83 / sqrt(8,192) ≈ 0.92 % | | 15 | 32,768 | \~32 KB | 0.83 / sqrt(32,768) ≈ 0.46 % | * Memory used grows linearly with *m*. * Error decreases as the inverse square root of *m*. * To halve the error, we must quadruple the number of registers. **Examples of thresholds and steady‑state for common precisions:** * **p = 10** (*m* = 1,024 registers, \~1 KB): * Sparse limit: 2^(10+5) = 32,768 distincts (error ≈ 0%). * Plateau begins at \~5·m = 5,120 distincts. * Steady‑state error: 0.83/√1,024 ≈ 2.6%. * **p = 12** (*m* = 4,096 registers, \~4 KB): * Sparse limit: 2^(12+5) = 131,072 distincts. * Plateau begins at \~5·m = 20,480 distincts. * Steady‑state error: 0.83/√4,096 ≈ 1.3%. * **p = 15** (*m* = 32,768 registers, \~32 KB): * Sparse limit: 2^(15+5) = 1,048,576 distincts. * Plateau begins at \~5·m = 163,840 distincts. * Steady‑state error: 0.83/√32,768 ≈ 0.46%. ## Error Behavior Over Cardinality The relative error of an HLL++ sketch evolves through three phases: * **Exact Counting Phase** * Cardinality ≤ 2^(p+5). The sketch operates in sparse mode with error close to 0%. * **Bias Ramp Phase** * Cardinality between 2^(p+5) and approximately 5·m. The sketch is in dense mode. Systematic bias increases gradually from near 0 up to the maximum bound. * **Steady-State Phase** * Cardinality ≥ 5·m. Relative error stabilizes at the theoretical bound of 0.83 / sqrt(m). Once in this phase, adding more distinct items does not change the sketch's register distribution shape—so the error remains effectively constant. In other words, after enough uniques, the sketch "plateaus," and error fluctuations are limited to the estimator's minimal random variance around that fixed bound. ### Example for p = 15 (m = 32,768) * **Exact in sparse mode (≤ 1,048,576 items)** - For up to 2^(p+5) = 2^20 = 1,048,576 distinct elements, HLL++ uses a compact "sparse" representation that records individual hashes directly—resulting in an exact count (0% error). * **Transition to dense mode & rising error** - Beyond 1,048,576 distinct elements, it switches to the full register array of size m = 2^p = 32,768. In this *dense* regime, the Relative Standard Error (RSE) gradually increases from 0% to its theoretical limit. * **Asymptotic error plateau (≈ 0.46%)** - Once in dense mode, the RSE quickly converges to RSE ≈ 0.83/√m = 0.83/√32,768 ≈ 0.46%, and remains at approximately 0.46% regardless of further increases in distinct count. ## Merging Sketches Over Time or Partitions To compute distinct counts across multiple segments (for example, daily sketches), we merge sketches by taking the element-wise maximum of their registers. This operation is associative and idempotent. Merged sketches preserve the same error bound as individual sketches and do not add additional error. ## Mathematical Formulation 1. **Raw Estimate** * *M\[i]* is the value of register *i*. * *alpha\_m* is a bias correction constant dependent on *m*. This is the core HyperLogLog "raw" estimator for the number of distinct elements in a multiset. You take each register value M\_i, compute $2^{-M_i}$, sum those values, invert that sum, and multiply by m^2 (where m is the number of registers) to get a harmonic-mean-based estimate. This is then scaled by the bias-correction constant α\_m. In effect, this formula transforms the observed distribution of leading-zero counts into an approximate count of unique items. 2. **HIP Estimator** * Maintains a running sum of inverse probabilities for each new distinct element. * Produces lower variance and a smaller worst-case error constant (0.83 instead of 1.04). 3. **Relative Standard Error (RSE)** RSE ≈ 0.83/√m ## Best Practices and Limitations * Choose precision *p* based on the maximum cardinality we expect and the error tolerance. (This will be done for you so you don't have to worry about it) * Once a sketch is created with precision *p*, we cannot increase its precision later. We can only merge or downsample to lower precision. * Long keys or complex objects increase CPU and memory in sparse mode but do not affect dense mode size. * Plan for a noise floor equal to the sketch's relative error when designing experiments or setting thresholds. * Please reach out if you believe you require higher precision. ## Conclusion and Recommendations HLL++ sketches provide efficient, bounded- error approximations for `count_distinct` queries. They trade a small, predictable error for large savings in memory and compute. * Memory scales as O(m) rather than O(n). * Error scales as O(1/√m). * Merging sketches is exact for unions, with no extra error. Select the smallest precision *p* that meets your accuracy requirements to minimize storage and computation costs. ## Choosing Precision for Your Cardinality When you know the maximum number of unique items (cardinality) ahead of time, pick a precision *p* so that **5 · 2^p** exceeds that cardinality. This ensures the sketch is in its steady‑state phase, where the relative error remains at the bound of 0.83/√(2^p). * **Up to 10,000 uniques** * *p* = 12 (m = 4,096 registers) → memory \~4 KB * Steady state from \~20,480 distincts (5·m) onward * Relative error ≈ 0.83/√4,096 ≈ 1.3 % * **Up to 50,000 uniques** * *p* = 13 (m = 8,192 registers) → memory \~8 KB * Steady state from \~40,960 distincts onward * Relative error ≈ 0.83/√8,192 ≈ 0.92 % * **Up to 100,000 uniques** * *p* = 15 (m = 32,768 registers) → memory \~32 KB * Steady state from \~163,840 distincts onward * Relative error ≈ 0.83/√32,768 ≈ 0.46 % * **Up to 500,000 uniques** * *p* = 16 (m = 65,536 registers) → memory \~64 KB * Steady state from \~327,680 distincts onward * Relative error ≈ 0.83/√65,536 ≈ 0.32 % * **Up to 1,000,000 uniques** * *p* = 17 (m = 131,072 registers) → memory \~128 KB * Steady state from \~655,360 distincts onward * Relative error ≈ 0.83/√131,072 ≈ 0.23 % * **General rule** 1. Compute *m* = 2^p 2. Ensure **5·m ≥ expected cardinality** 3. Verify memory footprint (*m* bytes) fits your budget 4. Confirm relative error (0.83/√m) meets your accuracy target ## Conclusion count\_distinct gives you a simple, consistent way to measure breadth at scale. Define once, use everywhere, and merge across time and partitions with stable behavior. You get decision-ready answers to “How many unique X?” without custom ETL or one-off SQL. # Creating Custom Metrics Source: https://docs.statsig.com/metrics/create Create custom Statsig metrics from raw events using metric types such as event count, user count, sum, ratio, and funnel for experiment analysis. Custom metrics are computed by Statsig from your raw events. To create custom metrics, navigate to **Metrics** from the left-hand navigation panel, then to the **Metrics Catalog** tab. Tap on the **Create** button. Create custom metrics interface Statsig supports six types of custom metrics: | Metric Type | Description | Examples | | -------------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | Event Count | **Total count of events** filtered by the *Value*, *Metadata*, or *User Object* properties of an event type | **Add to Cart** event filtered by category type | | User Count | **Number of unique users** that trigger events filtered by the *Value*, *Metadata*, or *User Object* of an event type | **Active Users** based on their views of a product category | | Aggregation | **Sum or Average** of the *Value*, *Metadata*, and *User Object* property of an event type | **Total Revenue** | | Ratio | **Rates** (e.g. cart conversion rate, purchase rate), **Normalized Values** (e.g. sessions per user, items per cart) | **Cart Conversion Rate**, **Sessions per User** | | Funnel | **Funnels**- funnel of multiple events with conversion tracking | **Sign-up Funnel**, **Checkout Funnel** | | Count Distinct | **Count Distinct**- Number of unique values for a given field, often approximated with sketches | **Unique Songs** listened per user | Statsig computes custom metrics on a per day basis for your **Metrics** dashboard, and rolled up for the duration of the experiment in your **Pulse Results** delivered with your Feature Gates and Experiments. After you create a custom metric, it will not populate until the next day (and will not backfill to previous days). Statsig will only calculate it moving forward from the creation date. Statsig offers a feature allowing users to set a daily **max cap** for specific metrics. With this capability, users can define maximum caps for various unit types associated with a metric. Any value surpassing the set cap will automatically be adjusted to match it. For instance, if you determine that purchases greater than \$10,000 per day on your E-commerce platform should not skew analysis, any transaction exceeding this threshold will be adjusted downward to this limit, ensuring the integrity of your experiment analysis. Capped metrics are available for event count and aggregation (sum) metric types. When creating your custom metric, you can preview a given custom metric's value at the bottom of the page. To view metric preview, tap **View Output Preview** to see what your metric would have looked like based on its components’ values over the last 7 days, in both chart and table form. Aggregation metric setup screen with output preview chart and table ## Examples ### 1. Event Count Metrics Here's an example of setting up a custom event metric to count the **number of add\_to\_cart events** filtered by a metadata property called *value*, which carries the price of the item added to the cart. As this example specifies the ID Type as *userID*, Statsig will compute this metric as part of the test group that the corresponding user is assigned to in an experiment. Event count metric filtering add_to_cart events with value greater than 100 If you select the ID Type as *stableID*, Statsig will compute this metric in the test group that the corresponding device is assigned to in an experiment. When you select more than ID Type, Statsig will compute this metric for each type of ID Type that you specify. ### 2. User Count Metrics The example below creates a metric to count the **number of unique users** who viewed a product in the *toys* category that was priced under \$10. User count metric configuration counting viewers of toys category under ten dollars **Time Window** Two time window options are available for user count metrics * Daily Participation Rate: It counts the total number of days that a user has the selected event, divided by the number of days the user is in the experiment. The result is a metric value between 0 and 1, which represents the probably of a user having the event on a daily basis. It works best for events that are expected to occur repeatedly for a given user. * One Time Event: This checks if a user has the selected event at least once during their time in the experiment. The result is a binary metric with value 0 or 1 for each user. This is ideal for events that are only expected once per user, such as sign up events. * Custom Attribution Window: Allows you to define a custom window after exposure to count an event towards a metric calculation. ### 3. Aggregation Metrics The example below shows a **Total Revenue** metric that sums the *value* associated with all purchase events. Sum aggregation configuration for Total Revenue metric **\[In Beta] Currency Sum** eCommerce customers can span multiple currency types and typically log payments in the currency and amounts the user actually pays. Currency sums allow for payments that are recorded in different currencies to be converted to one common currency. You will need to log two pieces of metadata in your event - a currency code and a value. Example metadata - `_{"currency_code": "USD", "currency_value": 123.45}_` Statsig supports currency sums upon request. Please ask in [Slack](https://statsig.com/slack). ### 4. Ratio Metrics The example below shows the creation of a **Cart Conversion Rate** metric. Here we use the unique users who triggered the *purchase event* as the numerator and the unique users who triggered the *add to cart* event in the denominator. Note that when calculating the numerator, we filter to only include users who also had the denominator event in the same day. So in the case of this metric, a user who only has *purchase event* on a given day without an *add to cart* on that same day will not count towards the numerator. Ratio metric setup for cart conversion rate This pattern also applies to **click through rates (CTR)** in any part of a step-wise product journey (aka funnels). Statsig recommends using unique users in both the numerator and denominator for defining these kinds of metrics. As an example, when a user reloads a page multiple times but clicks only once, this corresponds to a 100% CTR (1 out of 1). Similarly, a user who loads a page once but clicks multiple times on a button should only count as 1 out of 1. This also solves for cases where users see an important button such as "Sign-up" multiple times a day, and we would still consider it a success if they click just once. The example below shows creating a metric for **Items per Cart**. You can track the number of unique items added to a cart if you log an *add\_to\_cart* event for each item. For the numerator, select total event count. For the denominator, select unique users. As this metric is computed daily and only for users with a non-zero denominator, this metric can generate ratios such as 1/1, 2/1, 2/1, and 5/1 for individual users. When aggregated, this translates to 10/4 = 2.5 items per cart on average per day. Items per cart metric configuration using event counts **A Word of Caution**: In experimentation, ratio metrics are a frequent source of misleading information. It's possible to see an increase in **click through rate** alongside a net *decrease* in total clicks (the opposite may also happen). This situation can occur if the number of unique users viewing a button (denominator) decreases. As a best practice, Statsig recommends tracking the numerator and denominator as independent metrics when monitoring ratio indicator. Ratio metrics are often subject to statistical noise and can be tricky to use for obtaining a statistically significant result. In addition, for the numerator in ratios, we exclude units which don't have a denominator value. ### 5. Funnel Metrics You can create a custom funnel metric, from either the Custom Metrics Creation wizard in the Metrics Catalog or via the **Charts** tab. Funnel metric creation wizard **Important Note**: Statsig handles funnel metrics differently between our Cloud and Warehouse Native platforms. Funnel metrics using our Cloud platform are *always* unordered, meaning that funnel steps can be completed in any order, and they have a time window of *one day*. This means that for the metric to record a completion, all steps must be triggered by a user within 24 hours. In contrast, funnel metrics on Warehouse Native have the option to be set with strict ordering and custom time windows. ### Components of Funnel Metrics Funnel metrics have a few components: 1. **Lineage**: Surfaces the events used to generate the funnel 2. **Metric Value**: Metric value represents the overall funnel conversion rate, or the percentage of users who complete a funnel (trigger the end event) relative to all users who start the funnel (trigger the starting event) 3. **Conversion rate between stages**: This set of metrics track the percentage of users who triggered an event N relative to all users that triggered event N-1 in the funnel Funnel lineage and stage conversion breakdown view After funnels are created and populated, you can view your funnel metric much like any other metric in Pulse. Additionally, you can expand the funnel metric to view Pulse performance at each step in the funnel. In the example below, the **Square** variant shows a lift in the **overall funnel conversion rate**. Expanding the metrics to examine the entire funnel reveals two key insights: * Both the **Square** and **Circle** variants show a lift in top-of-funnel DAU (*Land Page View Start DAU*). However, only the **Square** variant shows statistically significant increase in end-of-funnel DAU (*Purchase Event End DAU*). * The overall funnel conversion rate improvement for **Square** is primarily due to the higher conversion from *Checkout Event* to *Purchase Event* stages in the funnel. Funnel experiment results showing Square variant performance ### 6 Count Distinct Metrics Sketch-based count distinct metrics are currently in beta, please reach out to Statsig support if you would like this enabled. * **What it is:** A high-performance way to estimate the number of distinct values using compact sketch data structures. Example use cases include trying to determine unique songs per user or unique products purchased by user. * **Benefit:** Delivers much faster computation and uses significantly less memory compared to full exact distinct counting, especially at high cardinality. ### Custom Metrics & Dimensions For Custom Metrics that are composed of a single event (event count/ event dau, aggregation, etc.), the dimensions of the source event will be automatically pulled through into the Custom Metric and be exposed as dimensions of that Custom Metric. However, if your Custom Metric is composed of 2+ different events, any associated dimension(s) of the source input events will *not* be pulled through as dimensions of the Custom Metric and are also not queryable today via the "Explore" tab in Pulse Results. # Creating Metric Tags Source: https://docs.statsig.com/metrics/create-metric-tags Organize and group Statsig metrics using tags to create reusable collections for easier experiment scorecard setup, monitoring, and team-level views. # Tagging Metrics To create a metric tag, click on **Metrics** in the left hand navigation menu, and click on the [Metrics Catalog](https://console.statsig.com/4TLCtqzctSqusYcQljJLJE/metrics/metrics_catalog) tab. Metrics catalog navigation In the **Metrics Catalog** tab, click on the **Tags** filter icon in the upper right-hand corner, and then tap **Manage** in the drop-down. This will take you to the tag manager within Project Settings for all tags in your project. Tags filter management Tap on the **Create New Tag** in the upper right-hand corner of the tag manager tab to create a new tag. Enter a name and description for your new tag. Click on **Create**. Create new tag dialog From now on, you can quickly add this tag to any metric using the **+** icon when you hover on the metric. To see what metrics are associated with a given tag, you can reference the **Tags** tab within Project Settings. Metrics catalog with tag management interface Once you've tagged your metrics, you can zoom into **Metric Lifts** in Pulse for the tagged metrics to focus on the results that matter the most to you. Pulse results showing tagged metric lifts ## Core Tag As part of every Statsig Project, a **Core** tag is auto-created and pre-populated with a subset of your app’s User Metrics. User Metrics are a suite of classic growth metrics, such as MAU, WAU, DAU, L7, 28d retention, etc. that Statsig calculates daily based on your logged events. The following User Metrics are added to the Core tag by default (though you can remove them or swap them for other, more relevant business metrics as needed): * DAU/ WAU/ MAU * New users * Daily/ Weekly/ Monthly user stickiness The Core tag is meant to serve as a collection of your most important business metrics, which you want to monitor with every new feature rollout and experiment. The Core tag is automatically added to all new feature gates and experiments at the point of creation. In an experiment, the Core tag is added to the Scorecard, under **Secondary Metrics**. In a feature gate, the Core tag is added to the **Monitoring Metrics** section. ## Configuring your Core Metrics Given the special treatment the Core tag receives, it is recommended to spend a bit of time curating the set of metrics included in this tag collection at Project set-up. To do this, go to **Metrics** → **Metrics Catalog** and add the **Core** tag to any relevant metrics. To see which metrics already have the **Core** tag applied, filter by the **Core** tag in the Metrics Catalog by tapping the filter icon in the upper right-hand corner, then **Tags**, then selecting the **Core** tag.  Remove the Core tag from any unwanted metrics in the resulting filtered listview. Core metrics filter view ## Setting Rollout Alert Thresholds for your Core Metrics Often when rolling out a new feature or experiment, you may want to be notified if any new rollouts or experiments negatively impact a key business metric. Statsig enables you to set Rollout Alerts at the per-metric level, whereby you will be alerted if any currently running experiment or feature gate regresses the metric beyond the set threshold. Given the importance of Core Metrics, it is recommended to configure Rollout Alerts for Core Metrics via the metrics Catalog. For more on Rollout Alert configuration, see [here](/metrics/rollout-alerts). # User Flows Source: https://docs.statsig.com/metrics/create-user-flows Visualize customer journeys through your application using Statsig User Flows to see common paths users take and where they drop off between events. Statsig's **User Flows** enable you to visualize customer journeys through your application. To create a **User Flow**, 1. Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) 2. On the left-hand navigation panel, select **Metrics** and then click on the **Charts** tab 3. Click on the **Create** button 4. Enter the chart name and start event of the User Flow that you want to visualize 5. Select the additional events that you want to include in the User Flow (Statsig will automatically determine the sequence of events) 6. Enter the number of days that users typically take to complete these events 7. You can also optionally enter the depth of the flow to define the scope of your analysis 8. Click on **Create** to finish User flow creation interface Assuming there's sufficient data for the User Flow, you'll see the chart appear in the console within a few minutes. # Funnel Metrics Source: https://docs.statsig.com/metrics/create-user-funnels Create custom funnel metrics in Statsig to track user conversion rates through multi-step processes like signup, checkout, and onboarding flows. You can create a custom **funnel metric**, from either the **Custom Metrics** creation wizard in the **Metrics Catalog** or via the **Charts** tab. Funnel metric creation interface ## Components of Funnel Metrics Funnel metrics have a few components: 1. **Lineage**: Surfaces the events used to generate the funnel 2. **Metric Value**: Metric value represents the overall funnel conversion rate, or the percentage of users who complete a funnel (trigger the end event) relative to all users who start the funnel (trigger the starting event) 3. **Roll-up Window**: Funnel metrics are calculated on a daily basis. 4. **Conversion Rate between Stages**: This set of metrics track the percentage of users who triggered an event N relative to all users that triggered event N-1 in the funnel Funnel components diagram After funnels are created and populated, you can view your funnel metric much like any other metric in Pulse. Additionally, you can expand the funnel metric to view Pulse performance at each step in the funnel. ## User-based Funnel Metrics When counting distinct events, funnel metrics add the number of events per day over the analysis period. When counting distinct users, funnel metrics add the number of distinct users per day over the analysis period. Suppose a funnel consists of events A, B, and C, in that order. User-based funnel metrics count the number of distinct users who triggered events A, B, and C on a given day. If you're tracking funnel conversion over multiple days, the daily granularity of funnel metrics may not be a good fit for your analysis. ## Example In the example below, the **Square** variant shows a lift in the **overall funnel conversion rate**. Expanding the metrics to examine the entire funnel reveals two key insights: * Both the **Square** and **Circle** variants show a lift in top-of-funnel DAU (*Land Page View Start DAU*). However, only the **Square** variant shows statistically significant increase in end-of-funnel DAU (*Purchase Event End DAU*). * The overall funnel conversion rate improvement for **Square** is primarily due to the higher conversion from *Checkout Event* to *Purchase Event* stages in the funnel. Funnel experiment results showing Square variant performance # Custom DAU Metric Creation Guide Source: https://docs.statsig.com/metrics/custom-dau Step-by-step guide to creating custom Daily Active User (DAU) metrics in Statsig tailored to your product definition of an active user and key actions. # Custom DAU Metric Creation Guide This guide will walk you through the steps to create a custom DAU metric. Follow the instructions carefully to ensure successful creation of your metric. ### **Step 1: Navigate to the Metrics Catalog** To begin, go to the "Metrics Catalog" in the left navigation bar and click on "Create" button. Picture1.png Picture2.png ### **Step 2: Name your metric** Name your metric. For example, I am naming it "Add to Cart DAU". Picture3.png ### **Step 3: Choose the metric type** Since we want to create a DAU metric, choose the metric type as "Unit Count". Picture4.png ### **Step 4: Select the ID type** Select the ID type you want to count. Here I am selecting "User ID" which it is the most common use-case for DAU metrics. Picture5.png ### **Step 5: Choose the event** Next, choose the event that you want to use for your metric. In this case, I select the event "Add to Cart". Picture6.png ### **Step 6: Submit the metric** Submit the metric and your custom DAU metric is now created! Picture7.png # Custom Metrics Source: https://docs.statsig.com/metrics/custom-metrics Create custom Statsig metrics by filtering and aggregating events based on event metadata and properties, including value, country, and platform filters. # Custom Metrics You can create **Custom Metrics** using the custom events you ingest by filtering or aggregating events based on event metadata. You must include metadata along with the custom events when you log these events with Statsig. For example, in addition to tracking overall **event\_count** for all events of type *purchase\_event*, you may want to filter these metrics only for events where your users purchase a specific product category. Statsig supports six types of custom metrics: | Metric Type | Description | Examples | | -------------- | ------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | Event Count | **Total count of events** filtered by the *Value*, *Metadata*, or *User* properties of an event type | **Add to Cart** event filtered by category type | | User Count | **Number of unique users** that trigger events filtered by the *Value*, *Metadata*, or *User* properties of an event type | **Active Users** based on their views of a product category | | Aggregation | **Sum or Average** of the *Value*, *Metadata*, or *User* property of an event type | **Total Revenue** | | Ratio | **Rates** (e.g. cart conversion rate, purchase rate), **Normalized Values** (e.g. sessions per user, items per cart) | **Cart Conversion Rate**, **Sessions per User** | | Funnel | **Funnels**- funnel of multiple events with conversion tracking | **Sign-up Funnel**, **Checkout Funnel** | | Count Distinct | **Count Distinct**- Number of unique values for a given field, often approximated with sketches | **Unique Songs** listened per user | Sketch-based count distinct metrics are in beta. Please reach out to Statsig support to join our beta. It's worth noting that the "average" in aggregation is the average of event value (average revenue per purchase per user), instead of the average of exposed units (average revenue per user). The latter is defined by sum. See **Metrics 201** topic, [Creating Custom Metrics](/metrics/create), to learn how to create custom metrics for your product. # Deprecating Event_dau Metric Source: https://docs.statsig.com/metrics/deprecate-event-dau Important changes to auto-generated event_dau metrics in Statsig and how to migrate to user count metrics to maintain DAU tracking for your custom events. ### Deprecation Details From Wednesday, October 16, 2024, we will stop auto-generating new event\_dau metrics for incoming events into Statsig. We will continue to auto-generate an event\_count metric for each logged event as we do today. **Note: This change will only affect Statsig Cloud customers, Warehouse Native customers will not be impacted.** * Any existing event\_dau metrics that have been used in a gate, experiment, dashboard, other Custom Metrics will NOT be affected by this change. * Existing event\_dau metrics that have been archived or never been used in another config will NO longer exist. See ‘Next Steps’ if you want to retain these metrics. * Going forward, new event\_dau metrics will need to be created manually as a Custom Metric. See [this guide](/metrics/custom-dau) to learn how to create a DAU metric. If you have any questions or concerns, please don’t hesitate to reach out! ### Motivation for This Change Historically, we have auto generated an event\_count and event\_dau metric for every incoming event into Statsig. After working closely with hundreds of customers, we have seen that auto generating two metrics for every event causes confusion and clutter inside projects. The proposed change will lead to cleaner Metrics Catalog and faster Console performance, while still retaining your ability to create event\_dau metrics for the events you care about most. ### Next steps If you wish to keep any unused event\_dau metrics going forward, you can earmark those metrics by performing any of the actions below: * Adding a Tag (RECOMMENDED) * Adding a description * Referencing in a gate/experiment/dashboard These actions will mark your unused metric as active, signaling us that you don’t want them to be deprecated. # Running Analysis Across Unit Types, AKA Cluster Experiments Source: https://docs.statsig.com/metrics/different-id Run Statsig experiment analysis when the assignment unit differs from the analysis unit, including how to configure cross-ID mappings and interpret results. There are two common scenarios where the experiment assignment unit differs from the analysis unit: 1. Measuring session-level metrics for a user-level experiment. Ratio metrics are commonly used to solve this (this doc). 2. Measuring logged-in metrics (eg. revenue) on a logged-out experiment. There are two solutions: a. Running the experiment at the [device-level](/guides/first-device-level-experiment), with device-level metrics collected even after the user is logged-in. b. Using [ID resolution](/statsig-warehouse-native/features/id-resolution). We will explain how to set up the first scenario with Warehouse Native in this doc. Workflow diagram for analyzing metrics at different ID levels ## Example: Organizations and Users Scenario: * Your metrics source has both `org_id` and `user_id`. * The relationship between `org_id` and `user_id` is 1-to-many. A single `org_id` can be associated with multiple users (`user_id`), but a `user_id` is only associated with a single `org_id`. * Your experiment is assigned at the `org_id` level. * You are interested in understanding the treatment effect at the `user_id` level, such as revenue per user. ### 1. Setup the metric source with `org_id` as an ID type. * In this table, each row of data should have both `org_id` and `user_id`. Metric source table setup with org_id and user_id fields ### 2. Choose your assignment source, where the unit of assignment is `org_id`. Assignment source configuration selecting org_id unit ### 3. Define your metric of revenue per `user_id`. * Your denominator should be `count distinct user_id` instead of `unit count`, because the latter is equivalent to `count distinct org_id` in an `org_id` level experiment. Metric definition screenshot showing revenue per user_id formula ### 4. Set up the experiment with `org_id` Experiment setup specifying org_id as unit type ## Statistics in the backend In the Stats Engine, we utilize the delta method to calculate variance and confidence intervals. * For mean metrics, we record a value indicating the number of observations per exposed unit in the records column of the staging data. This acts as the denominator or cluster-size value for delta calculations. * For general ratio metrics, we monitor the two-component metrics (the ratio and the denominator) as independent metrics and combine them during the pulse analysis to derive a single metric from them. For more information on how we apply the delta method, visit: [Statsig - Delta Method Methodology.](/stats-engine/methodologies/delta-method). The reason we choose to use the delta method is to account for the covariance between the numerator and the denominator (i.e. more users per org is correlated with more revenue). See section 3 of [this paper](https://alexdeng.github.io/public/files/kdd2018-dm.pdf) for details. This approach is also relevant for analyzing event-level outcomes, such as average purchase value, where randomization occurs at the user level, and each user may experience multiple session events. # Metric Directionality Source: https://docs.statsig.com/metrics/directionality Configure metric directionality in Statsig so increases or decreases in metric values are correctly labeled positive or negative in experiment results. # Setting Metric Directionality By default, increases to a metric value are assumed to be 'good', and are represented by a green color in Experiment results. However for many metrics, it’s actually a good thing when the metric decreases and similarly bad to see an increase (e.g. page load times or many performance metrics) For these types of metrics, you can set the desired direction you want to see a metric move and experiment results will update the color-coding in metric lifts accordingly. To do so, go to any metric detail view page and click the Edit Pencil next to the 'Directionality' section in the right rail of the page to see the “Set Metric Directionality” option. Metric directionality configuration interface # How Metrics Work on Statsig Source: https://docs.statsig.com/metrics/how-metrics-work Understand the fundamentals of how Statsig calculates metrics, including event ingestion, aggregation, exposure joins, and how they power experiment results. # How Metrics Work on Statsig **Warehouse Native users**: You're viewing the Cloud docs for this page. Metrics and experiments behave differently in Warehouse Native. Read more in [About Warehouse Native](/statsig-warehouse-native/introduction#how-warehouse-native-works). A metric in Statsig is a numeric value for each user on a given day. This value can be aggregated across the entire user base or a subset, such as the test or control group of an experiment. For example, say one user made two purchases on September 1st, and another made only one. These values can be aggregated across multiple users to calculate the total number of purchases across all users on September 1st. By default, Statsig computes metrics from logged raw events in the production environment. When testing experiments in lower environments (such as development or staging) with **Enable for Environments**, you can track cumulative exposures and metric results collected from those environments. This allows you to validate your experiment setup before launching to production but production data is prioritized for final Pulse result analyses. ## Two Sources of Statsig Metrics There are two fundamental sources of metrics in Statsig: 1. **Raw Events** * Statsig [auto-generates certain metrics](/metrics/metrics-from-events) such as **event\_count** and user accounting metrics from these events * You can also define [custom metrics](/metrics/create) using your logged raw events 2. **Precomputed Metrics** - You can provide these pre-computed values to Statsig Statsig's Stats Engine joins these metrics with your exposure events from feature gates and experiments to compute experiment results and analytics. Statsig data flow architecture diagram **How are events and metrics billed?** Each event (or a row when importing from your data warehouse) is billed once, regardless of how many experiments the event is used in. # Ingesting Metrics Source: https://docs.statsig.com/metrics/ingest Import precomputed metrics from Snowflake, BigQuery, Redshift, and other data warehouses into Statsig using built-in connector integrations and scheduled syncs. Statsig can ingest your precomputed product and business metrics using our data warehouse connector (Metrics Imports). Integrations like [Snowflake](/integrations/data-imports/snowflake), [BigQuery](/integrations/data-imports/bigquery) and [Redshift](/integrations/data-imports/redshift) are supported. Statsig does not automatically process these metrics until you mark them as ready, as it's possible you might land data out of order. Once you are finished loading data for a period, you mark the data as ready by hitting the `mark_data_ready` API: ``` curl --location --request POST ‘https://api.statsig.com/v1/mark_data_ready’ \ --header ‘statsig-api-key: {your statsig server secret}’ \ --header ‘Content-Type: application/json’ \ --data-raw ‘{ “timestamp”: 1647975283, “type”: “metrics” } ``` The timestamp provided should be: * A unix timestamp * Represents the latest point in time for which all metrics before have been uploaded. * Any future calls to this API with an earlier timestamp will be invalid. * We don’t guarantee correct behavior if metrics are provided with an earlier timestamp after this API is called. Statsig processes metrics as a full day in the PST timezone; we will wait until a full day is marked as ready before processing that day. ## Debugging Precomputed Metrics All precomputed metrics generate Metric Detail View pages. However, these Detail View pages take a few hours to generate post-ingestion. The fastest way to start seeing and debugging your precomputed metrics is via the **Metrics Logstream** on the **Metrics Catalog** tab within **Metrics**. Metrics catalog displaying logstream entries for ingested metrics The **Metrics Logstream** will surface all ingested, precomputed metrics in real-time as they are ingested, enabling you to check metric name, value, ID, ID type, and ingestion date. Pro tip- we often find that customers get tripped up on ensuring their precomputed metrics have the right ID type, so pay extra attention to this column! We've also exposed the ability to include test metrics, tagged with **isTest**, which you can toggle on/ off for debugging purposes in the **Metrics Logstream**. Note that this **isTest** flag is only available for precomputed metrics ingested via Statsig's APIs. Support for this flag via our integrations with Snowflake, BigQuery, and Redshift is coming soon. To mark a batch of metrics as test metrics the isTest parameter needs to be set to true as part of the data for the request as seen below. ```bash theme={null} curl \ “https://events.statsigapi.net/v1/log_custom_metric” \ --header “statsig-api-key: ” \ --header “Content-Type: application/json” \ --request POST \ --data “{"isTest": true, “metrics": [{"user_id": "1237", "metric_name": "test_metric", "id_type": "user_id", "metric_value": 90}, {"user_id": "4568", "metric_name": "ratio", "id_type": "stable_id", "numerator": 3, "denominator": 15}]}” ``` Metrics logstream UI showing isTest flag toggle Finally, it's important to note that **Metrics Logstream** only appears if you're actively ingesting precomputed metrics, so if you're not seeing it appear at the bottom of your **Metrics Catalog** this means there is likely a connection or schema issue and Statsig is not receiving your precomputed metrics. # Metrics User Guide Source: https://docs.statsig.com/metrics/introduction Overview of Statsig's metrics system, covering raw events, custom metrics, precomputed warehouse metrics, and real-time analytics for product teams. # Metrics User Guide Statsig combines data from any of your existing data sources to give you a complete view of your product metrics as well as the impact new features and experiments have on these metrics. Statsig automatically creates these metrics from the **raw events** that you log from your application as well as raw and transformed events you send to Statsig via a data collector. Statsig can also reuse your existing **precomputed metrics** by natively integrating and pulling data from your cloud data warehouse. You can also choose to ingest metrics into Statsig using an HTTP endpoint or Azure Blob storage. Data integration architecture diagram The **Metrics User Guide** consists of [Metrics 101](/metrics/101), [201](/metrics/201), and [301](/metrics/301) level guides that step through basic concepts to real-time analytics in Statsig. ## Metrics 101 - An Overview * [Introduction](/metrics/101) * [Raw Events](/metrics/raw-events) * [Raw Event Metrics](/metrics/raw-event-metrics) * [Custom Metrics](/metrics/custom-metrics) * [Precomputed Metrics](/metrics/precomputed-metrics) * [Pulse Metrics](/metrics/pulse) * [Metric Dimensions](/metrics/metric-dimensions) # Metrics 201 - Diving deeper * [Introduction](/metrics/201) * [Creating Metrics](/metrics/create) * [Tagging Metrics](/metrics/create-metric-tags) * [User Metrics](/metrics/user) * [Assignment ID != Analysis ID](/metrics/different-id) # Metrics 301 - Advanced Topics * [Introduction](/metrics/301) * [Metrics Explorer](/product-analytics/overview) * [User Funnels](/metrics/create-user-funnels) * [User Flows](/metrics/create-user-flows) Statsig combines product experimentation with real-time analytics to help you achieve 360° **product observability**. Product observability dashboard overview Get started with [Metrics 101](/metrics/101). # Local Metrics Source: https://docs.statsig.com/metrics/local-metrics Create local metrics scoped to a single Statsig experiment or feature gate without adding ad hoc metrics to the project-wide catalog. ## Overview Local Metrics are metrics that are scoped to an individual config, i.e. a specific experiment or gate. They are created in the context of this config with the goal of being able to capture how that metric trends in the context of that config, without adding that new metric to the Project-wide Metrics Catalog forever more. ## Creating a Local Metric You can create a Local Metric from two places within your config- 1. **Setup**- As you're setting up your gate or experiment and adding Primary/ Secondary/ Monitoring metrics, you may find that you want to add a metric that doesn't yet exist to your Scorecard. In this scenario, you can simply tap, **+ Create New Local Metric** to enter into the Local Metric creation flow. Setup tab showing + Create New Local Metric button 2. **Pulse**- If you've already started your gate/ experiment rollout, it's not too late to create and add a new Local Metric to your scorecard. From your Scorecard in Pulse, tap **Edit Primary Metrics** (or Secondary/ Monitoring metrics depending on where you want to add your new Local Metric), and then **+ Create New Local Metric**. Pulse scorecard with Edit Primary Metrics and Create New Local Metric options Entering into the Local Metric creation flow from either entry point will kick off the Local Metric creation wizard. If you're a heavy user of Metrics Explorer, this will feel quite familiar. The wizard allows you to select event(s), add filters, and preview the output values. Local metric creation wizard with event selection and filters When you are ready to save your Local Metric, you can choose to save it to either the Primary/ Secondary Metrics section of your Scorecard (for experiments), or the Monitoring Metrics section of your feature gate rollout. Local metric placement selector for primary scorecard Monitoring metrics section showing new local metric entry Once you've created a Local Metric, you can tap on the Local Metric in your Scorecard to view its configuration in the Local Metric wizard. ## Types of Local Metrics The types of Local Metrics you can create are very similar to Custom Metrics (accessible via the Metrics Tab), with a few exceptions. Here are the supported types of Local Metrics: | Metric Type | Description | Examples | | ----------- | --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | Event Count | **Total count of events** filtered by the *value* and *metadata* properties of an event type | **Add to Cart** event filtered by category type | | User Count | **Number of unique users** that trigger events filtered by the *value* and *metadata* of an event type | **Active Users** based on their views of a product category | | Aggregation | **Sum or Average** of the *value* of an event type | **Total Revenue** | | Ratio | **Rates** (e.g. cart conversion rate, purchase rate), **Normalized Values** (e.g. sessions per user, items per cart) | **Cart Conversion Rate**, **Sessions per User** | The one type of Custom Metric that you cannot (yet) create as a Local Metric are funnels. ## Lifecycle of Local Metrics By default, Local Metrics are scoped to the config they're created in the context of, and will only live for the lifecycle of that config. This means once you make a decision on your experiment or launch your feature gate, your Local Metric will no longer be computed. Local Metrics do NOT show up in your Project Metrics Catalog, and are not searchable in top-line search. While you can't convert a Local Metric into a "global" metric (i.e. a Metrics Catalog metric) today, this conversion flow is coming soon. In the meantime, you can recreate the same metric definition as a Custom Metric in the Metrics Catalog if you want this metric to live on in a more global capacity outside the scope of your gate/ experiment. # Event Property Source: https://docs.statsig.com/metrics/metric-dimensions Break down Statsig metrics into dimensions using event properties and metadata fields for detailed segment-level analysis in experiments and dashboards. Statsig enables you to breakdown metrics into a single set of non-overlapping dimensions for deep dive analyses. For example, you can breakdown an event such as **add-to-cart** into product categories such as *sports*, *toys*, *appliances*, *electronics*. To do this, you would simply log **add-to-cart** events and provide the product category in the event's **value** field. See the [Statsig SDK reference guide](/client/javascript-sdk#event-logging) to learn more. Statsig enables you to define up to four custom dimensions for an event (one via the **value** field and three via **metadata** fields). To configure these custom event fields, go to **Metrics** --> **Events**, select the event you want to configure, and then go to the **Setup** tab for that event. Event property configuration Providing custom dimensions with logged events allows you to break down the impact on the total **add-to-cart** events by category in Pulse as shown below. This enables you to zero in on the category that's most impacted by your experiment. Pulse results breakdown by product category Statsig recommends keeping the number of distinct dimensions in your logging less than 8. This is because tracking dimensions involves additional computational and storage resources, so we prioritize notable dimensions that make up a significant portion of your traffic. As a rule of thumb, we track dimensions that make up >5% of your total event volume. We don't track lower frequency dimensions since they generally have larger degrees of statistical noise and wider confidence intervals. We will bucket them in an "Other" bucket. Statsig also supports metric dimensions for custom metrics that are **Aggregations**. To set this up, log your dimension as a **value** and the number to be tallied as a metadata field. This is the old metric dimension page. We renamed it to Event Property. # Metric Family Source: https://docs.statsig.com/metrics/metric-family Use metric families in Statsig to manage variants of the same underlying metric, like a daily, weekly, or windowed version, from one configuration entry. It can be difficult to managing a large catalog of metrics that are a slight variant of one another. A common challenge we hear is the difficulty of ensuring that each metric variant inherit the same changes when the metrics need to be updated. Metric Families let you create metric variants as child metrics of a parent metric. When a parent metric is updated, those changes automatically cascade to its child metrics, keeping all related metric definitions consistent and in sync. The table below lists all supported metric types along with the modifications that can be applied to their child metrics. | Available Metric Type | Modifiable Configs | | | --------------------- | --------------------------------------------------------------------------------------------------------------------------------- | - | | SUM
    COUNT |
    • Add metric source filter
    • Outlier Management
    • Variance Reduction
    • Cohorts and Delayed Data
    | | ## Getting Started To create a child metric look for the “Create Child Metric” option in the drop down menu found in the top right corner of a metric. Creating a child metric You will be able to track the child metrics for a given parent metrics in the top right metric family icon Tracking tree of child metrics ## Deleting and Archiving When a parent metric is deleted or archived, all of its child metrics are deleted or archived as well. Re-enabling the parent will automatically restore those child metrics. Child metrics disabled through the parent cannot be restored individually—they can only be re-enabled by restoring the parent metric. Metric deletion confirmation message If a child metric is deleted or archived independently of its parent, it can be restored on its own—but it will return as a standalone metric, without its original parent-child relationship. ## In Experiments Clicking on a Parent Metric option in Experiment setup will add all of it's child metrics as well. You can remove any metrics you don’t need in the experiment Adding parent metric to an experiment will add all child metrics with it # Precomputed Metrics Source: https://docs.statsig.com/metrics/precomputed-metrics Import precomputed metrics from cloud data warehouses like Snowflake, BigQuery, and Redshift into Statsig for use in experiment analysis and scorecards. ## Importing Precomputed Metrics ### Importing Precomputing Metrics from your Data Warehouse Statsig integrates natively with cloud data warehouses such as [Snowflake](/data-warehouse-ingestion/snowflake), [BigQuery](/data-warehouse-ingestion/bigquery), [Redshift](/data-warehouse-ingestion/redshift) to ingest any of your existing metrics for computing experiment results. See [Data Warehouse Ingestion](/data-warehouse-ingestion/introduction) to get started. ## Debugging Precomputed Metrics Statsig creates a metric detail page for all precomputed metrics that you import from your data warehouse. These metric detail pages take a few hours to generate post-import or ingestion. The fastest way to start seeing and debugging your precomputed metrics is via the **Metrics Logstream** in the **Metrics Catalog** tab within **Metrics**. Metrics logstream showing imported precomputed metrics The **Metrics Stream** will surface all ingested, precomputed metrics in real-time as they are ingested, enabling you to check metric name, metric value, unit identifier, ID type, and ingestion date. **Tip**: Customers can trip up on ensuring that their precomputed metrics have the right ID type. Pay extra attention to this column! Finally, the **Metrics Stream** only appears if you're actively ingesting precomputed metrics. If you're not seeing it appear at the bottom of your **Metrics Catalog**, Statsig likely is not receiving your precomputed metrics due to a connection issue or an invalid schema. # Pulse Metrics Source: https://docs.statsig.com/metrics/pulse How different Statsig metric types are computed and interpreted in Pulse experiment results, including confidence intervals, lift, and significance flags. Experiments with Statsig use **Pulse** to compute and communicate results. The metric type is important in computing and interpreting the final result. Most metric types are aggregated across all users in the group; however, some metric types that use ratios are only aggregated across [participating users](/experiments/interpreting-results/participating-units) (users that have non-null value for that metric). We'll walk through the various types of metrics available in experiments and how to interpret their pulse results. ## Pulse Statistics by Metric Type | Metric Type | Total | Mean | Units | | ----------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | | event\_count | Sum of events (99.9% winsorization applied) | Average events per user (99.9% winsorization applied) | All users | | event\_dau | Sum of event DAU (distinct user-day pairs) | Average event\_dau value per user per day. Note that we call this "Event Participation Rate" as this can be interpreted as the probability a user is DAU for that event. | All users | | sum | Total sum of values (99.9% winsorization) | Average value per user (99.9% winsorization) | All users | | mean | Overall mean value | Overall mean value | [Participating users](/experiments/interpreting-results/participating-units) | | event\_user | Count of distinct users that have had the event. | Average metric value per user per day. Depending on Rollup Mode, can be a one-time event or daily participation rate. | All users | | ratio | Not shown | Overall ratio: sum(numerator values)/sum(denominator values) | [Participating users](/experiments/interpreting-results/participating-units) | | funnel | Not shown | Overall ratio: sum(numerator values)/sum(denominator values) | [Participating users](/experiments/interpreting-results/participating-units) | | user: dau, wau, mau\_28day | Not shown | Average metric value per user per day. The probability that a user is xAU | All users | | user: new\_dau, new\_wau, new\_mau\_28day | Count of distinct users that are new xAU at some point in the experiment | Fraction of users that are new xAU | All users | | user: retention metrics | Overall average retention rate | Overall average retention rate | [Participating users](/experiments/interpreting-results/participating-units) | | user: L7, L14, L28 | Not shown | Average L-ness value per user per day | All users | | count\_distinct | Total number of unique values | Average number of unique values per user | All users | **Some example metric breakdowns in Pulse:** pulse 1 pulse 2 ## Event Count and Event DAU in Pulse **event\_dau Legacy Support**: event\_dau metrics are now in legacy support only and are no longer created for new events. Existing event\_dau metrics will continue to be available for any of your new experiments and will continue to be computed daily. For all new events, you should create an event\_user metric to measure daily active users. From [Metrics 101](/metrics/101) and [Auto-generated Metrics](/metrics/raw-event-metrics), * [**event\_count**](/metrics/raw-event-metrics#event-count-metric) measures the volume of the activity based on count of events triggered * [**event\_dau**](/metrics/raw-event-metrics#event-dau-metric) measures unique daily users who triggered a given event For example, the table below shows the **event\_count** and **event\_dau** metrics for two event types,*Page Views* and *Add to Cart*, for three users over three days. Event count and event DAU metrics table Over the duration of an experiment, Pulse results measure the change in: * the **mean** event\_count, or the average event count per user * the **mean** event\_dau, or average active days per user; we call this the **Daily Event Participation Rate** For example, the table below shows the **Total event\_count**, **Total Units**, and **Mean event\_count** over the same three days as above, now in the context of an experiment. Experiment metrics table showing total event count and mean values Similarly, the table below shows the **Total event\_dau**, **Total Units**, and **Mean event\_dau** over the same three days of the experiment. Alice was 'active' on three days for the *Page View* event and on one day for the *Add to Cart* event. Therefore, average event*dau for Alice is 3/3 for the \_Page View* event and 1/3 for the *Add to Cart* event. In other words, Alice's **daily participation rate** is 1.00 for the *Page View* event and 0.33 for the *Add to Cart* event so far in the experiment. Statsig aggregates this average event\_dau for each user in the experiment, with each user weighted equally. Event DAU metrics table showing daily participation rates To measure the change in engagement for a call to action link or button, use event\_count to measure the change in average clicks per user, and use event\_dau to measure the change in users’ daily participation rate for the click. **Event Count and Event DAU in Custom Metrics**: When creating a custom ratio metric, use event\_count to include all events (counting all events triggered by the same user). Use event\_user (or event\_dau, if available) to count unique active users on a given day (all events triggered by the same user are counted as one). ## Winsorization To reduce the impact of outliers, Statsig caps *event\_count* and *sum* metric types at the 99.9th percentile (by default). This mitigates the risk of bots and extreme values significantly swaying experiment results. The winsorization 99.9th percentile is computed using all non-zero and non-null values of the metric, and then all values of exceeding this limit are replaced with it. Warehouse Native (WHN) allows for more customization of winsorization by metric and by percentile. ## Frequently Asked Questions **1. Is it possible for a ratio metric to move in the opposite direction than both the numerator and denominator metrics?** Yes, it is possible for the ratio to rise while both the numerator and denominator metrics decline. For example, this happens when the denominator is falls more than the numerator. As a best practice, Statsig recommends tracking the numerator and denominator as independent metrics when monitoring ratio metric. Ratio metrics are often subject to statistical noise and can be tricky to use for obtaining a statistically significant result. **2. For ratio metrics, how does Statsig determine *participating users*?** Ratio metrics are computed only for users that have a non-zero value in the denominator, i.e. the user must have triggered the denominator event on a given day to be included in the daily ratio. Users that don't trigger the denominator event during an experiment are not included in the test vs. control comparison of a ratio metric. **3. What is the difference in metrics between One-Time Event vs Daily Participation Rate?** The distinction between these in only relevant in the context of an experiment. Daily participation rate counts the number of *days* a user has that event, divided by the number of *days* the user has been in the experiment. One time event is a binary metric that checks whether the user has that event *at least once* during their time in the experiment. # Auto-generated Metrics Source: https://docs.statsig.com/metrics/raw-event-metrics How Statsig automatically generates event_count metrics from each custom event you log so you can monitor activity and use them in experiment scorecards. # Auto-generated Metrics Metrics are critical for monitoring the health and usage of your product as well as the impact of new features and experiments. Statsig automatically generates an "event\_count" metric for each uniquely named **custom event** that you log. Auto-generated metrics are created from production environment events, and any newly logged custom event should have an event\_count metric created within 24 hours of logging their first events. When testing experiments in lower environments (such as development or staging) with **Enable for Environments**, you can track cumulative exposures and metric results collected from those environments. This allows you to validate your experiment setup before launching to production but production data is prioritized for final Pulse result analyses. This auto-generated metric consists of three elements: 1. **Roll-up Window** - Statsig computes metrics from custom events aggregated over a 24-hour day, with the hours depending on your company's setting. These hours do not change with daylight saving time. This prevents some days from having 23 and 25 hours which can cause a +/-4% change to some metrics on a biannual basis. 2. **Unit Identifier** - While you can record custom events with and without a unique user identifier, Statsig requires a unit identifier (usually a user\_id) to track a user across multiple events and sessions to support Experiments, Pulse (experiment results), and Autotune. If you don't have access to a user\_id when logging a custom event, create a temporary identifier to track users at a session or device-level. 3. **Metric Value** - Statsig automatically computes values for **event\_count**, which measures the number of times an event is triggered. Up until October 16, 2024, Statsig also auto-computed values for an **event\_dau** metric that measures the number of unique users that triggered the event. While Statsig no longer auto-compute an **event\_dau** metric for every logged event, you can create your own metrics that function like **event\_dau** via [Custom Metrics](/metrics/custom-dau). Please see the [event\_dau deprecation details](/metrics/deprecate-event-dau) for more information. | Metric | Automatic | Dimensions | Possible Values | Description | Example | | -------------------------------------------------- | --------- | ---------- | --------------- | -------------------------------------------------------------------------- | ------------------------------ | | event\_count | Yes | Yes | 0, 1, 2,... | Counts the number of events triggered on a given day | Number of page views | | event\_dau (Legacy support as of October 16, 2024) | Yes | Yes | 0, 1 | Marks each user as 1 or 0 based on whether they triggered the event or not | Unique users who viewed a page | When you click on an event type in the **Events** tab, you will see a detailed view of the event, including any metrics linked to that event. Click on these metrics to visit the detail pages for these metrics. Event metrics dashboard view ## Event Count Metric Event count is the simplest metric in your Statsig Project. For every event recorded, Statsig automatically creates an **event\_count** metric based on the number of times Statsig receives that event each day. In experiments, Statsig calculates this value for each user, and each user can have values ranging from 0, 1, 2,... etc. You will find an **event\_count** metric for each event type that you record in the Statsig console. The name of the metric matches the name of the raw event and its metric type is marked as **event\_count**. Event count metric configuration interface ## User Accounting Metrics Statsig automatically derives a number of **User Accounting** metrics based on any exposure or custom event triggered by a user on a given day. **User Accounting** metrics start with a definition of a daily active user (DAU). By default, Statsig considers a user as a DAU if they trigger any event, gate check, or experiment check on a given day. A DAU is a binary designation that's assigned to every user. A user can be either a DAU for a given day, or not (i.e. inactive). You can customize this definition of DAU to exclude or include specific exposure and custom events from your application. | Metric | Automatic | Dimensions | Possible Values | Description | Example | | ------ | --------- | ---------- | -------------------------- | --------------------------------------------------------------------- | ------------------ | | user | Yes | No | Depends on specific metric | Counts users that trigger any exposure or custom event on a given day | Daily Active Users | Like the **event\_dau** metric (Deprecated as of October 16, 2024), Statsig computes **User Accounting Metrics** for each unit ID that you define in your Statsig Project. For example, given User IDs, DAU counts the number of distinct users that triggered the event. Given Stable IDs, DAU counts the number of distinct devices running your application. See [User Accounting Metrics](/metrics/user) for the full list of user accounting metrics and learn how to customize the definition of a DAU. Note that Statsig's default day starts at GMT-8 (Pacific Standard Time), and does not follow daylight savings time. Auto-generated **User Accounting Metrics** are not supported today for data warehouse ingestions. ## Metrics Catalog The **Metrics Catalog** tab allows you to quickly search and tag your metrics. Tags enable organizing your metrics and creating collections of metrics that are associated in some way. For example, you could tag a set of metrics focused on a product area, business function, or business objective. You can also create a loose collection of guardrail metrics that teams can add to every experiment to ensure they are causing no unexpected effects in other parts of the business. Once you create a tagged collection of metrics, you can easily pull up this set of metrics when viewing your experiment results and zoom into the context that you want to focus on. Metrics catalog interface ## Event DAU Metric (Legacy Support Only) See deprecation notice above. Like **event\_count**, Statsig formerly created an **event\_dau** metric that measures the number of unique users who trigger a specific event on a given day. Each user can have a value of 1 or 0 corresponding to active or inactive, based on whether they trigger an event or not, on a given day. This metric counts the number of users who are marked as active ("1") or not ("0"). It's important to note that an **event\_dau** metric produces a single value per user per day. When the metric is aggregated for users across the duration of an experiment, it is known as the "Event Participation Rate" as this can be interpreted as the probability a unit is DAU for that event. As such, **event\_dau** metrics are always between 0 and 1 for a user, since they are computed as "# Days with the Event" / "# Days Being Considered". **Tip**: Sometimes you might want a metric similar to **event\_dau** but not normalized by a number of days. If you're looking for a metric that measures if the user has an event over the entire duration of the experiment, try a custom metric set to metric type "Unit Count" with "One-Time Event" rollup mode. This metric works well in experimentation as it minimizes outliers, has tighter confidence intervals, and enables a simple measure to describe a user's breadth of activity across different events. Statsig computes the **event\_dau** for each unit ID that you define in your Statsig Project. For example, given User IDs, **event\_dau** counts the number of distinct users that triggered the event. Given Stable IDs, *event\_dau* counts the number of distinct devices using your application. You will find an **event\_dau** metric for each event type that you record with Statsig. The name of the metric matches the name of the raw event and its metric type is marked as **event\_dau**. Event DAU metric configuration interface # Raw Events Source: https://docs.statsig.com/metrics/raw-events Learn how Statsig uses exposure events and custom events from your application to compute metrics and generate experiment results. # Raw Events Statsig uses the raw events emitted by your application to compute a wide range of product metrics. These events contain the context you need to understand user behavior and infer their intentions. ## Types of Raw Events Statsig records two types of raw events from your application: 1. **Exposure events** track which users are assigned to control and test groups. This data allows Statsig to generate test results so you can evaluate the impact of new features and experiments. Exposure events also allow Statsig to assess the health of an experiment so you can always make key decisions based on trustworthy experiments. To generate experiment results, Statsig will require you to provide exposure events at a minimum. 2. **Custom events** track user actions and as any events that get triggered in the course of using your application, including events that capture performance (e.g. latency) or capture data for analytics (e.g. session start). These events allow Statsig to assess overall user engagement in your application (e.g. daily active users, weekly stickiness) as well as changes in user behavior as you roll out new features and experiments. > **Note:** When logging custom events, avoid using dot (.) notation in metadata keys. Keys containing dots are interpreted as nested paths during JSON parsing (e.g., by JSON\_VALUE), which can cause the values to be parsed as NULL. ## Unit Identifiers You must include at least one unit identifier when you record any raw events with Statsig. This unit identifier is essential for two reasons: 1. Ensure that your users receive a consistent application experience when they're allocated to control or test groups in an experiment 2. Join exposure events with all custom events triggered by a given user to compute experiment results ## Ingesting Raw Events There are three ways to send raw events into Statsig. Raw events ingestion methods diagram 1. Integrate with Statsig's [client](/client/introduction) or [server](/server/introduction) SDKs or [HTTP](/http-api) API 2. Set up Statsig as a destination in a data connector such as [Segment](/integrations/data-connectors/segment#configuring-incoming-events), [mParticle](/integrations/data-connectors/mparticle#configuring-incoming-events), [RudderStack](/integrations/data-connectors/rudderstack#configuring-incoming-events) and [Census](/integrations/data-connectors/census#configuring-incoming-events) 3. Import from your data warehouse such as [Snowflake](/integrations/data-imports/snowflake#direct-ingestion-from-snowflake), [BigQuery](/integrations/data-imports/bigquery), and [Redshift](/integrations/data-imports/redshift#direct-ingestion). When processing events, event names that contain this regex/character set are dropped `"\\[\]{}<>#=;&$%|\u0000\n\r` ## Raw Events in Console As you ingest custom events, you'll see these listed in the **Metrics** section under the **Events** tab in the Statsig console. Events tab list view showing recent custom events You can toggle between a list view or chart view of your events to view the trend line over time. Chart view of events showing trend line over time Statsig provides two unit identifiers by default: **User ID** and **Stable ID**. Select a unit identifier from the drop down to view all events that include the selected unit identifier. See the [guide to create custom ID type](/guides/experiment-on-custom-id-types#step-1---add-companyid-as-a-new-id-type-in-your-project-settings) to create additional unit identifiers for your project. Unit identifier dropdown for filtering events by user or stable ID ## Billing Statsig bills you for the [two types of raw events](/metrics/raw-events#types-of-raw-events) outlined above. We only bill for production environment events. 1. An **Exposure Event** is recorded for billing when you check a user for assignment in a Feature Gate or Experiment, or check for a value using a Dynamic Config. Note that: * Statsig does not bill you for duplicate checks for the same user on the same Feature Gate, Experiment, or Dynamic Config within an hour. * Statsig does not bill you for checks against Features Gates that are disabled. * Statsig also does not bill you for checks for users who are in not participating in an experiment due to the allocation or targeting you have configured. 2. A **Custom Event** is recorded for billing when you log an event using the Statsig SDK (or import from your data warehouse, or ingest from your data collector). Each event may contain multiple unit identifiers and used in multiple experiments. Note that: * Statsig bills custom events only once, regardless of the number of experiments where these events are used. # Rollout Alerts Source: https://docs.statsig.com/metrics/rollout-alerts Get notified in Slack and email when Statsig experiments or feature rollouts cause metric regressions beyond preset thresholds so you can react quickly. ## Overview On Statsig Cloud, Rollout Alerts are evaluated every 24 hours. Alerts only trigger if the cumulative metric delta is statistically significant lower / higher your threshold, which helps reduce alert noisiness. On Statsig Warehouse Native, Rollout Alerts are evaluated every time Metric Results are loaded in a feature gate / experiment. Loading Results on the first day will help provide more real-time visibility during this window. Alerts only trigger if the metric delta is statistically significant lower/higher than your threshold, which helps reduce alert noisiness. Finally, all stats methodologies you've enabled for your experiment/gate rollout (CUPED, Sequential Testing, etc.) will be applied to alert calculations. Rollout Alerts do not alert at the **topline metric value** level, but rather at **the experiment/feature gate level**. This means that even if you have an experiment allocated to 10% of your users, but the metric change within that 10% allocation breaches the set threshold, you will be alerted. All alerts you receive will be in the context of a specific experiment or feature gate and to debug/resolve the alert you will be directed to the offending experiment or gate in question. ## Setting up a Rollout Alert To set up a metric alert, go to the **Metrics** tab —> **Metrics Catalog** and search for the desired metric. Metrics catalog entry showing alert configuration option Once in the Metric Detail View, go to the **Alerts** tab, and tap **+ Create Alert**. As part of Metric Alert configuration, you will be asked to configure the following inputs- * **% Change-** The metric delta threshold after which you want to trigger an automated alert. As noted above, this delta is *in the context of the feature gate rollout or experiment the metric is being measured in* and is not a top-line metric value change across your whole user-base. * **Minimum Participating Units-** Set a minimum threshold for the number of unique units emitting the metric in each test group before triggering an alert. We surface the 25th / 50th / 75th percentiles of metric-emitting unit counts per group across your gates and experiments to help you choose a reasonable threshold. * **Direction-** Positive or negative, depending on whether you want to be alerted based on your metric *exceeding* vs. *dropping below* a target threshold. * **Subscribers-** By default, all creators of a feature gate/experiment with a metric that has an alert configured will be notified if an alert is fired. You can also add additional, global subscribers to a metric alert, who will be notified if *any* feature gate or experiment regresses the metric beyond the target threshold. Note that in a large project adding yourself as a global subscriber of a metric risks being noisy. Metric alert creation panel specifying threshold, units, and subscribers Once a Rollout Alert has been configured for a given metric, you will see an “alert” alarm bell icon next to the metric inline in the Metrics Catalog. You can also filter for metrics with alerts set on them via the standard metrics filtering affordance next to the search bar. ## Determining the Right Threshold To help you configure the right threshold for your Rollout Alert, there is a preview of how much the metric has moved in the context of any feature gates or experiments containing that metric in the Scorecard. Scorecard showing metric delta preview with alert threshold guidance To see how a given metric has trended over a longer period of time, hover over the metric delta for a given feature gate/experiment, and tap on a data point to view more details. This will open the time-series for the metric, with a configurable date range picker. Metric trend hover showing detailed data point information Metric time-series chart with selectable date range ## Alert UX If a Rollout Alert is triggered, all subscribers and the relevant gate/experiment creator(s) will receive a notification via email, in the Statsig Console, and via Slack for users who have configured Slack notifications for their Statsig accounts. Alert notification card showing metric exceeding threshold Slack notification preview for rollout alert Tapping on **View Alert** will take you into the Diagnostics page of the offending feature gate or experiment that triggered the Rollout Alert. Diagnostics page showing call-to-action after alert triggers Once in the Diagnostics section, scroll to the **Rollout Alerts** section and select the active alert. You will see the metric delta trend with the threshold overlayed. To resolve the alert, tap **Resolve** inline, which will give you the option to either resolve the alert (+ provide an explanation), or snooze the alert for a specified period of time. The alert will show as resolved both inline in the **Rollout Alerts** section, as well as on the top of the Pulse page for the given gate/experiment. Rollout alerts section with resolve and snooze controls ## Viewing Alert History To view alert history, go to **Metrics** tab —> **Metrics Catalog** and select the metric you want to view alert history for. Then go to **Alerts** —> **…** menu in the **Experiment and Gate Alerts** section, and select **View Alert History**. You will be able to see all instances of the alert firing, being resolved, or being snoozed, as well as reason provided for resolution, who resolved the alert, and at what time. Alert history table listing firing and resolution events Alert resolution dialog capturing notes and snooze options # User Accounting Metrics (DAU/WAU/etc.) Source: https://docs.statsig.com/metrics/user Understand and customize how Statsig calculates standard user engagement metrics like daily active users, retention, stickiness, and L7 engagement. ## DAU (Daily Active User) Definition Statsig automatically creates a standard set of user accounting metrics to track common product-wide engagement metrics like daily active users (DAU), new users, and retention. We also track more sophisticated metrics like L-ness, retention, and stickiness. All of these Standard User Accounting Metrics rely on a company-wide definition of a daily active user, which is defined by default in Statsig as [users that the Statsig SDK has logged](/concepts/user) any custom event (i.e. log\_event) for. You can always [customize your company's definition of a DAU](#customizing-the-dau-definition). **Warehouse Native Users**: You're currently viewing a feature designed for Statsig Cloud users. Warehouse Native customers typically have multiple datasets that uniquely affect how they define active users. Read about [Retention metrics in Warehouse Native](/statsig-warehouse-native/metrics/retention). ## Notation and Conventions * It's common to denote the first day a (new) user was active as Day Zero (D0), and the subsequent days as D1, D2, D3... etc. * A weekly active user is someone who has been active within the last 7 days (0-6 days). This includes users who were active on each of the 7 days, and users who have only been active on a single day. The same definition applies to a monthly active user. * A user with a single session that spans midnight with qualifying events at 11:59p and 12:01a will qualify a user as being a daily active user on both days. * We reserve the right to limit tracking to 100M unique IDs per unit type for 1 year. ## Default User Accounting Metrics in Statsig ### General User Metrics | **Metric Name** | **Type** | **Description** | | -------------------------- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `daily_active_user` | Count | Users who were active on a given calendar day (DAU). | | `weekly_active_user` | Count | Users who were active at least once in the past 7 days (WAU). | | `monthly_active_user` | Count | Users who were active at least once in the past 28 days (MAU). | | `new_dau` | Count | Users who became active for the first time on a specific day. | | `new_wau` | Count | Users who became active for the first time within the last 7 days. | | `new_mau_28d` | Count | Users who became active for the first time within the last 28 days. | | `daily_user_stickiness` | Stickiness (Rolling) | Fraction of the prior day's users who are active on the next day. Rolling day-to-day repeat engagement (**not** DAU/MAU or DAU/WAU). | | `weekly_user_stickiness` | Stickiness (Rolling) | Fraction of the previous week's users who have been active within the last 7 days. This metric tracks rolling week-over-week repeat engagement (**not** WAU/MAU). The previous week is defined as 8-14 days before the metric date. | | `monthly_user_stickiness` | Stickiness (Rolling) | Fraction of the previous month's users who have been active within the last 28 days. This metric tracks rolling month-over-month repeat engagement (**not** DAU/MAU). The previous month is defined as 29-56 days before the metric date. | | `d1_retention_rate` | Retention (Rolling) | % of new users from 1 day ago who were active at least once today. Rolling Day 2 window retention. | | `WAU @ D14 Retention Rate` | Retention (Rolling) | % of new users from 13 days ago who were active at least once in days 8–14. Rolling Week 2 window retention. | | `MAU @ D56 Retention Rate` | Retention (Rolling) | % of new users from 56 days ago who were active at least once during days 29–56. Rolling Month 2 retention. | | `L7` | L-ness | Average number of days a user was active in the last 7 days (value range: 0–7). | | `L14` | L-ness | Average number of days a user was active in the last 14 days (value range: 0–14). | | `L28` | L-ness | Average number of days a user was active in the last 28 days (value range: 0–28). | These user metrics can be very useful in understanding the long-term behavior of your users. However, several of these metrics do not behave well as daily experimentation metrics. This is because metrics like L7 are highly correlated across days. For example, a user who is L7 = 7 on a given day can either be L7 = 6 or L7 = 7 the following day. This is not a true daily independent variable. Metrics like this can be more likely to trigger false positive and false negative results. This generally applies to stickiness and L-ness metrics. ## Customizing the DAU Definition You can customize the definition of DAU within the Statsig Console and specify or exclude a set of Statsig and custom events. You can find this in the Project Settings. If you have privileges, you can edit this. DAU definition configuration interface There are several options for defining an active user using log events: 1. You can specify the set of events which will qualify a user as a daily active user. By default, all events are included. i. Excluding specific events - Some companies may want to exclude specific events (eg. events that aren't considered significant user interactions, homepage\_visit or notification\_sent). You can Expand and uncheck events you do not want to include. You can also toggle whether future events (not shown on the list) should be included or excluded. ii. Include specific events - Some companies prefer to have a very narrow definition of an active user (eg. event = login). This can be accomplished by selecting the set of events, and turning off "Include New Events by Default". Changes to the DAU definition will go in place on the date of the change. Historic data will remain unchanged. We do not currently support the ability to backfill. We do recommend setting your DAU definition before running any experiments or rolling out any features. # User Property Source: https://docs.statsig.com/metrics/user-property Break down Statsig experiment results by user properties like subscription tier, country, or platform to gain deeper insights into heterogeneous effects. It is helpful to be able to break down Pulse results by user properties like Free vs Paid, or OS type. Setting User Property lets you slice data by properties like this. You set user property when you create the User object you use with the Statsig SDK. These are frozen when a user is first exposed to a feature gate or experiment - in case your experiment ends up changing these properties (e.g. convert a Free user to Paid). User property configuration interface You can run custom queries on your Pulse results (Explore tab) to group by or filter by these User Property. Pulse results custom query interface # Verified Metrics Source: https://docs.statsig.com/metrics/verified Mark trusted, curated Statsig metrics as verified so teams can quickly identify reliable metrics to use in experiment scorecards and analytics dashboards. This is an Enterprise feature, please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. Verified metrics are a way to identify metrics that are curated by your company and known to be trustworthy. When users set up experiments and search for metrics, they'll see a verified icon next to verified metrics. Verified metrics interface showing verified icon You choose to verify metrics from the overflow menu on the Metric. Metric overflow menu with verify option Admins can control which roles are able to verify metrics. Admin role permissions for metric verification # Alerts Overview Source: https://docs.statsig.com/product-analytics/alerts-overview Overview of Statsig product analytics alerts, including topline alerts, anomaly detection, and rollout alerts that notify your team of metric shifts. # Alerts Statsig offers two types of alerts on the platform today: 1. **[Topline Metric Alerts](/product-analytics/alerts/topline_alerts)** - Monitor a metric’s overall performance, independent of experiments or gates. These alerts keep you informed about topline trends and highlight when key product or business metrics move in a concerning direction. 2. **[Rollout Alerts](/product-analytics/alerts/rollout_alerts)** - Monitor how a metric behaves in the context of a feature gate or experiment rollout. These alerts help you confirm that no critical metrics regress when introducing a new change. # Dashboards Source: https://docs.statsig.com/product-analytics/dashboards Build Statsig product analytics dashboards to consume, share, and save the insights that matter most for your product team in one organized view. ## Creating a Dashboard There are two ways to create a dashboard: 1. Navigate to the Dashboards tab and click Create. Here, you have the option to choose one of Statsig's [Dashboard Templates](#dashboard-templates) or create a custom new one Dashboard creation interface 2. You can also create a Dashboard directly from Metrics Explorer. To do this, once you have finished building a chart: 1. Click "Export to Dashboard" at the top right corner of the chart 2. Name the Chart 3. Select "Create New Dashboard" from the Dashboard Destination selector 4. Finally, give your new Dashboard a name Export to dashboard from Metrics Explorer ### Adding Charts, Feature Gates, and Experiments to a Dashboard Dashboards are designed to help teams share and absorb product insights of all types. To that end, it is possible to add Metric Charts and keep track of ongoing pulse results from A/B tests & feature launches. There are several types of dashboard widgets you can add or create including: * **Charts:** Create a new chart directly from a dashboard or export a chart created in Metrics Explorer to a dashboard. Supported charts include: * Drilldown Charts * Funnel Charts * Retention Charts * Distribution Charts * User Journey Charts * **Text**: Annotate dashboards with context or create section headers for better readability. * **Single Value:** Highlight a hero metric with clarity by adding a single value to the dashboard. * **Experiment, Feature Flag:** Get a quick snapshot of an experiment or feature flag. * **Funnel Metrics:** Visualize custom funnel metrics. ### To add a widget to a dashboard: 1. Click the "Add Widget" button 2. Select the type of widget you would like to add 3. Configure the widget, e.g. select a chart type and then select events and metrics you want to track 4. Save the widget to the dashboard ## Exploring your Dashboard ### Edit Date Ranges By default, the charts and widgets on a dashboard are synced with the date range set for the entire dashboard. To update this default date range, click the pencil icon in the top right corner of the dashboard. In the settings that appear, you can modify the dashboard's title, description, and "Default Date Lookback Range." This selection will determine the date range that is automatically applied each time you open the dashboard. Choosing the "Chart Default" option allows each chart to revert to the date range originally set when it was first added to the dashboard, offering greater customization to your dashboard. Dashboard settings and date range configuration You can also change the date range of the dashboard on the fly by modifying the date picker on the top right of the dashboard. Applying changes to this selector will synchronize all charts and widgets on the dashboard. Note that any changes made here will not be saved the next time you open up the dashboard. Dashboard date picker interface ### Exporting your Dashboard If you want to share a static version of your dashboard, print it, or save it for any other purpose, you can easily export a copy of your dashboard with the present data by clicking the settings dropdown "..." in the top right corner, and then clicking on the "Export as PDF" button. After a few seconds, a PDF of your dashboard will be generated and downloaded automatically. Export dashboard menu ### Cloning your Dashboard If you want to duplicate or clone any of your dashboards, open the desired dashboard, and click the settings dropdown "..." in the top right corner, then click on the "Clone" button. This will bring up the dialog for you to clone your dashboard, and take you there upon success. Dashboard clone dialog ### Filtering your Dashboard You can click on the filters button below the dashboard name to add a global filter to your dashboard. The filter will be applied across all eligible widgets and you can quickly view updated results across all widgets, rather than having to filter each widget individually. You can also use free-form text to apply filters for more generic values, such as filtering emails that contain '@gmail.com'. Filters added to your dashboard will be applied to your widgets. When you expand a dashboard widget which has a dashboard filter applied, it will show the filter as an "Inherited Dashboard filter". Any changes made to this will only temporarily reflect and will not be saved. Dashboard global filter interface #### Default Dashboard Filters Default dashboard filters allow you to pin commonly used filters directly to your dashboards, making it easier to analyze different views of your data without rebuilding filters from scratch. These filters appear at the top of your dashboard and apply across all eligible widgets, enabling quick comparison across different dimensions like company, region, or platform. To configure default dashboard filters: 1. Navigate to your dashboard and click the settings cog ⚙️ 2. Scroll to "Default Filters" and configure the filters you want to pin to the dashboard 3. Click Save Once configured, the pinned filters will appear at the top of your dashboard. You can quickly swap values to see how different users, cohorts, or properties impact the same set of charts. When you change a filter value, all charts on the dashboard update automatically without needing to reconfigure each widget individually. This feature is particularly useful for comparing trends across different segments of your data. Instead of duplicating dashboards or manually editing filters on each widget, you can reuse the same dashboard with dynamic filtering to perform scoped analysis more efficiently. ### Refreshing your Dashboard Widgets To ensure your dashboard data is up to date, you can refresh dashboard widgets in several ways: **Manual Refresh**: Simply click the refresh button shown in the image below to refresh all dashboard widgets at once. Dashboard refresh button **Automatic Dashboard Refreshes**: Dashboards can now be automatically refreshed on a schedule with results cached for faster loading and a snappier experience. You can configure a refresh frequency for each dashboard (e.g. hourly, daily) and automatically cache results in the background. Once set, queries for that dashboard will run on the specified schedule and store the results. When someone opens the dashboard, they'll see the most recent cached data instantly, instead of triggering fresh queries. To configure automatic dashboard refreshes: 1. Navigate to your dashboard and click the settings cog ⚙️ 2. Scroll to "Schedule Dashboard Refresh" and set the interval 3. Click Save This feature helps dashboards load faster and stay up to date without manual effort, especially helpful for shared dashboards or recurring check-ins where you want fresh data ready without delay. ### Dashboard Subscriptions Dashboard subscriptions send a scheduled snapshot of a dashboard to your team—so stakeholders can stay up to date without needing to open Statsig. **What gets delivered**: A subscription delivers a snapshot of the dashboard at send time, including: * Dashboard title and link * All widgets currently on the dashboard * The dashboard’s default date range (and any pinned default dashboard filters, if configured) Depending on your workspace setup, subscriptions can be delivered via Email and/or Slack. **Creating a Subscription**: 1. Open the dashboard you want to subscribe to 2. Click the settings menu (...) in the top right corner 3. Select "Add/Manage Dashboard Subscription" 4. Configure your delivery channel (email, Slack), recipients (emails or Slack channel) and schedule (e.g. daily, weekly) 5. Click Save ### Organize your Dashboard A well-made dashboard helps easily convey a narrative around what information is most important and the relationship between items on the dashboard. To facilitate this, we make it easy to move and resize dashboard widgets in place. Each dashboard is constructed as a grid over which you can place, move, and resize dashboard widgets. Move dashboard items around the grid by placing your mouse on empty space in the widget, and then click and hold to drag the widget around. Resize the widget by clicking and holding the bottom right edge of the widget dragging to the desire size. ### View and Edit a Chart in a Dashboard All of the charts we support in Metrics Explorer can be added to a dashboard. In addition, dashboard charts are not static. To dive into a chart on the dashboard, click the \[ ] icon. Once expanded, you get the full power of Metrics Explorer, allowing you to modify the overall query, the date range and the chart title. These modifications enable further exploration without permanently altering the chart on the dashboard. If you want to save changes to a chart on the dashboard, configure the chart as desired and click "Save" to update the existing chart, or "Save As" to create a new chart on your dashboard. Chart editing interface in dashboard ## Tips ### Dashboard Templates Dashboard templates are a great way to reduce the time to insights. Statsig compiles industry-standard metrics to ensure your data visualization is focused on critical success indicators. You can start with a template and add on any additional insights you find helpful. Currently, Statsig offers the following templates: * Product Growth * Use Statsig's built-in metrics like DAU, WAU, Stickiness, etc. to track long-term product health * Feature Success * Input a feature usage event, like form\_submit or purchase\_completed, to evaluate the success of your feature rollout with usage and retention metrics * Optionally, you can add a feature adoption event or related feature flag to broaden the scope of your tracking * B2B SaaS Topline Metrics * Input a key feature event and user subscription events to monitor product usage, user retention and subscription conversion rates over time * Web Analytics * Available with Autocapture, track essential web metrics like page views, clicks, and sessions duration. See more on Autocapture [here](/guides/sidecar-experiments/measuring-experiments#using-autocapture) * Create from Gate or Experiment * Input your Gate or Experiment of interest to generate a focused dashboard with related monitoring metrics * Create from Tag * Utilize Statsig's project tagging to create and easily maintain a dashboard with the metrics, experiments, and feature flags you care about ### Finding Dashboards Once you've created a dashboard, you may want to quickly find the charts that matter to you. Heading to the Dashboards tab will give you several ways to find a dashboard. 1. Find dashboards you've created quickly by navigating to the Dashboards tab, clicking into the search box, and selecting "My dashboards". 2. Navigate to the Dashboards tab and click the filter icon to scope to Dashboards with specific tags or created by specific team members. 3. Navigate to the Dashboards tab and simply search for the name of the dashboard you would like to find. 4. Anywhere within Statsig you can bring up global search with "cmd+k" and type in the name of the dashboard. # Distribution Source: https://docs.statsig.com/product-analytics/distribution Visualize the range of user experiences across your product with Statsig distribution charts to spot outliers, percentile shifts, and skewed engagement. ## Overview Distribution charts in Metric Explorer help you visualize the range of user experiences across your product. These charts are a great way to generalize central tendencies, evaluate product health, and identify outlying behavior. Distribution chart interface in Metrics Explorer ### Use Cases * **Analyze the Spectrum of Experiences:** Study the distribution of event values to identify any trends * **Measure Feature Engagement:** Visualize how often each user is interacting with a critical event * **Monitor Product Performance:** Ensure that your product isn't experiencing performance issues like unusual latency times ## Creating a Distribution Chart ### Step 1: Choose a metric or event The first step to creating a distribution chart is to decide if you want to use a metric or an event. The distribution of a metric will show you the aggregated property value per unit ID. For example, if your metric aggregation is by count, the chart will display the number of times each user has triggered the event. The distribution of an event will show you the range of data stored under the property "Value" for each logged event. Note that the distribution of events is only displayed if the data under "Value" is numeric. ### Step 2: Refine your bucket size As a default, the bucket size will be set to 1 with a minimum value of 0 and a maximum value of 10. Note that the buckets are always lower inclusive and upper exclusive. If your range exceeds the initial max bucket value, we exhibit the data as "10+". You can refine your bucket sizes, minimums, and maximums to find the best view of your data. ## Interpreting your Distribution Chart ### Distribution of a metric Under distribution of a metric, there are two important factors to consider: the unit ID and the metric value. The distribution will always represent the metric values per unit ID. The X-axis represents the metric value. This could be a count of events or the sums and averages of a property value, depending on the configuration of your metric. The Y-axis is the unit ID of the metric. Depending on the construction, this can be users, stable IDs or a custom ID you've constructed. ### Distribution of an Event The distribution of an event will always display the range of data under the "Value" property of your event. Thus, the X-axis is the "Value" and the Y-axis is the number of events. # Metric Drilldown Charts Source: https://docs.statsig.com/product-analytics/drilldown Use Statsig drilldown as a versatile tool for understanding customer behavior and trends within your product across segments, time periods, and properties. The Metric Drilldown chart in Metrics Explorer is a versatile tool for understanding customer behavior and trends within your product. Designed for clarity and depth, it allows you to analyze key metrics and user behavior over time. Importantly, it also allows you to delve several layers deeper into your metrics by filtering to interesting properties or cohorts, as well as the ability to group-by these same properties to compare behaviors between groups. ## Use Cases * **Trend Analysis Over Time**: Gain insights into how specific metrics evolve over time. Visualizing product data in Metrics Explorer allows you to track and compare key performance indicators and user behavior, and helps understand long-term trends and short-term fluctuations in how users engage with your product and your product's performance. * **Identify interesting cohorts**: Define and explore interesting cohorts by zooming in on users who performed certain events at frequencies you define. * **Understand how Targeted Feature Launches, A/B tests, and Experiments affect usage:** Split any metric out by Experiment Group or Feature Gate Group to compare how those metrics perform for different groups. Leverage automatically generated annotations on charts for important decisions such as Feature or Experiment launches to help correlate those decisions with changing trends. * **Segmentation and Comparison**: Dissect metrics to understand how different user segments or product features perform. This is crucial for identifying which areas are providing value for your users and those which may need more attention or improvement. It is also useful in understanding how different segments interact with your product, and for identifying unique trends or needs within these groups. * **Filtering**: Focus on specific segments or cohorts that are of particular interest. This filtering capability allows for a more targeted analysis, helping you to understand the behaviors and needs of specific user groups. * **Statistical Understanding:** Understand how the average, median, or other percentile value (e.g. p99, p95) of a metric changes over time. * **Dynamic Metric Creation with Formulas**: Craft new metrics on the fly using custom formulas. This flexibility is useful in deriving ad-hoc insights with minimal effort. * **Flexible Visualization Options**: Choose from a range of visualization formats, like line charts, bar charts, horizontal bar charts, and stacked bar charts, to best represent your data. The right visualization can make complex data more understandable and actionable. * **Event Samples for Debugging**: Quickly access and analyze a metric's underlying sample events, and the granular user-level information attached to the event. This feature is particularly useful for troubleshooting and understanding the root causes of trends or anomalies in your data. * **Detailed Data Control**: Adjust the granularity of your data analysis, from high-level overviews to detailed breakdowns. Use features like rolling averages to smooth data for more accurate trend analysis and decision-making. * **Debug Experiments**: Breakdown your experiment's first exposures to understand how certain properties or groups (feature gates, experiments, holdouts) affect your experiment. * **View Sample Ratio Mismatch (SRM)**: See the SRM of your experiments over time and drill down into event and user metadata fields to understand how certain properties (country, browser, etc.) or groups (feature gates, experiments, holdouts) can affect your experiment SRM. * **Debug Feature Gates**: Breakdown your feature gate's first exposures per rule to understand how certain properties or groups (feature gates, experiments, holdouts) affect your feature gate. * **Analyze Dimensions Across Metrics**: Quickly identify top or bottom performers across multiple metrics by viewing your data in a table and sorting by any metric–column combination. # Using the Metric Drilldown Chart ## Selecting Metrics and Events In Metrics Explorer you can choose events, custom-metrics, auto-generated metrics, or experiment exposures to explore. You can add several metrics and events or exposures to plot on a single chart. ### Events When selecting an event, the total number of times the event occurred (Count) on a given data point (hour, day, etc) will be plotted by default. You can also choose different ways to aggregate event data. The full list is as follows: * **Count**: Plot the number of times the event occurred within the given time range per data point. * **Unique**: Plot the number of unique ids (generally UserIDs) that performed the event in the given time range per data point. Event aggregation options interface * When viewing data on uniques (e.g. unique users) at daily granularity, you can choose to have the value of each daily data point represent the number of unique weekly users (unique users over previous 7 days). This enables you to get a sense of how weekly usage is changing day over day. Weekly unique users configuration option * **Average**: Plot the average of a selected event property value within the given time range per data point. Note this only works for properties that have numerical values. Average event property aggregation interface * **Sum**: Plot the sum of a selected event property value within the given time range per data point. Note this only works for properties that have numerical values. Sum event property aggregation interface * **Percentiles**: Plot the value of a selected event property value at the selected percentile within the given time range per data point. Percentiles aggregation configuration interface * **Unique Values**: Plot the count of distinct values for any property across events or users within the given time range per data point. This aggregation helps answer questions like "How many different referrers drove traffic last week?" or "How many SKUs were added to carts today?" by counting unique property values rather than event occurrences. * **Count per User**: Plot the frequency distribution of how often users perform a specific event, showing statistics like average, median, or percentile values per user within the given time range per data point. This aggregation helps analyze user engagement patterns by measuring how many times each user performed an event, then applying summary statistics across those users. ### Exposures Selecting an experiment or gate exposure plots its first exposures over your selected date-range. First exposures are the first time a unique id (set on the experiment or gate) was exposed to each of your experiment groups or each of your gate's rules. ### Understanding First Exposures in Feature Gates When a gate is checked for a user, an exposure is created for the rule who's conditions they've met. If the user is exposed to multiple different rules at different times, the first exposure from each rule is kept. We recommend grouping by rule to see each rule's exposures separately. ### Metrics Selecting a custom Metric or auto-generated Metric plots the value of that metric over your selected date-range. **Viewing and Modifying the Metric Definition** You can easily view the definition of the metric directly below the metric name. You can also modify your metric plot on the fly by making ad-hoc edits to the event based definition shown. This allows you to plot new metrics on the fly, based on metrics you have already defined. ### Comparing Multiple Metrics, Events, or Exposures You can compare multiple metrics or events by plotting them on the same chart. To add multiple metrics click the "+" icon and select "Metric". Then select the metric of interest. When multiple metrics are plotted, you can scope to a single metric by clicking anywhere on that metric's row in the table below the chart. Clicking on the row again undoes the scoping. You can scope to a specific set of metrics by using the check boxes next to the metric names in the table. ### Filtering It's often useful when exploring data to narrow your exploration down in order to find an interesting insight. Filters allow you to scope to events & metrics logged by users with specific properties. To add a filter to an event or metric, click the filter icon and select the event or user property you would like to filter to. ### Adding formulas A powerful tool for on-the-fly analysis of trends and user behavior are formulas. They allow you to dynamically combine and transform plotted events and metrics to answer new questions as they arise using mathematical expressions. To add a formula, hover over the "+" icon in the Metrics section and select Formula. This will give you a free-form text field in which you can use the label of each plotted Metric or Event (each plotted metric is labeled with a letter) as variables in your expression. For example if you had * Metric A, which plotted number of unique purchasers * Metric B, which plotted total purchases You could plot purchases per purchaser as "B/A". ## Drilling down In addition to plotting metrics, you may want to drill into your metrics to identify unique groups that tell you something interesting about how people are leveraging your product. We support several features that make finding these types of insights easy. ### Group-By Leveraging a Group-By makes it easy to disaggregate plotted metrics and events by a selected property or group. Doing so allows you to compare how an action or user behavior may correlate with a specific property. Adding a Group-By will split the plotted metric(s) into several plots. By default we only show the top ten groups by value on the chart, but you can select more groups. You can select 50 groups when the chart is set to daily granularity. A metric can be grouped-by event properties, user profile properties, experiment group, or feature gate group. Group-By limits can be added by first adding a group-by, then moving to the summary table below the charts, and clicking the "Top X series" dropdown button. From there you can select how many groups you want to see at once. You can use this to further drill down on your top X categories (up to 50). This feature is available for line charts, stacked-line charts, bar charts, and stacked-bar charts. Group-by top series selection dropdown When you have a Group-By applied, you can view the results as raw numbers, or as a percentage. **Feature Gate and Experiment Groups** At Statsig we believe in the power of experimentation. To that end, you can also select one of your Feature Gate or Experiments in order to split out a metric by the different groups in the selected test. **Adding a Group-By** To add a group by, hover over the "+" icon in the Drilldown section and select Group By. You can then select the property or experiment group to split the metric out by. **Quickly hiding or isolating lines** When performing a Group By, it's often useful to isolate lines to show data for a specific set of groups. You can do this by clicking in the row representing that group in the table view under the cart. Clicking on a group that is currently isolated will once again show all groups. You can uncheck and check groups in the table view to scope to a custom set of groups. ### Define and Compare Event-Based Cohorts Building a useful group of users to analyze often requires a bit more precision than just comparing by different property values. For example you may want to understand the behavior of your power users and compare them against your general set of users. To support this we allow you to define event-based cohorts. You can select an event of interest, and then define a frequency criteria for how often users in the cohort performed the event. These criteria include: * Performed the event ***at least*** some number of times * Performed the event ***at most*** some number of times * Performed the event ***exactly*** some number of times You can also define the window in which a user performed the event for inclusion, as well as filter to some property in order to be as granular as needed when defining the cohort. You can also save cohorts to be able to easily reuse them. ## Exploring a Metric Drilldown Chart ### Selecting chart granularity The Metric Drilldown chart gives the flexibility to view data at the granularity you need. You can view data at the daily, hourly, 30 minutes, 5 minute, or 1 minute granularity. This granularity corresponds with the interval between x-axis values. Viewing data at granularities less than hourly limits the analysis time window to 1 day. The default chart granularity is daily. You can change this by selecting the "Daily" drop down in the top right of your chart and selecting the granularity you prefer. Note that when viewing data on uniques (e.g. unique users) at daily granularity, you can choose to have the value of each daily data point represent the number of unique weekly users (unique users over previous 7 days). This enables you to get a sense of how weekly usage is changing day over day. ### Setting the date range The default date range of a chart is 14 days. To select the date range of a chart click on the "Last 14 days" dropdown and select one of the quick date ranges, or select a custom range you prefer. ### Changing your unit type The default formatting / unit type is numbers. To see your data as a percentage, time value, or measurement of space (bytes, bits, etc.), click on the settings cog in the top right corner of your chart, and then select format unit type. ### Smoothing out the data with rollups Metrics like daily usage often have seasonality effects which can make longer-term trends harder to see at a glance. To help with this, techniques such as modifying each data point on a daily chart to represent a 7 day rolling average is useful. We support the following rollups to smooth out data, each of which can be rolled up over 3, 7, 14, 28, 48, or 60 data points: * Rolling average: Replaces each data point with the average of the preceding number of selected data points. * Rolling sum: Replaces each data point with the sum of the preceding number of selected data points. * Cumulative Sum: Replaces each data point with the sum of all preceding data points, including itself. This results in a continuously increasing total, where each value represents the accumulated sum of all previous values in the dataset. ### Selecting the chart visualization Metrics Drilldown offers many ways to visualize your data, including: * **Line:** Useful when plotting one or more metrics over time. * **Stacked Line:** Useful when comparing groups to understand the relative proportion a certain group has of a metric or event. * **Bar:** Useful when comparing the total value of two metrics over the entire date range. * **Horizontal Bar:** Ideal for grouped data comparisons, especially when you have longer label names. Makes it easy to identify top and bottom performers across any business dimension like user types, locations, or platforms. * **Donut:** Useful for visualizing the proportional breakdown of a whole into distinct categories at a single point in time. Perfect for showing how different segments (like countries, user types, or feature groups) contribute to your total metric value. Apply a Group-By to any metric to see the breakdown as a donut chart. * **World Map:** Visualize your metrics geographically by country when using a country-based Group-By. This view makes it easy to spot geographic trends and understand how your product performs across different regions. * **Single Value:** Display key metrics at a glance for quick summaries. Perfect when you need to highlight a single important number or KPI without the complexity of a full chart. * **Data Table:** Compare multiple metrics across groups in a structured table format. Ideal for detailed analysis when you need to examine exact values and perform side-by-side comparisons of different segments. Both donut charts and world maps work with any metric when you apply a Group-By. Simply select your metric, add a Group-By for the property you want to analyze (such as country for geographic analysis), and choose your preferred visualization from the chart type selector. ### Zooming in Often when digging for insight, you may want to quickly zoom in on a certain portion of the date range to view things with more granularity. To help with this, you can simply use your mouse to click at one end of the date range you want to zoom in on, and hold the mouse button down while moving to the other end of the date range of interest. Letting go of the mouse button will then zoom in on that portion of the chart. To reset your zoom, click "Reset Zoom" in the top right of the chart. ### Sharing your insights Once you've arrived at an insight you find interesting and want to share you have two options for sharing: * **Share via URL:** Simply copy the URL. This is a quick and easy way to share a query as it currently is defined. * **Create a share link:** If you would like to share a shorter, cleaner, URL, clicking the "…" button in the top right of the chart and then clicking "Share Link" copies a shortened link to the query as currently configured to the clipboard. * **Share to Dashboard:** Clicking the "…" button and selecting "Export to Dashboard" allows you to either save your chart to an existing dashboard, or create a new dashboard where you can save the chart. Note sharing a chart via URL or shortened link essentially shares a "snapshot" of the chart as currently defined when the link was copied. Any subsequent changes will not be captured via the share link. # Funnel Charts Source: https://docs.statsig.com/product-analytics/funnels Statsig funnels provide a granular understanding of what portion of users complete each step of a journey you define, including time-to-convert analysis. ## Overview Funnel Charts in Metrics Explorer provide a granular understanding of what portion of users are completing each step of a journey you define through your product or service. These charts are useful for understanding user behavior, identifying bottlenecks, and helping you develop insight-driven product changes that help users convert through your product more successfully. ### Example Use Cases * **Conversion Analysis**: Monitor the progression of users through stages like sign-up, adding to cart, and purchase completion. * **Identifying Drop-off Points**: Pinpoint where users drop-off of a process, allowing targeting improvements at these points. * **Comparing User Segments**: Observe how different user segments move through the funnel, highlighting variations in behavior based on demographics, user types, or other criteria. * **Product Optimization**: Determine which features or steps effectively move users to the next stage, and which require improvements. * **Experiment Analysis**: Understand the conversion rates before and after first exposure to an experiment. ## Defining a User Funnel ### Step 1: Add Steps to your Funnel To define a funnel, select a series of a events that represent different parts of a product flow that you are interested in understanding. To do so: 1. Go to **Metrics Explorer** under **Analytics** in the Navigation Bar, and switch over to the Funnel Charts view. 2. Add steps to your funnel using the "**+**" icon. **Optionally add filters** to funnel steps to target specific event or user properties. For WHN, you would add steps directly from your **Metric Source**. For Statsig Cloud, you can leverage events or metrics for the steps. **Combining Multiple Events into one step** You can also combine multiple events into a single step. This is especially useful when there are more than 1 qualifying events that are indicative of a meaningful yet single portion of your funnel. Combined events are done so using OR logic. To do this: 1. On the step in question click the "…" button and select "**Combine Events**". 2. Select the an additional event to add. 3. Add any desired filters to each of the events in the funnel step. Repeat these steps as needed. You can define a step as a combination of up to 5 events. Funnel step configuration interface **Filtering to the first time a user performed an event** It is sometimes useful to understand the first-time user experience, which may have a significant impact on things such as long term retention or may meaningfully differ from general flows through the product. To help analyze first-time experience, you can filter the events in a funnel to the first time ever a user (or other unit ID) performed an event. To do so: 1. Click "…" next to the event in question. 2. Select the "**Filter to First Time**". Filter to first time event option **Renaming Steps** When sharing a Funnel Chart you have created, such as by saving it to a dashboard, the event names as logged may not always be easily readable or meaningfully understood. In these cases you may want to rename the steps in your funnel for better legibility. To do so: 1. Click the "…" next to the event name you would like to rename. 2. Click "**Rename Funnel Step**". 3. Give the funnel step a new easily readable name. Note this doesn't change the name of the underlying event, and only applies to the funnel being configured. ### Step 2: Select a Graph Type We support 3 graph types for understanding conversion funnels: * **Conversion Rate**: This is a standard funnel view that offers a step-by-step breakdown of where users are dropping off in your funnel, and the number of people converting through each step. * **Conversion Rate Over Time**: This shows how the overall conversion rate of your funnel has changed over time. This is useful in understanding how new features and product changes have made an effected your funnel conversion rate. * **Time to Convert**: This graph type helps you understand distributions in time it takes to complete your funnel. ### Step 3: Choose an ID Type In general, you want to construct funnels to understand the rate at which individual users make it through each step of the flow you've defined. This means creating a funnel using a userID. However, some interesting funnels may involve several people in an organization using your service. For example, you may want to understand the end-to-end success of company onboarding to your service, where different people are involved in purchasing, deploying, and then using your service. To this end, Statsig allows you to perform individual or group analytics. This enables you to analyze the success of your funnel at the user level, or the success of your funnel for whole groups such as organizations or companies. You can choose any of the ID types defined in your Statsig project to create a funnel over. If you have an experiment exposure selected, make sure that the ID type selected matches that of the experiment. Group analytics is not a paid add-on at Statsig and is included at no-extra cost for all tiers. ### Step 4: Define the Conversion Window Once you have defined your funnel, you can limit your analysis to Users (or other Unit IDs) that converted within a specified time frame. Users who start the funnel but do not convert within this time frame will be count as dropped off. ### Step 5: Drilldown To better understand how conversion varies between different groups of users, you leverage the **Group By** feature to split out funnels by properties, experiment groups, or feature flag groups. To this, click the **"+"** to the right of "Group By" and select the property, experiment, or feature flag you would like to split out the funnel conversion analysis by. ## Advanced Funnel Analysis ### Ordered or Unordered Funnels In general, the power of funnels lies in understanding whether or not users completed a specific set of events in a specific order. This is the most common scenario, and you can achieve this by toggling "Ordered" on. This is the default. Ordered funnels require that a user completes the selected events in the specified order to be counted as converted. The user may still perform other events between the specified events, including events in the funnel, and still be counted as converted. For example, if an ordered funnel is defined with events A, B, C, and D, the following sequence will count as converted: A→B→B→A→C→D. Sometimes, you just want to know whether a user has completed all a given set of events, regardless of the order they completed them in. You can achieve this by toggling "Ordered" off in the advanced settings. Unordered funnels only require that the user completes the specified events within the given time range to be counted as converted. ### Unique Users (or Unit ID) vs Total Conversions You can choose whether your funnel analysis is defined by the total number of conversion that occur, or the number of unique users who convert. The default is Unique Users. ### Daily Aggregation When toggled on, this calculates funnel conversions per calendar day. A given unit with funnel conversions on multiple days is counted as multiple conversions. This is toggled on by default. ## Interpreting your User Funnel ### Conversion rate vs. Number of Conversions At the top right of the funnel chart, you'll see the Conversion Rate vs. Conversions selector. This selector allows you to switch between viewing the y-axis of your funnel as a conversion rate or the number of conversions. This feature is particularly powerful when used in conjunction with a group by. Toggling between conversion rate and number of conversions, you can see both the relative and absolute scales in conversions across different user groups. ### Conversion Summary and Table Under each funnel step, you'll see a quick summary that helps demystify your funnel data. Specifically, these metrics tell you: * The percentage and number of units that have converted relative to the first step of the funnel * The percentage and number of units that have dropped off, again this number is relative to the first step of the funnel * The average time it takes for each user to convert Next, in the conversion table, you'll find the percentage and number of units that have converted relative to the first step of the funnel and relative to the previous step. Again, this feature is particularly useful when grouping by a certain property in which you want to compare conversions between user groups. Funnel conversion summary table ### User Exploration Finally, by clicking any bar in the funnel, you have the option to download all users in that segment who dropped off or converted as a CSV file. You also have the option to view session streams to get a granular, event-by-event understanding of a user's experience before and after the data point in question. User exploration options interface ## Conversion Drivers Conversion Drivers identify statistically significant factors that correlate with funnel conversion or drop-off at each step. This analysis surfaces event properties, user properties, and intermediary events that influence user progression through your funnel. Conversion Drivers Interface To access Conversion Drivers: 1. Click on any funnel step 2. Select **"View Drop-Off & Conversion Drivers"** 3. Configure the analysis scope using the dropdown filters Conversion Drivers Entry Point ### Analysis Configuration You can configure which data types to include in the analysis: * **Event Properties**: Attributes attached to events (e.g., platform, plan\_type, referral\_code) * **User Properties**: User-level attributes (e.g., country, account\_age, signup\_method) * **Intermediary Events**: Events that occurred between the selected funnel steps ### Driver Metrics Each identified driver displays: * **Conversion Likelihood**: Expressed as a multiplier of the funnel's conversion rate (e.g., users with platform::Android are 1.2x as likely to convert) * **Conversion Rate**: Percentage of users with this factor who converted * **Participant Share**: Percentage of total funnel participants who had this factor ### Detailed Analysis Click on any driver to access the drilldown view, which provides: * **Conversion Matrix**: Side-by-side comparison of conversion outcomes for users with and without the factor * **Correlation Coefficient**: Statistical measure of the factor's association with funnel completion (also known as the [phi coefficient](https://en.wikipedia.org/wiki/Phi_coefficient)) You can group your funnel by any identified driver by clicking the group by option. This reconfigures the funnel chart to show conversion performance segmented by the selected property. ### Use Cases Conversion Drivers are useful for: * Exploratory analysis when investigating funnel performance without predefined hypotheses * Root cause analysis of conversions/drop-offs * Validating assumptions about user segment behavior * Monitoring funnel performance changes over time Conversion Drivers require a Pro plan subscription or Enterprise plan with the Advanced Analytics package. # Lifecycle Source: https://docs.statsig.com/product-analytics/lifecycle Use Statsig lifecycle analysis to understand how users start, return, remain, and churn over time so you can target the right cohorts for growth experiments. ## Overview Lifecycle charts in Metrics Explorer help you understand how usage changes over time by classifying your unique units (for example user\_id) within each time interval based on whether they used an event recently, returned after a gap, continued to use it, or churned. Life Cycle chart interface in Metrics Explorer ### Use Cases * **Track new-user ramp after a launch:** See whether adoption is growing week over week after shipping a new feature * **Monitor churn and reactivation:** Understand whether users are falling off (and whether they come back) * **Compare retention health across releases:** Spot changes in “stickiness” and reactivation patterns over time ## Creating a Lifecycle Chart ### Step 1: Choose an event (or a compatible metric) The first step to creating a lifecycle chart is to decide if you want to use a metric or an event. Lifecycle is designed for a single underlying event and count-style metrics that are composed of a single count type event ### Step 2: Choose your unit (unique units) Select what you want to count uniquely (for example user\_id, stable\_id, or another unit). The chart reports how many unique units fall into each category for each time interval. ### Step 3: Choose your interval (granularity) Pick an interval unit (day / week / month) and a number of intervals per bucket (1–48). Each bar on the chart represents one interval bucket, and the chart shows one data point per bucket across the selected date range (max 1 year). Life Cycle chart interface in Metrics Explorer ### Step 4: (Optional) Add filters Apply filters to focus on a specific segment (for example platform, country, app version, or a feature-related property). ## Interpreting your Lifecycle Chart Each interval bucket classifies unique units into four categories (mutually exclusive): **New:** Used during this interval, and did not use at any point earlier within the lookback window (up to 1 year before this interval). **Resurrected:** Used during this interval, did not use in the immediately previous interval, and did use earlier within the lookback window. **Recurring:** Used during this interval and the immediately previous interval. **Dormant:** Used in the immediately previous interval, but not during this interval (often displayed below the axis to make churn visually obvious). ### Reading the chart **X-axis:** Time, grouped into your chosen interval buckets. **Y-axis:** Count of unique units. **Stacked bars:** Show how the composition of usage changes over time (new vs resurrected vs recurring), while the churn component (dormant) highlights drop-off between adjacent intervals. # Product Analytics Overview Source: https://docs.statsig.com/product-analytics/overview Understand how users experience and interact with your product through Statsig product analytics, including funnels, retention, dashboards, and user flows. ## Metrics Explorer Metrics Explorer provides a way for you to gain data-driven insights which can serve as the inspiration and basis for new insight-driven features. Once you implement these features, you can release and measure their impact with confidence, leveraging Statsig feature gates, A/B tests, or a full-blown experiment to understand how they affect core metrics. ### Charts in Metrics Explorer In Metrics Explorer, charts are the primary means of analysis, providing a versatile and in-depth look at your data. You can switch between different chart types from the top left section of the interface, while maintaining context on the selected events. This allows you to maintain focus on a particular metric or event while exploring it from various angles. We offer the following chart types: * [Metric Drilldown](/product-analytics/drilldown) * [Funnels](/product-analytics/funnels) * [Retention](/product-analytics/retention) * [Distribution](/product-analytics/distribution) * [User Journeys](/product-analytics/user-journeys) * [Lifecycle](/product-analytics/lifecycle) Get started with Metrics Explorer charts by navigating to the **Analytics** section of the Statsig console navigation menu and selecting [Metrics Explorer](https://console.statsig.com/metrics/explore). ## Dashboards [Dashboards](/product-analytics/dashboards) are a great way to share important and interesting insights and information with your team. Any chart that you have built can be saved to a dashboard. You can also save a snapshot of Feature Gates, A/B tests, and experiments to dashboards as well. You can also automatically sync experiments and feature gate snapshots to a dashboard, making it easy to keep people update to date with relevant launches and experiments as they emerge. These dashboards are auto-populated with any entity matching any tag you specify. ### Charts in Dashboards All of the charts we support in Metrics Drilldown can be added to dashboards. In addition, dashboard charts are not static. Once expanded using the \[ ] icon, charts can be modified for further exploration without making a persisted change to the dashboard itself. You can also persist changes to charts on a dashboard by clicking the edit icon or button and saving changes. Get started with Dashboards by navigating the Dashboard in the Statsig Left Navigation, and clicking on Dashboards. # Retention Chart Source: https://docs.statsig.com/product-analytics/retention Statsig retention analysis helps you understand how effectively your product maintains user interest and engagement over time across cohorts and segments. ## Overview Retention charts in Metrics Explorer help you understand how effectively your product or service maintains user interest and engagement over time. It's a great way to measure product-market fit and critical for overall product growth. By analyzing user retention patterns, you can identify whether your product or certain features are resonating with your audience, and which areas might need improvement. This insight is invaluable for making strategic decisions aimed at enhancing user experience and boosting long-term engagement. Retention chart interface showing user engagement over time ### Use Cases For our Retention Chart in Metrics Explorer, you can unlock a deeper understanding of user engagement and loyalty over time. This chart type is essential for tracking how well your product retains users after their initial interaction. * **Long-Term User Engagement Tracking**: Monitor user retention over various periods like days, weeks, or months. This helps in understanding the duration for which users stay engaged with your product after their initial interaction or a specific trigger event. This helps provide insights into the long-term appeal of your product or specific features. * **Stickiness of Specific Features:** Understand which of your product features users derive value from and correlate with them coming back to use your service. * **Sub-population Analysis:** Uncover the engagement of different sub-populations of your users by filtering your retention analysis to groups of interest. ## Creating a Retention Chart Retention charts are primarily defined by a start event, a return event, and a return window. This allows you to answer the question, "For users who did some starting event, what percentage of those users performed the return event within the given conversion window, over successive days or weeks?" Below we describe how you can create your own retention chart. ### Step 1: Define a Start Event The start event (or trigger event) is what indicates that a user has started using the product or service. Depending on the question you are trying to answer, this is often some "top-of-funnel" event such as sign-up, any active event, or an event that represents using the feature you are interested in understanding the retention of. If you're interested in understanding the retention of a specific group of users, you can optionally add a filter to scope your analysis. ### Step 2: Define a Return Event The return event is the event that indicates the user is getting the value you expect from your service. Depending on the question you are trying to answer, this can be simply coming back to your product (another active event), or use of a specific feature. Once again, you can add a filter to the selected event based on user or event properties. ### Step 3: Choose an ID Type Statsig allows you to perform individual or group analytics. This enables you to analyze the retention of users, or the retention of whole groups such as organizations or companies. ### Step 4: Choose a Return Time Window This step allows you to choose whether you want to understand daily or weekly retention. For example, when you don't expect an user to engage in a behavior on a daily basis to be considered an "active" user of that feature, it may be more helpful to use a weekly return window. This allows you to answer the question, "for users who perform the Start event, how many perform the return event within 1 *week.*" Choosing daily retention defines each of your retention cohorts as users who performed the start event on a given calendar day. The chart will then show you over your selected time range (e.g. 30 days) what percent of users in the cohort came back on that day, for each day in your time range. Choosing weekly retention defines each of your retention cohorts as user who performed the start event within a calendar week. Each cohort compared will span successive weeks. The chart will then show you over your selected time range (e.g. 30 days) what percent of users in the cohort returned during that week, for each week in your team range. ### Step 5 (Optional): Choose a Chart Granularity for Weekly Retention Charts When you select weekly retention we optionally allow you to change to the granularity of the chart. You can do this by selecting "Daily" from the granularity dropdown in the top left of the chart. Note, this is distinct from the Return Time Window. When you create a weekly retention chart and select "Daily Granularity", the chart now shows on a given day, what percent of the cohort performed the return event within the last week. E.g. day 12 weekly granularity will indicate what percent of the cohort performed the return event between the 7 days spanning day 6 and day 12, inclusive. ## Understanding a Retention Chart [Retention charts](https://www.youtube.com/watch?v=mqlHpYimik8) in Statsig are divided into two main sections, the **retention graph (j-curve)** and the **retention table (triangle chart)**. ### Retention Table (Triangle Chart) A Retention Table, or Triangle Chart, visually represents how well you are retaining users after their first interaction with your product or service. It tracks cohorts of users based on their first engagement date (as defined by your start event) and shows the percentage of these users returning (as defined by your return event) over subsequent days or weeks. The leftmost column of the chart identifies these cohorts. Across the top, you'll see time intervals (days, weeks) after the initial engagement. The chart's body is filled with percentages, each representing the proportion of users from a cohort who were active at a specific time interval. When reading the chart, a vertical analysis (looking down a column) allows you to compare the retention rates of different cohorts at the same stage of their lifecycle. In contrast, a horizontal analysis (reading across a row) shows how a single cohort's retention evolves over time. High numbers in the first few columns suggest strong initial user engagement, while consistent percentages over longer intervals indicate successful long-term retention strategies. The columns in the chart start at day/week 0, which is earliest day/week in the time range. The right-most column will have retention for the most recent day/week. When viewing retention chart analyzing daily retention for a given cohort, each subsequent column indicates the percentage of the cohort that performed the return event on that calendar day. When viewing retention chart analyzing weekly retention with weekly granularity for a given cohort, each subsequent column indicates the percentage of the cohort that performed the return event within that week. When viewing retention chart analyzing weekly retention with daily granularity for a given cohort, each subsequent column indicates the percentage of the cohort that performed the return event on that calendar day or within the 6 days prior. ### Retention Graph A Retention Graph typically plots the percentage of retained users against time, offering a visual representation of user retention. The X-axis represents time, segmented into daily or weekly intervals since the users' first interaction with the product. The Y-axis indicates the percentage of the original users who remain active (performed the return event). **Reading the Graph** The Retention Graph provides insights into user behavior by illustrating the rate at which users disengage over time. * A **declining curve** followed by a flattening of the curve is common, suggesting a drop in user interest after the first interaction and then stabilization among a core user group. The higher percentage at which the curve flattens out, the better the retention and health of the product. * A **smiling curve** shows a decline in retention initially, followed by an upward trend at later stages. This pattern often results from successful re-engagement strategies or product improvements that bring back users who had previously disengaged * A **continuously declining curve** indicates a consistent loss of users over time. This trend suggests that the product or service is not retaining its user base effectively, often pointing to issues in user satisfaction or engagement. Key aspects to observe: 1. **Initial Drop-off Rate:** The steepness of the initial decline indicates how many users stop using the service after their first experience. 2. **Long-term Engagement:** The slope in the later stages of the graph shows the long-term user engagement. A flatter slope at this stage means better user retention. 3. **Trends Over Time:** Comparing multiple graphs over different time periods can reveal the impact of product changes or external factors on retention. ## Scoping to Specific Cohorts You can view plots for specific or many cohorts. By default only the "All cohorts" retention curve is plotted. To scope to a single specific cohort, click anywhere on the row in the table associated with that cohorts. To scope to many specified cohorts, leverage the check boxes on the left most portion of the table to select the cohorts that you would like to compare retention curves of. # Topline Alerts Source: https://docs.statsig.com/product-analytics/topline-alerts Configure Statsig topline alerts to get notified in Slack or email when key product metrics shift beyond a fixed threshold or relative change in value. Topline Alerts are in a limited beta. Please reach out if you would like these enabled for your Project. Topline Alerts are available in Statsig Cloud and Statsig Warehouse Native with support available for three types of Topline Alerts: | Alert Condition Type | When to Use | Example | | -------------------- | ----------------------------------------------------------------------- | ----------------------------------------------------------- | | **Threshold** | Use when you want to stay above or below a fixed number. | “Alert me when P90 latency spikes above 15 seconds.” | | **Change** | Use when the absolute size of the change matters. | “Alert me when hourly P90 latency increases by 10 seconds.” | | **Change (%)** | Use when the relative size of the change matters more than raw numbers. | “Alert me when hourly P90 latency increases by 50%.” | ## Creating a Topline Alert Go to Analytics → Topline Alerts in the product menu. Click +Create and give the new alert a name.

    Pick the data you want to monitor.

    • On Statsig Warehouse Native: Select a Metric Source, filter, and group by your desired dimensions.
    • On Statsig Cloud: Select the event, aggregation, filters, and group-by conditions for this alert.
    We're setting up an alert to monitor P90 latency for mex\_query events, filtering out internal employee queries and grouping by the hadGroupBy dimension. Select events or metrics for a Topline Alert

    The preview shows how your metric is trending with the current setup. Confirm values look correct, or open the metric in Metrics Explorer for deeper analysis.

    Topline Alert preview showing trending metric

    The preview updates along with each change. Define the:

    • Condition type (threshold, change, change %)
    • Directionality
    • Alert and Warn values
    • Evaluation window
    On Warehouse Native: Define the evaluation frequency, lookback, and max delay—they directly influence warehouse compute costs. Configure alert conditions for a Topline Alert

    Notifications go to email, the Statsig Console, and Slack (if connected). Project-wide defaults live in Settings.

    • Draft a clear, actionable message subscribers receive when the alert fires.
    • Add subscribers.
    • Set alert priority.
    • Configure re-notification rules if alerts should resend while conditions hold.

    Once saved, triggered alerts appear at the top of the page. From here you can:

    • View samples of the event.
    • Open the trend in Metrics Explorer.
    • Mute the alert temporarily if it is noisy or already under investigation.
    Topline Alerts table with active alerts
    ## Diagnostics Head to the Diagnostics tab to review alert history, inspect samples, jump into Metrics Explorer, or mute noisy alerts. Diagnostics tab showing alert history *** ## Interested in More? * 👉 Check out how to [create a Topline Alert on log lines](/infra-analytics/topline-alerts-logs) * 🔔 Learn how to set up [team Slack notifications](/integrations/slack/#team-notifications) # User Journeys Source: https://docs.statsig.com/product-analytics/user-journeys Statsig user journeys show the many paths users take through your product so you can better understand the end-user experience and identify drop-off points. ## Overview The User Journeys chart shows you the many paths users are taking through your product so you can better understand the end-user experience. User journeys sankey diagram visualization ### Use cases * **Event Pathways:** Follow along the most common paths users are taking through your product * **Identifying Drop-off Points:** Pinpoint where users are dropping off within key product flows * **Discovering Unexpected Behavior:** Uncover surprising usages and iterate on your product ## Creating a User Journeys Chart ### Step 1: Choose a starting or ending event The selected event will indicate where you want your analysis to begin. You can choose whether you want to explore a journey that either starts or ends with a specific event. This allows you to understand where people go from some starting event, or where people come from for some ending event. Starting events are often some "top-of-funnel" event such as sign-up or log-in but can also be an event trigger for any new feature you introduce into your product. You can optionally add filters to the event to scope your analysis to a specific group. ### Step 2: Choose an ID type Statsig allows you to perform individual or group analytics. This enables you to analyze the paths of registered users with User IDs, Stable IDs, or your own custom IDs. ### Step 3: Define your measurement criteria Under the "Measured as" section, you can further refine your analysis by choosing between total conversions or unique conversions. Under total conversions, users can re-enter the flow every time they trigger the start event. Under unique conversions, the users will only show up in the path once. Next, you can specify the duration of your observation. Defaulted to 1 hour, you can change the length of time users have to convert to the next event. This is can help validate if users are passing through flows at a pace you expect. You can gain additional context by breaking the User Journey out by an event property. This allows you to choose a specific event property and view the user journey granularity expanded by the different property values of the select property. This is similar to a group-by. For example, you can view user journeys split out by operating system. Finally, you can further filter down your exploration by excluding user paths that include a specified event. When editing the measurement criteria, remember to click "Run Query" to confirm your selections. ### Step 4: Adjust your chart visualization Finally, at the top bar of the chart, you can edit the conversion percent threshold, hide events, and expand your time window. The conversion percent threshold sets the minimum percentage of conversions needed to be considered relevant. If below the threshold, the event will be consolidated under "Others". The "Hide Events" selector allows you to filter out extraneous events you don't want to see. Finally, adjusting the time selector allows you to view user journeys during the time window you most care about. ## Using User Journeys After selecting your start event, you will see a sankey diagram of all next steps users took. By clicking any node, the graph will continue on to display the next events users triggered. Notice that under each event is the total number of events counted and the percentage of users who triggered this particular event. You can continue clicking on nodes to explore user flows through your product. # Users Tab Source: https://docs.statsig.com/product-analytics/users-tab The Statsig Users tab gives an event-by-event level understanding of how individual users interact with your product, useful for debugging and support. ## Overview The Users section of Statsig gives you an event-by-event level understanding of how users (and other Unit IDs) are leveraging your product. This allows you to diagnose issues and understand user behavior at an extremely granular level with event logs, session streams, and session replays. It is also a singular place to manage feature rollout and experiment overrides. ## Exploring the Users Tab When you enter the Users tab, you need to input a specific ID. Be sure to also select the matching Unit ID type to the ID you are looking for. There are also many context specific entry points to the Users Tab within Statsig. A Users Tab query surfaces the following information: * **Properties-** Additional context around this ID including email, country, OS, and Browser. For customers with user profiles, you can see all of the user profile properties we have for a given ID. * **Log History & Event Details-** History of events and exposures for this ID, limited to 5000 rows. These can be viewed as an event log stream or a session stream * **Overrides-** The ability to manually override an ID into a given feature gate or experiment variant * **Session Replays-** A sample of session replays the user has triggered ## Events ### Log Streams Under the Events tab, you can see into the events and exposures this user triggered by seeing their event log stream. Here, you can diagnose what features this user was most interested in or what experiment group they were in. Event log stream interface ### Session Streams You can also switch to a session-centric view of users events by switching from "**Log Stream**" to "**Session Stream**" to get an in-depth look at all events the user has triggered during each of their sessions. You can dive further into these two explorations by changing the time window, hiding noisy or uninteresting events, and filtering to view only events or experiment / gate exposures. Session stream interface showing user events grouped by session ## Overrides Under the Overrides tab, you have the ability to override an ID into a given experiment variant or feature gate rollout. If there is already an active override for an ID on a particular entity, this will be surfaced within this section (and can be modified inline). Any overrides set in the Users Tab will be synced to the "Overrides" section of the entity in question (and can be edited/ removed from this interface as well). ## Session Replays Under Session Replays, you can quickly jump to a sample of session replays the particular user has triggered to get even more insight into their behavior. To learn more, check out our documentation on [Session Replays](/session-replay/overview). # Manage an Ongoing Release Pipeline Source: https://docs.statsig.com/release-pipeline/actions Control and manage Statsig release pipelines in progress using actions like approve, pause, skip, and abort to safely steer rollouts through stages. # Managing Release Actions Once a Release Pipeline is triggered, you can control its progression using the following actions. These controls allow you to safely test, pause, fast-track, or halt propagation of feature gate and config changes across pipeline phases. Release pipeline management actions ### Approve **What it does:** Kick off a phase that requires a manual approval before rollout begins. This is useful when human verification is required before changes move forward. **How to use it:** 1. The pipeline will automatically pause at the last completed phase awaiting approval 2. You will receive a notification prompting you to take action 3. Go to the Release Pipeline status page 4. Click the ⋯ menu and select 'Approve' 5. Confirm the action — the next phase will begin rolling out ### Pause **What it does:** Stops the bake timer between two phases. Useful if you want to delay rolling out the next phase in order to investigate the current phase. :::note Pause does not stop the current phase — only the timer to move to the next phase. ::: **How to use it:** 1. Go to the Release Pipeline status page 2. Click the ⋯ menu and select 'Pause' 3. Confirm to pause the bake timer ### Unpause **What it does:** Resumes a previously paused bake timer, allowing the pipeline to move to the next phase after the remaining wait time. **How to use it:** 1. Go to the Release Pipeline status page 2. Click the ⋯ menu and select 'Unpause' 3. Confirm to resume the bake timer ### Skip **What it does:** Immediately skips the current phase and moves rollout to the next defined phase. Useful for fast-tracking safe changes. **How to use it:** 1. Go to the Release Pipeline status page 2. Click the ⋯ menu and select 'Skip' 3. Confirm to resume the bake timer ### Abort **What it does:** Halts the release process. The feature gate or dynamic config will revert to its pre-release state, and no further changes will propagate. **Important:** Aborting a release is an irreversible action. Once aborted, you will need to trigger a new release to restart the process with any modifications. **How to use it:** 1. Go to the Release Pipeline status page 2. Click the ⋯ menu and select 'Abort' 3. Confirm to resume the bake timer ### Full Roll Out **What it does:** Skips all intermediate phases and releases the latest version across all environments and custom attributes once. **How to use it:** 1. Go to the Release Pipeline status page 2. Click the ⋯ menu and select 'Full Roll Out' 3. Confirm to resume the bake timer # Create and Manage Release Pipelines Source: https://docs.statsig.com/release-pipeline/create-and-manage Create, configure, and manage Statsig Release Pipelines for controlled, multi-stage feature rollouts with approvals, soak times, and rollback safeguards. ## Creating a New Pipeline To create a new Release Pipeline: 1. Log into the [Statsig console](https://console.statsig.com) 2. Navigate to **Settings** > **Feature Management** 3. Under Release Pipelines, click the **Create** button 4. Enter a descriptive name for your pipeline 5. Click **Create** to proceed to the configuration page Create release pipeline interface ## Configuring Phases Each pipeline consists of one or more phases, with each phase representing a distinct release target. ### Adding Phases For each phase in your pipeline: 1. Add one or more release rules 2. Select a required **Environment** for the rule 3. Optionally add custom field conditions for more precise targeting Phase configuration ### Setting Phase Transitions Control how your phases progress with these transition options: | Transition Type | Description | | ------------------ | -------------------------------------------------------------------------------- | | **Require Review** | Requires manual approval from an authorized user before starting the phase | | **Time Interval** | Automatically proceeds to the next phase after a specified duration (in minutes) | :::note You can combine both options in a single phase. When both are used, the time interval will only begin counting down after the required approval is given. ::: Phase transition conditions ## Managing Existing Pipelines ### Updating a Pipeline To modify an existing pipeline: 1. Click on the pipeline name from the list 2. Make your desired edits to any section 3. Click **Save** to apply your changes **Important:** Pipelines with active rollouts currently in progress cannot be modified until those rollouts complete or are aborted. ### Viewing Pipeline References There are two ways to see which feature gates and dynamic configs are currently using a pipeline: 1. Project Settings * Navigate to Project Settings * Click on the Feature Management menu in the left-rail * Navigate to Release Pipelines section * Click on the **References** column against each Release Pipeline * This will show all feature gates and dynamic configs that are currently attached to a specific pipeline Reference 2. Feature Gates / Dynamic Configs Page * Navigate to Feature Gates / Dynamic Config list view * Filter by 'Release Pipeline' current status * This will show all feature gates and dynamic configs with an ongoing release pipeline Feature gates and dynamic configs with release pipeline status ## Opting Out Environments from Release Pipelines By default, all environments will trigger Release Pipelines when changes are made. However, you can configure specific environments to be exempt from this behavior. When an environment is opted out from Release Pipelines: * Changes made exclusively to that environment will not trigger a Release Pipeline * This allows for quick environment-specific adjustments without initiating the full release process ### How to Opt Out an Environment To exclude an environment from triggering Release Pipelines: 1. Navigate to **Settings** in the Statsig console 2. Under **Keys & Environments**, select **Environments** 3. Click on the environment you wish to opt out 4. Unselect the **Pipeline-required Environment** option 5. Click **Save** to apply your changes Environment opt-out setting # Release Pipeline Overview Source: https://docs.statsig.com/release-pipeline/overview Learn how Statsig Release Pipelines enable sophisticated multi-stage rollout strategies across environments with approvals, soak periods, and auto-rollback. Release Pipelines enable sophisticated, multi-stage rollout strategies that respect your infrastructure boundaries, providing greater control over feature deployments. ## When to Use Release Pipelines Consider implementing Release Pipelines when you need to: * Implement infrastructure-aware deployment strategies beyond what traditional feature flags offer * Safely roll out changes in complex, distributed systems with minimal risk * Deploy progressively across environments (dev → staging → prod) * Target specific infrastructure segments with precision (e.g., prod-us-west → prod-us-east → prod-eu) * Control progression between stages with time intervals or manual approvals * Monitor each deployment stage before proceeding to the next * Enable instant rollbacks if issues arise during any phase **Especially valuable for:** * Platform engineering teams managing multi-environment, multi-region infrastructure * DevOps practitioners implementing progressive delivery strategies * SREs responsible for maintaining system reliability across complex deployments * Organizations with mission-critical services that cannot afford downtime ## Key Concepts | Concept | Description | | -------------------- | ----------------------------------------------------------------------------------------------- | | **Release Pipeline** | Defines the complete release strategy for a feature gate or dynamic config change | | **Phase** | A distinct stage in the pipeline with specific conditions targeting designated release segments | | **Trigger** | An event initiated when a feature gate or dynamic config starts using a Release Pipeline | | **Action** | Controls that can be applied to an active Trigger to manage the release process | ## Getting Started Release Pipeline is currently only supported in Statsig's [Server Core SDKs](https://www.statsig.com/blog/introducing-statsig-server-core-v0-1-0). Legacy SDKs will continue to work but will not get the full features of release pipelines. Follow these tutorials to begin implementing Release Pipelines: * [Create and Manage Release Pipelines](/release-pipeline/create-and-manage) * [Trigger a Release Pipeline](/release-pipeline/trigger) * [Manage an Ongoing Release Pipeline](/release-pipeline/actions) ## Current Limitations * Experiments are not supported in release pipeline * Resalt/Disable/Delete/Archive/Launch actions won’t trigger a release pipeline * [Scheduled rollouts](/feature-flags/scheduled-rollouts) are not supported to work in conjunction with release pipeline # Trigger a Release Pipeline Source: https://docs.statsig.com/release-pipeline/trigger Attach Statsig Release Pipelines to feature gates and triggers to start controlled rollouts automatically when conditions or approvals are met. A Release Pipeline is activated when you make changes to a feature gate or dynamic config that has a pipeline attached to it. ## Attaching a Pipeline Before triggering a release, you must first attach a pipeline to your feature gate or dynamic config. The Statsig console offers two methods for attaching a Release Pipeline. ### During Feature Creation You can select a Release Pipeline directly in the creation modal when setting up a new feature gate or dynamic config: Attaching a pipeline during creation ### To an Existing Feature For an existing feature gate or dynamic config, you can attach a Release Pipeline through the sidebar settings: Attaching a pipeline via sidebar :::note You must have at least one Release Pipeline created before it will appear in the dropdown menu. See [Create and Manage Pipelines](/release-pipeline/create-and-manage) for instructions on creating pipelines. ::: ## Starting a Rollout When a Release Pipeline is attached, making changes to your feature gate or dynamic config will automatically initiate the pipeline process: 1. Make your desired changes to the feature gate or dynamic config 2. Click **Save** to commit the changes 3. A confirmation dialog will appear, informing you that changes will progress through the pipeline 4. Review the information and click **Confirm** to proceed The system will then begin the rollout following the phases defined in the attached pipeline. Confirmation dialog when triggering a pipeline ## Viewing Release Status Once a Release Pipeline is triggered, you can monitor its progress: 1. At the top of the feature gate or dynamic config page, a status banner will appear 2. This banner displays the current phase and overall progress through the pipeline Status banner showing release progress For information about controlling an ongoing release, including approvals and aborts, see [Managing Release Actions](/release-pipeline/actions). ## Frequently Asked Questions **Q: Can I attach different Release Pipelines to different feature gates?**\ A: Yes, each feature gate or dynamic config can use a different pipeline based on its specific rollout needs. However, a single feature gate or dynamic config can only have one Release Pipeline attached at a time. **Q: What happens if I need to cancel a release in progress?**\ A: You can abort an ongoing release using the actions menu in the release details view. See [Managing Release Actions](/release-pipeline/actions) for complete instructions on how to abort a release. # Client vs Server SDKs Source: https://docs.statsig.com/sdks/client-vs-server Compare Statsig client and server SDKs to choose the right SDK for your platform based on security, latency, identity, and supported features. Statsig offers client and server SDKs to enable experimentation and feature management across different parts of your application. This document outlines when you should choose each. ## Overview **Client SDKs** run in code that executes on end-user devices, like a website, mobile app, video game, or smart TV app. **Server SDKs** run on your servers (typically in the cloud), like a web server or API server. Many customers **deploy both Server and Client SDKs**, letting them gate features on both the client- and server-side for the most control. While this is common, you can get value from Statsig by starting with just one. ## Usage Client and Server SDKs follow a similar pattern of setup and usage in-code: 1. **Initialize:** Setup the SDK and download the latest values 2. **Check an experiment/gate:** Reference those values to assign an experiment or flag a feature 3. **Logging custom events:** log important app metrics to power your analyses After initialization, both Client and Server SDKs evaluate experiments/gates *without a network request*, and typically in less than 1ms. Checks in the Statsig SDKs are designed to be very efficient. ## Conceptual Differences: * Data Privacy: The *Server* SDK is presumed to be a secure, multi-user environment, so Server SDKs have access to the full ruleset describing each experiment and gate. Client SDKs fetch only the value for a single user, avoiding exposing the definition of your configurations. | | Server | Client | | -------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Privacy | Your servers are presumed to be a secure, multi-user environment, so Server SDKs have access to the definition of all configurations in your project | All evaluations are precomputed on Statsig servers and sent down to you client applications. Names are obfuscated, but a savvy user may be able to glean information from the raw response | | Evaluation Performance | Evaluations are done real-time, without a network request. Very complex configurations can take longer to compute, but in practice, this is rarely an issue. | As all evaluation is precomputed, gate and experiment checks are effectively a dictionary lookup with some computation used for creating and flushing exposure events | | Initialization Performance | The SDK will make an upfront request for configuration files, then continually poll for any changes to your configurations, updating its internal state when a change is detected | Client SDKs download configurations when you initialize, before which, the SDK may not have usable values. The values aren't updated mid-session unless you explicitly call updateUser. Additional options are available for performant initialization, see [Initializing](/client/concepts/initialize) | | Users | Server SDKs are designed to run against multiple users, and all SDK methods require a user object for evaluation/logging | Client SDKs are designed to be run with one user at a time. All evaluations are loaded once up front during initialization, and every event logged uses that user object | | Infrastructure | Server SDKs require you to host your own backend services | Client SDKs run entirely on the client and utilize Statsig's servers | ## Usage Differences: | | Server | Client | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Initializing | Requires only a secret key, downloads the entire ruleset and syncs it in the background | Requires a client key and a user object. Before/during initialization, the SDK will attempt to fallback to cached values. | | Checking an Experiment | Requires a user object which is evaluated locally (without a network request) against a ruleset persisted in memory | Does not require a user object, uses a dictionary lookup for values fetched during initialize() | | User Identifiers | Pass any and all useful user identifiers | Pass any useful identifiers, the SDK also generates a "StableID", Statsig's anonymous ID you can use to experiment on a user per-device | | Logging Events | Requires a user object | Does not require a user object. Note, there is some risk of adblocking log events on client SDKs, which can be minimized by setting up a [Custom Proxy](../custom_proxy) | | Flushing Events | Batched and flushed by the SDK every 60 seconds | Batched and flushed by the SDK every 10 seconds | | Updating Configurations | Poll Statsig servers for updates every 10 seconds by default (configurable), Streaming possible with some Server SDKs and the [Statsig Forward Proxy](/server/concepts/forward_proxy) | Configuration persists until next `initialize` or `updateUser` call, recommended to call `initialize` at the start of each user session | ## Difference in initialize/update logic: **Client SDKs:** * Configuration persists until next `initialize` or `updateUser` call * Recommended to call `initialize` at the start of each user session Client SDKs **Server SDKs:** * Poll Statsig servers for updates every 10 seconds by default (configurable) * Some SDKs support grpc streaming updates via connecting to the [Statsig Forward Proxy](/server/concepts/forward_proxy) Server SDKs ## Available SDKs ### Client SDKs * [Javascript](/client/javascript-sdk) * [React](/client/React) * [React Native](/client/ReactNative) * [Expo](/client/Expo) * [iOS](/client/iosClientSDK) * [Android](/client/Android) * [.Net](/client/DotNet) * [Unity](/client/Unity) * [Roku](/client/Roku) * [C++](/client/CPP) * [Dart/flutter](/client/Dart) ### Server SDKs * [Node.js](/server/nodejsServerSDK) * [Java](/server/javaSdk) * [Python](/server/pythonSDK) * [Go](/server/golangSDK) * [Ruby](/server/rubySDK) * [.NET Core](/server-core/dotnetCoreSDK) * [C++](/server-core/cpp-core) * [PHP](/server/phpSDK) * [Rust](/server/rustSDK) For more detailed information on each SDK, please refer to their respective documentation pages. Got questions? Join the Statsig engineering and product team on the [Statsig Slack channel](https://statsig.com/slack) and ask away! # SDK Debugging Source: https://docs.statsig.com/sdks/debugging Troubleshoot Statsig SDK feature gate and experiment evaluations using built-in diagnostics, evaluation reasons, exposure logs, and targeted debug logging. ## Debugging Tools When a user sees an unexpected value, start with the tooling built into the Statsig console and SDKs. The sections below outline where to look and what each signal means. ### Diagnostics & Log Stream Every gate, config, experiment, and layer has both a **Setup** and **Diagnostics** tab. The diagnostics view highlights pass/fail rates and bucketing counts so you can spot anomalies over time. Diagnostics tab showing pass and fail counts Scroll to the log stream to inspect individual evaluations. Entries arrive within seconds for both production and non-production environments. Log stream showing recent exposures Enable **Show non-production logs** in the diagnostics view to surface checks coming from test keys and development builds. ### Logging Levels and Expected Information Statsig SDKs emit runtime logs across four verbosity levels: * `Debug` – deep tracing meant for onboarding or issue triage. * Missing gate/config warnings when a definition is unavailable. * Step-by-step messages that follow SDK initialization and evaluations. * `Info` – healthy lifecycle events for day-to-day operation. * Initialization summaries with source and SDK version details. * Notifications when the configuration store is populated. * `Warning` – recoverable issues that might affect functionality. * Non-critical errors automatically handled by the SDK. * gRPC reconnection attempts or similar transient network events. * `Error` – critical failures that block expected behavior. * Initialization timeouts or outright failures. * Fallback notices that indicate gRPC is unavailable or misconfigured. ## Evaluation Details Click any exposure in the log stream to drill into the precise rule, user attributes, evaluation reason, SDK metadata, and server timestamps. That detail is often enough to pinpoint why a user received a specific value. Evaluation details modal with rule match information ### Evaluation Reasons Evaluation reasons answer two questions: where the SDK sourced its definitions and why a particular value was returned. Use them to distinguish between healthy results, overrides, and error states. Evaluation reason popover showing source and reason #### Data Source | Source | Description | Type | Debugging Suggestions | | ------------------------------------------------ | ----------------------------------------------------------------- | ------- | ------------------------------------------------------------ | | `Network` | Values fetched during initialization from Statsig servers. | Normal | — | | `Bootstrap` | Supplied during bootstrap (often from a server SDK). | Normal | — | | `Prefetch` | Loaded via the `prefetchUsers` API (JS only). | Normal | — | | `NetworkNotModified` | Network request succeeded but cached values were already current. | Normal | — | | `Sticky` (legacy) | Persisted from a sticky evaluation. | Normal | — | | `LocalOverride` (legacy) | Set locally via override APIs. | Normal | — | | `Cache` | Served from local cache because network values were unavailable. | Warning | Ensure `initialize()` has completed before checks run. | | `InvalidBootstrap` / `BootstrapPartialUserMatch` | Bootstrap values were generated for a different user profile. | Error | See [Fixing InvalidBootstrap](#invalid-bootstrap). | | `BootstrapStableIDMismatch` | StableID differed between bootstrap and runtime user. | Error | See [BootstrapStableIDMismatch](#bootstrapstableidmismatch). | | `Error` | A generic evaluation failure that was logged to Statsig. | Error | Ask for help in [Slack](https://statsig.com/slack). | | `Error:NoClient` (JS) | No Statsig client was found in context. | Error | Wrap checks in `` or equivalent. | | `Unrecognized` (legacy) | The definition was missing from the initialize payload. | Error | Confirm the config exists and the correct API key is used. | | `NoValues` | Initialization ran but failed to retrieve values. | Error | Verify client key and network connectivity. | | `Loading` | Initialization is still in progress. | Error | Await `initializeAsync()` or guard checks until ready. | | `Uninitialized` | Initialization never started. | Error | Call `initializeAsync()`/`initializeSync()` explicitly. | | `UAParserNotLoaded` | UA parsing was disabled while targeting relies on it. | Error | Remove UA-based targeting or re-enable parsing. | | `CountryLookupNotLoaded` | Country lookups were disabled while targeting relies on them. | Error | Avoid IP-based targeting or re-enable lookups. | #### Reason (new SDKs only) | Reason | Description | Type | Debugging Suggestions | | --------------- | ----------------------------------------------------------------------------- | ------ | ---------------------------------------------------------------------------------------------- | | `Recognized` | The definition was present and matched the current values. | Normal | — | | `Sticky` | Result persisted because `keepDeviceValue` was set. | Normal | — | | `LocalOverride` | Value came from a developer override. | Normal | — | | `Unrecognized` | Definition missing from the payload. | Error | Confirm targeting and [Target Apps](/sdk-keys/target-apps). | | `Filtered` | Definition filtered from `/initialize` because the default value was `false`. | Error | Check [client bootstrapping](/client/concepts/initialize#bootstrapping-overview) or targeting. | #### Data Source | Source | Description | Type | Debugging Suggestions | | ------------------------ | ------------------------------------------------------------- | -------- | --------------------------------------------------------------------------------------- | | `Network` | Configurations loaded from Statsig servers at initialization. | Normal | — | | `Bootstrap` | Server SDK bootstrapped with precomputed values. | Normal | — | | `DataAdapter` | Values retrieved from a data adapter or external store. | Normal | Review your [data adapter setup](/server/concepts/data_store#dataadapter-or-datastore). | | `LocalOverride` (legacy) | Value supplied via server-side override APIs. | Normal | — | | `StatsigNetwork` | Proxy/streaming fell back to Statsig APIs directly. | Fallback | Revisit your [proxy configuration](/server/concepts/forward_proxy/). | | `Uninitialized` | SDK has not completed initialization. | Error | Ensure initialization succeeds before evaluations. | | `Unrecognized` (legacy) | Definition missing from the cached payload. | Error | Confirm configuration exists and the API key is correct. | #### Reason (new SDKs only) | Reason | Description | Type | Debugging Suggestions | | --------------- | ------------------------------------------------ | ------ | -------------------------------------------------------- | | `LocalOverride` | Result supplied by a developer override. | Normal | — | | *None* | Successful evaluation with the expected payload. | Normal | — | | `Unrecognized` | Definition missing from the payload. | Error | Confirm configuration exists and the API key is correct. | | `Unsupported` | SDK lacks support for a condition/operator. | Error | Upgrade to the latest SDK version. | | `Error` | Generic evaluation failure captured by the SDK. | Error | Ask for help in [Slack](https://statsig.com/slack). | ### Evaluation Times Evaluation timestamps reveal whether an SDK is serving fresh definitions. An up-to-date LCUT (Last Config Updated Time) indicates the SDK has the latest changes. If LCUT lags far behind, users may be seeing stale values—either because a browser tab stayed open or because a server integration is unable to sync. | Time Field | Description | SDKs | | ------------ | ----------------------------------------------------------- | ---------------------------------------- | | `LCUT` | Time of the most recent config change reflected in the SDK. | JavaScript (incl. React & RN), iOS, Dart | | `receivedAt` | Timestamp when the client obtained the current values. | JavaScript (incl. React & RN), iOS, Dart | | Time Field | Description | SDKs | | ---------------- | ------------------------------------------------------- | --------------------------- | | `initTime` | LCUT captured when the server SDK initialized. | All server SDKs except Rust | | `configSyncTime` | Latest LCUT received from the network or data adapter. | All server SDKs except Rust | | `serverTime` | Current server clock time when the evaluation was made. | All server SDKs except Rust | ### Mocking Statsig / Local Mode Use the following tools to validate code paths without hitting the network: * **Local mode:** Set `localMode` to `true` so the SDK skips network calls and returns default values—ideal for tests and offline environments. * **Override APIs:** Call `overrideGate` and `overrideConfig` to force specific values for an individual user or globally. Combine both techniques to exercise each branch of your application safely before shipping. ### Client SDK Debugger Inspect the exact values a client SDK is using by opening the Client SDK Debugger. It exposes the active user object and every gate/config tied to that client. * **JavaScript / React:** Use the [Chrome extension](https://github.com/statsig-io/statsig-sdk-debugger-chrome-extension). * **iOS:** Call `Statsig.openDebugView()` in [v1.26.0](https://github.com/statsig-io/statsig-kit/releases/tag/v1.26.0) or later. * **Android:** Call `Statsig.openDebugView()` in [v4.29.0](https://github.com/statsig-io/android-sdk/releases/tag/4.29.0) or later. Accounts that sign in to the Statsig console via Google SSO are currently unsupported by the Chrome extension. | Landing | Gates List | Gate Details | Experiment Details | | ------------------------------------------ | ---------------------------------------- | ------------------------------------------ | ------------------------------------------------ | | Client debugger landing view | Client debugger gates list | Client debugger gate details | Client debugger experiment details | ## FAQs For SDK-specific edge cases, check each SDK’s FAQ or contact us in the [Statsig Slack community](https://statsig.com/slack). ### Invalid Bootstrap `InvalidBootstrap` occurs when a client SDK is bootstrapped with values generated for a different user profile. Always ensure the bootstrap user exactly matches the runtime user. ```js theme={null} // Server side const userA = { userID: 'user-a' }; const bootstrapValues = Statsig.getClientInitializeResponse(userA); // Client side const bootstrapValues = await fetchStatsigValuesFromMyServers(); const userB = { userID: 'user-b' }; // <-- Different from userA await Statsig.initialize('client-key', userB, { initializeValues: bootstrapValues }); ``` Even subtle differences count as a mismatch—adding `customIDs` or other attributes results in a distinct user object. ```js theme={null} const userA = { userID: 'user-a' }; const userAExt = { userID: 'user-a', customIDs: { employeeID: 'employee-a' } }; ``` ### BootstrapStableIDMismatch `BootstrapStableIDMismatch` is similar but focuses on `stableID`. Client SDKs generate a stable ID automatically when one is not provided, so mixing empty user objects between server and client code can cause drift. ```js theme={null} // Server side const userA = {}; const bootstrapValues = Statsig.getClientInitializeResponse(userA); // Client side const bootstrapValues = await fetchStatsigValuesFromMyServers(); const userB = { stableID: '12345' }; // <-- Server user lacked a stableID await Statsig.initialize('client-key', userB, { initializeValues: bootstrapValues }); ``` Even if both sides start with `{}`, the client-generated stable ID may not match the server’s, leading to the same warning. ```js theme={null} const userC = {}; // Client SDK auto-generates a stableID await Statsig.initialize('client-key', userC, { initializeValues: bootstrapValues }); ``` ### Environments SDKs inherit their environment from initialization options. If none is provided, the SDK defaults to production. To verify which environment a user evaluated under, open the diagnostics log stream and inspect the `statsigEnvironment` property attached to the exposure. ### Maximizing Event Throughput Python SDK v0.45.0+ introduces tunables that help handle high event volume without drops. The SDK batches events and retries failures in the background. When throughput spikes, adjust these options to reduce the chance of dropped events: * **`eventQueueSize`** – Number of events flushed per batch. Larger batches increase throughput but use more memory; keep the value under \~3000 to avoid oversized payloads. * **`retryQueueSize`** – Number of batches kept for retrying. The default is 10; raise it to retain more data at the cost of memory. Tune both settings to match your traffic profile and keep pipelines healthy. # SDK Overview Source: https://docs.statsig.com/sdks/getting-started Get started with a Statsig SDK by choosing your platform, installing the package, initializing with your API key, and evaluating your first gate. Statsig's SDKs are the in-code tool you'll use to show experiment variants, flag your features, and log your key business metrics. Statsig's SDKs: * **Abstract the complexity away:** we handle caching, retry mechanisms, networking, etc. * **Encourage best practice:** built around parameters & language-specific best practice * **Broad deployment support:** 30+ SDKs across clients, servers, the edge, and more. *** ## When Will I Use the Statsig SDKs? ### 1. **Targeting & Assignment** Decide in-code who sees new features and experiment variants. Target based on any **user attributes** (e.g., location, device type) or **environment-level attributes**, letting you fine-tune rollouts and experiment enrollment. ### 2. **Logging Events** Log your key business metrics to power experiment & product analytics. Your events are piped into Statsig without any extra effort. For web-based platforms (e.g., JavaScript, React), Statsig also supports **[Autocaptured Events](/webanalytics/overview)**. *** ## Choosing the Right SDK: Client vs. Server Statsig offers both **client-side** and **server-side** SDKs, each suited to different use cases: * **Client-Side SDKs**: Designed for user-facing applications where events are logged directly from the browser or mobile app. These SDKs work in real-time, providing instant feedback on user behavior. * **Server-Side SDKs**: Ideal for backend services, allowing you to control experiments, feature flags, and log server-side events. Server-side SDKs offer more control, especially for use cases involving system-level actions or business logic. For a detailed comparison, refer to the [Client vs Server SDK Overview](/sdks/client-vs-server). Additionally, for frameworks like **Next.js** that bridge client and server-side logic, we offer guides like the [Next.js SDK](/client/Next) to help you integrate. *** ## Explore SDKs Statsig offers SDKs for a wide variety of platforms to suit any codebase or deployment preference: ### Client SDKs Browser JavaScript Client-Side React Bare React Native SDK Next.js SSR, SSG & Client-Side Angular bindings for Javascript SDK iOS, MacOS, tvOS SDK Android Kotlin/Java SDK Client SDK for .NET framework Roku Brightscript SDK Unity game engine SDK Flutter/Dart Mobile App SDK C++ client-side SDK ### Server Side SDKs Node.js server SDK Java server SDK Python server SDK Go server SDK Ruby server SDK .NET server SDK PHP server SDK Rust server SDK C++ server SDK *** ## Next Steps: Installing Your SDK 1. **Select Your SDK**: Choose the client or server SDK that fits your platform from the lists above. 2. **Follow the Installation Guide**: Each SDK has an installation guide that walks you through the setup, ensuring a smooth integration with your app. 3. **Start Experimenting**: Once integrated, you can begin setting up feature flags, running experiments, and logging events for analysis. If you run into any questions or need help with installation, feel free to reach out via our [Slack Community](https://statsig.com/slack). # How Evaluation Works Source: https://docs.statsig.com/sdks/how-evaluation-works How Statsig SDKs evaluate feature gates, experiments, and dynamic configs, including rule ordering, conditions, and exposure logging behavior. ## Evaluation's importance The essential function of the Statsig SDKs is reliable, consistent, incredibly performant allocation of users to the correct bucket in your experiment or feature gate. Understanding how we accomplish this can help you answer questions like: * Why do I have to pass every user attribute, every time? * Why do I have to wait for initialization to complete? * When do you decide each users' bucket? ## How Evaluation Works Evaluation in Statsig is deterministic. Given the same user object and the same state of the experiment or feature gate, Statsig always returns the same result, even when evaluated on different platforms (client or server). Here's how it works: 1. **Salt Creation**: Each experiment or feature gate rule generates a unique salt. 2. **Hashing**: The user identifier (e.g., userId, organizationId) is passed through a SHA256 hashing function, combined with the salt, which produces a large integer. 3. **Bucket Assignment**: The large integer is then subjected to a modulus operation with 10000 (or 1000 for layers), assigning the user to a bucket. 4. **Bucket Determination**: The result defines the specific bucket out of 10000 (or 1000 for layers) where the user is placed. This process ensures a randomized but deterministic bucketing of users across different experiments or feature gates. The unique salt per-experiment or feature gate rule ensures that the same user can be assigned to different buckets in different experiments. This also means that if you rollout a feature gate rule to 50% - then back to 0% - then back to 50%, the same 50% of users will be re-exposed, **so long as you reuse the same rule** - and not create a new one. See [here](/faq/#when-i-change-the-rollout-percentage-of-a-rule-on-a-feature-gate-will-users-who-passed-continue-to-pass). For more details, check our open-source SDKs [here](https://github.com/statsig-io/node-js-server-sdk/blob/main/src/Evaluator.ts). This is not generally recommended, but for advanced use cases - e.g. a series of related experiments that needs to reuse the control and test buckets, we now expose the ability to copy and set the salts used for deterministic hashing. This is meant to be used with care and is only available to Project Administrators. It is available in the Overflow (...) menu in Experiments. ## Evaluation Order When evaluating gates, experiments, and layers, the SDK iterates through a list of rules generated by the server. Rules are evaluated sequentially and the first matching rule determines the result. Overrides always take precedence because they appear first in the rule list. Each step uses the hash-based bucketing described above. Layer allocation and group assignment use different salts, so a user's position in the layer is independent of their group assignment within the experiment. ### Experiments When an experiment is evaluated (you call `getExperiment`), it follows this evaluation order: 1. **ID overrides** — Specific user/unit IDs mapped to a group 2. **Conditional overrides** — Segment or gate-based overrides, evaluated in order 3. **Layer holdouts** — If the experiment is in a layer, layer-level holdout gates are checked 4. **Holdout gates** — Experiment-level holdout gates; users in a holdout receive default values 5. **Experiment exclusion** — Mutual exclusion segments that prevent users from being in multiple experiments 6. **Start status** — If the experiment is not started, users receive default values (with optional non-production environment exceptions) 7. **Layer allocation** — For experiments in a layer, the user's bucket (based on the layer's universe salt) must fall within the experiment's allocated segments. This is checked **before** targeting. 8. **Targeting gate** — Users who fail the targeting gate receive default values. This is checked **after** layer allocation. 9. **Group assignment** — The user's bucket (based on the experiment salt) determines which group they fall into. Groups are cumulative ranges across 1000 buckets. ### Layers When a layer is evaluated (you call `getLayer`), it follows this evaluation order: 1. **Override rules** — ID overrides from all experiments in the layer 2. **Layer holdout gates** — Holdout gates attached to the layer 3. **Experiment allocation** — Each experiment in the layer has a `configDelegate` rule. The user's bucket determines which experiment they are delegated to. 4. **Delegated experiment evaluation** — Once delegated, the experiment's own evaluation runs (start status, targeting gate, group bucketing as described above) If no experiment allocation rule matches, the user receives the layer's default values. ### Holdouts Holdout gates evaluate in this order: 1. **Experiment exclusion** — Exclusion segments (if applicable) 2. **ID overrides** — Specific user/unit IDs 3. **Population targeting gate** — If the holdout has a targeting gate, users who fail it are not held out 4. **Holdout percentage** — The pass percentage on the holdout rule determines the holdout rate ### Gates When a feature gate is evaluated (you call `checkGate`), it follows this evaluation order: 1. **ID overrides** — Specific user/unit IDs mapped to pass/fail 2. **Conditional overrides** — Segment or gate-based overrides 3. **Holdout rules** — If the gate has holdouts attached 4. **Rules** — The gate's targeting rules, evaluated in the order they appear in the console. Each rule has its own conditions and pass percentage. ## When Evaluation Happens Evaluation happens when the gate or experiment is checked on Server SDKs. To be able to do this, Server SDKs hold the entire ruleset of your project in memory - a representation of each gate or experiment in JSON. On client SDKs, we evaluate all of the gates/experiments when you call initialize - on our servers. All of the above logic holds true for both SDKs. In both, the user's assignment bucket is not sent to Statsig until you call the getExperiment/checkGate method in the SDK. ## What this means: * **Performant Evaluation:** no evaluations require a network request, and we focus on evaluation performance, meaning that checks take \<1ms after evaluation. * **The SDKs don't "remember" user attributes, or previous evaluations:** we rely on you to pass all of the necessary user attributes consistently - and we promise if you do, we'll provide the same value. A common assumption is that Statsig tracks of a list of all ids and what group they were assigned to for experiments/gates. While our data pipelines track users exposed to each variant to compute experiment results, we do not cache previous evaluations and maintain distributed evaluation state across client and server SDKs. That won't scale - we've even talked to customers doing this in the past, and were paying more for Redis to maintain that state than they ended up paying for Statsig. * **Server SDKs can handle multiple users:** because they hold the ruleset in memory, Server SDKs can evaluate any user. Without a network request. This means you'll have to pass a user object into the getExperiment method on Server SDKs, whereas on client SDKs you pass it into initialize(). * **We ensure each user receives the same bucket:** our ID-based hashing assignment guarantees consistency. If you make a change in console that could affect user bucketing on an experiment, we'll provide warning. ## Evaluation of null/empty unitIDs Note, we do not apply any filtering/ business logic before we assign an individual userID. This means that even a null or empty unitID will be bucketed depending on the salt. # Identify Users Source: https://docs.statsig.com/sdks/identify-users How to identify users in Statsig SDKs using user IDs, custom IDs, and user properties so feature gates and experiments evaluate consistently. ## Why identify users? When you run an experiment, rollout a feature, or log events, Statsig needs to know who the user is to determine: * Targeting: whether a feature gate should pass or fail for a given user * Experiment bucketing: which group a user belongs to * Analytics: How many unique users triggered an event This is achieved by passing a User Object to Statsig SDKs. User Object is also needed for experiment analysis - users are allocated to the test or control group by an ID, and the Statsig stats engine uses the same ID on the users' events to determine the difference in metrics between test and control. ## Basic User Object Start by defining a basic user object: ```jsx Basic User Object theme={null} { "userID": "u_123", // required for most setups "email": "user@example.com" // optional } ``` Later, you can layer in more details when you need finer targeting. To learn more about enriching User Object with more attributes, scroll down to [Enriched Attributes](#enriched-attributes). ## Identifying users in Client SDKs In client SDKs, the SDK is "initialized" for a single User Object at any one time. If you try to call a method like checkGate or getExperiment before your SDK is initialized, you won't get expected results - and you'll see warnings in the Statsig Console when you look at the diagnostics tab of any gate or experiment. Initialization requires a network request, which you fire by calling the `initialize()` method, or in React, `useClientAsyncInit()`. This means that in client apps with asynchronous logic, you'll want to wait for initialization to complete before you check a gate. For example, in React: ```jsx React Example theme={null} export function App() { const { client, isLoading } = useClientAsyncInit( YOUR_CLIENT_KEY, user={ userID: "u_123"} // user object ); if (isLoading) { return
    Loading...
    ; } return ( ); } ```
    This also means that if you initialize with a certain user object, and the user object changes (e.g., the user logs in and now has a `userID`) then you'll need to call the `updateUser` method, which calls to the Statsig Server to refresh the values. For example, in React: ```jsx Update User theme={null} const { user, updateUserSync } = useStatsigUser(); return

    Current User: { user.userID }

    ; ```
    If you don't update the user before checking gates or experiments, they'll evaluate using the previous user object you initialized with, which can lead to unexpected behavior. Any attribute added to the user object is targetable in the Statsig console - if you'd like to target users based on their `email`, `companyID`, or other attributes - add them to your User object. ## Identifying users in Server SDKs In Server SDKs, you pass a User Object with each evaluation or event call such as logEvent, checkGate or getFeatureGate. The Server SDK holds the necessary rules in memory to enable correct evaluation. Every time you call one of these methods - you should pass all of the attributes needed to evaluate your gate or experiment. The Statsig SDKs don't "remember" or enrich previous attributes seen. Example (Node.js) ```jsx Node.js Example theme={null} import { Statsig } from "statsig-node"; await Statsig.initialize(""); const user = { userID: "u_123", custom: { plan: "premium" } }; const gate = await Statsig.checkGate(user, "beta_feature"); if (gate) { console.log("Gate passed"); } ``` ## Enriched attributes Statsig attempts to enrich the user object with useful attributes to give you more targeting power. Examples of what Statsig can enrich for you: * `stableID`: an anonymous identifier which identifies a unique device on client SDKs * `country`: derived from IP or device locale * `appVersion`: pulled from mobile SDKs * `browser` / `OS`: captured from user agent * and more... To learn about all available fields — including `custom` attributes, `customIDs`, and `privateAttributes` — see the [StatsigUser object](/concepts/user) docs. # Get started with the Statsig SDK Source: https://docs.statsig.com/sdks/quickstart Get data flowing into Statsig with only a few lines of code using a Statsig SDK to log events, evaluate feature gates, and run your first experiment. If you're looking for a more detailed guide, check out the [SDK Overview](/sdks/getting-started) or read about choosing between [client or server SDKs](/sdks/client-vs-server). ```bash theme={null} npm install @statsig/react-bindings ``` Next, update your app's default function (Usually App.tsx or Layout.tsx) so that the StatsigProvider wraps around all child components. ```tsx theme={null} import { StatsigProvider } from "@statsig/react-bindings"; // [!code ++] export default function RootLayout({ children }: { children: React.ReactNode }) { return ( Loading...
    }> // [!code ++] {children} // [!code ++] ); } ``` This example assumes you're using client-side React, if you're Server-Side Rendering, you'd be better served by our [Next.js docs](/client/Next). Create a client API key in the [Statsig console Settings](https://console.statsig.com/api_keys). Copy and paste it to replace `` in the code snippet from the previous step. First, create a gate on the [Feature Gates page](https://console.statsig.com/gates) in console, then check it in-code: ```jsx theme={null} const { client } = useStatsigClient(); return (
    Gate is {client.checkGate('check_user') ? 'passing' : 'failing'}.
    ); ```
    First, create an Experiment on the [Experiments page](https://console.statsig.com/experiments) in console ```jsx theme={null} const { client } = useStatsigClient(); const experiment = client.getExperiment('my_experiment_name'); return (
    Headline Parameter: {experiment.get('my_experiment_parameter_name', 'fallback_value')}.
    ); ```
    You can use Events to power metrics in your experiment or gates. Events don't need to be set up in console first, just add to your code: ```jsx theme={null} const { client } = useStatsigClient(); return ```
    ## Next steps Congratulations! You've successfully set up the Statsig SDK in React. Continue on to the tutorials, or jump in to the full [Next.js](/client/Next) or [React](/client/React) SDK libraries. In the `` section of your website, paste the following code snippet: ```html theme={null} ``` Create a client API key in the [Statsig console Settings](https://console.statsig.com/api_keys). Copy and paste it to replace `` in the code snippet from the previous step. First, create a gate on the [Feature Gates page](https://console.statsig.com/gates) in console, then check it in-code: ```jsx theme={null} window.Statsig.instance().checkGate("my_feature_gate_name"); ``` You'll want to wait for the SDK to initialize before checking a gate to ensure it has fresh values, one way to accomplish this is waiting for the ["values\_updated"](/client/javascript-sdk#client-event-emitter) event. First, create an Experiment on the [Experiments page](https://console.statsig.com/experiments) in console ```jsx theme={null} window.Statsig.instance().getExperiment("my_experiment_name").get('my_experiment_parameter_name'); ``` You'll want to wait for the SDK to initialize before getting an experiment to ensure it has fresh values, one way to accomplish this is waiting for the ["values\_updated"](/client/javascript-sdk#client-event-emitter) event. You can use Events to power metrics in your experiment or gates. Events don't need to be set up in console first, just add to your code: ```jsx theme={null} window.Statsig.instance().logEvent("my_checkout_event_name", "event_value_item_1234", {"event_metadata": "my_metadata"}) ``` ## Next steps Congratulations! You've set up the Statsig JavaScript snippet. You can now start: * Start [recording events](/webanalytics/overview) * Watch [session replays](/session-replay/overview) * Run [experiments](/experiments/overview) * Use [feature flags](/feature-flags/overview) See the [Javascript SDK reference](/client/javascript-sdk) for more info. ```shell theme={null} pip install statsig-python-core ``` ```python theme={null} from statsig_python_core import Statsig, StatsigOptions options = StatsigOptions() options.environment = "development" statsig = Statsig("", options) statsig.initialize().wait() statsig.shutdown().wait() ``` Create a server secret key in the [Statsig console Settings](https://console.statsig.com/api_keys). Copy and paste it to replace `` in the code snippet from the previous step. First, create a gate on the [Feature Gates page](https://console.statsig.com/gates) in console, then check it in-code: ```python theme={null} user_object = StatsigUser(user_id="123", email="testuser@statsig.com") //add any number of other attributes gate_value = statsig.check_gate(user_object, "my_feature_gate_name"): ``` First, create an Experiment on the [Experiments page](https://console.statsig.com/experiments) in console ```python theme={null} user_object = StatsigUser(user_id="123", email="testuser@statsig.com" my_experiment_object = statsig.get_experiment(user_object, "my_experiment_name") my_experiment_parameter_value = my_experiment_object.get_string('my_experiment_parameter_name') ``` You can use Events to power metrics in your experiment or gates. Events don't need to be set up in console first, just add to your code: ```python theme={null} user_object = StatsigUser(user_id="123", email="testuser@statsig.com" statsig.log_event( user=user_object, event_name="my_checkout_event_name", value="SKU_12345" ) ``` ## Next steps Congratulations! You've set up the Statsig SDK in Python. Continue on to our tutorials, or jump in to the full [Python SDK Reference.](/server-core/python-core) ```bash theme={null} npm i @statsig/statsig-node-core ``` ```jsx theme={null} // Basic initialization const statsig = new Statsig(""); await statsig.initialize(); // or with StatsigOptions const options: StatsigOptions = { environment: "staging" }; const statsigWithOptions = new Statsig("secret-key", options); await statsigWithOptions.initialize(); ``` Create a server secret key in the [Statsig console Settings](https://console.statsig.com/api_keys). Copy and paste it to replace `` in the code snippet from the previous step. First, create a gate on the [Feature Gates page](https://console.statsig.com/gates) in console, then check it in-code: ```js theme={null} const userObject = new StatsigUser({ userID: "123", email="testuser@statsig.com" }); const is_gate_enabled = statsig.checkGate(userObject, "my_feature_gate_name"): ``` First, create an Experiment on the [Experiments page](https://console.statsig.com/experiments) in console ```js theme={null} const userObject = new StatsigUser({ userID: "123", email="testuser@statsig.com" }); const myExperimentObject = statsig.getExperiment(userObject, "my_experiment_name") const myExperimentParameterValue = myExperimentObject.getValue('my_experiment_parameter_name') ``` You can use Events to power metrics in your experiment or gates. Events don't need to be set up in console first, just add to your code: ```js theme={null} userObject = StatsigUser(user_id="123", email="testuser@statsig.com" statsig.logEvent( userObject, "my_checkout_event_name", "SKU_12345" //value for the event ); ``` ## Next steps Congratulations! You've successfully set up the Statsig SDK in Node.js. Continue on to the tutorials, or jump in to the full [Node.js](/server-core/node-core) SDK library. ## Explore SDKs Statsig offers SDKs for a wide variety of platforms to suit any codebase or deployment preference: ### Client SDKs Browser JavaScript Client-Side React Bare React Native SDK Next.js SSR, SSG & Client-Side Angular bindings for Javascript SDK iOS, MacOS, tvOS SDK Android Kotlin/Java SDK Client SDK for .NET framework Roku Brightscript SDK Unity game engine SDK Flutter/Dart Mobile App SDK C++ client-side SDK ### Server Side SDKs Node.js server SDK Java server SDK Python server SDK Go server SDK Ruby server SDK .NET server SDK PHP server SDK Rust server SDK C++ server SDK ### Integrations Webflow integration Shopify integration Segment data connector Rudderstack connector Hightouch integration mParticle connector Framer integration Slack notifications View more integrations # SDK Support Policy Source: https://docs.statsig.com/sdks/support Statsig SDK support reference, including supported runtimes, version policies, deprecation timelines, and how to file SDK issues with the team. To ensure the security, performance, and reliability of our SDKs, we only provide official support for SDK versions that are less than **one year old** from their release date, unless otherwise specified (some updates may require a different timeline or migration path). ## Supported Versions * We provide support (bug reports, issue investigation, and compatibility assistance) for SDK versions released **within the past 12 months**. * Versions older than one year are considered **unsupported**. While they may continue to function, we do not guarantee their behavior or compatibility with our backend services. * We strongly recommend upgrading to the latest version to benefit from new features, performance improvements, and critical fixes. ## Versioning and Releases * All SDKs follow [Semantic Versioning (SemVer)](https://semver.org/), using the format `MAJOR.MINOR.PATCH`, **unless otherwise specified**. * **MAJOR**: breaking changes * **MINOR**: backwards-compatible new features * **PATCH**: backwards-compatible bug fixes * Release notes are published in the GitHub Releases section of each SDK’s repository. These notes contain details on changes, improvements, and upgrade instructions. ## Open Source and Community Contributions * All SDKs are open source and licensed under the [ISC License](https://opensource.org/licenses/ISC), allowing developers to freely use, modify, and distribute them. * We welcome and encourage community contributions. ## Need Help? For help with upgrading, reviewing release notes, or contributing, please refer to the SDK’s GitHub repository or reach out in Slack. # Target Apps Source: https://docs.statsig.com/sdks/target-apps Use target apps in Statsig SDKs to scope feature gates, experiments, and dynamic configs to specific applications inside a single project. Target Apps are an Enterprise-only feature. Reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. ## Motivation There are two main attributes you can add to SDK Keys which restrict their access to certain configs or config definitions - Environments and Target Apps. A “Target App” is a user defined abstraction tied to entities in your Statsig Project which can be linked to one or more SDK Keys. Target apps should be things like your "Android" or "iOS" apps, or could be more restricted to specific services like "Search" and "Feed" which may span both client and server usage. Defining and using Target Apps is a best practice when using Statsig at large scale, and has two main benefits: * Performance: removing unused entities from the payload to your Statsig SDK integration will speed up the initialization time and reduce the amount of data that is stored in any local caches or data stores * Security: you may not want to expose certain gates/experiments/configs to the client side, or to the client key which is easily found in your client app. Specifying a target app for your client key and ONLY linking that to client specific gates/configs/experiments hides those from prying eyes When using target apps, first consider if you should instead use separate projects for your Statsig integrations: * If you need to share gates/experiments/metrics across apps or services, it is recommended to use a Target App as entities are restricted within a single Statsig project. * If your services or apps always have distinct gates/experiments/metrics, using a separate Statsig project might be a better solution. If you have an existing Statsig project with multiple SDK integrations, Statsig will help you to bootstrap your Target Apps by suggesting existing entities to tag which have historically recorded checks from that specific SDK. ## Configuring Target Apps ### Project Setup Start by going to **Settings** → [**Target Applications**](https://console.statsig.com/settings/apps). Project Settings Screenshot 2025-09-04 at 11 48 55 PM Create a target app, or use one that is suggested based on your current SDK exposure logs. Project Settings ### Associating a Feature Gate with a Target App If you wish to verify the suggestions you added in the initial setup, or add additional entities to the target app, you can do so from that entity itself. For example, navigate to the **Feature Gates** tab, and then select a specific gate. In the right rail for that gate, you can view and edit the target app(s) applied to that gate. Project Settings Repeat this flow for the Dynamic Configs and Experiments that are used by this Target App. ### Associating an SDK Key with a Target App Navigate to the **Settings** → [**Keys & Environments**](https://console.statsig.com/api_keys) page. You're ready to restrict a certain key to a certain target app. Select the SDK Key you wish to restrict to the set of Target App entities. Under **Actions**, trigger the dropdown menu Project Settings Then select the target app(s) to apply to that key. Project Settings When [bootstrapping](/client/concepts/initialize#2-bootstrap-initialization) a Client SDK from a Server SDK, you'll need the server SDK to have access to all of the gates/experiments/configs needed both for server and client. If you have separate target apps for client and server SDKs, you should apply both target apps to the server key you're using. Then, to filter the boostrapping response to a specific target app, add a client key with that target app applied to the getClientInitializeResponse call. ### Debugging To debug SDK evaluations which may be impacted by your Target App configuration, use the **Diagnostics** tab for any entity. In this Feature Gate example, the Exposure Stream at the bottom will show the Target App associated with any checks if the SDK key used to check it has a Target App applied. If any exposures for that entity do not have a matching Target App, it will be flagged with a warning. # User (StatsigUser) Object Source: https://docs.statsig.com/sdks/user Reference for the StatsigUser object used by SDKs, including fields like userID, email, country, custom properties, and private attributes. ## Introduction to the StatsigUser object The user(StatsigUser) object is the sole input you provide to SDKs to target gates and assign users to experiments. If you'd like to target on an attribute, you'll need to add it to your user object, so assembling the data is an important part of SDK setup. **We recommend providing as much info as practical,** as every additional field can enrich your analyses and expand targeting possibilities. Statsig also infers some information about each user based on other traits (for example, we resolve IP Addresses into countries), read on for more details. ## Usage Example in Node: ```jsx Node.js Example theme={null} //Create a user object const user = new StatsigUser({ userID: "12345", email: "vincent@statsig.com"}); //Use it in getExperiment() const my_experiment = statsig.getExperiment(user, "my_experiment_name") //<- any attribute you pass, you can target on ``` ### I passed that attribute before - why do I need to pass it again? If you want to see a specific user's historical attributes for analytics purposes, you can see that in the statsig console via [User Profiles](/product-analytics/users-tab). However, you cannot use a user's historical attributes for targeting or gate/experiment evaluation at runtime. Statsig evaluates gates and experiment buckets based **only on the information you provide when you check something.** Statsig's promise of evaluating every gate or experiment in milliseconds is dependent on having all information available at request time, rather than querying previous data, or storing massive amounts of historical data in memory. ## User Attributes All user attributes can be explicitly supplied, and some can be inferred from a user's device or connection. Supplying one will always override an inferred value. | Key | Description | Example | Client SDK Support | Auto-infer | | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------ | ---------- | | `userID` | ID representing a unique user. This ID will be used to guarantee consistency of targeting for Feature Gates and Experiments and will be used to evaluate experiment results. If `User ID` doesn't exist yet, leave this empty; a `Stable ID` persisted locally will be used for evaluations. | `your_user_id` | All | | | `email` | Email of the user. | `marcos@statsig.com` | All | | | `userAgent` | User agent of the browser. This will be decoded to determine the Browser and Operating System of the user's context. Will be inferred if not provided. | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.40 Safari/537.36` | Web | ✔ | | `ip` | IP address of the user. Inferred from the request to /initialize if not provided | `192.168.1.101` | All | ✔ | | `country` | 2-letter country code of the user. This can be supplied or inferred, and we can target based on the country code in both cases. When inferred, the country code follows ISO-3166. | `US` | All | ✔ | | `locale` | Locale of the user. When using the Android or iOS SDK, this will be inferred if not provided. | `en_US` | Mobile | ✔ | | `appVersion` | Version of the app the user is using. When using the Android or iOS SDK, this will be inferred if not provided. | `1.0.1` | Mobile | ✔ | | `systemName` | When using our Android/iOS SDKs, this will be automatically assigned, but you can also provide an explicit operating system to override. | `Android` | All | ✔ | | `systemVersion` | When using our Android/iOS SDKs, this will be automatically assigned, but you can also provide an explicit OS version to override. | `15.4` | All | ✔ | | `browserName` | When using our Web SDK, this will be automatically assigned, but you can also provide an explicit Browser Name to override. | `Chrome` | Web | ✔ | | `browserVersion` | When using our Web SDK, this will be automatically assigned, but you can also provide an explicit Browser Version to override. | `45.0` | Web | ✔ | | `custom` | Dictionary that can contain key/value pairs that can be used for Feature Gate targeting. The content of this dictionary will be stored and available after targeting. | `{subscriber: "yes", ...}` | All | | | `privateAttributes` | Dictionary that can contain key/value pairs that can be used for Feature Gate targeting. The content of this dictionary will **not** be stored after being used for targeting and will be removed from any `log_event` calls. | `{sensitive_field: "sensitive_information", ...}` | All | | | `customIDs` | Dictionary that can contain key/value pairs used as the randomization unit ID for experiments that are set up using these IDs instead of the `User ID`. | `{account_id: "23456555", company_id: "company_xyz"}` | All | | ### How to override the Stable ID Client SDKs generate a `stableID` for logged-out or device-level targeting. If you already manage your own durable device identifier, override the SDK-generated value before or during initialization so Statsig uses your identifier for Stable ID-based evaluations. ```tsx theme={null} import { StatsigClient, StatsigUser } from '@statsig/js-client'; const user: StatsigUser = { userID: 'a-user', customIDs: { stableID: 'my-custom-stable-id', }, }; const client = new StatsigClient('client-sdk-key', user); await client.initializeAsync(); ``` ```kotlin theme={null} val options = StatsigOptions(overrideStableID = "my-custom-stable-id") Statsig.initialize(app, "client-sdk-key", user, options = options) ``` ```swift theme={null} let options = StatsigOptions(overrideStableID: "my-custom-stable-id") Statsig.initialize(sdkKey: "client-sdk-key", user: user, options: options) ``` When you override the Stable ID in a client SDK, that value is persisted locally and reused on later sessions unless local storage is cleared or the app is reinstalled. All user attributes can be explicitly supplied, and some can be inferred from a provided IP Address or User Agent. Supplying one will always override an inferred value. | Attributes | Description | Example | Auto Infer | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ---------- | | `userID` | ID representing a unique user. This ID will be used to guarantee consistency of targeting for Feature Gates and Experiments and will be used to evaluate experiment results. | `your_user_id` | | | `email` | Email of the user | `marcos@statsig.com` | | | `userAgent` | User agent of the browser. This will be decoded to determine the Browser and Operating System of the user's context | `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.40 Safari/537.36` | | | `ip` | IP address of the user | `192.168.1.101` | | | `country` | 2 letter country code of the user | `US` | ✔, from IP | | `locale` | Locale of the user | `en_US` | ✔, from IP | | `appVersion` | Version of the app the user is using | `1.0.1` | | | `custom` | Dictionary that can contain key/value pairs that can be used for Feature Gate targeting. The content of this dictionary will be stored and available after targeting | `{skill_level: "5", is_subscriber:"false" ...}` | | | `privateAttributes` | Dictionary that can contain key/value pairs that can be used to evaluate feature gate conditions and segment conditions. The content of this dictionary will **not** be stored after used for targeting and will be removed from any log\_event calls | `{sensitive_field: "sensitive_information", ...}` | | | `customIDs` | Dictionary that can contain key/value pairs used as the randomization unit ID for experiments that are set up using these IDs instead of the `User ID` | `{account_id: "23456555", company_id: "company_xyz"}` | | ### How to override "Operating System" and "Browser" explicitly Operating system and Browser are two default targeting options on Statsig, and if you set the userAgent field, server SDKs will parse out the OS/Browser information to evaluate them. If you prefer to explicitly setup this, you can in two places: either top-level in the user object (which typing may not allow in languages), or in the "custom" object. You need to provide this information under the following keys: * Operating System: os\_name * OS Version: os\_version * Browser Name: browser\_name * Browser Version: browser\_version As an example, you could set this either of these two ways in the user object: ```json Example theme={null} { "userID": "uuid", "os_name": "Android", // top level "custom": { "os_name": "iOS" } } ``` If either of these fields is explicitly set, it will take precedence over inferring the value from the `userAgent` field. ### Have sensitive user PII data that should not be logged? On the StatsigUser object, there is a field called privateAttributes, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in privateAttributes will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to privateAttributes on the user and that's it! ### Time-based conditions All SDKs (both server and client-side) support unix timestamps in milliseconds to evaluate time based conditions (After time, before time). Without knowing all possible variations of DateTime formats, we have to normalize on something, so it's best to convert your DateTime field into a standard format for evaluation. We have added support for ISO timestamps to some server SDKs. * The `java-server` sdk, as of `v1.6.0` supports [DateTime fields in the format of `ISO_INSTANT`](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ISO_INSTANT) * The `go` sdk, as of `v1.12.1` supports [DateTime fields in the format of `RFC3339`](https://pkg.go.dev/time#pkg-constants) * The `python` sdk supports the usage of epoch time in seconds. using `time.time()` may include sub-second components so you should use round the value to an integer We have added support for ISO timestamps to server SDKs supported by server core (Java, Python, PHP, Rust, Node). * [DateTime fields in the format of `ISO_INSTANT`](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html#ISO_INSTANT) * [DateTime fields in the format of `RFC3339`](https://pkg.go.dev/time#pkg-constants) ### Why is an ID always required for server SDKs? In Server SDKs, a StatsigUser with a userID (or customID) is required for checkGate, getConfig, and getExperiment. In short, it's *always* better to pass the user ID if it's available: users will get a stable experience, and any events will be attributed to the correct users so you can accurately measure downstream metrics. Still don't want to pass an ID? Here are our suggestions for different use cases: 1. If you plan to only use one/off feature gates, or non-percent-based rules (like countries): While you're still losing functionality, you can pass any non-empty identifier, hard coded string, or **a random ID less than 100 if you do not have the actual user ID.** Don't pass a purely random ID - as we won't be able to dedupe your events, you'll explode your event usage, and your Statsig bill. 2. If you want to rollout a feature partially, check for regressions, then roll out to everyone: You must pass an ID in your checkGate/getConfig/getExperiment calls, as well as any logEvent calls you make. Otherwise, we're not able to attribute the events you log to the correct users who saw or didn't see your new feature, or calculate metrics correctly to help you see any regressions. 3. If you want to run an A/B experiment to decide whether to ship a new feature: You should also **pass the persistent user IDs**, for the same reason mentioned in 2, above. 4. If you want to pass a userID for the above reasons, but don't have a logged in user (e.g. you are optimizing the login flow): Set a stable identifier (we provide one in client SDKs!) as a cookie or in local storage and use that with each call to Statsig. We hope this is helpful. If you have a use case that is not covered in these scenarios, or have any question at all, feel free to join our [Slack community](https://statsig.com/slack) and drop a question/comment there! ### Pass all IDs when you have them A common mistake is to accidentally expose users without a userID to a user-ID based experiment. All of them will be bucketed into one group (whichever the "null" userID goes to), polluting your results. If you run experiments on multiple ID types (or you might one day), it's best to pass every identifier available. # Adding ID Lists Source: https://docs.statsig.com/segments/add-id-list Create and manage ID List segments in Statsig to target specific users by their identifiers, such as user IDs, emails, or custom IDs uploaded as a list. ## Adding ID Lists ### What is an ID List? An ID List enables you to define a reusable audience segment using user identifiers like `userID`, `stableID`, or `organizationID`. You can manage ID Lists by manually adding/removing IDs, uploading CSVs, or replacing the entire list. ### Creating an ID List Segment 1. Navigate to the **Segments** section in the [Statsig Console](https://console.statsig.com). 2. Click **Create New Segment**. 3. Toggle the segment type from **Conditional** to **ID List**. Create New Segment 4. Select the **ID type** you want to build from. Create New Segment ### Managing IDs Once inside your ID List segment, you have a variety of options: * **Manual Entry**: Enter IDs directly into the input box. Create New Segment * **Upload CSV**: Import a list of IDs from a CSV. * **Bulk Actions**: Choose how your input affects the existing list: * **Add**: Add IDs to the list * **Remove**: Remove matching IDs from the list * **Replace**: Clear all IDs and replace with a new list You can also sync in an ID list Segment from sources like an [Amplitude Cohorts](https://help.amplitude.com/hc/en-us/articles/4789303290011) or [Segment Audiences](/integrations/data-connectors/segment#syncing-statsig-segment-id-lists-with-segment-personas-audiences) or from a custom source using the [Console API](/console-api/segments) There is a hard limit of 10 million IDs across all ID Lists in your project. ### Keep ID Lists Small When an ID List is small (less than 1000 IDs), it is synchronized in the same process with other feature gates, experiments, and conditional segments, which has high consistency and reliability. As the ID List goes over 1000, there is a separate process to synchronize them to scale efficiently and not negatively impact the other entities. As a result, they are not downloaded in the main initialization process (in the initialization path of server SDKs, or the local evaluation version of our client SDKs), and change propagation takes longer. We recommend always keeping your ID Lists to be no more than 1000 IDs in each, unless you are okay with the caveats mentioned above. ### Large ID Lists To target hundreds of thousands of users (or millions), the more performant pattern is to set an attribute on the User object passed to the Statsig SDK. e.g. pass in user\_type:Paid or user\_type:Trial in the User object, instead of having ID Lists that enumerate the Paid and Trial users. This does require your app to have this information when calling the Statsig SDK, and you're better able to control how/where to cache this information. If you have a use-case that requires leveraging ID lists larger than 10,000 IDs, please reach out to our team via Slack. Large ID Lists is currently in limited beta, please contact Statsig Support to enable. Note this is not available for Free/ Pro users at this time. # Adding Rules Source: https://docs.statsig.com/segments/add-rule Create rules in Statsig segments that define which users are included based on user attributes, custom properties, country, app version, and more. ## Create a rule for a segment A rule defines the criteria for which users are included in a segment. To add a rule to a segment, * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Segments** * Select the segment where you want to add a rule * Click the **Add New Rule** button * Select the criteria to target a set of users. For example, select Email and the Contains any of operator, and enter the email domain of your company to target only internal employees Segment rule configuration interface * Click the **Add Rule** button * Click the **Save Changes** button at the top right of the **Rules** section # Create a segment Source: https://docs.statsig.com/segments/create-new Step-by-step guide to creating a new segment in the Statsig console for targeting specific user groups in feature gates, experiments, and dynamic configs. To create a segment, * Log into the Statsig console at [https://console.statsig.com](https://console.statsig.com) * On the left-hand navigation panel, select **Segments** * Click on the **Create** button * Enter a name and description for your segment * Select the type of segment you want to create: **Conditional** or **ID List**, and click **Create** (Conditional is selected by default) Segment creation interface For **Conditional Segments**, you can **[Add Rules](/segments/add-rule)** to complete the segment's definition. For **ID List Segments**, you can **[Add an ID List](/segments/add-id-list)** to complete the segment's definition. # Using a segment Source: https://docs.statsig.com/segments/implement Use Statsig segments as reusable targeting building blocks across feature gates, experiments, and dynamic configs so audience definitions stay consistent. You can use a segment to target a set of users in a feature gate or a dynamic config as follows: * Select the feature gate or dynamic config where you want to target a set of users * Click the **Add New Rule** button * In the criteria, select the **User in Segment** option * Select the segment that you want to use and enter a name Segment targeting configuration interface * Click the **Add Rule** button * Click the **Save Changes** button at the top right of the **Rules** section # Segments Source: https://docs.statsig.com/segments/overview Create and use Statsig segments to define reusable sets of users for targeting across feature gates, experiments, and dynamic configs in one place. ## What is a segment? A segment defines a reusable set of users. You can define a segment based on common user attributes such as location, client device, browser, client application version, or simply using user IDs. ## When to use segments? Segments are ideal for targeting a commonly identified set of users across features or dynamic configs. For example, you may create a segment of users that represent your team members or company employees to test a new feature with your colleagues before launching to external users. You may also create different segments of users for English and Spanish speaking countries to deliver a dynamically configured, localized application experience to each geographically identified segment of users. To get started, see the Statsig guide to [create your first segment](/guides/first-segment). The following tutorials show how you can perform common tasks with segments. * [Create a segment](/segments/create-new) * [Create rules for a segment](/segments/add-rule) * [Use a segment](/segments/implement) # C++ Server SDK Source: https://docs.statsig.com/server-core/cpp-core Statsig's next-generation C++ Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for backend services. Cpp Core on Github Cpp Core Working Repo ## Setup the SDK ## Installation All core logic are written in rust, and we don't require rust environment from you, so we pre-built all binaries (attached as assets with each release). Our CMakeLists.txt file handle all the complexity for you. If you encounter any installation or build issue, please reach back to us! ```CMake theme={null} FetchContent_Declare( Statsig GIT_REPOSITORY https://github.com/statsig-io/statsig-cpp-core.git GIT_TAG 0.12.2-rc.1 ) FetchContent_MakeAvailable(Statsig) target_include_directories(CppApp PRIVATE ${statsig_SOURCE_DIR}/include) target_include_directories(CppApp PRIVATE ${statsig_SOURCE_DIR}/src) target_link_libraries(CppApp PRIVATE Statsig) ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. ```cpp theme={null} #include statsig_cpp_core::StatsigOptionsBuilder optionsBuilder; ptionsBuilder.environment = "development"; ptionsBuilder.specs_url = api_v2 + "/download_config_specs"; optionsBuilder.log_event_url = api + "/log_event"; optionsBuilder.id_lists_url = api + "/get_id_lists"; statsig_cpp_core::StatsigOptions options = optionsBuilder.build(); Statsig statsig("server-secret-key", options); statsig.initializeBlocking().wait(); ``` `initialize` will perform a network request. After `initializeBlocking` completes, virtually all SDK operations will be synchronous. The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```cpp theme={null} if (statsig.checkGate(user, "a_gate")) { // Gate is on, enable new feature } else { // Gate is off } ``` You can also disable exposure logging for this evaluation: ```cpp theme={null} FeatureGateEvaluationOptions options; options.disable_exposure_logging = true; bool gateValue = statsig.check_gate(user, "a_gate", options); ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```cpp theme={null} // Get a dynamic config for a specific user DynamicConfig config = statsig.getDynamicConfig(user, "a_config"); // Access config values (We will provide accessors) std::string product_name = config.value.get("product_name", "Awesome Product v1"); double price = config.value.get("price", 10.0); // Access evaluation details such as rule id statsig_cpp_core::EvaluationDetails detail = config.details; std::cout << config.rule_id << std::endl; // The ID of the rule that served this config std::cout << config.id_type << std::endl; // The type of the evaluation (experiment, config, etc) // Advanced Usage: // You can disable exposure logging for this specific check DynamicConfigEvaluationOptions options; options.disable_exposure_logging = true; config = statsig.getDynamicConfig(user, "a_config", options); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```cpp theme={null} // Get a experiment for a specific user statsig_cpp_core::Experiment exp = statsig.getExpriment(user, "an_experiment"); // Access config values (We will provide accessors) auto product_name = exp.value.get("product_name", "Awesome Product v1"); auto price = exp.value.get("price", 10.0); // Access evaluation details such as rule id statsig_cpp_core::EvaluationDetails detail = exp.details; std::cout << exp.rule_id << std::endl; // The ID of the rule that served this config std::cout << exp.id_type << std::endl; // The type of the evaluation (experiment, config, etc) // Advanced Usage: // You can disable exposure logging for this specific check ExperimentEvaluationOptions options; options.disable_exposure_logging = true; config = statsig.getExperiment(user, "an_experiment", options); ``` ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```cpp theme={null} FeatureGate gate = statsig.getFeatureGate(user, "example_gate"); std::cout << gate.rule_id << std::endl; std::cout << gate.value << std::endl; ``` The `get_feature_gate()` method returns a FeatureGate object that provides: * `value`: The boolean gate value * `rule_id`: The ID of the rule that served this gate * `id_type`: The type of the evaluation * `evaluation_details`: Additional metadata about the evaluation ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```cpp theme={null} statsig.log_event( user, "add_to_cart", { {"price", "9.99"}, {"item_name", "diet_coke_48_pack"} } ); ``` The `log_event` method supports multiple overloads: * `log_event(user, event_name)` * `log_event(user, event_name, string_value)` * `log_event(user, event_name, string_value, metadata)` We will add support for numerical value and metadata ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. ```cpp theme={null} statsig_cpp_core::UserBuilder builder; builder.setUserID(j["userID"]); builder.setCustomIDs(j["customIDs"]); builder.setCountry(j["customIDs"]); statsig_cpp_core::User user = builder.build(); // UserBuilder also supports deserialize from json ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. ### Available Options ```cpp theme={null} std::optional specs_url; std::optional id_lists_url; std::optional log_event_url; std::optional output_log_level; std::optional environment; bool enable_id_lists = false; bool disable_all_logging = false; bool disable_country_lookup = false; bool disable_network = false; ``` ### Proxy and Custom Network Routing The C++ Server Core SDK currently documents endpoint overrides rather than a dedicated outbound proxy config. Use `specs_url`, `log_event_url`, and `id_lists_url` to route Statsig network calls through your own endpoints or proxy layer. ### Example Usage ```cpp theme={null} statsig_cpp_core::StatsigOptionsBuilder optionsBuilder; optionsBuilder.environment = "development"; optionsBuilder.specs_url = api_v2 + "/download_config_specs"; optionsBuilder.log_event_url = api + "/log_event"; optionsBuilder.id_lists_url = api + "/get_id_lists"; statsig_cpp_core::StatsigOptions options = optionsBuilder.build(); ``` ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```cpp theme={null} statsig.shutdownBlocking(); ``` The `shutdown()` method will: * Flush any pending events to Statsig servers * Gracefully shutdown the SDK, cleaning up resources * Wait for all operations to complete It's recommended to call `shutdown()` before your application exits to ensure all events are sent. ## Reference ### API Methods * `check_gate(user: StatsigUser, gate_name: str, options: Optional[FeatureGateEvaluationOptions] = None) -> bool` * `get_dynamic_config(user: StatsigUser, config_name: str, options: Optional[DynamicConfigEvaluationOptions] = None) -> DynamicConfig` * `get_experiment(user: StatsigUser, experiment_name: str, options: Optional[ExperimentEvaluationOptions] = None) -> DynamicConfig` * `get_layer(user: StatsigUser, layer_name: str, options: Optional[LayerEvaluationOptions] = None) -> Layer` * `get_feature_gate(user: StatsigUser, gate_name: str, options: Optional[FeatureGateEvaluationOptions] = None) -> FeatureGate` * `log_event(user: StatsigUser, event_name: str, value: Optional[Union[str, float]] = None, metadata: Optional[Dict[str, str]] = None) -> None` * `shutdown() -> AsyncResult[None]` ### Fields Needed Methods The following methods return information about which user fields are needed for evaluation: * `get_gate_fields_needed(gate_name: str) -> List[str]` * `get_dynamic_config_fields_needed(config_name: str) -> List[str]` * `get_experiment_fields_needed(experiment_name: str) -> List[str]` * `get_layer_fields_needed(layer_name: str) -> List[str]` These methods return a list of strings representing the user fields that are required to properly evaluate the specified gate, config, experiment, or layer. # .NET Server SDK Source: https://docs.statsig.com/server-core/dotnet-core Statsig's next-generation .NET Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for C# backends. .NET Core on Github, NuGet Package ## Setup the SDK ## Installation ```bash theme={null} dotnet add package Statsig.Dotnet ``` Or add the package reference to your `.csproj` file: ```xml theme={null} ``` ### Requirements: * **.NET 8.0** or later * **Windows, macOS, or Linux** (x64 and ARM64 supported) After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. ```csharp theme={null} using Statsig; var statsig = new Statsig.Statsig("server-secret-key"); await statsig.Initialize(); ``` You can also provide custom options: ```csharp theme={null} var options = new StatsigOptionsBuilder() .SetSpecsSyncIntervalMs(10000) .SetDisableAllLogging(false) .Build(); var statsig = new Statsig("server-secret-key", options); await statsig.Initialize(); ``` For shared instance usage: ```csharp theme={null} var sharedStatsig = Statsig.NewShared("server-secret-key", options); await sharedStatsig.Initialize(); var statsig = Statsig.Shared(); ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .SetEmail("user@example.com") .Build(); var gateValue = statsig.CheckGate(user, "new_feature_gate"); if (gateValue) { // Gate is on, enable new feature } else { // Gate is off } ``` You can also disable exposure logging for this evaluation: ```csharp theme={null} var options = new EvaluationOptions(disableExposureLogging: true); var gateValue = statsig.CheckGate(user, "new_feature_gate", options); ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .Build(); var config = statsig.GetDynamicConfig(user, "product_config"); var productName = config.Get("product_name", "Default Product"); var price = config.Get("price", 9.99); var isEnabled = config.Get("enabled", false); var features = config.Get>("features", new List()); Console.WriteLine($"Config Name: {config.Name}"); Console.WriteLine($"Group Name: {config.GroupName}"); Console.WriteLine($"Rule ID: {config.RuleID}"); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .Build(); var experiment = statsig.GetExperiment(user, "button_color_test"); var buttonColor = experiment.Get("color", "blue"); var fontSize = experiment.Get("font_size", 14); var showBorder = experiment.Get("show_border", true); Console.WriteLine($"Experiment Name: {experiment.Name}"); Console.WriteLine($"Group Name: {experiment.GroupName}"); Console.WriteLine($"Rule ID: {experiment.RuleID}"); Console.WriteLine($"Button Color: {buttonColor}"); ``` ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .Build(); var layer = statsig.GetLayer(user, "user_prefs_layer"); var theme = layer.Get("theme", "light"); var language = layer.Get("language", "en"); var notifications = layer.Get("notifications_enabled", true); Console.WriteLine($"Layer Name: {layer.Name}"); Console.WriteLine($"Allocated Experiment: {layer.AllocatedExperimentName}"); Console.WriteLine($"Group Name: {layer.GroupName}"); Console.WriteLine($"Rule ID: {layer.RuleID}"); ``` Note: Layer parameter access automatically logs exposure events unless disabled with `EvaluationOptions`. ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .Build(); statsig.LogEvent(user, "button_clicked"); statsig.LogEvent(user, "purchase_completed", 29.99); statsig.LogEvent(user, "page_view", "homepage", new Dictionary { ["referrer"] = "google", ["campaign"] = "summer_sale" }); statsig.LogEvent(user, "video_watched", 120, new Dictionary { ["video_id"] = "abc123", ["quality"] = "1080p" }); ``` The `LogEvent` method supports multiple overloads: * `LogEvent(user, eventName)` * `LogEvent(user, eventName, stringValue, metadata)` * `LogEvent(user, eventName, intValue, metadata)` * `LogEvent(user, eventName, doubleValue, metadata)` ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .Build(); var gate = statsig.GetFeatureGate(user, "new_feature_gate"); Console.WriteLine($"Gate Name: {gate.Name}"); Console.WriteLine($"Gate Value: {gate.Value}"); Console.WriteLine($"Rule ID: {gate.RuleID}"); Console.WriteLine($"ID Type: {gate.IDType}"); if (gate.EvaluationDetails != null) { Console.WriteLine($"Config Sync Time: {gate.EvaluationDetails.ConfigSyncTime}"); Console.WriteLine($"Init Time: {gate.EvaluationDetails.InitTime}"); Console.WriteLine($"Reason: {gate.EvaluationDetails.Reason}"); } ``` ### Parameter Stores Sometimes you don't know whether you want a value to be a Feature Gate, Experiment, or Dynamic Config yet. If you want on-the-fly control of that outside of your deployment cycle, you can use Parameter Stores to define a parameter that can be changed into at any point in the Statsig console. Parameter Stores are optional, but parameterizing your application can prove very useful for future flexibility and can even allow non-technical Statsig users to turn parameters into experiments. ```csharp theme={null} # Get a Parameter Store by name param_store = statsig.getParameterStore(user, "my_parameter_store") ``` ### Retrieving Parameter Values Parameter Store provides methods for retrieving values of different types with fallback defaults. * **GetBool(string, default(bool))**: Pulls key value of type boolean * **GetString(string, "")**: Pulls key value of type string * **GetLong(string, default(long))**: Pulls key value of type long * **GetDouble(string, default(double))**: Pulls key value of type double * **GetList(string, new List\<>())**: Pulls key value of type list * **GetDictionary(string, new Dictionary\())**: Pulls key value of type Dictionary ### Evaluation Options You can disable exposure logging when retrieving a parameter store: ```csharp theme={null} var store = statsig.GetParameterStore(user, name!, options); if (store == null) { throw new Exception($"Parameter store {name} not found"); } var x = store.GetBool(paramName, default(bool)); var y = store.GetString(paramName, ""); var z = store.GetDictionary(paramName, new Dictionary()); ``` ## Using Shared Instance In some applications, you may want to create a single Statsig instance that can be accessed globally throughout your codebase. The shared instance functionality provides a singleton pattern for this purpose: ```csharp theme={null} var sharedStatsig = Statsig.NewShared("server-secret-key"); await sharedStatsig.Initialize(); // Later, anywhere in your codebase var statsig = Statsig.Shared(); // Use the shared instance var result = statsig.CheckGate(user, "my_gate"); ``` The shared instance is useful for: * Singleton pattern usage across your application * Dependency injection scenarios * Avoiding multiple SDK instances Remember to clean up the shared instance on shutdown: ```csharp theme={null} var statsig = Statsig.Shared(); await statsig.FlushEvents(); await statsig.Shutdown(); Statsig.RemoveSharedInstance(); ``` ## Manual Exposures By default, the SDK will automatically log an exposure event when you check a gate, get a config, get an experiment, or call get() on a parameter in a layer. However, there are times when you may want to log an exposure event manually. For example, if you're using a gate to control access to a feature, but you don't want to log an exposure until the user actually uses the feature, you can use manual exposures. All of the main SDK functions (`CheckGate`, `GetDynamicConfig`, `GetExperiment`, `GetLayer`) accept an optional `EvaluationOptions` parameter. When `disableExposureLogging` is set to `true`, the SDK will not automatically log an exposure event. You can then manually log the exposure at a later time using the corresponding manual exposure logging method: ```csharp theme={null} var result = statsig.CheckGate(user, "a_gate_name", new EvaluationOptions(disableExposureLogging: true)); ``` ```csharp theme={null} statsig.ManuallyLogGateExposure(user, "a_gate_name"); ``` ```csharp theme={null} var config = statsig.GetDynamicConfig(user, "a_dynamic_config_name", new EvaluationOptions(disableExposureLogging: true)); ``` ```csharp theme={null} statsig.ManuallyLogDynamicConfigExposure(user, "a_dynamic_config_name"); ``` ```csharp theme={null} var experiment = statsig.GetExperiment(user, "an_experiment_name", new EvaluationOptions(disableExposureLogging: true)); ``` ```csharp theme={null} statsig.ManuallyLogExperimentExposure(user, "an_experiment_name"); ``` ```csharp theme={null} var layer = statsig.GetLayer(user, "a_layer_name", new EvaluationOptions(disableExposureLogging: true)); var paramValue = layer.Get("a_param_name", "fallback_value"); ``` ```csharp theme={null} statsig.ManuallyLogLayerParameterExposure(user, "a_layer_name", "a_param_name"); ``` ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. `StatsigUser` represents the user context for feature flag evaluation. Use `StatsigUserBuilder` to create user instances: ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .SetEmail("user@example.com") .SetIP("192.168.1.1") .SetUserAgent("Mozilla/5.0...") .SetCountry("US") .SetLocale("en-US") .SetAppVersion("1.2.3") .SetCustomIDs(new Dictionary { ["employee_id"] = "emp_456", ["team_id"] = "team_789" }) .AddCustomID("department_id", "dept_123") .SetCustomProperties(new Dictionary { ["subscription_tier"] = "premium", ["account_age_days"] = 365, ["is_beta_user"] = true }) .AddCustomProperty("last_login", DateTime.UtcNow) .SetPrivateAttributes(new Dictionary { ["internal_user_score"] = 0.85, ["risk_level"] = "low" }) .AddPrivateAttribute("pii_hash", "abc123def456") .Build(); ``` ## Builder Methods * **SetUserID(string)**: Set the primary user ID * **SetEmail(string)**: Set user email * **SetIP(string)**: Set user IP address * **SetUserAgent(string)**: Set browser user agent * **SetCountry(string)**: Set user country * **SetLocale(string)**: Set user locale * **SetAppVersion(string)**: Set app version * **SetCustomIDs(Dictionary\)**: Set all custom IDs * **AddCustomID(string, string)**: Add a single custom ID * **SetCustomProperties(Dictionary\)**: Set all custom properties * **AddCustomProperty(string, object)**: Add a single custom property * **SetPrivateAttributes(Dictionary\)**: Set all private attributes * **AddPrivateAttribute(string, object)**: Add a single private attribute * **Build()**: Create the StatsigUser instance Remember to dispose of StatsigUser instances when done: ```csharp theme={null} using var user = new StatsigUserBuilder() .SetUserID("user_123") .Build(); ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. `StatsigOptions` can be configured using the `StatsigOptionsBuilder` pattern: ```csharp theme={null} var options = new StatsigOptionsBuilder() .SetSpecsURL("https://custom-api.statsig.com/v1/download_config_specs") .SetLogEventURL("https://custom-api.statsig.com/v1/rgstr") .SetEnvironment("production") .SetSpecsSyncIntervalMs(30000) .SetEventLoggingMaxQueueSize(1000) .SetWaitForCountryLookupInit(true) .SetWaitForUserAgentInit(true) .SetDisableCountryLookup(false) .SetDisableUserAgentParsing(false) .SetDisableAllLogging(false) .SetInitTimeoutMs(3000) .SetFallbackToStatsigApi(false) .SetEnableIDLists(true) .SetIDListsURL("https://custom-api.statsig.com/v1/get_id_lists") .SetIDListsSyncIntervalMs(60000) .SetGlobalCustomFields(new Dictionary { ["app_version"] = "1.2.3", ["build_number"] = "456" }) .Build(); var statsig = new Statsig("server-secret-key", options); ``` ## Available Options * **SetSpecsURL(string)**: Override the default specs download endpoint * **SetLogEventURL(string)**: Override the default event logging endpoint * **SetEnvironment(string)**: Set the environment tier (e.g., "production", "staging") * **SetSpecsSyncIntervalMs(int)**: How often to sync configuration specs (default: 10000ms) * **SetEventLoggingMaxQueueSize(int)**: Maximum events to queue before flushing * **SetWaitForCountryLookupInit(bool)**: Wait for country lookup initialization * **SetWaitForUserAgentInit(bool)**: Wait for user agent parsing initialization * **SetDisableCountryLookup(bool)**: Disable automatic country detection * **SetDisableUserAgentParsing(bool)**: Disable user agent parsing * **SetDisableAllLogging(bool)**: Disable all event logging * **SetInitTimeoutMs(int)**: Maximum time in milliseconds to wait for SDK initialization (default: 3000ms) * **SetFallbackToStatsigApi(bool)**: Fallback to Statsig API when custom adapters fail (default: false) * **SetEnableIDLists(bool)**: Enable ID list targeting * **SetIDListsURL(string)**: Override the default ID lists endpoint * **SetIDListsSyncIntervalMs(int)**: How often to sync ID lists (default: 60000ms) * **SetGlobalCustomFields(Dictionary\)**: Global custom fields for all events * **SetSpecAdapterConfig(SpecAdapterConfig)**: Configure a custom spec source such as [Statsig Forward Proxy](/infrastructure/forward-proxy) * **SetProxyConfig(ProxyConfig)**: Configuration for connecting through a proxy server ### Proxy and Custom Network Routing The `.NET` Server Core SDK uses `SetProxyConfig(ProxyConfig)` for a standard outbound HTTP proxy. If you are routing config sync through [Statsig Forward Proxy](/infrastructure/forward-proxy) or another custom spec source, use `SetSpecAdapterConfig(SpecAdapterConfig)`. If you only need custom Statsig endpoints, use `SetSpecsURL`, `SetLogEventURL`, and `SetIDListsURL`. ```csharp theme={null} var proxyConfig = new ProxyConfig { ProxyHost = "proxy.example.com", ProxyPort = 8080, ProxyAuth = "username:password", // Optional ProxyProtocol = "http", // Optional: "http" or "https" CaCertPath = "/etc/ssl/certs/corporate-ca.pem" // Optional }; var options = new StatsigOptionsBuilder() .SetProxyConfig(proxyConfig) .Build(); var statsig = new Statsig("server-secret-key", options); ``` ### ProxyConfig Properties * **ProxyHost** (string): The hostname or IP address of the proxy server * **ProxyPort** (int): The port number of the proxy server * **ProxyAuth** (string, optional): Authentication credentials in the format "username:password" * **ProxyProtocol** (string, optional): The protocol to use for the proxy connection ("http" or "https") * **CaCertPath** (string, optional): Path to a PEM CA bundle for outbound TLS ### Statsig Forward Proxy Example ```csharp theme={null} var specAdapterConfig = new SpecAdapterConfig( adapterType: "network_grpc_websocket", specsUrl: "http://forward-proxy.internal:50051" ); var options = new StatsigOptionsBuilder() .SetSpecAdapterConfig(specAdapterConfig) .SetFallbackToStatsigApi(true) .Build(); ``` `EvaluationOptions` allows you to customize the behavior of feature flag evaluations: ```csharp theme={null} var options = new EvaluationOptions(disableExposureLogging: true); var gateValue = statsig.CheckGate(user, "feature_gate", options); var config = statsig.GetDynamicConfig(user, "product_config", options); var experiment = statsig.GetExperiment(user, "button_test", options); var layer = statsig.GetLayer(user, "user_prefs_layer", options); ``` ## Options * **DisableExposureLogging**: When `true`, prevents automatic exposure event logging for this evaluation. Useful when you want to evaluate a feature flag without affecting analytics or experiment results. ## Use Cases * **Internal Tools**: Check flag values for debugging without affecting user metrics * **Conditional Logic**: Evaluate flags as part of complex logic where exposure should be logged manually later When exposure logging is disabled, you can manually log exposures later using the manual exposure methods: ```csharp theme={null} var options = new EvaluationOptions(disableExposureLogging: true); var gateValue = statsig.CheckGate(user, "feature_gate", options); if (shouldLogExposure) { statsig.ManuallyLogGateExposure(user, "feature_gate"); } ``` ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```csharp theme={null} await statsig.FlushEvents(); await statsig.Shutdown(); statsig.Dispose(); ``` For shared instances: ```csharp theme={null} var statsig = Statsig.Shared(); await statsig.FlushEvents(); await statsig.Shutdown(); Statsig.RemoveSharedInstance(); ``` ## Methods * **FlushEvents()**: Immediately flush any pending events to Statsig servers * **Shutdown()**: Gracefully shutdown the SDK, flushing events and cleaning up resources * **Dispose()**: Release native resources (implements IDisposable) It's recommended to call `FlushEvents()` before `Shutdown()` to ensure all events are sent, and always call `Dispose()` or use `using` statements to properly clean up resources. ## Client SDK Bootstrapping | SSR If you are using the Statsig client SDK in a browser or mobile app, you can bootstrap the client SDK with the values from the server SDK to avoid a network request on the client. This is useful for server-side rendering (SSR) or when you want to reduce the number of network requests on the client. ```csharp theme={null} var user = new StatsigUserBuilder() .SetUserID("user_123") .Build(); var initResponse = statsig.GetClientInitializeResponse(user); var options = new ClientInitResponseOptions { HashAlgorithm = "sha256", ClientSDKKey = "client-sdk-key", IncludeLocalOverrides = false }; var customInitResponse = statsig.GetClientInitializeResponse(user, options); ``` The `GetClientInitializeResponse` method returns a JSON string containing the initialization data needed by client-side SDKs. This enables server-side rendering and reduces client initialization time. ## ClientInitResponseOptions * **HashAlgorithm**: Hash algorithm for response integrity (default: "djb2") * **ClientSDKKey**: Client SDK key to include in response * **IncludeLocalOverrides**: Whether to include local overrides in the response (default: false) ### Working with IP or UserAgent Values The server SDK will not automatically use the `ip`, or `userAgent` for gate evaluation as Statsig servers would, since we don't have access to the request headers. If you'd like to use the attributes we derive from these properties, like Browser Name/Version, OS Name/Version & Country, you must manually set the `ip` and `userAgent` fields on the user object when calling `GetClientInitializeResponse`. ### Working with IDs To ensure that the server SDK evaluates each config accurately, they need access to all user attributes that the client SDK leverages. We recommend passing all of these attributes to the server SDK - using tools like Cookies if needed to ensure they're attached on first requests. If the user objects on the client and server aren't identical, modern SDKs will throw an InvalidBootstrap warning. Client SDKs also auto-generate a StableID, and it's important to manage the lifecycle of this ID to be sure that it is consistent on client and server side. Managing this with a cookie is often easiest, see [Keeping StableID Consistent](/client/javascript-sdk-stable-id#keeping-stableid-consistent). If StableID differs between Client and Server, you'll see a BootstrapStableIDMismatch warning, and checks with this warning won't contribute to your experiment analyses. ### getClientInitializeResponse and the legacy JS SDK If you are migrating from the legacy JS Client, you will need to make some updates to how your server SDK generates values. The default hashing algorithm was changed from `sha256` to `djb2` for performance and size reasons. ## Local Overrides Local Overrides are a way to override the values of gates, configs, experiments, and layers for testing purposes. This is useful for local development or testing scenarios where you want to force a specific value without having to change the configuration in the Statsig console. ```csharp theme={null} statsig.OverrideGate("test_gate", true); statsig.OverrideDynamicConfig("test_config", new Dictionary { ["color"] = "red", ["size"] = 42, ["enabled"] = true }); statsig.OverrideExperiment("test_experiment", new Dictionary { ["variant"] = "treatment", ["multiplier"] = 1.5 }); statsig.OverrideExperimentByGroupName("test_experiment", "treatment_group"); statsig.OverrideLayer("test_layer", new Dictionary { ["theme"] = "dark", ["font_size"] = 16 }); statsig.OverrideParameterStore("testing123", new Dictionary() { ["brush_color"] = "blue", ["monochromatic"] = true, ["weight"] = 42, ["gradient"] = 3.14, ["pen_sizes"] = [1, 2, 3, 4, 5], ["artwork"] = new Dictionary() { ["nesting"] = "treatment" } }); ``` You can also specify a user ID for targeted overrides: ```csharp theme={null} statsig.OverrideGate("test_gate", true, "user_123"); statsig.OverrideDynamicConfig("test_config", new Dictionary { ["special_feature"] = true }, "user_123"); ``` Local overrides are useful for: * Testing specific configurations during development * QA testing with known values * Debugging feature flag behavior * Integration testing with predictable results Note: Overrides persist for the lifetime of the Statsig instance and affect all evaluations unless a specific user ID is provided. ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for user-specific configurations. This enables you to persist user assignments across sessions, ensuring consistent experiment groups even when the user returns later. This is particularly useful for client-side A/B testing where you want to ensure users always see the same variant. ```csharp theme={null} using System.Collections.Generic; using Statsig; // Implement PersistentStorage to control how user stickiness is stored. public class MyPersistentStorage : PersistentStorage { private readonly Dictionary> _store = new(); public override IDictionary Load(string key) { // Load persisted values for this user from your backing store. return _store.TryGetValue(key, out var configs) ? new Dictionary(configs) : new Dictionary(); } public override void Save(string key, string configName, StickyValues data) { // Persist the sticky assignment (database, Redis, etc.). if (!_store.TryGetValue(key, out var configs)) { configs = new Dictionary(); _store[key] = configs; } configs[configName] = data; } public override void Delete(string key, string configName) { // Remove the persisted value for this config/user. if (_store.TryGetValue(key, out var configs)) { configs.Remove(configName); } } } ``` ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. Data stores allow you to customize how the SDK fetches and caches feature specifications, enabling advanced use cases like using Redis or other distributed caches. ```csharp Csharp theme={null} using System.Threading.Tasks; using Statsig; public class MyDataStore : DataStore { public Task Initialize() { // Perform any initialization needed for your data store. return Task.CompletedTask; } public Task Shutdown() { // Clean up resources. return Task.CompletedTask; } public DataStoreResponse Get(string key) { // Retrieve data for the given key. // This is called during SDK evaluation. return Task.FromResult(null); } public void Set(string key, string value, long? time = null) { // Store data for the given key. // Called when SDK receives updates from Statsig. return Task.CompletedTask; } public bool SupportsPollingUpdatesFor(string key) { // Return true if your store supports polling updates for this key, false otherwise. return Task.FromResult(false).CompletedTask; } } // Use data store var options = new StatsigOptionsBuilder() .SetDataStore(new MyDataStore()) .Build(); var statsig = new Statsig("server-secret-key", options); await statsig.Initialize(); ``` ## Performance Benefits The .NET Core SDK leverages Statsig's high-performance Rust evaluation engine through FFI bindings, details: * Native Rust evaluation engine handles all rule processing * .NET wrapper provides familiar C# APIs and type safety * Automatic memory management between .NET and Rust boundaries * Thread-safe operations across the FFI boundary ## Async/Await Support All network operations are fully async: ```csharp theme={null} await statsig.Initialize(); await statsig.FlushEvents(); await statsig.Shutdown(); ``` Evaluation methods are synchronous for optimal performance: ```csharp theme={null} var result = statsig.CheckGate(user, "gate_name"); ``` ## Thread Safety The Statsig instance is thread-safe and can be used concurrently across multiple threads. Consider using the shared instance(singleton) pattern for application-wide usage: ```csharp theme={null} var sharedStatsig = Statsig.NewShared("server-secret-key"); await sharedStatsig.Initialize(); var statsig = Statsig.Shared(); ``` # Elixir Server SDK Source: https://docs.statsig.com/server-core/elixir-core Statsig's next-generation Elixir Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for BEAM apps. Elixir Core on Github, Hex Package ## Setup the SDK To use the SDK, add the following to your `mix.exs`: ```elixir theme={null} {:statsig_elixir, "~> 0.8.0"}, ``` SDK is written using `rustler_precompiled`, which will download precompiled binary by default. But there is also an option to build Rust code within your project by cloning [statsig-server-core](https://github.com/statsig-io/statsig-server-core) ```bash theme={null} cd statsig-server-core/statsig-elixir/ # set the environment variable FORCE_STATSIG_NATIVE_BUILD="true" mix compile ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. Statsig.ex is using GenServer to manage the actual implementation of statsig instance (which is written in Rust). Which requires you add Statsig into your Supervision Tree. ```elixir theme={null} # Initializing, with StatsigOptions sdk_key = "secret-key******" # your secret key statsig_options = %StatsigOptions{enable_id_lists: true} # Add to your supervision tree statsig_spec = %{id: Statsig, start: {Statsig, :start_link, [sdk_key, statsig_options]}} children = [ # Other Apps statsig_spec ] res = Supervisor.start_link(children, opts) # Or directly initialize the GenServer {:ok,_} = Statsig.start_link(sdk_key, statsig_options) Statsig.initialize() ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```elixir theme={null} user = %StatsigUser{ user_id: "test_user_123" } {:ok, check_gate} = Statsig.check_gate("test_public", user) # check_gate will be a boolean ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```elixir theme={null} # Get a dynamic config for a specific user user = %StatsigUser{ user_id: "test_user_123" } {:ok, config} = Statsig.get_dynamic_config("a_config", user) # Access the values in the dynamic config: param_value = DynamicConfig.get_param_value(config, "header_text") # Disable exposure options = %Statsig.DynamicConfigEvaluationOptions{ disable_exposure_logging = true } {:ok, config} = Statsig.get_dynamic_config("a_config", user, options) ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```elixir theme={null} # Values via get_layer user = %StatsigUser{ user_id: "test_user_123" } {:ok, layer} = Statsig.get_layer("user_promo_experiments", user) {:ok, title_string_value} = Layer.get(layer, "title", "Welcome to Statsig!") {:ok, discount_float_value} = Layer.get(layer, "discount", 0.1) # Via get_experiment {:ok, experiment} = Statsig.get_experiment("user_promo_experiment", user) title_exp = Experiment.get_param_value(experiment, "new_user_promo_title") # Disable exposure options = %Statsig.ExperimentEvaluationOptions{ disable_exposure_logging = true } {:ok, experiment} = Statsig.get_experiment("user_promo_experiment", user, options) ``` If you are using layer to get value -- get param value. It will return primitive types: boolean, string, and numbers, for more complex type, SDK will return json serialized values. ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```elixir theme={null} {:ok, feature_gate} = Statsig.get_feature_gate(user, "example_gate") # access the value, or the name off of the feature_gate object options = %Statsig.FeatureGateEvaluationOptions{ disable_exposure_logging = true } {:ok, feature_gate} = Statsig.get_feature_gate(user, "example_gate", options) ``` ### Parameter Stores Sometimes you don't know whether you want a value to be a Feature Gate, Experiment, or Dynamic Config yet. If you want on-the-fly control of that outside of your deployment cycle, you can use Parameter Stores to define a parameter that can be changed into at any point in the Statsig console. Parameter Stores are optional, but parameterizing your application can prove very useful for future flexibility and can even allow non-technical Statsig users to turn parameters into experiments. *Parameter stores are not yet available for this sdk. Need it now? Let us know in Slack.* ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```elixir theme={null} Statsig.log_event(user, "test_event", 1, %{"metadata_1" => "value"}) ``` ### Sending Events to Log Explorer You can forward logs to Logs Explorer for convenient analysis using the Forward Log Line Event API. This lets you include custom metadata and event values with each log. Sending events to Log Explorer is not yet available for this sdk. Need it now? Let us know in Slack. ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. ```elixir theme={null} user = %Statsig.User{ user_id: "a-user-id", email: "user@example.com", ip: "192.168.1.1", user_agent: "Mozilla/5.0...", country: "US", locale: "en_US", app_version: "1.0.0", custom: %{ # Custom fields "plan" => "premium", "age" => 25 }, custom_ids: %{ # Custom ID types "stable_id" => "stable-id-123" }, private_attributes: %{ # Private attributes not forwarded to integrations "email" => "private@example.com" } } ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. The environment you're operating in (e.g., Production) The type of logs you'd like exposed. (e.g., Debug) How long it should take for initialization to time out. How often events should flush to the Statsig servers. Maximum number of events to queue before forcing a flush. * Default is `2000` * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * See also `event_logging_max_pending_batch_queue_size` Maximum number of event batches to hold in buffer to retry. * Default is `100`. * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * eg: 2000 \* 100 means the SDK can process 200k event per second before events start getting dropped * See also `event_logging_max_queue_size`. The URL events should be logged to How often specs should sync from the Statsig servers The URL Statsig should download specs from Advanced configuration for fetching specs from custom sources, including [Statsig Forward Proxy](/infrastructure/forward-proxy). Enable ID list download. **Required to be `true` when using segments with more than 1000 IDs.** See [ID List segments](/segments/add-id-list) for more details. The URL ID lists should be downloaded from How often ID lists should be synced. Block initialization call until country lookup is initialized Block initialization call until user\_agent is initialized When `true`, disables all event logging. To improve memory usage, disable using country lookup. Turn off all network requests including get\_dcs and log\_events To improve memory usage, disable using user agent parsing. *** ```elixir theme={null} # Initialize StatsigOptions with custom parameters statsig_options = %StatsigOptions{ enable_id_lists: true } # Pass the options object into statsig.initialize() {:ok, _} = Statsig.start_link(sdk_key, statsig_options) Statsig.initialize() ``` ### Proxy and Custom Network Routing The Elixir Server Core SDK does not currently document a dedicated outbound HTTP proxy config. For custom network routing, use `spec_adapter_configs` for [Statsig Forward Proxy](/infrastructure/forward-proxy) or set `specs_url`, `log_event_url`, and `id_lists_url` directly. ```elixir theme={null} statsig_options = %StatsigOptions{ spec_adapter_configs: [ %Statsig.SpecAdapterConfig{ adapter_type: "network_grpc_websocket", specs_url: "http://forward-proxy.internal:50051", authentication_mode: "none" } ], fallback_to_statsig_api: true } ``` ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```elixir theme={null} Statsig.shutdown() ``` ## Client SDK Bootstrapping | SSR If you are using the Statsig client SDK in a browser or mobile app, you can bootstrap the client SDK with the values from the server SDK to avoid a network request on the client. This is useful for server-side rendering (SSR) or when you want to reduce the number of network requests on the client. ```elixir theme={null} # Get client initialize response for a user {:ok, response} = Statsig.get_client_init_response_as_string(user) # Pass values to a client SDK to initialize without a network request ``` ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for user-specific configurations. This enables you to persist user assignments across sessions, ensuring consistent experiment groups even when the user returns later. This is particularly useful for client-side A/B testing where you want to ensure users always see the same variant. Not supported at this time. ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. Not supported at this time. ## Custom Output Logger The Output Logger interface allows you to customize how the SDK logs messages. This enables integration with your own logging system and control over log verbosity. Not supported at this time. ## Observability Client The Observability Client interface allows you to monitor the health of the SDK by integrating with your own observability systems. This enables tracking metrics, errors, and performance data. For more information on the metrics emitted by Statsig SDKs, see the [Monitoring documentation](/sdk_monitoring). Not supported at this time. ## FAQ See the guide on [device level experiments](/guides/logging-events#device-level-events) # Go Server Core SDK (Beta) Source: https://docs.statsig.com/server-core/go-core Statsig's next-generation Go Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for Go backends. Go Core on Github ## Setup the SDK via the `go get` CLI: You can install the latest version of the SDK: ``` go get github.com/statsig-io/statsig-go-core@latest ``` Or, add a dependency on the most recent version of the SDK in go.mod: ``` require ( github.com/statsig-io/statsig-go-core v0.10.2-beta ) ``` See the [Releases tab in GitHub](https://github.com/statsig-io/statsig-server-core/releases) for the latest versions. :::note `go get` requires a Go module project with a go.mod file. ::: After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. ```go expandable theme={null} import ( "log" statsig "github.com/statsig-io/statsig-go-core" ) options, err := statsig.NewOptionsBuilder(). WithOutputLogLevel("DEBUG"). Build() if err != nil { log.Fatalf("failed to build options: %v", err) } s, err := statsig.NewStatsigWithOptions("secret-key", options) if err != nil { log.Fatalf("failed to create Statsig client: %v", err) } s.Initialize() ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() // Basic check if s.CheckGate(user, "a_gate") { // Gate is on, enable new feature } else { // Gate is off } // With options (e.g., disable automatic exposure logging) options := &statsig.FeatureGateEvaluationOptions{DisableExposureLogging: true} s.CheckGateWithOptions(user, "a_gate", options) ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() // You can disable exposure logging for this specific check dynamicConfigOptions := &statsig.DynamicConfigEvaluationOptions{DisableExposureLogging: false} dynamic_config := "a_config" config := s.GetDynamicConfigWithOptions(user, dynamic_config, dynamicConfigOptions) // Access the config values str_val := config.GetString("str_val", "default_str_val") // returns String int_val := config.GetNumber("number_val", 100) // returns float64 bool_val := config.GetBool("bool_val", false) // returns bool interface_val := config.GetSlice("interface_val", []any{"default_interface_val"}) // returns []any map_val := config.GetMap("map_val", map[string]any{"default": "value"}) // returns map[string]interface{} // The config object also provides metadata about the evaluation fmt.Println(config.RuleID) // The ID of the rule that served this config fmt.Println(config.IDType) // The type of the evaluation (experiment, config, etc) ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() // Getting values with GetExperiment experimentOptions := &statsig.ExperimentEvaluationOptions{DisableExposureLogging: false} experiment := s.GetExperimentWithOptions(user, "a_test_experiment", experimentOptions) str_val := experiment.GetString("str_val", "default_str_val") // returns String int_val := experiment.GetNumber("number_val", 100) // returns float64 // Getting values with GetLayer layerOptions := &statsig.LayerEvaluationOptions{DisableExposureLogging: false} layer := s.GetLayerWithOptions(user, "a_test_experiment", layerOptions) str_val := layer.GetString("str_val", "default_str_val") // returns String int_val := layer.GetNumber("number_val", 100) // returns float64 ``` ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() options := &statsig.FeatureGateEvaluationOptions{DisableExposureLogging: false} featureGate := s.GetFeatureGateWithOptions(user, "a_gate", options) ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() event := statsig.EventPayload{ EventName: "sample_event", Value: "event", Metadata: map[string]string{ "val_1": "log val 1", }, } s.LogEvent(user, event) ``` ## Manual Exposures By default, the SDK will automatically log an exposure event when you check a gate, get a config, get an experiment, or call get() on a parameter in a layer. However, there are times when you may want to log an exposure event manually. For example, if you're using a gate to control access to a feature, but you don't want to log an exposure until the user actually uses the feature, you can use manual exposures. All of the main SDK functions (`CheckGate`, `GetDynamicConfig`, `GetExperiment`, `GetLayer`) accept an optional options parameter with `DisableExposureLogging` field. When this is set to `true`, the SDK will not automatically log an exposure event. You can then manually log the exposure at a later time using the corresponding manual exposure logging method: ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() options := &statsig.FeatureGateEvaluationOptions{DisableExposureLogging: true} result := s.CheckGateWithOptions(user, "a_gate_name", options) ``` ```go theme={null} s.ManuallyLogFeatureGateExposure(user, "gate_name") ``` ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() options := &statsig.DynamicConfigEvaluationOptions{DisableExposureLogging: true} config := s.GetDynamicConfigWithOptions(user, "config_name", options) ``` ```go theme={null} s.ManuallyLogDynamicConfigExposure(user, "config_name") ``` ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() options := &statsig.ExperimentEvaluationOptions{DisableExposureLogging: true} experiment := s.GetExperimentWithOptions(user, "experiment_name", options) ``` ```go theme={null} s.ManuallyLogExperimentExposure(user, "experiment_name") ``` ```go theme={null} user, _ := statsig.NewUserBuilderWithUserID("a-user").Build() options := &statsig.LayerEvaluationOptions{DisableExposureLogging: true} layer := s.GetLayerWithOptions(user, "layer_name", options) paramValue := layer.GetString("param_name", "fallback") ``` ```go theme={null} s.ManuallyLogLayerParamExposure(user, "layer_name", "param_name") ``` ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. ### Parameters Custom URL for fetching feature specifications. Custom URL for logging events. Environment parameter for evaluation. How often events are flushed to Statsig servers (in milliseconds). Maximum number of events to queue before forcing a flush. How often the SDK updates specifications from Statsig servers (in milliseconds). Controls the verbosity of SDK logs. Disables country lookup based on IP address. Set to `true` to improve performance if country-based targeting is not needed. Disables user agent parsing. Set to `true` to improve performance if device/browser-based targeting is not needed. When set to true, the SDK will wait for country lookup data (e.g., GeoIP or YAML files) to fully load during initialization. This may slow down by \~1 second startup but ensures that IP-to-country parsing is ready at evaluation time. When set to true, the SDK will wait until user agent parsing data is fully loaded during initialization. This may slow down by \~1 second startup but ensures that parsing of the user's userAgent string into fields like browserName, browserVersion, systemName, systemVersion, and appVersion is ready before any evaluations. Enables downloading ID lists; set to false to skip ID list syncing. Custom URL for fetching ID lists. How often the SDK syncs ID lists (in milliseconds). When set to true, the SDK will not log any events or exposures. When set to true, the SDK will not make any network requests. JSON string of custom fields that should be appended to every evaluation. Internal reference for a custom observability client created via the SDK. Internal reference for a custom data store adapter created via the SDK. Internal reference for a persistent storage adapter created via the SDK. Maximum time in milliseconds to wait for SDK initialization to complete. If initialization takes longer than this timeout, the SDK will continue to operate but may return default values until initialization completes. When set to true, the SDK will fallback to using the Statsig API directly if custom adapters (like local file adapters) fail to load configurations. ### Proxy and Custom Network Routing The Go Server Core SDK currently documents endpoint overrides rather than a dedicated outbound proxy config. Use `WithSpecsUrl(...)`, `WithLogEventUrl(...)`, and `WithIdListsUrl(...)` to route Statsig network calls through your own endpoints or proxy layer. *** ### Example Options Usage ```go expandable theme={null} import ( statsig "github.com/statsig-io/statsig-go-core" ) // Initialize StatsigOptions with custom parameters options, err := statsig.NewOptionsBuilder(). WithSpecsUrl("https://example.com/specsUrl"). WithLogEventUrl("https://example.com/logUrl"). WithEnvironment("production"). WithEventLoggingFlushIntervalMs(2000). WithEventLoggingMaxQueueSize(5000). WithSpecsSyncIntervalMs(1000). WithOutputLogLevel("DEBUG"). WithDisableCountryLookup(true). WithDisableUserAgentParsing(true). WithWaitForCountryLookupInit(false). WithEnableIdLists(true). WithIdListsSyncIntervalMs(60000). WithInitTimeoutMs(3000). WithFallbackToStatsigApi(false). Build() if err != nil { // handle options build error } // Pass the options object when initializing the Statsig client s, err := statsig.NewStatsigWithOptions("secret-key", options) if err != nil { // handle init error } s.Initialize() ``` ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```go theme={null} // Method signature func (s *Statsig) Shutdown() {} // example usage s, err := statsig.NewStatsig("secret-key") s.Initialize() s.Shutdown() ``` ### Flush Events ```go theme={null} // Method signature func (s *Statsig) FlushEvents() {} // example usage s, err := statsig.NewStatsig("secret-key") s.Initialize() s.FlushEvents() ``` ## FAQ ##### Installation FAQs #### How do I fix undefined symbol errors/linker errors? You may need to reset your environment variables to those found in the statsig.env or statsig.env.ps1 file which was generated during the post-install script. Running the command sets the environment variables for the current terminal session. Users may need to add in the variables to their .bashrc, .zshrc, or .ps1 file to load them automatically. # Server Core Overview Source: https://docs.statsig.com/server-core/index Statsig Server Core is the second generation of Server SDKs with improved performance, a unified evaluation engine, and consistent APIs across languages. ## Statsig Server Core Statsig Server Core is our second generation of Server SDKs: a full rewrite with a shared, performance-focused core library that enables superior across-the-board perf and feature maturity across our Server SDKs. Server Core SDKs include: * **Faster evaluation:** A shared, performance-focused Rust evaluation engine, that evaluates 3-5x as our pure native code SDKs. * **Superior non-eval performance:** more efficient network and CPU usage, plus other performance optimizations always arriving to each SDK. * **Features new to Server SDKs:** Parameter Stores, Contextual Multi-Armed Bandits, and more. * **Quality-of-life improvements from certain SDKs:** Observability Interface, streaming config changes (from the Statsig Forward Proxy) and more ### Availability across SDKs Server Core SDKs are available on an opt-in basis, with native SDKs still available and supported in all languages. If you're just getting started with Statsig, we recommend choosing a Server Core SDK if its convenient for you, but we continue to support bug fixes in our [Legacy SDKs](/server-core/legacy-sdks), and will until we set end-of-support dates for these SDKs. At that time - we'll also provide guidance to make migration as easy as possible. | SDK | Status | Package | Migration Guide | | ---------------------------------- | -------------- | ------------------------------------------------------------------------------------ | -------------------------------------------- | | [Node](/server-core/node-core) | Stable | [npm](https://www.npmjs.com/package/@statsig/statsig-node-core) | [Link](/server-core/migration-guides/node) | | [Python](/server-core/python-core) | Stable | [PyPI](https://pypi.org/project/statsig-python-core/) | [Link](/server-core/migration-guides/python) | | [Java](/server-core/java-core) | Stable | [Maven Central](https://central.sonatype.com/artifact/com.statsig/javacore/overview) | [Link](server-core/migration-guides/java) | | [PHP](/server-core/php-core) | Stable | [Packagist](https://packagist.org/packages/statsig/statsigsdk) | | | [Rust](/server-core/rust-core) | Stable | [Crates.io](https://crates.io/crates/statsig-rust) | | | [Elixir](/server-core/elixir-core) | Stable | | | | [C++](/server-core/cpp-core) | Stable | | | | [.NET](/server-core/dotnet-core) | Stable | [NuGet](https://www.nuget.org/packages/Statsig.Dotnet) | | | Ruby | In Development | | | | [Go](/server-core/go-core) | Beta | | | ### Technical differences **Build process:** Statsig Server Core uses a core library written in Rust, with bindings written to other languages. In the vast majority of cases, this results in an unchanged development experience with superior performance, but given that the Rust code must compile into a binary usable in your development and deployment environments, some development snafus exist: * **Choosing the right build:** In most cases, the SDK's package manager will automatically install the versions you need. The notable exception is in Java - where our SDK will print out the right build for you, if it's not included at runtime. * **Managing lockfiles:** If your deployment and development environments require different builds (as is common), you'll want to include both versions in a lockfile like package-lock.json. * **Untested environments:** Certain environments, like the edge, aren't friendly to this new build process - and for now, we recommend using our [native SDKs](/server-core/legacy-sdks) for the time being. **New Configuration Spec:** Server Core uses a smaller "ruleset", or configuration spec. If you use the spec directly, your logic and parsing will have to change. **Event Logging** Server core SDKs, starting in v0.4.0+, have a new event logging architecture. This is designed to stream events freely to statsig servers during normal operation, and throttle/drop events SDK side during outages on the event logging endpoint to enable the service to spin up healthy before processing steady-state qps. We expose the following parameters to tune this implementation ``` - event_logging_max_queue_size: Controls batch size (default 2000). Note that exceeding the backend request size limit (10MB) will drop requests - event_logging_max_pending_batch_queue_size: Controls max pending batches (default: 20). The tradeoff here is increased memory usage to buffer events when requests fail if you increase it, and losing additional events if you decrease it - disable_all_logging: Completely disables event logging ``` ``` +----------------+ +----------------+ +----------------+ | Event Sources | | Event Queue | | Flush Triggers | |----------------| |----------------| |----------------| | - Gate Checks |---->| - Pending | | - Scheduled | | - Config Reads | | Events | | (Time-based) | | - Custom Events| | - Batches |<----| - Manual | +----------------+ +----------------+ | - Shutdown | | +----------------+ v +----------------+ | Flush Process | |----------------| | - Batch Events | | - Send to API | | - Process | | Response | +----------------+ / \ / \ v v +---------------------------+---------------------------+ | On Request Failure | On Request Success | |---------------------------|---------------------------| | - Double interval | - Halve interval | | (max: 60000ms) | (min: 1000ms) | | - Retry errors | - No specific limit | | up to 5 times | flush mechanism | +---------------------------+---------------------------+ ``` ### Support If you have trouble with a Server Core SDK - we'd love to hear your feedback. Reach out in our [Slack](https://statsig.com/slack) and we'll do our best to improve your experience! # Java Server SDK Source: https://docs.statsig.com/server-core/java-core Statsig's next-generation Java and Kotlin Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for JVM. Java Core on Github Migrating from the Legacy Java SDK? See our [Migration Guide](/server-core/migration-guides/java). ## Setup the SDK ## Requirements * Java 8 or higher (Java 8 support added in version 0.4.3) * Compatible with all platforms listed in the Supported OS and Architecture Combinations section, including: * macOS (x86\_64, arm64) * Windows (x86\_64) * Amazon Linux 2 and 2023 (x86\_64, arm64) ## Overview The Statsig Java SDK can be installed in two ways: **Recommended: Single JAR installation** (since version 0.4.0) * Use the "uber" JAR which contains both the core library and popular platform-specific native libraries in a single package * Simplifies dependency management and deployment across different environments **Advanced: Two-part installation** 1. The platform-independent core library 2. An OS/architecture-specific native library for your specific platform ## Installation Steps ### Recommended: Using the Uber JAR (All-in-One) Since version 0.4.0, Statsig provides an "uber" JAR that contains both the core library and native libraries for popular supported platforms in a single package. This is the recommended approach for most users. ```groovy Gradle theme={null} repositories { mavenCentral() } dependencies { implementation 'com.statsig:javacore:X.X.X:uber' // Uber JAR with all native libraries } ``` ```xml Maven theme={null} com.statsig javacore X.X.X uber ``` You can find the latest version on [Maven Central](https://central.sonatype.com/artifact/com.statsig/javacore). The uber JAR includes native libraries for: * Linux (x86\_64, arm64) * macOS (x86\_64, arm64) * Windows (x86\_64) This approach eliminates the need to specify the exact platform and simplifies deployment across different environments. ### Advanced: Platform-Specific Installation If you need more control over dependencies or want to minimize the JAR size for a specific platform, you can use the platform-specific installation approach. ```groovy Gradle theme={null} repositories { mavenCentral() } dependencies { implementation 'com.statsig:javacore:X.X.X' // Replace X.X.X with the latest version } ``` ```xml Maven theme={null} com.statsig javacore X.X.X ``` You can find the latest version on [Maven Central](https://central.sonatype.com/artifact/com.statsig/javacore). You need to add the appropriate OS/architecture-specific dependency. Choose one of the following methods: **Method 1: Automatic Detection** Run the following code to detect your system and get the appropriate dependency: ```java theme={null} import com.statsig.*; // All StatsigOptions are optional, feel free to adjust them as needed StatsigOptions options = new StatsigOptions.Builder().build(); Statsig statsig = new Statsig("your-secret-key", options); ``` You'll receive output similar to: ``` For Linux with arm64 architecture, add the following to build.gradle: implementation 'com.statsig:javacore::aarch64-unknown-linux-gnu' For Linux with x86_64 architecture, add the following to build.gradle: implementation 'com.statsig:javacore::x86_64-unknown-linux-gnu' ``` **Method 2: Manual Configuration** ```groovy Gradle theme={null} dependencies { implementation 'com.statsig:javacore:X.X.X' // Core SDK (from Step 1) implementation 'com.statsig:javacore:X.X.X:YOUR-OS-ARCHITECTURE' // OS/architecture-specific dependency } ``` ```xml Maven theme={null} com.statsig javacore X.X.X com.statsig javacore X.X.X YOUR-OS-ARCHITECTURE ``` Replace `YOUR-OS-ARCHITECTURE` with one of the supported combinations from the Supported OS and Architecture Combinations section. **Docker Considerations for Alpine Linux** When using Alpine Linux or other musl-based Docker containers, you need to install additional compatibility packages for the native libraries to work properly. Add the following to your Dockerfile: ```dockerfile theme={null} RUN apk add --no-cache libgcc gcompat ``` The Statsig Java Core SDK automatically detects musl-based systems and will use the appropriate musl-compatible native libraries (e.g., `x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl`). Docker base images where the Java Core SDK has been tested and verified: | Docker Base Image | Description | | ------------------------------------------- | -------------------------------------- | | amazoncorretto:21 | Amazon Corretto 21 JDK | | amazoncorretto:21-alpine-jdk | Amazon Corretto 21 JDK on Alpine Linux | | amazonlinux:2 | Amazon Linux 2 | | public.ecr.aws/amazonlinux/amazonlinux:2023 | Amazon Linux 2023 | | azul/zulu-openjdk-alpine:21 | Azul Zulu OpenJDK 21 on Alpine Linux | | azul/zulu-openjdk:21 | Azul Zulu OpenJDK 21 | | eclipse-temurin:21-jdk | Eclipse Temurin 21 JDK | | eclipse-temurin:21-jdk-alpine | Eclipse Temurin 21 JDK on Alpine Linux | After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. ```java Java theme={null} import com.statsig.*; // All StatsigOptions are optional, feel free to adjust them as needed StatsigOptions options = new StatsigOptions.Builder() .setSpecsSyncIntervalMs(10000) .setEventLoggingFlushIntervalMs(10000) .setOutputLoggerLevel(OutputLogger.LogLevel.INFO) .build(); Statsig myStatsigServer = new Statsig(SECRET_KEY, options); myStatsigServer.initialize().get(); ``` ```kotlin Kotlin theme={null} import com.statsig.* // All StatsigOptions are optional, feel free to adjust them as needed val options = StatsigOptions.Builder() .setSpecsSyncIntervalMs(10000) .setEventLoggingFlushIntervalMs(10000) .setOutputLoggerLevel(OutputLogger.LogLevel.INFO) .build() val myStatsigServer = Statsig(SECRET_KEY, options) myStatsigServer.initialize().get() ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Getting Started ### Quick Start Example Create a new Gradle or Maven project with the following structure: ``` my-statsig-app/ ├── build.gradle (or pom.xml) └── src/main/java/ExampleApp.java ``` ```groovy build.gradle theme={null} plugins { id 'java' id 'application' } repositories { mavenCentral() } dependencies { implementation 'com.statsig:javacore:X.X.X:uber' } application { mainClass = 'ExampleApp' } ``` ```xml pom.xml theme={null} 4.0.0 com.example my-statsig-app 1.0-SNAPSHOT com.statsig javacore X.X.X uber org.codehaus.mojo exec-maven-plugin 3.0.0 ExampleApp ``` Replace `X.X.X` with the latest version from [Maven Central](https://central.sonatype.com/artifact/com.statsig/javacore). ```java ExampleApp.java theme={null} import com.statsig.*; public class ExampleApp { public static void main(String[] args) throws Exception { // Initialize Statsig StatsigOptions options = new StatsigOptions.Builder().build(); Statsig statsig = new Statsig("YOUR_SERVER_SECRET_KEY", options); statsig.initialize().get(); try { // Check a feature gate boolean isEnabled = statsig.checkGate("user123", "my_feature_gate"); System.out.println("Feature gate is " + (isEnabled ? "enabled" : "disabled")); // Get a config DynamicConfig config = statsig.getConfig("user123", "my_config"); System.out.println("Config value: " + config.getString("some_parameter", "default_value")); } finally { // Always shutdown Statsig when done statsig.shutdown(); } } } ``` ```kotlin ExampleApp.kt theme={null} import com.statsig.* fun main() { // Initialize Statsig val options = StatsigOptions.Builder().build() val statsig = Statsig("YOUR_SERVER_SECRET_KEY", options) statsig.initialize().get() try { // Check a feature gate val isEnabled = statsig.checkGate("user123", "my_feature_gate") println("Feature gate is ${if (isEnabled) "enabled" else "disabled"}") // Get a config val config = statsig.getConfig("user123", "my_config") println("Config value: ${config.getString("some_parameter", "default_value")}") } finally { // Always shutdown Statsig when done statsig.shutdown().get() } } ``` Replace `YOUR_SERVER_SECRET_KEY` with your actual server secret key from the [Statsig Console](https://console.statsig.com/). ```bash Gradle theme={null} ./gradlew run ``` ```bash Maven theme={null} mvn compile exec:java ``` If everything is set up correctly, you should see output related to your feature gate and configuration. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```java Java theme={null} String userID = "user_id"; boolean result = statsig.checkGate(userID, "my_feature_gate"); // with StatsigUser StatsigUser user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build(); boolean gateResult = statsig.checkGate(user, "my_feature_gate"); ``` ```kotlin Kotlin theme={null} val userID = "user_id" val result = statsig.checkGate(userID, "my_feature_gate") // with StatsigUser val user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build() val gateResult = statsig.checkGate(user, "my_feature_gate") ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```java Java theme={null} String userID = "user_id"; DynamicConfig config = statsig.getConfig(userID, "my_config"); String name = config.getString("name", ""); int size = config.getInt("size", 10); // with StatsigUser StatsigUser user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build(); DynamicConfig dynamicConfig = statsig.getConfig(user, "my_config"); ``` ```kotlin Kotlin theme={null} val userID = "user_id" val config = statsig.getConfig(userID, "my_config") val name = config.getString("name", "") val size = config.getInt("size", 10) // with StatsigUser val user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build() val dynamicConfig = statsig.getConfig(user, "my_config") ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```java Java theme={null} String userID = "user_id"; // Getting an Experiment Experiment experiment = statsig.getExperiment(userID, "my_experiment"); String expName = experiment.getString("experiment_param", ""); // Getting a Layer Layer layer = statsig.getLayer(userID, "my_layer"); String layerValue = layer.getString("layer_param", "default"); // with StatsigUser StatsigUser user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build(); Experiment experimentWithUser = statsig.getExperiment(user, "my_experiment"); Layer layerWithUser = statsig.getLayer(user, "my_layer"); ``` ```kotlin Kotlin theme={null} val userID = "user_id" // Getting an Experiment val experiment = statsig.getExperiment(userID, "my_experiment") val expName = experiment.getString("experiment_param", "") // Getting a Layer val layer = statsig.getLayer(userID, "my_layer") val layerValue = layer.getString("layer_param", "default") // with StatsigUser val user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build() val experimentWithUser = statsig.getExperiment(user, "my_experiment") val layerWithUser = statsig.getLayer(user, "my_layer") ``` ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```java Java theme={null} String userID = "user_id"; FeatureGate gate = statsig.getFeatureGate(userID, "my_feature_gate"); System.out.println("Gate name: " + gate.name); System.out.println("Gate value: " + gate.value); System.out.println("Rule ID: " + gate.ruleID); // with StatsigUser StatsigUser user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build(); FeatureGate gateWithUser = statsig.getFeatureGate(user, "my_feature_gate"); ``` ```kotlin Kotlin theme={null} val userID = "user_id" val gate = statsig.getFeatureGate(userID, "my_feature_gate") println("Gate name: ${gate.name}") println("Gate value: ${gate.value}") println("Rule ID: ${gate.ruleID}") // with StatsigUser val user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build() val gateWithUser = statsig.getFeatureGate(user, "my_feature_gate") ``` ### Parameter Stores Sometimes you don't know whether you want a value to be a Feature Gate, Experiment, or Dynamic Config yet. If you want on-the-fly control of that outside of your deployment cycle, you can use Parameter Stores to define a parameter that can be changed into at any point in the Statsig console. Parameter Stores are optional, but parameterizing your application can prove very useful for future flexibility and can even allow non-technical Statsig users to turn parameters into experiments. ```java Java theme={null} String userID = "user_id"; ParameterStore store = statsig.getParameterStore(userID, "my_param_store"); String stringValue = store.getString("string_param", "default"); int intValue = store.getInt("int_param", 0); boolean boolValue = store.getBoolean("bool_param", false); double doubleValue = store.getDouble("double_param", 0.0); // with StatsigUser StatsigUser user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build(); ParameterStore storeWithUser = statsig.getParameterStore(user, "my_param_store"); ``` ```kotlin Kotlin theme={null} val userID = "user_id" val store = statsig.getParameterStore(userID, "my_param_store") val stringValue = store.getString("string_param", "default") val intValue = store.getInt("int_param", 0) val boolValue = store.getBoolean("bool_param", false) val doubleValue = store.getDouble("double_param", 0.0) // with StatsigUser val user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build() val storeWithUser = statsig.getParameterStore(user, "my_param_store") ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```java Java theme={null} import java.util.HashMap; import java.util.Map; String userID = "user_id"; String eventName = "my_custom_event"; // Simple event statsig.logEvent(userID, eventName); // Event with value statsig.logEvent(userID, eventName, 10.5); // Event with metadata Map metadata = new HashMap<>(); metadata.put("key1", "value1"); metadata.put("key2", "value2"); statsig.logEvent(userID, eventName, metadata); // Event with value and metadata statsig.logEvent(userID, eventName, 10.5, metadata); // with StatsigUser StatsigUser user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build(); statsig.logEvent(user, eventName, 10.5, metadata); ``` ```kotlin Kotlin theme={null} val userID = "user_id" val eventName = "my_custom_event" // Simple event statsig.logEvent(userID, eventName) // Event with value statsig.logEvent(userID, eventName, 10.5) // Event with metadata val metadata = mapOf( "key1" to "value1", "key2" to "value2" ) statsig.logEvent(userID, eventName, metadata) // Event with value and metadata statsig.logEvent(userID, eventName, 10.5, metadata) // with StatsigUser val user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build() statsig.logEvent(user, eventName, 10.5, metadata) ``` ### Sending Events to Log Explorer You can forward logs to Logs Explorer for convenient analysis using the Forward Log Line Event API. This lets you include custom metadata and event values with each log. ```java Java theme={null} import java.util.HashMap; import java.util.Map; String userID = "user_id"; Map payload = new HashMap<>(); payload.put("log_level", "error"); payload.put("message", "Something went wrong"); payload.put("timestamp", System.currentTimeMillis()); statsig.forwardLogLineEvent(userID, payload); // with StatsigUser StatsigUser user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build(); statsig.forwardLogLineEvent(user, payload); ``` ```kotlin Kotlin theme={null} val userID = "user_id" val payload = mapOf( "log_level" to "error", "message" to "Something went wrong", "timestamp" to System.currentTimeMillis() ) statsig.forwardLogLineEvent(userID, payload) // with StatsigUser val user = StatsigUser.builder() .userID("user_id") .email("my_user@example.com") .build() statsig.forwardLogLineEvent(user, payload) ``` ## Using Shared Instance In some applications, you may want to create a single Statsig instance that can be accessed globally throughout your codebase. The shared instance functionality provides a singleton pattern for this purpose: ```java Java theme={null} import com.statsig.*; // Create a shared instance that can be accessed globally StatsigServer statsig = Statsig.newShared("secret-key"); statsig.initialize().get(); // Access the shared instance from anywhere in your code StatsigServer sharedStatsig = Statsig.shared(); boolean isFeatureEnabled = sharedStatsig.checkGate(user, "feature_name"); // Check if a shared instance exists if (Statsig.hasSharedInstance()) { // Use the shared instance } // Remove the shared instance when no longer needed Statsig.removeShared(); ``` ```kotlin Kotlin theme={null} import com.statsig.* // Create a shared instance that can be accessed globally val statsig = Statsig.newShared("secret-key") statsig.initialize().get() // Access the shared instance from anywhere in your code val sharedStatsig = Statsig.shared() val isFeatureEnabled = sharedStatsig.checkGate(user, "feature_name") // Check if a shared instance exists if (Statsig.hasSharedInstance()) { // Use the shared instance } // Remove the shared instance when no longer needed Statsig.removeShared() ``` The shared instance functionality provides a singleton pattern where a single Statsig instance can be created and accessed globally throughout your application. This is useful for applications that need to access Statsig functionality from multiple parts of the codebase without having to pass around a Statsig instance. * `Statsig.newShared(sdkKey, options)`: Creates a new shared instance of Statsig that can be accessed globally * `Statsig.shared()`: Returns the shared instance * `Statsig.hasSharedInstance()`: Checks if a shared instance exists (useful when you aren't sure if the shared instance is ready yet) * `Statsig.removeShared()`: Removes the shared instance (useful when you want to switch to a new shared instance) `hasSharedInstance()` and `removeShared()` are helpful in specific scenarios but aren't required in most use cases where the shared instance is set up near the top of your application. Also note that only one shared instance can exist at a time. Attempting to create a second shared instance will result in an error. ## Manual Exposures By default, the SDK will automatically log an exposure event when you check a gate, get a config, get an experiment, or call get() on a parameter in a layer. However, there are times when you may want to log an exposure event manually. For example, if you're using a gate to control access to a feature, but you don't want to log an exposure until the user actually uses the feature, you can use manual exposures. Manual exposures allow you to control when exposure events are logged. This is useful when you want to delay exposure logging until certain conditions are met. ```java Java theme={null} String userID = "user_id"; // Check gate without logging exposure boolean result = statsig.checkGateWithExposureLoggingDisabled(userID, "my_feature_gate"); // Manually log the exposure when ready statsig.manuallyLogGateExposure(userID, "my_feature_gate"); // Works with configs too DynamicConfig config = statsig.getConfigWithExposureLoggingDisabled(userID, "my_config"); statsig.manuallyLogConfigExposure(userID, "my_config"); // And with experiments/layers Experiment experiment = statsig.getExperimentWithExposureLoggingDisabled(userID, "my_experiment"); statsig.manuallyLogExperimentExposure(userID, "my_experiment"); Layer layer = statsig.getLayerWithExposureLoggingDisabled(userID, "my_layer"); statsig.manuallyLogLayerParameterExposure(userID, "my_layer", "parameter_name"); ``` ```kotlin Kotlin theme={null} val userID = "user_id" // Check gate without logging exposure val result = statsig.checkGateWithExposureLoggingDisabled(userID, "my_feature_gate") // Manually log the exposure when ready statsig.manuallyLogGateExposure(userID, "my_feature_gate") // Works with configs too val config = statsig.getConfigWithExposureLoggingDisabled(userID, "my_config") statsig.manuallyLogConfigExposure(userID, "my_config") // And with experiments/layers val experiment = statsig.getExperimentWithExposureLoggingDisabled(userID, "my_experiment") statsig.manuallyLogExperimentExposure(userID, "my_experiment") val layer = statsig.getLayerWithExposureLoggingDisabled(userID, "my_layer") statsig.manuallyLogLayerParameterExposure(userID, "my_layer", "parameter_name") ``` ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. The `StatsigUser` object represents a user in your application and contains fields used for feature evaluation. ```java Java theme={null} import com.statsig.*; import java.util.HashMap; import java.util.Map; // Build a user with various fields Map customFields = new HashMap<>(); customFields.put("plan", "premium"); customFields.put("age", 25); Map privateAttributes = new HashMap<>(); privateAttributes.put("internal_id", "123456"); StatsigUser user = StatsigUser.builder() .userID("user_123") .email("user@example.com") .ip("192.168.1.1") .userAgent("Mozilla/5.0...") .country("US") .locale("en_US") .appVersion("1.2.3") .custom(customFields) .privateAttributes(privateAttributes) .build(); // Use the user for evaluation boolean result = statsig.checkGate(user, "my_feature_gate"); ``` ```kotlin Kotlin theme={null} import com.statsig.* // Build a user with various fields val customFields = mapOf( "plan" to "premium", "age" to 25 ) val privateAttributes = mapOf( "internal_id" to "123456" ) val user = StatsigUser.builder() .userID("user_123") .email("user@example.com") .ip("192.168.1.1") .userAgent("Mozilla/5.0...") .country("US") .locale("en_US") .appVersion("1.2.3") .custom(customFields) .privateAttributes(privateAttributes) .build() // Use the user for evaluation val result = statsig.checkGate(user, "my_feature_gate") ``` ## Private Attributes Private attributes are fields that will be used for evaluation but will not be logged in events sent to Statsig servers. ```java Java theme={null} Map privateAttributes = new HashMap<>(); privateAttributes.put("sensitive_field", "sensitive_value"); StatsigUser user = StatsigUser.builder() .userID("user_123") .privateAttributes(privateAttributes) .build(); ``` ```kotlin Kotlin theme={null} val privateAttributes = mapOf( "sensitive_field" to "sensitive_value" ) val user = StatsigUser.builder() .userID("user_123") .privateAttributes(privateAttributes) .build() ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. Environment parameter for evaluation. Custom URL for fetching feature specifications. How often the SDK updates specifications from Statsig servers (in milliseconds). Turn this on if you are proxying `download_config_specs` / `get_id_lists` and want to fall back to the default Statsig endpoint to increase reliability. Custom URL for logging events. If true, the SDK will not collect any logging within the session, including custom events and config check exposure events. Required to be `true` when using segments with more than 1000 IDs. If true, the SDK will not parse User-Agent strings into `browserName`, `browserVersion`, `systemName`, `systemVersion`, and `appVersion` when needed for evaluation. If true, the SDK will not parse IP addresses (from `user.ip`) into country codes when needed for evaluation. How often events are flushed to Statsig servers (in milliseconds). Maximum number of events to queue before forcing a flush. An adapter with custom storage behavior for config specs. Can also continuously fetch updates in place of the Statsig network. Configuration for connecting through an outbound HTTP proxy. Advanced configuration for fetching specs from multiple sources or protocols, including [Statsig Forward Proxy](/infrastructure/forward-proxy) over gRPC websocket streaming. Each `SpecAdapterConfig` can set: * `adapterType`: one of `SpecAdapterType.DATA_STORE`, `SpecAdapterType.NETWORK_HTTP`, or `SpecAdapterType.NETWORK_GRPC_WEBSOCKET` * `specsUrl`: optional endpoint override for the adapter * `initTimeoutMs`: optional initialization timeout in milliseconds * `authenticationMode`: optional transport auth mode via `AuthenticationMode.NONE`, `AuthenticationMode.TLS`, or `AuthenticationMode.MTLS` * `caCertPath`, `clientCertPath`, `clientKeyPath`, `domainName`: optional TLS and mTLS fields for gRPC websocket connections Interface to use persistent storage within the SDK. Set the logging level for the SDK. Options: `NONE`, `ERROR`, `WARN`, `INFO`, `DEBUG`. Interface to integrate observability metrics exposed by the SDK. *** ```java Java theme={null} // Example usage StatsigOptions options = new StatsigOptions.Builder() .setEnvironment("staging") .setSpecsSyncIntervalMs(10000) .setEventLoggingFlushIntervalMs(5000) .setOutputLoggerLevel(OutputLogger.LogLevel.INFO) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); ``` ```kotlin Kotlin theme={null} // Example usage val options = StatsigOptions.Builder() .setEnvironment("staging") .setSpecsSyncIntervalMs(10000) .setEventLoggingFlushIntervalMs(5000) .setOutputLoggerLevel(OutputLogger.LogLevel.INFO) .build() val statsig = Statsig("secret-key", options) statsig.initialize().get() ``` ### Proxy and Custom Network Routing Use `setProxyConfig(ProxyConfig)` if your service needs a standard outbound HTTP proxy for Statsig network calls. Use `setSpecAdapterConfigs(...)` if you are routing spec downloads through [Statsig Forward Proxy](/infrastructure/forward-proxy). ```java theme={null} ProxyConfig proxyConfig = new ProxyConfig(); proxyConfig.setProxyHost("proxy.example.com"); proxyConfig.setProxyPort(8080); proxyConfig.setProxyProtocol("https"); proxyConfig.setCaCertPath("/etc/ssl/certs/corporate-ca.pem"); // Optional StatsigOptions options = new StatsigOptions.Builder() .setProxyConfig(proxyConfig) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); ``` `ProxyConfig` supports `proxyHost`, `proxyPort`, `proxyAuth`, `proxyProtocol`, and `caCertPath`. Set `caCertPath` when your environment requires a custom PEM CA bundle for outbound TLS. ### Statsig Forward Proxy Example ```java theme={null} import java.util.Collections; SpecAdapterConfig forwardProxyConfig = new SpecAdapterConfig() .setAdapterType(SpecAdapterType.NETWORK_GRPC_WEBSOCKET) .setSpecsUrl("http://forward-proxy.internal:50051") .setAuthenticationMode(AuthenticationMode.NONE); StatsigOptions options = new StatsigOptions.Builder() .setSpecAdapterConfigs(Collections.singletonList(forwardProxyConfig)) .setFallbackToStatsigApi(true) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); ``` ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```java Java theme={null} // Shutdown flushes all pending events and stops background tasks statsig.shutdown(); // Or with a timeout (blocks until shutdown completes or timeout) statsig.shutdown().get(5, TimeUnit.SECONDS); ``` ```kotlin Kotlin theme={null} // Shutdown flushes all pending events and stops background tasks statsig.shutdown() // Or with a timeout (blocks until shutdown completes or timeout) statsig.shutdown().get(5, TimeUnit.SECONDS) ``` ## Local Overrides Local Overrides are a way to override the values of gates, configs, experiments, and layers for testing purposes. This is useful for local development or testing scenarios where you want to force a specific value without having to change the configuration in the Statsig console. Local overrides allow you to override feature gate and config values for testing purposes. ```java Java theme={null} import java.util.HashMap; import java.util.Map; // Override a gate statsig.overrideGate("my_feature_gate", true); // Override a config Map configOverride = new HashMap<>(); configOverride.put("key", "value"); configOverride.put("number", 42); statsig.overrideConfig("my_config", configOverride); // Override an experiment Map experimentOverride = new HashMap<>(); experimentOverride.put("variant", "test"); statsig.overrideExperiment("my_experiment", experimentOverride); // Override an experiment to a particular groupname statsig.overrideExperimentByGroupName("my_experiment", "a_group_name"); statsig.overrideExperimentByGroupName("my_experiment", "a_group_name", "user_123"); // Override a layer Map layerOverride = new HashMap<>(); layerOverride.put("layer_param", "override_value"); statsig.overrideLayer("my_layer", layerOverride); // Clear all overrides statsig.clearAllOverrides(); // Clear specific override statsig.clearGateOverride("my_feature_gate"); statsig.clearConfigOverride("my_config"); ``` ```kotlin Kotlin theme={null} // Override a gate statsig.overrideGate("my_feature_gate", true) // Override a config val configOverride = mapOf( "key" to "value", "number" to 42 ) statsig.overrideConfig("my_config", configOverride) // Override an experiment val experimentOverride = mapOf( "variant" to "test" ) statsig.overrideExperiment("my_experiment", experimentOverride) // Override an experiment to a particular groupname statsig.overrideExperimentByGroupName("my_experiment", "a_group_name") statsig.overrideExperimentByGroupName("my_experiment", "a_group_name", "user_123") // Override a layer val layerOverride = mapOf( "layer_param" to "override_value" ) statsig.overrideLayer("my_layer", layerOverride) // Clear all overrides statsig.clearAllOverrides() // Clear specific override statsig.clearGateOverride("my_feature_gate") statsig.clearConfigOverride("my_config") ``` ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for user-specific configurations. This enables you to persist user assignments across sessions, ensuring consistent experiment groups even when the user returns later. This is particularly useful for client-side A/B testing where you want to ensure users always see the same variant. Persistent storage allows the SDK to persist sticky experiment assignments for users, ensuring they see consistent experiment variants across sessions. ```java Java theme={null} import com.statsig.*; import java.util.HashMap; import java.util.Map; class MyPersistentStorage implements PersistentStorage { private Map> storage = new HashMap<>(); @Override public Map load(String key) { // Load persisted sticky values for the given key // Key format is "{userId}:userID" or "{customId}:{idType}" return storage.get(key); } @Override public void save(String key, String configName, StickyValues data) { // Save sticky values for a specific experiment/config storage.computeIfAbsent(key, k -> new HashMap<>()).put(configName, data); } @Override public void delete(String key, String configName) { // Delete sticky values for a specific experiment/config Map values = storage.get(key); if (values != null) { values.remove(configName); } } } // Use persistent storage StatsigOptions options = new StatsigOptions.Builder() .setPersistentStorage(new MyPersistentStorage()) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); ``` ```kotlin Kotlin theme={null} import com.statsig.* class MyPersistentStorage : PersistentStorage { private val storage = mutableMapOf>() override fun load(key: String): Map? { // Load persisted sticky values for the given key // Key format is "{userId}:userID" or "{customId}:{idType}" return storage[key] } override fun save(key: String, configName: String, data: StickyValues) { // Save sticky values for a specific experiment/config storage.getOrPut(key) { mutableMapOf() }[configName] = data } override fun delete(key: String, configName: String) { // Delete sticky values for a specific experiment/config storage[key]?.remove(configName) } } // Use persistent storage val options = StatsigOptions.Builder() .setPersistentStorage(MyPersistentStorage()) .build() val statsig = Statsig("secret-key", options) statsig.initialize().get() ``` ### Helper methods The `PersistentStorage` interface provides helper methods for working with user-based storage keys: ```java theme={null} // Get persisted values for a user using the helper method Map values = persistentStorage.getValuesForUser(user, "userID"); // Generate a storage key from a user and ID type String key = PersistentStorage.getStorageKey(user, "userID"); // Returns "{userId}:userID" String customKey = PersistentStorage.getStorageKey(user, "companyID"); // Returns "{companyId}:companyID" ``` ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. Data stores allow you to customize how the SDK fetches and caches feature specifications, enabling advanced use cases like using Redis or other distributed caches. ```java Java theme={null} import com.statsig.*; class MyDataStore implements DataStore { @Override public String getDataSync(String key) { // Synchronously fetch data for the given key // This is called during SDK evaluation return null; } @Override public CompletableFuture setData(String key, String data) { // Store data for the given key // Called when SDK receives updates from Statsig return CompletableFuture.completedFuture(null); } @Override public CompletableFuture initialize() { // Perform any initialization needed for your data store return CompletableFuture.completedFuture(null); } @Override public void shutdown() { // Clean up resources } } // Use data store StatsigOptions options = new StatsigOptions.Builder() .setDataStore(new MyDataStore()) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); ``` ```kotlin Kotlin theme={null} import com.statsig.* import java.util.concurrent.CompletableFuture class MyDataStore : DataStore { override fun getDataSync(key: String): String? { // Synchronously fetch data for the given key // This is called during SDK evaluation return null } override fun setData(key: String, data: String): CompletableFuture { // Store data for the given key // Called when SDK receives updates from Statsig return CompletableFuture.completedFuture(null) } override fun initialize(): CompletableFuture { // Perform any initialization needed for your data store return CompletableFuture.completedFuture(null) } override fun shutdown() { // Clean up resources } } // Use data store val options = StatsigOptions.Builder() .setDataStore(MyDataStore()) .build() val statsig = Statsig("secret-key", options) statsig.initialize().get() ``` ## Custom Output Logger The Output Logger interface allows you to customize how the SDK logs messages. This enables integration with your own logging system and control over log verbosity. Custom output logger allows you to redirect SDK logs to your own logging system. ```java Java theme={null} import com.statsig.*; class MyOutputLogger implements OutputLogger { @Override public void log(LogLevel level, String message) { // Route SDK logs to your logging system switch (level) { case ERROR: System.err.println("[ERROR] " + message); break; case WARN: System.out.println("[WARN] " + message); break; case INFO: System.out.println("[INFO] " + message); break; case DEBUG: System.out.println("[DEBUG] " + message); break; } } } // Use custom output logger StatsigOptions options = new StatsigOptions.Builder() .setOutputLogger(new MyOutputLogger()) .setOutputLoggerLevel(OutputLogger.LogLevel.INFO) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); ``` ```kotlin Kotlin theme={null} import com.statsig.* class MyOutputLogger : OutputLogger { override fun log(level: OutputLogger.LogLevel, message: String) { // Route SDK logs to your logging system when (level) { OutputLogger.LogLevel.ERROR -> println("[ERROR] $message") OutputLogger.LogLevel.WARN -> println("[WARN] $message") OutputLogger.LogLevel.INFO -> println("[INFO] $message") OutputLogger.LogLevel.DEBUG -> println("[DEBUG] $message") else -> {} } } } // Use custom output logger val options = StatsigOptions.Builder() .setOutputLogger(MyOutputLogger()) .setOutputLoggerLevel(OutputLogger.LogLevel.INFO) .build() val statsig = Statsig("secret-key", options) statsig.initialize().get() ``` ## Observability Client The Observability Client interface allows you to monitor the health of the SDK by integrating with your own observability systems. This enables tracking metrics, errors, and performance data. For more information on the metrics emitted by Statsig SDKs, see the [Monitoring documentation](/sdk_monitoring). Observability client allows you to monitor SDK performance and emit custom metrics. ```java Java theme={null} import com.statsig.*; class MyObservabilityClient implements ObservabilityClient { @Override public void emitMetric(String metricName, double value, Map tags) { // Send metric to your monitoring system System.out.println("Metric: " + metricName + " = " + value + ", tags: " + tags); } @Override public void startTimer(String operationName) { // Start timing an operation } @Override public void endTimer(String operationName, Map tags) { // End timing and emit duration metric } } // Use observability client StatsigOptions options = new StatsigOptions.Builder() .setObservabilityClient(new MyObservabilityClient()) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); ``` ```kotlin Kotlin theme={null} import com.statsig.* class MyObservabilityClient : ObservabilityClient { override fun emitMetric(metricName: String, value: Double, tags: Map) { // Send metric to your monitoring system println("Metric: $metricName = $value, tags: $tags") } override fun startTimer(operationName: String) { // Start timing an operation } override fun endTimer(operationName: String, tags: Map) { // End timing and emit duration metric } } // Use observability client val options = StatsigOptions.Builder() .setObservabilityClient(MyObservabilityClient()) .build() val statsig = Statsig("secret-key", options) statsig.initialize().get() ``` ## FAQ The Java Core SDK supports Java 8 and higher. Java 8 support was added in version 0.4.3. The SDK supports: * Linux (x86\_64, arm64, musl variants) * macOS (x86\_64, arm64) * Windows (x86\_64) See the Tested Platforms section for verified Docker images. The uber JAR is recommended for most use cases as it includes native libraries for all popular platforms and simplifies deployment. Use platform-specific JARs only if you need to minimize JAR size or have specific dependency requirements. For Alpine Linux (musl-based systems), install compatibility packages: ```dockerfile theme={null} RUN apk add --no-cache libgcc gcompat ``` The SDK will automatically use musl-compatible native libraries. The SDK initialization is asynchronous. Use `.get()` on the CompletableFuture to wait for initialization: ```java theme={null} Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); // Blocks until initialized ``` Always call `shutdown()` when your application terminates to flush pending events. Yes, you can create multiple Statsig instances with different configurations. You can also use the global singleton with `Statsig.getGlobalSingleton()` for convenience. Set the output logger level to DEBUG to see detailed logs: ```java theme={null} StatsigOptions options = new StatsigOptions.Builder() .setOutputLoggerLevel(OutputLogger.LogLevel.DEBUG) .build(); ``` ## Reference ### FeatureGate Class ```java theme={null} class FeatureGate { String name; // Gate name boolean value; // Evaluation boolean result String ruleID; // Rule ID for this gate EvaluationDetails evaluationDetails; // Evaluation details String rawJson; // Raw JSON string representation } ``` ### Experiment Class ```java theme={null} class Experiment { String name; // Name of the experiment String ruleID; // ID of the rule used in the experiment Map value; // Configuration values specific to the experiment String groupName; // The group name the user falls into EvaluationDetails evaluationDetails; // Details about how the experiment was evaluated String rawJson; // Raw JSON representation of the experiment } ``` **Methods for Experiment:** ```java theme={null} public String getString(String key, String fallbackValue) public boolean getBoolean(String key, Boolean fallbackValue) public double getDouble(String key, double fallbackValue) public int getInt(String key, int fallbackValue) public long getLong(String key, long fallbackValue) public Object[] getArray(String key, Object[] fallbackValue) public Map getMap(String key, Map fallbackValue) ``` ### DynamicConfig Class ```java theme={null} class DynamicConfig { String name; // Name of the config String ruleID; // ID of the rule used Map value; // Configuration values EvaluationDetails evaluationDetails; // Details about how the config was evaluated String rawJson; // Raw JSON representation } ``` **Methods for DynamicConfig:** ```java theme={null} public String getString(String key, String fallbackValue) public boolean getBoolean(String key, Boolean fallbackValue) public double getDouble(String key, double fallbackValue) public int getInt(String key, int fallbackValue) public long getLong(String key, long fallbackValue) public Object[] getArray(String key, Object[] fallbackValue) public Map getMap(String key, Map fallbackValue) ``` ### Layer Class ```java theme={null} class Layer { String name; // Layer name String ruleID; // Rule ID for this layer String groupName; // Group name Map value; // Layer values String allocatedExperimentName; // Allocated experiment name EvaluationDetails evaluationDetails; // Evaluation details String rawJson; // Raw JSON string representation } ``` **Methods for Layer:** ```java theme={null} public String getString(String key, String fallbackValue) public boolean getBoolean(String key, Boolean fallbackValue) public double getDouble(String key, double fallbackValue) public int getInt(String key, int fallbackValue) public long getLong(String key, long fallbackValue) public Object[] getArray(String key, Object[] fallbackValue) public Map getMap(String key, Map fallbackValue) ``` ### Fields Needed Methods (Enterprise Only) This is available for Enterprise contracts. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. These methods allow you to retrieve a list of user fields that are used in the targeting rules for gates, configs, experiments, and layers. ```java theme={null} // Get user fields needed for a gate evaluation List getFieldsNeededForGate(String gateName) // Get user fields needed for a dynamic config evaluation List getFieldsNeededForDynamicConfig(String configName) // Get user fields needed for an experiment evaluation List getFieldsNeededForExperiment(String experimentName) // Get user fields needed for a layer evaluation List getFieldsNeededForLayer(String layerName) ``` **Field Mapping** The fields returned by these methods correspond to the following user properties: ```java theme={null} // Field mapping between user properties and internal field names userID -> "u" email -> "e" ip -> "i" userAgent -> "ua" country -> "c" locale -> "l" appVersion -> "a" time -> "t" stableID -> "s" environment -> "en" targetApp -> "ta" // Custom fields are prefixed with "cf:" // Example: "cf:plan", "cf:age" ``` # Legacy Server SDKs Source: https://docs.statsig.com/server-core/legacy-sdks Reference for Statsig's long-lived legacy Server SDKs that are transitioning to the Server Core framework, including supported versions and migration timelines. Statsig is in the process of transitioning to the new Statsig Server Core architecture, which unifies all Server SDKs around a unified, performance optimized core package. We expect this model will support both superior performance and greater feature consistency across SDKs. However, the vast majority of customers continue to use Statsig's legacy SDKs, and we'll continue to support them for the foreseeable future, with the exception of major feature additions. ## Timelines We've begun announcing end-of-life for some Legacy Server SDKs. We encourage you to explore using the [Server Core SDKs](server-core/index), which often evaluate 5-10x faster than the legacy SDKs, are more feature complete, and have other performance advantages. See docs for our existing SDKs below: * [Node](/server/nodejsServerSDK) * [Python](/server/pythonSDK) * [Elixir/Erlang](/server/erlangSDK) (End of Life April 30th, 2026) * [Java](/server/javaSdk) * [Rust](/server/rustSDK) (End of Life April 30th, 2026) * [Go](/server/go) * [PHP](/server/php) * [.NET](/server/dotnet) # Node Server SDK Source: https://docs.statsig.com/server-core/node-core Statsig's next-generation Node.js Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for Node backends. Node Core on Github, NPM Package Migrating from the Legacy Node SDK? See our [Migration Guide](/server-core/migration-guides/node). ## Setup the SDK ```shell theme={null} npm i @statsig/statsig-node-core ``` The Node SDK is pre-built and compiled for different OSs & CPU architectures. Package managers will resolve the correct version automatically. If your service has locked dependencies with package-lock.json or pnpm-lock.yml, you'll need to include all versions you need. For example, if you develop locally on macOS, and deploy to linux, then you have to include: ``` dependencies { "@statsig/statsig-node-core-darwin-arm64": "0.1.0" // for macOS "@statsig/statsig-node-core-linux-x64-gnu": "0.1.0" // for linux x64 machines } ``` `statsig-node-core` uses native binary files that can't be packaged with webpack/esbuild. To prevent errors, take the following steps: In your `next.config.js` file, add the `@statsig/statsig-node-core` package to the `serverExternalPackages` array: ```jsx theme={null} const nextConfig = { serverExternalPackages: ['@statsig/statsig-node-core'], } ``` Add the `--packages=external` flag to your build script: ```shell theme={null} esbuild --packages=external ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. ```jsx theme={null} // Basic initialization import { Statsig, StatsigUser } from '@statsig/statsig-node-core'; //Or, in common JS, const { Statsig, StatsigUser } = require('@statsig/statsig-node-core'); const statsig = new Statsig("secret-key"); await statsig.initialize(); // or with StatsigOptions const options: StatsigOptions = { environment: "staging" }; const statsigWithOptions = new Statsig("secret-key", options); await statsigWithOptions.initialize(); ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```jsx theme={null} const user = new StatsigUser({ userID: "a-user" }); if (statsig.checkGate(user, "a_gate")) { // Gate is on, enable new feature } else { // Gate is off } ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```jsx theme={null} // Get the dynamic config const config = statsig.getDynamicConfig(user, "a_config"); // Get typed values using the getValue() method const itemName = config.getValue("product_name", "Awesome Product v1"); const price = config.getValue("price", 10.0); const shouldDiscount = config.getValue("discount", false); // Or access the entire value object directly const value = config.value; ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```jsx theme={null} // Or, via individual experiments const titleExp = statsig.getExperiment(user, "new_user_promo_title"); const priceExp = statsig.getExperiment(user, "new_user_promo_price"); const experimentTitle = titleExp.getValue("title", "Welcome to Statsig!"); const experimentDiscount = priceExp.getValue("discount", 0.1); // Get values via Layer const layer = statsig.getLayer(user, "user_promo_experiments"); const title = layer.getValue("title", "Welcome to Statsig!"); const discount = layer.getValue("discount", 0.1); ``` ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```jsx theme={null} const gate = statsig.getFeatureGate(statsigUser, "example_gate") console.log(gate.rule_id) console.log(gate.value) ``` ### Parameter Stores Sometimes you don't know whether you want a value to be a Feature Gate, Experiment, or Dynamic Config yet. If you want on-the-fly control of that outside of your deployment cycle, you can use Parameter Stores to define a parameter that can be changed into at any point in the Statsig console. Parameter Stores are optional, but parameterizing your application can prove very useful for future flexibility and can even allow non-technical Statsig users to turn parameters into experiments. ```jsx theme={null} const paramStore = statsig.getParameterStore(statsigUser, "my_parameters") const paramStoreValue = paramStore.getValue('my_parameter_value') ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```jsx theme={null} statsig.logEvent( user, "add_to_cart", null, { price: "9.99", item_name: "diet_coke_48_pack" } ); ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ### Sending Events to Log Explorer You can forward logs to Logs Explorer for convenient analysis using the Forward Log Line Event API. This lets you include custom metadata and event values with each log. ```jsx theme={null} const user = new StatsigUser({ userID: "a-user", custom: { service: "my-service", pod: "my-pod", namespace: "my-namespace", container: "my-container", // ...include any service-specific metadata } }); // levels: trace, debug, info, log, warn, error statsig.forwardLogLineEvent(user, "warn", "script failed to load", { cusom_metadata: "script_name:my-script" // ... include any event-specific metadata }); ``` ## Using Shared Instance In some applications, you may want to create a single Statsig instance that can be accessed globally throughout your codebase. The shared instance functionality provides a singleton pattern for this purpose: ```jsx theme={null} // Create a shared instance that can be accessed globally const statsig = Statsig.newShared("secret-key"); await statsig.initialize(); // Access the shared instance from anywhere in your code const sharedStatsig = Statsig.shared(); const isFeatureEnabled = sharedStatsig.checkGate(user, "feature_name"); // Check if a shared instance exists if (Statsig.hasSharedInstance()) { // Use the shared instance } // Remove the shared instance when no longer needed Statsig.removeShared(); ``` The shared instance functionality provides a singleton pattern where a single Statsig instance can be created and accessed globally throughout your application. This is useful for applications that need to access Statsig functionality from multiple parts of the codebase without having to pass around a Statsig instance. * `Statsig.newShared(sdkKey, options)`: Creates a new shared instance of Statsig that can be accessed globally * `Statsig.shared()`: Returns the shared instance * `Statsig.hasSharedInstance()`: Checks if a shared instance exists (useful when you aren't sure if the shared instance is ready yet) * `Statsig.removeShared()`: Removes the shared instance (useful when you want to switch to a new shared instance) `hasSharedInstance()` and `removeShared()` are helpful in specific scenarios but aren't required in most use cases where the shared instance is set up near the top of your application. Also note that only one shared instance can exist at a time. Attempting to create a second shared instance will result in an error. ## Manual Exposures By default, the SDK will automatically log an exposure event when you check a gate, get a config, get an experiment, or call get() on a parameter in a layer. However, there are times when you may want to log an exposure event manually. For example, if you're using a gate to control access to a feature, but you don't want to log an exposure until the user actually uses the feature, you can use manual exposures. All of the main SDK functions (`checkGate`, `getDynamicConfig`, `getExperiment`, `getLayer`) accept an optional `disableExposureLogging` parameter. When this is set to `true`, the SDK will not automatically log an exposure event. You can then manually log the exposure at a later time using the corresponding manual exposure logging method: ```jsx theme={null} const result = statsig.checkGate(aUser, 'a_gate_name', {disableExposureLogging: true}); ``` ```jsx theme={null} statsig.manuallyLogGateExposure(aUser, 'a_gate_name'); ``` ```jsx theme={null} const config = statsig.getDynamicConfig(aUser, 'a_dynamic_config_name', {disableExposureLogging: true}); ``` ```jsx theme={null} statsig.manuallyLogDynamicConfigExposure(aUser, 'a_dynamic_config_name'); ``` ```jsx theme={null} const experiment = statsig.getExperiment(aUser, 'an_experiment_name', {disableExposureLogging: true}); ``` ```jsx theme={null} statsig.manuallyLogExperimentExposure(aUser, 'an_experiment_name'); ``` ```jsx theme={null} const layer = statsig.getLayer(aUser, 'a_layer_name', {disableExposureLogging: true}); const paramValue = layer.get('a_param_name', 'fallback_value'); ``` ```jsx theme={null} statsig.manuallyLogLayerParameterExposure(aUser, 'a_layer_name', 'a_param_name'); ``` ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. ```typescript theme={null} const user = new StatsigUser({ userID: "a-user-id", email: "user@example.com", ip: "192.168.1.1", userAgent: "Mozilla/5.0...", country: "US", locale: "en_US", appVersion: "1.0.0", custom: { // Custom fields plan: "premium", age: 25 }, customIDs: { // Custom ID types stableID: "stable-id-123" }, privateAttributes: { // Private attributes not forwarded to integrations email: "private@example.com" } }); ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. ### Parameters Environment parameter for evaluation. Custom URL for fetching feature specifications. How often the SDK updates specifications from Statsig servers (in milliseconds). Turn this on if you are proxying `download_config_specs` / `get_id_lists` and want to fall back to the default Statsig endpoint to increase reliability. Custom URL for logging events. If true, the SDK will not collect any logging within the session, including custom events and config check exposure events. Required to be `true` when using segments with more than 1000 IDs. See [ID List segments](/segments/add-id-list). If true, the SDK will not parse User-Agent strings into `browserName`, `browserVersion`, `systemName`, `systemVersion`, and `appVersion` when needed for evaluation. When true, the SDK waits until user agent parsing data is fully loaded during initialization (\~1 second), ensuring parsing is ready before any evaluations. If true, the SDK will not parse IP addresses (from `user.ip`) into country codes when needed for evaluation. When true, the SDK waits for country lookup data (e.g., GeoIP or YAML files) to fully load during initialization (\~1 second), ensuring IP-to-country parsing is ready at evaluation time. How often events are flushed to Statsig servers (in milliseconds). Maximum number of events to queue before forcing a flush. An adapter with custom storage behavior for config specs. Can also continuously fetch updates in place of the Statsig network. See [Data Stores](#data-store). For example, see our 1P Redis implementation [statsig-node-redis](https://github.com/statsig-io/node-js-server-sdk-redis). Advanced settings to fetch from different sources (e.g., [statsig forward proxy](/infrastructure/forward-proxy), your own proxy server, data store) or to use different network protocols (HTTP vs gRPC streaming). Interface to integrate observability metrics exposed by the SDK (e.g., config propagation delay, initialization time). See [details](#observability-client). Interface to use persistent storage within the SDK. See [details](#persistent-storage). Configuration for connecting through a proxy server. Proxy server host. Proxy server port. Proxy authentication in the form "username:password". Protocol (e.g., "http", "https"). Optional path to a PEM CA bundle for outbound TLS. ### Proxy and Custom Network Routing Use `proxyConfig` if your service needs a standard outbound HTTP proxy. Use `specAdaptersConfig` if you are routing spec downloads through [Statsig Forward Proxy](/infrastructure/forward-proxy) or another custom spec source. ```typescript theme={null} // Example usage: const options = new StatsigOptions(); options.environment = "staging"; options.initTimeoutMs = 3000; options.proxyConfig = { proxyHost: "proxy.example.com", proxyPort: 8080, // proxyAuth can be set if authentication is required proxyProtocol: "https", caCertPath: "/etc/ssl/certs/corporate-ca.pem" }; const statsig = new Statsig("secret-key", options); await statsig.initialize(); ``` Set `caCertPath` when your environment requires a custom PEM CA bundle for outbound TLS. ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```jsx theme={null} await statsig.shutdown(); ``` ## Client SDK Bootstrapping | SSR If you are using the Statsig client SDK in a browser or mobile app, you can bootstrap the client SDK with the values from the server SDK to avoid a network request on the client. This is useful for server-side rendering (SSR) or when you want to reduce the number of network requests on the client. ## Client Initialize Response The Node Core SDK provides a method to generate a client initialize response that can be used to bootstrap client SDKs without requiring network requests. ```typescript theme={null} // Get client initialize response for a user const values = statsig.getClientInitializeResponse(user, options); // Pass values to a client SDK to initialize without a network request ``` The `getClientInitializeResponse` method accepts an optional `options` parameter with the following properties: ```typescript theme={null} export interface ClientInitializeResponseOptions { hashAlgorithm?: string; // Algorithm used for hashing gate/experiment names (default: 'djb2') clientSdkKey?: string; // Client SDK key to use for initialization includeLocalOverrides?: boolean; // Whether to include local overrides in the response featureGateFilter?: Set; // Filter to only include specific feature gates experimentFilter?: Set; // Filter to only include specific experiments dynamicConfigFilter?: Set; // Filter to only include specific dynamic configs layerFilter?: Set; // Filter to only include specific layers paramStoreFilter?: Set; // Filter to only include specific parameter stores } ``` The `hashAlgorithm` option specifies which algorithm to use for hashing gate and experiment names in the client initialize response. The default is `'djb2'` for better performance and smaller payload size. ```typescript theme={null} // Use djb2 hashing algorithm for better performance const values = statsig.getClientInitializeResponse(user, { hashAlgorithm: 'djb2', }); ``` The `clientSdkKey` option lets you filter the response to only the specific feature gates, experiments, dynamic configs, layers, or parameter stores that a particular client key has access to - effectively letting you apply [target apps](/sdk-keys/target-apps/). ```typescript theme={null} // Specify a client SDK key const values = statsig.getClientInitializeResponse(user, { clientSdkKey: 'client-key', }); ``` The filter options allow you to reduce the payload size by only including specific feature gates, experiments, dynamic configs, layers, or parameter stores in the response. ```typescript theme={null} // Only include specific feature gates and experiments const values = statsig.getClientInitializeResponse(user, { featureGateFilter: new Set(['my_gate_1', 'my_gate_2']), experimentFilter: new Set(['my_experiment']), }); ``` The `includeLocalOverrides` option determines whether to consider [local overrides](#local-overrides) you've set when evaluating each config in the response. ```typescript theme={null} // Include local overrides in the response const values = statsig.getClientInitializeResponse(user, { includeLocalOverrides: true, }); ``` Below is a complete example of using the client initialize response to bootstrap a client SDK. Note that you may choose to parallelize or inline the initialize response data with other requests to your server, to eliminate additional requests and latency. ```typescript theme={null} // Server-side code import { Statsig, StatsigUser } from '@statsig/node-core'; // Initialize the server SDK await Statsig.initialize('server-secret-key'); // In your API endpoint handler app.get('/statsig-bootstrap', (req, res) => { // Create a user object from the request const user = new StatsigUser({ userID: req.query.userID || '', email: req.query.email, ip: req.ip, userAgent: req.headers['user-agent'], }); // Generate the client initialize response with filters const values = Statsig.getClientInitializeResponse(user, { hashAlgorithm: 'djb2', featureGateFilter: new Set(['onboarding_v2', 'new_checkout']), experimentFilter: new Set(['pricing_experiment']), layerFilter: new Set(['ui_layer']), }); // Return the values to the client res.json({ statsigValues: values }); }); ``` ```typescript theme={null} // Client-side code using @statsig/js-client import { Statsig } from '@statsig/js-client'; // Fetch bootstrap values from your API const response = await fetch('/statsig-bootstrap'); const { statsigValues } = await response.json(); // Initialize the client SDK with the bootstrap values await Statsig.initialize({ sdkKey: 'client-sdk-key', initializeValues: statsigValues, }); ``` ## SDK Event Subscriptions The Statsig SDK provides an event subscription system that allows you to listen for evaluation events and lifecycle events in real-time. This feature is useful for debugging, analytics, custom logging, and integrating with external systems. ### Supported Events The SDK supports subscribing to the following evaluation events: * **`gate_evaluated`** - Fired when a feature gate is evaluated for a user * **`dynamic_config_evaluated`** - Fired when a dynamic config is retrieved for a user * **`experiment_evaluated`** - Fired when an experiment is evaluated for a user * **`layer_evaluated`** - Fired when a layer is evaluated for a user * **`specs_updated`** - Fired when the SDK updates its cached specs, including where the specs were loaded from * **`"*"`** - Subscribe to all evaluation events ### SDK Event Data Each event includes relevant context about the evaluation: * **Gate Evaluated Events** include: `gate_name`, `value` (boolean), `rule_id`, `reason` * **Dynamic Config Events** include: the full `dynamic_config` object with values and metadata * **Experiment Events** include: the full `experiment` object with variant assignment and parameters * **Layer Events** include: the full `layer` object with allocated experiment and parameters * **Specs Updated Events** include `source`, `source_api`, and `values` metadata, where `values.time` is the timestamp of the last update to the project in the Statsig console ### Use Cases Event subscriptions are particularly useful for: * **Debugging**: Monitor which features are being evaluated and their results * **Analytics**: Track feature usage patterns and user segments * **Custom Logging**: Send evaluation data to your own logging systems * **Integration**: Forward events to external analytics or monitoring tools * **Testing**: Verify that features are being evaluated as expected ### Best Practices * **Clean up subscriptions**: Always unsubscribe when you no longer need to listen for events to prevent memory leaks * **Handle event data carefully**: Event objects may contain sensitive user information depending on your configuration * **Use specific event types**: Subscribe to specific events rather than "\*" when possible for better performance * **Avoid heavy processing**: Keep event handlers lightweight to avoid impacting SDK performance ```javascript expandable theme={null} const statsig = new Statsig('server-secret-key'); // Subscribe to gate evaluation events const gateSubId = statsig.subscribe('gate_evaluated', (event) => { console.log('Gate evaluated:', { gateName: event.gate_name, value: event.value, ruleId: event.rule_id, reason: event.reason }); }); // Subscribe to dynamic config evaluation events const configSubId = statsig.subscribe('dynamic_config_evaluated', (event) => { console.log('Config evaluated:', { configName: event.dynamic_config.name, values: event.dynamic_config.value }); }); // Subscribe to experiment evaluation events const experimentSubId = statsig.subscribe('experiment_evaluated', (event) => { console.log('Experiment evaluated:', { experimentName: event.experiment.name, groupName: event.experiment.group_name, parameters: event.experiment.value }); }); // Subscribe to layer evaluation events const layerSubId = statsig.subscribe('layer_evaluated', (event) => { console.log('Layer evaluated:', { layerName: event.layer.name, allocatedExperiment: event.layer.allocated_experiment_name, parameters: event.layer.value }); }); // Subscribe to specs updated events const specsUpdatedSubId = statsig.subscribe('specs_updated', (event) => { console.log('Specs updated:', { source: event.data.source, sourceApi: event.data.source_api, projectLastUpdated: event.data.values.time, }); }); // Subscribe to all events const allEventsSubId = statsig.subscribe('*', (event) => { console.log('Event received:', event.event_name, event); }); // Unsubscribe from specific event types statsig.unsubscribe('gate_evaluated'); statsig.unsubscribe('specs_updated'); // Unsubscribe using subscription ID statsig.unsubscribeById(configSubId); // Unsubscribe from all events statsig.unsubscribeAll(); ``` ## Local Overrides Local Overrides are a way to override the values of gates, configs, experiments, and layers for testing purposes. This is useful for local development or testing scenarios where you want to force a specific value without having to change the configuration in the Statsig console. ```jsx theme={null} // Overrides the given gate to the specified value Statsig.overrideGate("a_gate_name", true); // Optional third parameter, overrides the gate only for a given ID Statsig.overrideGate("a_gate_name", true, "userID-123"); ``` ```jsx theme={null} // Overrides the given dynamic config to the provided value Statsig.overrideDynamicConfig("a_config_name", { key: "value" }); // Optional third parameter, overrides the dynamic config only for a given ID Statsig.overrideDynamicConfig("a_config_name", { key: "value" }, "userID-123"); ``` ```jsx theme={null} // Overrides the given experiment to the provided value Statsig.overrideExperiment("an_experiment_name", { key: "value" }); // Optional third parameter, overrides the experiment only for a given ID Statsig.overrideExperiment("an_experiment_name", { key: "value" }, "userID-123"); // Overrides the given experiment to a particular groupname Statsig.overrideExperimentByGroupName("an_experiment_name", "a_group_name"); // Alternatively, get the Experiment object for a given groupName const groupExp = statsig.getExperimentByGroupName("pricing_experiment", "premium_group"); const premiumPrice = groupExp.getValue("price", 9.99); ``` ```jsx theme={null} // Overrides the given layer to the provided value Statsig.overrideLayer("a_layer_name", { key: "value" }); // Optional third parameter, overrides the layer only for a given ID Statsig.overrideLayer("a_layer_name", { key: "value" }, "userID-123"); ``` ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for user-specific configurations. This enables you to persist user assignments across sessions, ensuring consistent experiment groups even when the user returns later. This is particularly useful for client-side A/B testing where you want to ensure users always see the same variant. ```typescript expandable PersistentStorageInterface.ts theme={null} export interface PersistentStorage { load: (key: string) => UserPersistedValues | null; save: (key: string, config_name: string, data: StickyValues) => void; delete: (key: string, config_name: string) => void; } export interface StickyValues { value: boolean; json_value: Record; rule_id: string; group_name: string | null; secondary_exposures: SecondaryExposure[]; undelegated_secondary_exposures: SecondaryExposure[]; config_delegate: string | null; explicit_parameters: string[] | null; time: number; configVersion?: number | undefined; } export type UserPersistedValues = Record; export interface SecondaryExposure { gate: string; gateValue: string; ruleId: string; } ``` ### Usage Example ```typescript expandable PersistentStorageUsage.ts theme={null} import { PersistentStorage, StickyValues, UserPersistedValues } from '@statsig/statsig-node-core'; class MyPersistentStorage implements PersistentStorage { private storage = new Map(); constructor() { this.load = this.load.bind(this); this.save = this.save.bind(this); this.delete = this.delete.bind(this); } load(key: string): UserPersistedValues | null { return this.storage.get(key) || null; } save(key: string, config_name: string, data: StickyValues): void { const existing = this.storage.get(key) || {}; existing[config_name] = data; this.storage.set(key, existing); } delete(key: string, config_name: string): void { const existing = this.storage.get(key); if (existing) { delete existing[config_name]; this.storage.set(key, existing); } } getUserPersistedValue(user: StatsigUser, idType: string): UserPersistedValues | null { const storageKey = this.getStorageKey(user, idType); if (storageKey !== null) { return this.load(storageKey); } return null; } private getStorageKey(user: StatsigUser, idType: string): string | null { const lowerCaseIdType = idType.toLowerCase(); if (lowerCaseIdType === "user_id" || lowerCaseIdType === "userid") { const id = user.userID; return id ? `${id}:userID` : null; } else if (user.customIDs) { const id = user.customIDs[idType]; return id ? `${id}:${idType}` : null; } return null; } } ``` Persistent storage support was added in version 0.6.1 of the Node.js SDK. ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. ```typescript theme={null} export interface DataStore { initialize?: () => Promise; shutdown?: () => Promise; get?: (key: string) => Promise; set?: (key: string, value: string, time?: number) => Promise; supportPollingUpdatesFor?: (key: string) => Promise; } export interface DataStoreResponse { result?: string; time?: number; } ``` For example, see our 1P implementation via Redis [statsig-node-redis](https://github.com/statsig-io/node-js-server-sdk-redis). ## Custom Output Logger The Output Logger interface allows you to customize how the SDK logs messages. This enables integration with your own logging system and control over log verbosity. ```typescript theme={null} interface OutputLoggerProvider { initialize?: () => void; debug?: (tag: string, message: string) => void; info?: (tag: string, message: string) => void; warn?: (tag: string, message: string) => void; error?: (tag: string, message: string) => void; shutdown?: () => void; } ``` ### Implementation Example ```typescript theme={null} import { OutputLoggerProvider } from '@statsig/statsig-node-core'; class CustomOutputLogger implements OutputLoggerProvider { initialize = () => { console.log('Logger initialized'); }; debug = (tag: string, message: string) => { console.debug(`[${tag}] ${message}`); }; info = (tag: string, message: string) => { console.info(`[${tag}] ${message}`); }; warn = (tag: string, message: string) => { console.warn(`[${tag}] ${message}`); }; error = (tag: string, message: string) => { console.error(`[${tag}] ${message}`); }; shutdown = () => { console.log('Logger shutdown'); }; } ``` ```typescript theme={null} import { Statsig, StatsigOptions } from '@statsig/statsig-node-core'; const customLogger = new CustomOutputLogger(); const options: StatsigOptions = { outputLoggerProvider: customLogger, outputLogLevel: 'info', // 'none' | 'debug' | 'info' | 'warn' | 'error' }; const statsig = new Statsig('secret-key', options); await statsig.initialize(); ``` * All methods in the `OutputLoggerProvider` interface are optional * The `tag` parameter indicates the SDK component or category generating the log message * Use `outputLogLevel` in StatsigOptions to control which log levels are actually called * The logger is automatically initialized when the Statsig client initializes and shut down when the client shuts down ## Observability Client The Observability Client interface allows you to monitor the health of the SDK by integrating with your own observability systems. This enables tracking metrics, errors, and performance data. For more information on the metrics emitted by Statsig SDKs, see the [Monitoring documentation](/sdk_monitoring). ```typescript theme={null} export interface ObservabilityClient { initialize?: () => void; increment?: (metricName: string, value: number, tags: Record) => void; gauge?: (metricName: string, value: number, tags: Record) => void; dist?: (metricName: string, value: number, tags: Record) => void; error?: (tag: string, error: string) => void; } ``` ## FAQs ### How do I run experiments for logged out users? ### Common Problems while installing 1. **Seeing SSL Error** Right now the binary files will look at certain versions of SSL. ```shell theme={null} // Try run this apt-get update && apt-get install libcurl4-openssl-dev -y && rm -rf /var/lib/apt/lists/* ``` 2. **Docker build failing with platform-specific dependencies** When building in Docker (Linux environment), the build may fail if your local `package-lock.json` or `yarn.lock` contains platform-specific dependencies for macOS. This happens because `npm install` locally on Mac pulls down Apple-specific variants, but Docker tries to use those same locked dependencies on Linux. **Solution:** Either install the Linux-specific variant during your Docker build step: ```dockerfile theme={null} RUN npm install @statsig/statsig-node-core-linux-x64-gnu ``` Or add both platform variants as dependencies in your `package.json`: ```json theme={null} "dependencies": { "@statsig/statsig-node-core": "X.Y.Z", // Common (Required) "@statsig/statsig-node-core-darwin-arm64": "X.Y.Z", // Mac Specific "@statsig/statsig-node-core-linux-x64-gnu": "X.Y.Z" // Linux Specific } ``` ## Reference * `checkGate(user: StatsigUser, gateName: string, options?: EvaluationOptions): boolean` * `getDynamicConfig(user: StatsigUser, configName: string, options?: EvaluationOptions): DynamicConfig` * `getExperiment(user: StatsigUser, experimentName: string, options?: EvaluationOptions): DynamicConfig` * `getLayer(user: StatsigUser, layerName: string, options?: EvaluationOptions): Layer` * `getFeatureGate(user: StatsigUser, gateName: string, options?: EvaluationOptions): FeatureGate` * `getParameterStore(user: StatsigUser, parameterStoreName: string, options?: EvaluationOptions): ParameterStore` * `getPrompt(user: StatsigUser, promptName: string, options?: EvaluationOptions): Prompt` * `getPromptSet(user: StatsigUser, promptSetName: string, options?: EvaluationOptions): PromptSet` * `logEvent(user: StatsigUser, eventName: string, value?: string | number | null, metadata?: Record): void` * `forwardLogLineEvent(user: StatsigUser, level: string, message: string, metadata?: Record): void` * `manuallyLogGateExposure(user: StatsigUser, gateName: string): void` * `manuallyLogDynamicConfigExposure(user: StatsigUser, configName: string): void` * `manuallyLogExperimentExposure(user: StatsigUser, experimentName: string): void` * `manuallyLogLayerParameterExposure(user: StatsigUser, layerName: string, parameterName: string): void` * `overrideExperimentByGroupName(experimentName: string, groupName: string, id?: string | null): void` * `getClientInitializeResponse(user: StatsigUser, options?: ClientInitializeResponseOptions): ClientInitializeResponse` * `shutdown(): Promise` The following methods return information about which user fields are needed for evaluation: * `getGateFieldsNeeded(gateName: string): string[]` * `getDynamicConfigFieldsNeeded(configName: string): string[]` * `getExperimentFieldsNeeded(experimentName: string): string[]` * `getLayerFieldsNeeded(layerName: string): string[]` These methods return an array of strings representing the user fields that are required to properly evaluate the specified gate, config, experiment, or layer. This can be useful for: * Optimizing user object creation by only including necessary fields * Understanding which user attributes affect a particular feature * Debugging evaluation issues # PHP Server SDK Source: https://docs.statsig.com/server-core/php-core Statsig's next-generation PHP Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for PHP applications. PHP Core on Github, Packagist Package Migrating from the Legacy PHP SDK? See our [Migration Guide](/server-core/migration-guides/php). ## Setup the SDK ## Installation ### 1. Install and Add as a Dependency You can install the new PHP Core SDK using composer: ```shell theme={null} composer require statsig/statsig-php-core ``` ### 2. PHP Configuration Requirements The PHP Core SDK uses the FFI extension to interface with the Rust core. You must enable FFI in your PHP configuration by setting `ffi.enable=true` in your php.ini file. For more information about this setting, see the [PHP manual for ffi.enable](https://www.php.net/manual/en/ffi.configuration.php#ini.ffi.enable). ### 3. Add Scripts & Cron Job Add post-install and post-update scripts in composer.json: ```json theme={null} // composer.json { "name": "awesome-php-project", ... "scripts": { ... "post-install-cmd": [ "cd vendor/statsig/statsig-php-core && php post-install.php" ], "post-update-cmd": [ "cd vendor/statsig/statsig-php-core && php post-install.php" ] } } ``` Next, you'll want to add a script to sync your Statsig configs and flush your events, see example files on Statsig's Github [here](https://github.com/daniel-statsig/statsig-php-core-slim-example/tree/main/bin). You'll also want to setup cron jobs to run these scripts periodically: ```shell theme={null} */10 * * * * /usr/bin/php /var/www/example.com/bin/StatsigSyncConfig.php 1>> /dev/null 2>&1 */1 * * * * /usr/bin/php /var/www/example.com/bin/StatsigFlushEvents.php 1>> /dev/null 2>&1 ``` Also, be sure to run the StatsigSyncConfig.php cron job at least once before proceeding. After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. You'll want to add your client secret key to the environment, by adding to a .env file, or directly on the command line: ```shell theme={null} export STATSIG_SECRET_KEY=secret-123456789 ``` ```php theme={null} // At the top of your file use Statsig\Statsig; use Statsig\StatsigOptions; use Statsig\StatsigLocalFileEventLoggingAdapter; use Statsig\StatsigLocalFileSpecsAdapter; //In the case of slim framework, in container builder definitions: Statsig::class => function (ContainerInterface $c) { $sdk_key = getenv("STATSIG_SECRET_KEY"); $options = new StatsigOptions( null, null, new StatsigLocalFileSpecsAdapter($sdk_key, "/tmp"), new StatsigLocalFileEventLoggingAdapter($sdk_key, "/tmp") ); $statsig = new Statsig($sdk_key, $options); $statsig->initialize(); return $statsig; }, ``` `StatsigLocalFile` Adapters rely on cron jobs and files. If you are seeing errors around file access, ensure your cron job has run at least one time before using Statsig. See [Add Scripts & Cron Job](#2-add-scripts--cron-job) `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```php theme={null} use Statsig\Statsig; use Statsig\StatsigUserBuilder; $user = StatsigUserBuilder::withUserID('my_user')->build(); $passed = $statsig->checkGate($user, 'my_gate'); ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```php theme={null} $user = StatsigUserBuilder::withUserID('my_user')->build(); $config = $statsig->getDynamicConfig($user, 'my_config'); ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```php theme={null} $user = StatsigUserBuilder::withUserID('my_user')->build(); $xp = $statsig->getExperiment($user, 'an_experiment'); ``` ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```php theme={null} $gate = $statsig->getFeatureGate($user, "example_gate"); ``` ### Parameter Stores Sometimes you don't know whether you want a value to be a Feature Gate, Experiment, or Dynamic Config yet. If you want on-the-fly control of that outside of your deployment cycle, you can use Parameter Stores to define a parameter that can be changed into at any point in the Statsig console. Parameter Stores are optional, but parameterizing your application can prove very useful for future flexibility and can even allow non-technical Statsig users to turn parameters into experiments. *Parameter stores are not yet available for this sdk. Need it now? Let us know in [Slack](https://statsig.com/slack).* ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```php theme={null} $user = StatsigUserBuilder::withUserID('my_user')->build(); $statsig->logEvent($user, 'an_experiment'); ``` ### Sending Events to Log Explorer You can forward logs to Logs Explorer for convenient analysis using the Forward Log Line Event API. This lets you include custom metadata and event values with each log. Sending events to Log Explorer is not yet available for this sdk. Need it now? Let us know in [Slack](https://statsig.com/slack). ## Using Shared Instance In some applications, you may want to create a single Statsig instance that can be accessed globally throughout your codebase. The shared instance functionality provides a singleton pattern for this purpose: ```php theme={null} use Statsig\Statsig; // Initialize the shared instance Statsig::initializeShared('your-server-secret-key', $options); // Access the shared instance from anywhere in your code $user = StatsigUserBuilder::withUserID('my_user')->build(); $gate = Statsig::shared()->checkGate($user, 'my_gate'); // Shutdown the shared instance when your application closes Statsig::shared()->shutdown(); ``` ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. ```php theme={null} use Statsig\StatsigUser; $user = new StatsigUser([ 'userID' => 'a-user-id', 'email' => 'user@example.com', 'ip' => '192.168.1.1', 'userAgent' => 'Mozilla/5.0...', 'country' => 'US', 'locale' => 'en_US', 'appVersion' => '1.0.0', 'custom' => [ // Custom fields 'plan' => 'premium', 'age' => 25 ], 'customIDs' => [ // Custom ID types 'stableID' => 'stable-id-123' ], 'privateAttributes' => [ // Private attributes not forwarded to integrations 'email' => 'private@example.com' ] ]); ``` ## Private Attributes You can define which attributes are considered "private" and should not be forwarded to any third-party integrations like the data connectors by setting the `privateAttributes` parameter in the `StatsigUser` constructor. The `privateAttributes` parameter is a key-value dictionary where keys are attribute names and values are the private values. Note that in the example user object above, for the key `"email"`, we have values for the top-level `email` field AND for the private attributes field with `"email"` as the key. These are distinct; you can have a value in the top-level `email` field that is not `private`, and a value in `private_attributes` that is `private`, or vice versa. ```php theme={null} $user = new StatsigUser([ 'userID' => 'a-user-id', 'email' => 'non_private@example.com', 'privateAttributes' => [ 'email' => 'private@example.com' ] ]); ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. Custom URL for fetching feature specifications. Custom URL for logging events. An adapter with custom storage behavior for config specs. For example, use `StatsigLocalFileSpecsAdapter` to store configs in the local filesystem. An adapter with custom event logging behavior. For example, use `StatsigLocalFileEventLoggingAdapter` to store events in the local filesystem. Environment parameter for evaluation. How often events are flushed to Statsig servers (in milliseconds). Maximum number of events to queue before forcing a flush. * Default is `2000` * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * See also `event_logging_max_pending_batch_queue_size` Maximum number of event batches to hold in buffer to retry. * Default is `100`. * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * eg: 2000 \* 100 means the SDK can process 200k event per second before events start getting dropped * See also `event_logging_max_queue_size`. How often the SDK updates specifications from Statsig servers (in milliseconds). Controls the verbosity of SDK logs. Disables country lookup based on IP address. Set to `true` to improve performance if country-based targeting is not needed. Disables user agent parsing. Set to `true` to improve performance if device/browser-based targeting is not needed. Maximum time in milliseconds to wait for SDK initialization to complete. If initialization takes longer than this timeout, the SDK will continue to operate but may return default values until initialization completes. When set to true, the SDK will fallback to using the Statsig API directly if custom adapters (like local file adapters) fail to load configurations. Enable/disable ID list functionality. **Required to be `true` when using segments with more than 1000 IDs.** See [ID List segments](/segments/add-id-list) for more details. Configuration for connecting through a proxy server. The hostname or IP address of the proxy server (e.g., `"proxy.example.com"` or `"192.168.1.100"`). The port number of the proxy server (e.g., `8080`, `3128`, `1080`). Authentication credentials for the proxy server in the format `"username:password"`. Only required if the proxy requires authentication. The protocol to use for the proxy connection. Supported values: `"http"`, `"https"`, `"socks5"`. ### Proxy and Custom Network Routing The PHP Server Core SDK supports a dedicated `proxy_config` for a standard outbound HTTP proxy. If you only need to route Statsig traffic to different endpoints, use `specs_url`, `log_event_url`, and `id_lists_url`. *** ### Performance Recommendations If you are experiencing performance issues, particularly with long initialization times, you can disable user agent parsing and country lookup to improve performance: * Set `disable_user_agent_parsing: true` if you don't need device or browser-based targeting * Set `disable_country_lookup: true` if you don't need country-based targeting These optimizations were added in response to performance issues identified in [PR #1119](https://github.com/statsig-io/private-statsig-server-core/pull/1119). *** ### Example Usage ```php theme={null} use Statsig\Statsig; use Statsig\StatsigOptions; use Statsig\ProxyConfig; use Statsig\StatsigLocalFileSpecsAdapter; use Statsig\StatsigLocalFileEventLoggingAdapter; // Create proxy configuration $proxyConfig = new ProxyConfig( proxyHost: "proxy.example.com", proxyPort: 8080, proxyAuth: "username:password", // Optional, only if authentication is required proxyProtocol: "http" ); // Initialize StatsigOptions with custom parameters $options = new StatsigOptions( specs_url: null, log_event_url: null, specs_adapter: new StatsigLocalFileSpecsAdapter($sdk_key, "/tmp"), event_logging_adapter: new StatsigLocalFileEventLoggingAdapter($sdk_key, "/tmp"), environment: "development", event_logging_flush_interval_ms: 60000, event_logging_max_queue_size: 1000, specs_sync_interval_ms: 600000, output_log_level: "INFO", disable_country_lookup: true, // For better performance wait_for_country_lookup_init: false, wait_for_user_agent_init: false, enable_id_lists: false, disable_network: false, id_lists_url: null, id_lists_sync_interval_ms: null, disable_all_logging: false, init_timeout_ms: 3000, fallback_to_statsig_api: false, use_third_party_ua_parser: null, persistent_storage: null, proxy_config: $proxyConfig // Add proxy configuration ); // Pass the options object into Statsig constructor $statsig = new Statsig($sdk_key, $options); $statsig->initialize(); ``` When using `StatsigLocalFile` Adapters, ensure your cron job has run at least one time before using Statsig. See [Add Scripts & Cron Job](#2-add-scripts--cron-job) ## Custom Adapters ### SpecsAdapterBase - Custom Configuration Sources The `SpecsAdapterBase` allows you to fetch Statsig configurations from custom sources instead of (or in addition to) Statsig's servers. This is useful when you want to: * Store configurations in your own database or cache (e.g., Redis, Memcached) * Implement custom caching strategies * Use Statsig in environments with restricted network access * Reduce latency by serving configs from a local source #### Implementation To create a custom specs adapter, extend the `SpecsAdapterBase` class and implement the required methods: ```php theme={null} redis = $redis; } public function setup(SpecsUpdateListener $listener): void { $this->listener = $listener; } public function start(): void { // Fetch initial specs when SDK starts $this->refreshSpecsFromRedis(); // Optionally, trigger a background job or set up a timer $this->refreshSpecsFromRedis(); } private function fetchSpecsFromRedis() { try { $specs = $this->redis->get('statsig_config_specs'); return $specs ?: null; } catch (Exception $e) { error_log("Failed to fetch specs from Redis: " . $e->getMessage()); return null; } } private function refreshSpecsFromRedis() { $specs = $this->fetchSpecsFromRedis(); if ($specs && $this->listener) { $timestamp = intval(microtime(true) * 1000); $this->listener->didReceiveSpecsUpdate($specs, "Redis", $timestamp); } } } ``` #### Usage ```php theme={null} use Statsig\Statsig; use Statsig\StatsigOptions; $redis = new Redis(); $redis->connect('127.0.0.1', 6379); $specsAdapter = new RedisSpecsAdapter($redis); $options = new StatsigOptions( specs_adapter: $specsAdapter ); $statsig = new Statsig('your-server-secret-key', $options); $statsig->initialize(); ``` #### Key Methods * **`setup(SpecsUpdateListener $listener)`**: Called during initialization to provide the listener for spec updates * **`start()`**: Called when the SDK starts. Fetch and provide initial configuration specs * **`shutdown()`**: Called when the SDK shuts down. Clean up resources * **`scheduleBackgroundSync()`**: Called to schedule periodic updates of configuration specs The `SpecsUpdateListener` provides: * **`didReceiveSpecsUpdate(string $specs, string $source, int $timestamp)`**: Notify the SDK of new specs * **`getCurrentSpecsInfo()`**: Get information about current specs *** ### EventLoggingAdapterBase - Custom Event Destinations The `EventLoggingAdapterBase` allows you to send events to custom destinations instead of or in addition to Statsig's servers. This is useful when you want to: * Send events to your existing analytics platform * Store events in a database for custom analysis * Forward events to multiple destinations * Implement custom batching or retry logic #### Implementation To create a custom event logging adapter, extend the `EventLoggingAdapterBase` class and implement the required methods: ```php theme={null} analyticsClient = $analyticsClient; } public function start(): void { $this->isStarted = true; // Initialize analytics client connection if needed $this->analyticsClient->connect(); } public function logEvents(LogEventRequest $request): bool { if (!$this->isStarted) { return false; } try { $events = $request->payload->events; $metadata = $request->payload->statsig_metadata; foreach ($events as $event) { // Transform Statsig event to analytics platform format $analyticsEvent = [ 'event_name' => $event['eventName'], 'user_id' => $event['user']['userID'] ?? null, 'timestamp' => $event['time'], 'properties' => array_merge( $event['metadata'] ?? [], ['statsig_metadata' => $metadata] ) ]; // Send to analytics platform $this->analyticsClient->track($analyticsEvent); } return true; } catch (Exception $e) { error_log("Failed to log events to analytics platform: " . $e->getMessage()); return false; } } public function shutdown(): void { $this->isStarted = false; // Clean up analytics client connection $this->analyticsClient->disconnect(); } } ``` #### Usage ```php theme={null} use Statsig\Statsig; use Statsig\StatsigOptions; $analyticsClient = new MyAnalyticsClient('api-key'); $eventAdapter = new AnalyticsEventAdapter($analyticsClient); $options = new StatsigOptions( event_logging_adapter: $eventAdapter ); $statsig = new Statsig('your-server-secret-key', $options); $statsig->initialize(); // Events will now be sent to your custom analytics platform $statsig->logEvent($user, 'button_clicked', ['button_id' => 'signup']); ``` #### Key Methods * **`start()`**: Called when the SDK starts. Initialize connections or resources * **`logEvents(LogEventRequest $request): bool`**: Process and send events. Return true on success, false on failure * **`shutdown()`**: Called when the SDK shuts down. Clean up resources The `LogEventRequest` contains: * **`event_count`**: Number of events in the request * **`retries`**: Number of retry attempts for this request * **`payload`**: `LogEventPayload` with events and metadata The `LogEventPayload` contains: * **`events`**: Array of event objects with user data, event names, and metadata * **`statsig_metadata`**: SDK metadata including version and environment information ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```php theme={null} // Method signature public function shutdown(): void // example usage try { $statsig->shutdown(); echo "Statsig instance has been successfully shutdown.\n"; } catch (Exception $e) { error_log($e->getMessage()); } ``` Alternatively, if you are operating in a serverless environment/cloud function, you may wish to leave Statsig running in case your function is recycled but flush the logs to Statsig servers. Or, if you need an async method to wait for logs to post before resolving, you can use: ```php theme={null} // Method signature public function flushEvents(): void // example usage try { $statsig->flushEvents(); echo "All events have been successfully flushed.\n"; } catch (Exception $e) { echo "Failed to flush events: " . $e->getMessage() . "\n"; } ``` ## Local Overrides Local Overrides are a way to override the values of gates, configs, experiments, and layers for testing purposes. This is useful for local development or testing scenarios where you want to force a specific value without having to change the configuration in the Statsig console. ```php theme={null} $statsig->overrideGate("a_gate_name", true); $statsig->overrideDynamicConfig("a_config_name", [ "key" => "value", ]); $statsig->overrideExperiment("an_experiment_name", [ "key" => "value", ]); $statsig->overrideExperimentByGroupName("an_experiment_name", "a_group_name"); $statsig->overrideLayer("a_layer_name", [ "key" => "value", ]); ``` You can also pass a third argument to scope an override to a specific ID: ```php theme={null} $statsig->overrideExperimentByGroupName("an_experiment_name", "a_group_name", "user_123"); ``` ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for user-specific configurations. This enables you to persist user assignments across sessions, ensuring consistent experiment groups even when the user returns later. This is particularly useful for client-side A/B testing where you want to ensure users always see the same variant. ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for experiment assignments. This ensures consistent user experiences across sessions by persisting experiment assignments. For more information on persistent assignments, see the [Persistent Assignment](/server/concepts/persistent_assignment) documentation. ```php expandable MyPersistentStorage.php theme={null} use Statsig\PersistentStorage; use Statsig\StickyValues; class MyPersistentStorage extends PersistentStorage { private array $storage = []; public function load(string $key): ?array { return $this->storage[$key] ?? null; } public function save(string $key, string $config_name, StickyValues $data): void { $this->storage[$key] ??= []; $this->storage[$key][$config_name] = $data->toArray(); } public function delete(string $key, string $config_name): void { unset($this->storage[$key][$config_name]); } } ``` ```php expandable PersistentStorageUsage.php theme={null} $persistent_storage = new MyPersistentStorage(); $options = new StatsigOptions( persistent_storage: $persistent_storage ); $statsig = new Statsig("secret-key", $options); $statsig->initialize(); $persisted_user = new StatsigUser("test-123"); $values = $persistent_storage->getValuesForUser($persisted_user, "userID") ?? []; $experiment = $statsig->getExperiment( $persisted_user, "active_experiment", ["user_persisted_values" => $values], ); ``` ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. Not supported at this time. ## Custom Output Logger The Output Logger interface allows you to customize how the SDK logs messages. This enables integration with your own logging system and control over log verbosity. ## Custom Output Logger The Output Logger interface allows you to customize how the SDK logs messages. This enables integration with your own logging system and control over log verbosity. Not supported at this time. ## Observability Client The Observability Client interface allows you to monitor the health of the SDK by integrating with your own observability systems. This enables tracking metrics, errors, and performance data. For more information on the metrics emitted by Statsig SDKs, see the [Monitoring documentation](/sdk_monitoring). ## Observability Client The Observability Client interface allows you to monitor the health of the SDK by integrating with your own observability systems. This enables tracking metrics, errors, and performance data. For more information on the metrics emitted by Statsig SDKs, see the [Monitoring](/infrastructure/monitoring) documentation. Not supported at this time. ## Notes on Beta Version The PHP SDK expects an adapter to be provided for both logging and saving config specs, given the stateless nature of PHP. In [our example](https://github.com/daniel-statsig/statsig-php-core-slim-example), we've provided simple file-based adapters. More mature implementations may choose a different, and more performant caching approach. If you need help setting this up, reach out to us in [Slack](https://statsig.com/slack). ## FAQ See the guide on [device level experiments](/guides/device-level-experiment) # Python Server SDK Source: https://docs.statsig.com/server-core/python-core Statsig's next-generation Python Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for Python services. Python Core on Github Migrating from the Legacy Python SDK? See our [Migration Guide](/server-core/migration-guides/python). ## Setup the SDK ## Installation ```shell theme={null} pip install statsig-python-core ``` ## Tested Platforms Docker base images where the Python Core SDK has been tested and verified: | Docker Base Image | Description | | ----------------------------------- | ---------------------------------------------------- | | `python:3.7-alpine` | Python 3.7 on Alpine Linux | | `python:3.7-buster` | Python 3.7 on Debian Buster | | `python:3.7-slim` | Python 3.7 slim variant | | `quay.io/pypa/manylinux2014_x86_64` | Manylinux 2014 x86\_64 for broad Linux compatibility | | `python:3.10-alpine` | Python 3.10 on Alpine Linux | | `python:3.10-slim` | Python 3.10 slim variant | After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. ```python theme={null} from statsig_python_core import Statsig, StatsigOptions # note, import statement has underscores while install has dashes options = StatsigOptions() options.environment = "development" statsig = Statsig("secret-key", options) statsig.initialize().wait() # If you're running this in a script, be sure to wait for shutdown at the end to flush event logs to statsig statsig.shutdown().wait() ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ### ⚠️ Warning: Process Forking and WSGI Servers **Important:** Never fork processes after calling `statsig.initialize()`. Doing so will put Statsig in an undefined state and may cause deadlock. The Python Core SDK uses internal threading and async runtime components that do not work correctly when copied across process boundaries. When a process forks after initialization, these components can become corrupted, leading to: * Deadlocks in event logging * Hanging initialization calls * Unpredictable SDK behavior * Silent failures in feature evaluation ### Initializing with WSGI servers For production deployments using WSGI servers like uWSGI or Gunicorn, ensure Statsig is initialized **after** the worker processes are forked, not in the main process. #### ✅ Correct: uWSGI example ```python expandable theme={null} # app.py from statsig_python_core import Statsig, StatsigOptions from flask import Flask app = Flask(__name__) statsig = None def init_statsig(): global statsig if statsig is None: options = StatsigOptions() options.environment = "production" statsig = Statsig("your-server-secret-key", options) statsig.initialize().wait() # Initialize in each worker process @app.before_first_request def before_first_request(): init_statsig() @app.route('/') def index(): # Use statsig here return "Hello World" ``` ```ini theme={null} # uwsgi.ini [uwsgi] module = app:app master = true processes = 4 # Statsig will be initialized in each worker process ``` #### ✅ Correct: Gunicorn example ```python theme={null} # gunicorn_config.py def post_fork(server, worker): # Initialize Statsig after worker process is forked from app import init_statsig init_statsig() # as defined above # ...app.py ``` ```bash theme={null} # Start Gunicorn with post-fork hook gunicorn --config gunicorn_config.py app:app ``` #### ✅ Correct: FastAPI example ```python expandable theme={null} from fastapi import FastAPI from statsig_python_core import Statsig, StatsigOptions app = FastAPI() statsig = None @app.on_event("startup") async def startup_event(): global statsig options = StatsigOptions() options.environment = "production" statsig = Statsig("your-server-secret-key", options) statsig.initialize().wait() @app.on_event("shutdown") async def shutdown_event(): if statsig: statsig.shutdown().wait() @app.get("/") async def root(): # Use statsig here return {"message": "Hello World"} ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```python theme={null} user = StatsigUser("a-user") if statsig.check_gate(user, "a_gate"): # Gate is on, enable new feature else: # Gate is off ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```python theme={null} # Get a dynamic config for a specific user config = statsig.get_dynamic_config(StatsigUser("my_user"), "a_config") # Access config values with type-safe getters and fallback values product_name = config.get_string("product_name", "Awesome Product v1") # returns String price = config.get_float("price", 10.0) # returns float should_discount = config.get_bool("discount", False) # returns bool quantity = config.get_integer("quantity", 1) # returns int64 # Advanced Usage: # You can disable exposure logging for this specific check options = DynamicConfigEvaluationOptions(disable_exposure_logging=True) config = statsig.get_dynamic_config(user, "a_config", options) # The config object also provides metadata about the evaluation print(config.rule_id) # The ID of the rule that served this config print(config.id_type) # The type of the evaluation (experiment, config, etc) ``` The `get_dynamic_config()` method returns a DynamicConfig object that allows you to: * Fetch typed values with fallback defaults using `get_string()`, `get_float()`, `get_boolean()`, and `get_integer()` * Access evaluation metadata through properties like `rule_id` and `id_type` * Configure evaluation behavior using `DynamicConfigEvaluationOptions` By default, Statsig logs exposures automatically when configs are evaluated. You can disable this for specific checks using the evaluation options. ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```Python theme={null} # Values via get_layer layer = statsig.get_layer(StatsigUser("my_user"), "user_promo_experiments") title = layer.get_string("title", "Welcome to Statsig!") discount = layer.get_float("discount", 0.1) # Via get_experiment title_exp = statsig.get_experiment(StatsigUser("my_user"), "new_user_promo_title") price_exp = statsig.get_experiment(StatsigUser("my_user"), "new_user_promo_price") title = title_exp.get_string("title", "Welcome to Statsig!") discount = price_exp.get_float("discount", 0.1) ``` ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```python theme={null} gate = statsig.get_feature_gate(user, "example_gate") print(gate.rule_id) print(gate.value) ``` ### Parameter Stores Sometimes you don't know whether you want a value to be a Feature Gate, Experiment, or Dynamic Config yet. If you want on-the-fly control of that outside of your deployment cycle, you can use Parameter Stores to define a parameter that can be changed into at any point in the Statsig console. Parameter Stores are optional, but parameterizing your application can prove very useful for future flexibility and can even allow non-technical Statsig users to turn parameters into experiments. ```python theme={null} # Get a Parameter Store by name param_store = statsig.get_parameter_store(user, "my_parameter_store") ``` ### Retrieving Parameter Values Parameter Store provides methods for retrieving values of different types with fallback defaults. ```python theme={null} # String parameters string_value = param_store.get_string("string_param", "default_value") # Boolean parameters bool_value = param_store.get_bool("bool_param", False) # Numeric parameters float_value = param_store.get_float("float_param", 0.0) integer_value = param_store.get_integer("integer_param", 0) # Complex parameters default_array = ["item1", "item2"] array_value = param_store.get_array("array_param", default_array) default_map = {"key": "value"} map_value = param_store.get_map("map_param", default_map) ``` ### Evaluation Options You can disable exposure logging when retrieving a parameter store: ```python theme={null} from statsig_python_core import ParameterStoreEvaluationOptions options = ParameterStoreEvaluationOptions(disable_exposure_logging=True) param_store = statsig.get_parameter_store(user, "my_parameter_store", options) ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```Python theme={null} statsig.log_event( user=StatsigUser("user_id"), # Replace with your user object event_name="add_to_cart", value="SKU_12345", metadata={ "price": "9.99", "item_name": "diet_coke_48_pack" } ) ``` ### Sending Events to Log Explorer You can forward logs to Logs Explorer for convenient analysis using the Forward Log Line Event API. This lets you include custom metadata and event values with each log. ```python theme={null} user = StatsigUser( user_id="a-user", custom={ "service": "my-service", "pod": "my-pod", "namespace": "my-namespace", "container": "my-container", # ...include any service-specific metadata } ) # levels: trace, debug, info, log, warn, error statsig.forward_log_line_event(user, "warn", "script failed to load", { "custom_metadata": "script_name:my-script" # ... include any event-specific metadata }) ``` ## Using Shared Instance In some applications, you may want to create a single Statsig instance that can be accessed globally throughout your codebase. The shared instance functionality provides a singleton pattern for this purpose: ```python theme={null} # Create a shared instance that can be accessed globally statsig = Statsig.new_shared("secret-key", options) statsig.initialize().wait() # Access the shared instance from anywhere in your code shared_statsig = Statsig.shared() is_feature_enabled = shared_statsig.check_gate(StatsigUser("user_id"), "feature_name") # Check if a shared instance exists if Statsig.has_shared_instance(): # Use the shared instance pass # Remove the shared instance when no longer needed Statsig.remove_shared() ``` The shared instance functionality provides a singleton pattern where a single Statsig instance can be created and accessed globally throughout your application. This is useful for applications that need to access Statsig functionality from multiple parts of the codebase without having to pass around a Statsig instance. * `Statsig.new_shared(sdk_key, options)`: Creates a new shared instance of Statsig that can be accessed globally * `Statsig.shared()`: Returns the shared instance * `Statsig.has_shared_instance()`: Checks if a shared instance exists (useful when you aren't sure if the shared instance is ready yet) * `Statsig.remove_shared()`: Removes the shared instance (useful when you want to switch to a new shared instance) `has_shared_instance()` and `remove_shared()` are helpful in specific scenarios but aren't required in most use cases where the shared instance is set up near the top of your application. Also note that only one shared instance can exist at a time. Attempting to create a second shared instance will result in an error. ## Manual Exposures By default, the SDK will automatically log an exposure event when you check a gate, get a config, get an experiment, or call get() on a parameter in a layer. However, there are times when you may want to log an exposure event manually. For example, if you're using a gate to control access to a feature, but you don't want to log an exposure until the user actually uses the feature, you can use manual exposures. All of the main SDK functions (`check_gate`, `get_dynamic_config`, `get_experiment`, `get_layer`) accept an optional `disable_exposure_logging` parameter. When this is set to `True`, the SDK will not automatically log an exposure event. You can then manually log the exposure at a later time using the corresponding manual exposure logging method: ```python theme={null} result = statsig.check_gate(aUser, 'a_gate_name', FeatureGateEvaluationOptions(disable_exposure_logging=True)) ``` ```python theme={null} statsig.manually_log_gate_exposure(aUser, 'a_gate_name') ``` ```python theme={null} config = statsig.get_dynamic_config(aUser, 'a_dynamic_config_name', DynamicConfigEvaluationOptions(disable_exposure_logging=True)) ``` ```python theme={null} statsig.manually_log_dynamic_config_exposure(aUser, 'a_dynamic_config_name') ``` ```python theme={null} experiment = statsig.get_experiment(aUser, 'an_experiment_name', ExperimentEvaluationOptions(disable_exposure_logging=True)) ``` ```python theme={null} statsig.manually_log_experiment_exposure(aUser, 'an_experiment_name') ``` ```python theme={null} layer = statsig.get_layer(aUser, 'a_layer_name', LayerEvaluationOptions(disable_exposure_logging=True)) paramValue = layer.get('a_param_name', 'fallback_value') ``` ```python theme={null} statsig.manually_log_layer_parameter_exposure(aUser, 'a_layer_name', 'a_param_name') ``` ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. ```python expandable theme={null} from statsig_python_core import StatsigUser user = StatsigUser( user_id="a-user-id", email="user@example.com", ip="192.168.1.1", user_agent="Mozilla/5.0...", country="US", locale="en_US", app_version="1.0.0", custom={ # Custom fields "plan": "premium", "age": 25 }, custom_ids={ # Custom ID types "stable_id": "stable-id-123" }, private_attributes={ # Private attributes not forwarded to integrations "email": "private@example.com" } ) ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. Custom URL for fetching feature specifications. How often the SDK updates specifications from Statsig servers (in milliseconds). Sets the maximum timeout for initialization requests (in milliseconds). Custom URL for logging events. When `true`, disables all event logging. When `true`, disables all network functions: event & exposure logging, spec downloads, and ID List downloads. Formerly called "localMode". How often events are flushed to Statsig servers (in milliseconds). Maximum number of events to queue before forcing a flush. * Default is `2000` * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * See also `event_logging_max_pending_batch_queue_size` Maximum number of event batches to hold in buffer to retry. * Default is `100`. * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * eg: 2000 \* 100 means the SDK can process 200k event per second before events start getting dropped * See also `event_logging_max_queue_size`. Enable/disable ID list functionality. **Required to be `true` when using segments with more than 1000 IDs.** See [ID List segments](/segments/add-id-list) for more details. If set to true, the SDK will NOT attempt to parse UserAgents (attached to the user object) into browserName, browserVersion, systemName, systemVersion, and appVersion at evaluation time, when needed for evaluation. When set to true, the SDK will wait until user agent parsing data is fully loaded during initialization. This may slow down by \~1 second startup but ensures that parsing of the user's userAgent string into fields like browserName, browserVersion, systemName, systemVersion, and appVersion is ready before any evaluations. If set to true, the SDK will NOT attempt to parse IP addresses (attached to the user object at user.ip) into Country codes at evaluation time, when needed for evaluation. When set to true, the SDK will wait for country lookup data (e.g., GeoIP or YAML files) to fully load during initialization. This may slow down by \~1 second startup but ensures that IP-to-country parsing is ready at evaluation time. Custom URL for fetching ID lists. How often the SDK updates ID lists from Statsig servers (in milliseconds). Whether to fallback to the Statsig API if custom endpoints fail. Environment parameter for evaluation. Controls the verbosity of SDK logs. Adapter / Interface to use persistent assignment within SDK. See [Persistent Assignment](/server/concepts/persistent_assignment/) for more details. Adapter to listen monitor the health of SDK. See [SDK Monitoring](https://docs.statsig.com/sdk_monitoring/) for more details. Custom data store implementation for storing and retrieving configuration data. Used for advanced caching or storage strategies. Maximum number of batches of events to hold in buffer to retry. Custom fields to include in all events logged by the SDK. Compression method for exposure logging. Options: "gzip", "dictionary" Configuration for connecting through a proxy server. The `ProxyConfig` object has the following properties: * `proxy_host`: Optional string specifying the proxy server host * `proxy_port`: Optional number specifying the proxy server port * `proxy_auth`: Optional string for proxy authentication (format: "username:password") * `proxy_protocol`: Optional string specifying the protocol (e.g., "http", "https") * `ca_cert_path`: Optional path to a PEM CA bundle for outbound TLS Advanced configuration for customizing how the SDK fetches feature specifications. Allows you to configure multiple spec adapters with different priorities and settings. Each `SpecAdapterConfig` object has the following properties: * `adapter_type`: String specifying the adapter type (e.g., "http", "grpc") * `specs_url`: Optional custom URL for fetching specifications * `init_timeout_ms`: Optional timeout for initialization (in milliseconds) **Migration Note:** This parameter was previously named `init_resources` in earlier versions. If you're upgrading from an older version, replace `init_resources` with `spec_adapter_configs`. ### Proxy and Custom Network Routing Use `proxy_config` if your service needs a standard outbound HTTP proxy. Use `spec_adapter_configs` if you need to route spec downloads through [Statsig Forward Proxy](/infrastructure/forward-proxy) or another custom spec source. ```python theme={null} from statsig_python_core import ProxyConfig, Statsig, StatsigOptions proxy_config = ProxyConfig( proxy_host="proxy.example.com", proxy_port=8080, proxy_protocol="https", ca_cert_path="/etc/ssl/certs/corporate-ca.pem", # Optional ) options = StatsigOptions() options.proxy_config = proxy_config statsig = Statsig("secret-key", options) statsig.initialize().wait() ``` Set `ca_cert_path` when your environment requires a custom PEM CA bundle for outbound TLS. ### Using spec\_adapter\_configs with Multiple Sources ```python theme={null} from statsig_python_core import StatsigOptions, SpecAdapterConfig # Configure multiple spec adapters with priority order # First source: Statsig CDN primary_adapter = SpecAdapterConfig( adapter_type="http", specs_url="https://api.statsigcdn.com/v2/download_config_specs", init_timeout_ms=3000 ) # Second source: Data Store adapter # The SDK will try this source if the primary source fails data_store_adapter = SpecAdapterConfig( adapter_type="data_store", init_timeout_ms=5000 ) options = StatsigOptions() options.spec_adapter_configs = [primary_adapter, data_store_adapter] options.environment = "production" statsig = Statsig("secret-key", options) statsig.initialize().wait() ``` ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```python theme={null} statsig.shutdown().wait() ``` ## Local Overrides Local Overrides are a way to override the values of gates, configs, experiments, and layers for testing purposes. This is useful for local development or testing scenarios where you want to force a specific value without having to change the configuration in the Statsig console. ```python theme={null} # Overrides the given gate to the specified value statsig.override_gate("a_gate_name", True) # Overrides the given dynamic config to the provided value statsig.override_dynamic_config("a_config_name", {"key": "value"}) # Overrides the given experiment to the provided value statsig.override_experiment("an_experiment_name", {"key": "value"}) # Overrides the given layer to the provided value statsig.override_layer("a_layer_name", {"key": "value"}) # Overrides the given experiment to a particular groupname, available for experiments only: statsig.override_experiment_by_group_name("an_experiment_name", "a_group_name") ``` ## Client SDK Bootstrapping | SSR If you are using the Statsig client SDK in a browser or mobile app, you can bootstrap the client SDK with the values from the server SDK to avoid a network request on the client. This is useful for server-side rendering (SSR) or when you want to reduce the number of network requests on the client. ## Client Initialize Response The Python Core SDK provides a method to generate a client initialize response that can be used to bootstrap client SDKs without requiring network requests. ```python theme={null} import json from statsig_python_core import Statsig, StatsigUser # Get client initialize response for a user response_data = statsig.get_client_initialize_response(user) response = json.loads(response_data) # Pass response to a client SDK to initialize without a network request ``` The `get_client_initialize_response` method accepts the following parameters: ```python theme={null} def get_client_initialize_response( user: StatsigUser, hash: Optional[str] = None, client_sdk_key: Optional[str] = None, include_local_overrides: Optional[bool] = None ) -> str: ``` * **`user`**: `StatsigUser` - The user to generate the initialize response for * **`hash`**: `Optional[str]` - Algorithm used for hashing gate/experiment names (default: 'djb2') * **`client_sdk_key`**: `Optional[str]` - Client SDK key to use for initialization * **`include_local_overrides`**: `Optional[bool]` - Whether to include local overrides in the response The `hash` parameter specifies which algorithm to use for hashing gate and experiment names in the client initialize response. The default is `'djb2'` for better performance and smaller payload size. Available options: * `'djb2'` (default) - DJB2 hashing algorithm for better performance * `'sha256'` - SHA-256 hashing algorithm * `'none'` - No hashing applied ```python theme={null} # Use djb2 hashing algorithm (default) response_data = statsig.get_client_initialize_response(user, hash='djb2') # Use SHA-256 hashing algorithm response_data = statsig.get_client_initialize_response(user, hash='sha256') # Disable hashing response_data = statsig.get_client_initialize_response(user, hash='none') ``` The `client_sdk_key` parameter lets you filter the response to only the specific feature gates, experiments, dynamic configs, layers, or parameter stores that a particular client key has access to - effectively letting you apply [target apps](/sdk-keys/target-apps/). ```python theme={null} # Specify a client SDK key response_data = statsig.get_client_initialize_response( user, client_sdk_key='client-key' ) ``` The `include_local_overrides` parameter determines whether to consider [local overrides](#local-overrides) you've set when evaluating each config in the response. ```python theme={null} # Include local overrides in the response response_data = statsig.get_client_initialize_response( user, include_local_overrides=True ) ``` Below is a complete example of using the client initialize response to bootstrap a client SDK. Note that you may choose to parallelize or inline the initialize response data with other requests to your server, to eliminate additional requests and latency. ```python theme={null} # Server-side code import json from statsig_python_core import Statsig, StatsigUser, StatsigOptions from flask import Flask, request, jsonify app = Flask(__name__) # Initialize the server SDK options = StatsigOptions() statsig = Statsig('server-secret-key', options) statsig.initialize().wait() # In your API endpoint handler @app.route('/statsig-bootstrap') def statsig_bootstrap(): # Create a user object from the request user = StatsigUser( user_id=request.args.get('userID', ''), email=request.args.get('email'), ip=request.remote_addr, user_agent=request.headers.get('User-Agent') ) # Generate the client initialize response response_data = statsig.get_client_initialize_response( user, hash='djb2', client_sdk_key='client-sdk-key' ) # Parse the JSON response statsig_values = json.loads(response_data) # Return the values to the client return jsonify({'statsigValues': statsig_values}) ``` ```javascript theme={null} // Client-side code using @statsig/js-client import { Statsig } from '@statsig/js-client'; // Fetch bootstrap values from your API const response = await fetch('/statsig-bootstrap'); const { statsigValues } = await response.json(); // Initialize the client SDK with the bootstrap values await Statsig.initialize({ sdkKey: 'client-sdk-key', initializeValues: statsigValues, }); ``` The method returns a JSON string containing the client initialize response. You'll need to parse this string to access the data: ```python theme={null} response_data = statsig.get_client_initialize_response(user) response = json.loads(response_data) # Access different parts of the response feature_gates = response.get('feature_gates', {}) dynamic_configs = response.get('dynamic_configs', {}) layer_configs = response.get('layer_configs', {}) ``` The response includes: * `feature_gates`: Feature gate evaluations for the user * `dynamic_configs`: Dynamic config and experiment evaluations * `layer_configs`: Layer evaluations * `has_updates`: Boolean indicating if there are updates * `time`: Timestamp of the response ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for user-specific configurations. This enables you to persist user assignments across sessions, ensuring consistent experiment groups even when the user returns later. This is particularly useful for client-side A/B testing where you want to ensure users always see the same variant. ```python theme={null} class PersistentStorage(PersistentStorageBaseClass): def __init__(): # When you initialize, remember to call super.__init__() super().__init__() self.load_fn = self.load self.save_fn = self.save self.delete_fn = self.delete def load(self, key: str) -> Optional[UserPersistedValues]: """ Load persisted values for a user from storage Args: key: A string key that uniquely identifies a user Returns: Dictionary mapping config names to their persisted values """ pass def save(self, key: str, config_name: str, data: StickyValues): """ Save a persistent value for a user Args: key: A string key that uniquely identifies a user config_name: The name of the config/experiment data: The values to persist """ pass def delete(self, key: str, config_name: str): """ Delete a persistent value for a user Args: key: A string key that uniquely identifies a user config_name: The name of the config/experiment to delete """ pass ``` ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. ```python theme={null} class DataStore(DataStoreBase): def initialize(self): """ Initialize the data store. Called when the Statsig client initializes. """ pass def shutdown(self): """ Clean up resources when the Statsig client shuts down. """ pass def get(self, key: str) -> Optional[DataStoreResponse]: """ Retrieve value from the data store. Args: key: The key to retrieve the value for Returns: DataStoreResponse containing the result and time """ pass def set(self, key: str, value: str, time: Optional[int] = None): """ Store a value in the data store. Args: key: The key to store the value under value: The value to store time: Optional timestamp """ pass def support_polling_updates_for(self, key: str) -> bool: """ Whether the data store supports polling for updates for the given key. Args: key: The key to check Returns: True if polling is supported, False otherwise """ return False ``` ## Custom Output Logger The Output Logger interface allows you to customize how the SDK logs messages. This enables integration with your own logging system and control over log verbosity. ## Output Logger The Output Logger Provider interface allows you to customize how the SDK logs internal messages. ```python theme={null} class OutputLoggerProvider(OutputLoggerProviderBase): def init(self): """ Initialize the logger. Called when the Statsig client initializes. """ pass def debug(self, tag: str, msg: str): """ Log a debug message. Args: tag: Category/component tag for the message msg: The message to log """ pass def info(self, tag: str, msg: str): """ Log an info message. Args: tag: Category/component tag for the message msg: The message to log """ pass def warn(self, tag: str, msg: str): """ Log a warning message. Args: tag: Category/component tag for the message msg: The message to log """ pass def error(self, tag: str, msg: str): """ Log an error message. Args: tag: Category/component tag for the message msg: The message to log """ pass def shutdown(self): """ Clean up resources when the Statsig client shuts down. """ pass ``` ## Observability Client The Observability Client interface allows you to monitor the health of the SDK by integrating with your own observability systems. This enables tracking metrics, errors, and performance data. For more information on the metrics emitted by Statsig SDKs, see the [Monitoring documentation](/sdk_monitoring). ```python theme={null} class ObservabilityClient(ObservabilityClientBase): def init(self): """ Initialize the observability client. Called when the Statsig client initializes. """ pass def increment(self, metric_name: str, value: float, tags: Optional[Dict[str, str]] = None): """ Report a counter metric. Args: metric_name: The name of the metric value: The amount to increment by tags: Optional tags to associate with the metric """ pass def gauge(self, metric_name: str, value: float, tags: Optional[Dict[str, str]] = None): """ Report a gauge metric. Args: metric_name: The name of the metric value: The current value tags: Optional tags to associate with the metric """ pass def dist(self, metric_name: str, value: float, tags: Optional[Dict[str, str]] = None): """ Report a distribution metric. Args: metric_name: The name of the metric value: The value to record tags: Optional tags to associate with the metric """ pass def error(self, tag: str, error: str): """ Report an error. Args: tag: Category/component tag for the error error: The error message """ pass def should_enable_high_cardinality_for_this_tag(self, tag: str) -> bool: """ Determine if high cardinality should be enabled for a tag. Args: tag: The tag to check Returns: True if high cardinality should be enabled, False otherwise """ pass ``` ## FAQ See the guide on [device level experiments](/guides/first-device-level-experiment). ## Reference ### API Methods * `check_gate(user: StatsigUser, gate_name: str, options: Optional[FeatureGateEvaluationOptions] = None) -> bool` * `get_dynamic_config(user: StatsigUser, config_name: str, options: Optional[DynamicConfigEvaluationOptions] = None) -> DynamicConfig` * `get_experiment(user: StatsigUser, experiment_name: str, options: Optional[ExperimentEvaluationOptions] = None) -> DynamicConfig` * `get_layer(user: StatsigUser, layer_name: str, options: Optional[LayerEvaluationOptions] = None) -> Layer` * `get_feature_gate(user: StatsigUser, gate_name: str, options: Optional[FeatureGateEvaluationOptions] = None) -> FeatureGate` * `get_parameter_store(user: StatsigUser, parameter_store_name: str, options: Optional[ParameterStoreEvaluationOptions] = None) -> ParameterStore` * `log_event(user: StatsigUser, event_name: str, value: Optional[Union[str, float]] = None, metadata: Optional[Dict[str, str]] = None) -> None` * `manually_log_gate_exposure(user: StatsigUser, gate_name: str) -> None` * `manually_log_dynamic_config_exposure(user: StatsigUser, config_name: str) -> None` * `manually_log_experiment_exposure(user: StatsigUser, experiment_name: str) -> None` * `manually_log_layer_parameter_exposure(user: StatsigUser, layer_name: str, parameter_name: str) -> None` * `override_experiment_by_group_name(experiment_name: str, group_name: str, id: Optional[str] = None) -> None` * `get_client_initialize_response(user: StatsigUser, options: Optional[ClientInitializeResponseOptions] = None) -> ClientInitializeResponse` * `shutdown() -> AsyncResult[None]` ### Fields Needed Methods The following methods return information about which user fields are needed for evaluation: * `get_gate_fields_needed(gate_name: str) -> List[str]` * `get_dynamic_config_fields_needed(config_name: str) -> List[str]` * `get_experiment_fields_needed(experiment_name: str) -> List[str]` * `get_layer_fields_needed(layer_name: str) -> List[str]` These methods return a list of strings representing the user fields that are required to properly evaluate the specified gate, config, experiment, or layer. # Rust Server SDK Source: https://docs.statsig.com/server-core/rust-core Statsig's next-generation Rust Server SDK built on the Server Core framework, with improved performance and a unified evaluation engine for Rust backends. Rust Core on Github, Cargo Package ## Setup the SDK To use the SDK, add the Statsig Rust package to your Cargo.toml file: ```toml theme={null} [dependencies] statsig-rust = "X.Y.Z" # Replace with the latest version ``` Or, you can use the cargo command: ```shell theme={null} cargo add statsig-rust ``` You can find the latest version and documentation at [crates.io/crates/statsig-rust](https://crates.io/crates/statsig-rust). After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Server Secret Keys should always be kept private. If you expose one, you can disable and recreate it in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a StatsigOptions to customize the SDK. ```rust theme={null} use statsig_rust::{Statsig, StatsigOptions}; use std::sync::Arc; // Simple initialization let statsig = Statsig::new("server-secret-key", None); statsig.initialize().await?; // Or with StatsigOptions let mut options = StatsigOptions::default(); options.environment = Some("development".to_string()); let statsig = Statsig::new("server-secret-key", Some(Arc::new(options))); statsig.initialize().await?; // Don't forget to shutdown when done statsig.shutdown().await?; ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ### Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```rust theme={null} use statsig_rust::{Statsig, StatsigUserBuilder}; let user = StatsigUserBuilder::new_with_user_id("a-user".to_string()).build(); if statsig.check_gate(&user, "a_gate") { // Gate is on, enable new feature } else { // Gate is off } ``` ### Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. For example: ```rust theme={null} use statsig_rust::{Statsig, StatsigUserBuilder, DynamicConfigEvaluationOptions}; use std::sync::Arc; // Get a dynamic config for a specific user let user = StatsigUserBuilder::new_with_user_id("my_user".to_string()).build(); let config = statsig.get_dynamic_config(&user, "a_config"); // Access config values with type-safe getters and fallback values let product_name = config.get_string("product_name", "Awesome Product v1"); // returns String let price = config.get_double("price", 10.0); // returns f64 let should_discount = config.get_bool("discount", false); // returns bool let quantity = config.get_int("quantity", 1); // returns i64 // Advanced Usage: // You can disable exposure logging for this specific check let mut options = DynamicConfigEvaluationOptions::default(); options.disable_exposure_logging = Some(true); let config = statsig.get_dynamic_config_with_options(&user, "a_config", &options); // The config object also provides metadata about the evaluation println!("{}", config.rule_id); // The ID of the rule that served this config println!("{}", config.id_type); // The type of the evaluation (experiment, config, etc) ``` ### Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but often recommend the use of [layers](/layers), which make parameters reusable and let you run mutually exclusive experiments. ```rust theme={null} use statsig_rust::{Statsig, StatsigUserBuilder}; // Values via get_layer let user = StatsigUserBuilder::new_with_user_id("my_user".to_string()).build(); let layer = statsig.get_layer(&user, "user_promo_experiments"); let title = layer.get_string("title", "Welcome to Statsig!"); let discount = layer.get_double("discount", 0.1); // Via get_experiment let title_exp = statsig.get_experiment(&user, "new_user_promo_title"); let price_exp = statsig.get_experiment(&user, "new_user_promo_price"); let title = title_exp.get_string("title", "Welcome to Statsig!"); let discount = price_exp.get_double("discount", 0.1); ``` ### Parameter Stores Sometimes you don't know whether you want a value to be a Feature Gate, Experiment, or Dynamic Config yet. If you want on-the-fly control of that outside of your deployment cycle, you can use Parameter Stores to define a parameter that can be changed into at any point in the Statsig console. Parameter Stores are optional, but parameterizing your application can prove very useful for future flexibility and can even allow non-technical Statsig users to turn parameters into experiments. ```jsx theme={null} let param_store = statsig.get_parameter_store("my_parameters"); let param_store_value = param_store.get(&user, "my_parameter_value", false); //false is fallback value println!("param_store_value: {}", param_store_value); ``` ### Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig—simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```rust theme={null} use statsig_rust::{Statsig, StatsigUserBuilder}; use std::collections::HashMap; use crate::evaluation::dynamic_value::DynamicValue; // Create a user let user = StatsigUserBuilder::new_with_user_id("user_id".to_string()).build(); // Create metadata hashmap let mut metadata = HashMap::new(); metadata.insert("price".to_string(), "9.99".into()); metadata.insert("item_name".to_string(), "diet_coke_48_pack".into()); // Log the event statsig.log_event( &user, "add_to_cart", Some("SKU_12345".into()), // value as DynamicValue Some(metadata) ); ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ### Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```rust theme={null} use statsig_rust::{Statsig, StatsigUserBuilder}; // Create a user let user = StatsigUserBuilder::new_with_user_id("user_id".to_string()).build(); // Get a feature gate let gate = statsig.get_feature_gate(&user, "example_gate"); // Access gate properties println!("{}", gate.rule_id); println!("{}", gate.value); // Boolean value of the gate ``` ## Using Shared Instance In some applications, you may want to create a single Statsig instance that can be accessed globally throughout your codebase. The shared instance functionality provides a singleton pattern for this purpose: ```rust theme={null} // Create a shared instance that can be accessed globally let statsig = Statsig::new_shared("server-secret-key", None).unwrap(); statsig.initialize().await?; // Access the shared instance from anywhere in your code let shared_statsig = Statsig::shared(); let is_feature_enabled = shared_statsig.check_gate(&user, "feature_name"); // Check if a shared instance exists if Statsig::has_shared_instance() { // Use the shared instance } // Remove the shared instance when no longer needed Statsig::remove_shared(); ``` The shared instance functionality provides a singleton pattern where a single Statsig instance can be created and accessed globally throughout your application. This is useful for applications that need to access Statsig functionality from multiple parts of the codebase without having to pass around a Statsig instance. * `Statsig::new_shared(sdk_key, options)`: Creates a new shared instance of Statsig that can be accessed globally * `Statsig::shared()`: Returns the shared instance * `Statsig::has_shared_instance()`: Checks if a shared instance exists (useful when you aren't sure if the shared instance is ready yet) * `Statsig::remove_shared()`: Removes the shared instance (useful when you want to switch to a new shared instance) `has_shared_instance()` and `remove_shared()` are helpful in specific scenarios but aren't required in most use cases where the shared instance is set up near the top of your application. Also note that only one shared instance can exist at a time. Attempting to create a second shared instance will result in an error. ## Manual Exposures By default, the SDK will automatically log an exposure event when you check a gate, get a config, get an experiment, or call get() on a parameter in a layer. However, there are times when you may want to log an exposure event manually. For example, if you're using a gate to control access to a feature, but you don't want to log an exposure until the user actually uses the feature, you can use manual exposures. All of the main SDK functions (`check_gate`, `get_dynamic_config`, `get_experiment`, `get_layer`) accept an options parameter with a `disable_exposure_logging` field. When this is set to `true`, the SDK will not automatically log an exposure event. You can then manually log the exposure at a later time using the corresponding manual exposure logging method: ```rust theme={null} result = statsig.check_gate_with_options(&user, 'a_gate_name', FeatureGateEvaluationOptions {disable_exposure_logging: true}); ``` ```rust theme={null} statsig.manually_log_gate_exposure(&user, 'a_gate_name') ``` ```rust theme={null} config = statsig.get_dynamic_config_with_options(&user, 'a_dynamic_config_name', DynamicConfigEvaluationOptions {disable_exposure_logging: true}); ``` ```rust theme={null} statsig.manually_log_dynamic_config_exposure(&user, 'a_dynamic_config_name') ``` ```rust theme={null} experiment = statsig.get_experiment_with_options(&user, 'an_experiment_name', ExperimentEvaluationOptions {disable_exposure_logging: true}); ``` ```rust theme={null} statsig.manually_log_experiment_exposure(&user, 'an_experiment_name') ``` ```rust theme={null} layer = statsig.get_layer_with_options(&user, 'a_layer_name', LayerEvaluationOptions {disable_exposure_logging: true}); paramValue = layer.get('a_param_name', 'fallback_value') ``` ```rust theme={null} statsig.manually_log_layer_parameter_exposure(&user, 'a_layer_name', 'a_param_name') ``` ## Statsig User The `StatsigUser` object represents a user in Statsig. You must provide a `userID` or at least one of the `customIDs` to identify the user. When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides userID, we also have email, ip, userAgent, country, locale and appVersion as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the custom field and be able to create targeting based on them. ### Private Attributes Private attributes are user attributes that are used for evaluation but are not forwarded to any integrations. They are useful for PII or sensitive data that you don't want to send to third-party services. ```rust theme={null} use statsig_rust::StatsigUserBuilder; use std::collections::HashMap; // Create a user with just a user ID let user = StatsigUserBuilder::new_with_user_id("user-123".to_string()) .build(); // Or create a user with custom IDs let mut custom_ids = HashMap::new(); custom_ids.insert("employee_id".to_string(), "emp-456".to_string()); let user_with_custom_ids = StatsigUserBuilder::new_with_custom_ids(custom_ids) .build(); // Create a user with several properties let mut custom_fields = HashMap::new(); custom_fields.insert("plan".to_string(), "premium".into()); custom_fields.insert("age".to_string(), 25.into()); let user = StatsigUserBuilder::new_with_user_id("user-123".to_string()) .email(Some("user@example.com".to_string())) .ip(Some("192.168.1.1".to_string())) .user_agent(Some("Mozilla/5.0...".to_string())) .country(Some("US".to_string())) .locale(Some("en-US".to_string())) .app_version(Some("1.0.0".to_string())) .custom(Some(custom_fields)) .build(); // Private Attributes (not forwarded to integrations) let mut private_attrs = HashMap::new(); private_attrs.insert("internal_id".to_string(), "emp-123".into()); let user_with_private = StatsigUserBuilder::new_with_user_id("user-123".to_string()) .email(Some("user@example.com".to_string())) .private_attributes(Some(private_attrs)) .build(); ``` ## Statsig Options You can pass in an optional parameter `options` in addition to `sdkKey` during initialization to customize the Statsig client. Here are the available options that you can configure. External data store for Statsig values. When true, disables all event logging. When `true`, disables all network functions: event & exposure logging, spec downloads, and ID List downloads. Formerly called "localMode". Enable/disable ID list functionality. **Required to be `true` when using segments with more than 1000 IDs.** See [ID List segments](/segments/add-id-list) for more details. If set to true, the SDK will NOT attempt to parse UserAgents (attached to the user object) into browserName, browserVersion, systemName, systemVersion, and appVersion at evaluation time, when needed for evaluation. When set to true, the SDK will wait until user agent parsing data is fully loaded during initialization. This may slow down by \~1 second startup but ensures that parsing of the user's userAgent string into fields like browserName, browserVersion, systemName, systemVersion, and appVersion is ready before any evaluations. If set to true, the SDK will NOT attempt to parse IP addresses (attached to the user object at user.ip) into Country codes at evaluation time, when needed for evaluation. When set to true, the SDK will wait for country lookup data (e.g., GeoIP or YAML files) to fully load during initialization. This may slow down by \~1 second startup but ensures that IP-to-country parsing is ready at evaluation time. Environment parameter for evaluation. Custom adapter for event logging. How often events are flushed to Statsig servers (in milliseconds). Maximum number of events to queue before forcing a flush. * Default is `2000` * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * See also `event_logging_max_pending_batch_queue_size` Maximum number of event batches to hold in buffer to retry. * Default is `100`. * event\_logging\_max\_queue\_size \* event\_logging\_max\_pending\_batch\_queue\_size is the upper limit on how many events are queued * eg: 2000 \* 100 means the SDK can process 200k event per second before events start getting dropped * See also `event_logging_max_queue_size`. Whether to fallback to the Statsig API if custom endpoints fail. Custom adapter for ID lists. How often the SDK updates ID lists from Statsig servers (in milliseconds). Custom URL for fetching ID lists. Sets the maximum timeout for initialization requests (in milliseconds). Custom URL for logging events. Client for collecting observability data. Controls the verbosity of SDK logs. Custom adapter for overrides. Configuration for specification adapters. Custom adapter for specifications. How often the SDK updates specifications from Statsig servers (in milliseconds). Custom URL for fetching feature specifications. Global custom fields to include with all evaluations. Configuration for connecting through a proxy server. The `ProxyConfig` struct has the following properties: * `proxy_host`: Option\ - Specifies the proxy server host * `proxy_port`: Option\ - Specifies the proxy server port * `proxy_auth`: Option\ - For proxy authentication (format: `"username:password"`) * `proxy_protocol`: Option\ - Specifies the protocol (e.g., `"http"`, `"https"`) * `ca_cert_path`: Option\ - Optional path to a PEM CA bundle for outbound TLS ### Example Usage ```rust theme={null} use statsig_rust::{Statsig, StatsigOptions}; use std::sync::Arc; // Initialize StatsigOptions with custom parameters let mut options = StatsigOptions::default(); options.environment = Some("development".to_string()); options.init_timeout_ms = Some(3000); options.disable_all_logging = Some(false); options.enable_id_lists = Some(true); options.output_log_level = Some(LogLevel::Info); // LogLevel enum, not a string // Pass the options object into Statsig::new() let statsig = Statsig::new("server-secret-key", Some(Arc::new(options))); statsig.initialize().await?; // Or, use the builder pattern for a more fluent interface let options = StatsigOptions::builder() .environment(Some("development".to_string())) .init_timeout_ms(Some(3000)) .disable_all_logging(Some(false)) .enable_id_lists(Some(true)) .specs_sync_interval_ms(Some(30000)) // Configure proxy settings .proxy_config(Some(ProxyConfig { proxy_host: Some("proxy.example.com".to_string()), proxy_port: Some(8080), proxy_protocol: Some("https".to_string()), proxy_auth: None, // Use Some("username:password".to_string()) if authentication is required ca_cert_path: Some("/etc/ssl/certs/corporate-ca.pem".to_string()), })) .build(); // Pass the options object into Statsig::new() let statsig = Statsig::new("server-secret-key", Some(Arc::new(options))); statsig.initialize().await?; ``` ### Proxy and Custom Network Routing Use `proxy_config` if your service needs a standard outbound HTTP proxy. If you need to route config sync through [Statsig Forward Proxy](/infrastructure/forward-proxy) or another custom source, use `spec_adapters_config` or `specs_url` instead. Set `ca_cert_path` when your environment requires a custom PEM CA bundle for outbound TLS. ## Shutting Statsig Down Because we batch and periodically flush events, some events may not have been sent when your app/server shuts down. To make sure all logged events are properly flushed, you should call `shutdown()` before your app/server shuts down: ```rust theme={null} statsig.shutdown().await?; ``` Alternatively, you can manually flush events without shutting down: ```rust theme={null} // Manually flush events to the server statsig.flush_events().await; ``` ## SDK Event Subscriptions The Statsig SDK provides an event subscription system that allows you to listen for evaluation events and lifecycle events in real-time. This feature is useful for debugging, analytics, custom logging, and integrating with external systems. ### Supported Events The SDK supports subscribing to the following evaluation events: * **`gate_evaluated`** - Fired when a feature gate is evaluated for a user * **`dynamic_config_evaluated`** - Fired when a dynamic config is retrieved for a user * **`experiment_evaluated`** - Fired when an experiment is evaluated for a user * **`layer_evaluated`** - Fired when a layer is evaluated for a user * **`specs_updated`** - Fired when the SDK updates its cached specs, including where the specs were loaded from * **`"*"`** - Subscribe to all evaluation events ### SDK Event Data Each event includes relevant context about the evaluation: * **Gate Evaluated Events** include: `gate_name`, `value` (boolean), `rule_id`, `reason` * **Dynamic Config Events** include: the full `dynamic_config` object with values and metadata * **Experiment Events** include: the full `experiment` object with variant assignment and parameters * **Layer Events** include: the full `layer` object with allocated experiment and parameters * **Specs Updated Events** include `source`, `source_api`, and `values` metadata, where `values.time` is the timestamp of the last update to the project in the Statsig console ### Use Cases Event subscriptions are particularly useful for: * **Debugging**: Monitor which features are being evaluated and their results * **Analytics**: Track feature usage patterns and user segments * **Custom Logging**: Send evaluation data to your own logging systems * **Integration**: Forward events to external analytics or monitoring tools * **Testing**: Verify that features are being evaluated as expected ### Best Practices * **Clean up subscriptions**: Always unsubscribe when you no longer need to listen for events to prevent memory leaks * **Handle event data carefully**: Event objects may contain sensitive user information depending on your configuration * **Use specific event types**: Subscribe to specific events rather than "\*" when possible for better performance * **Avoid heavy processing**: Keep event handlers lightweight to avoid impacting SDK performance ```rust theme={null} use statsig_rust::{Statsig, StatsigUserBuilder, sdk_event_emitter::SdkEvent}; let statsig = Statsig::new("server-secret-key", None)?; statsig.initialize().await?; // Subscribe to gate evaluation events let gate_sub_id = statsig.subscribe(SdkEvent::GATE_EVALUATED, |event| { if let SdkEvent::GateEvaluated { gate_name, value, rule_id, reason } = event { println!("Gate evaluated: {} = {}, rule: {}, reason: {}", gate_name, value, rule_id, reason); } }); // Subscribe to dynamic config evaluation events let config_sub_id = statsig.subscribe(SdkEvent::DYNAMIC_CONFIG_EVALUATED, |event| { if let SdkEvent::DynamicConfigEvaluated { dynamic_config } = event { println!("Config evaluated: {}", dynamic_config.name); } }); // Subscribe to experiment evaluation events let experiment_sub_id = statsig.subscribe(SdkEvent::EXPERIMENT_EVALUATED, |event| { if let SdkEvent::ExperimentEvaluated { experiment } = event { println!("Experiment evaluated: {} -> {}", experiment.name, experiment.group_name); } }); // Subscribe to layer evaluation events let layer_sub_id = statsig.subscribe(SdkEvent::LAYER_EVALUATED, |event| { if let SdkEvent::LayerEvaluated { layer } = event { println!("Layer evaluated: {}", layer.name); } }); let specs_updated_sub_id = statsig.subscribe(SdkEvent::SPECS_UPDATED, |event| { if let SdkEvent::SpecsUpdated { source, source_api, values, } = event { println!( "Specs updated from {} via {} (project last edited at {})", source, source_api, values.time ); } }); // Subscribe to all events let all_events_sub_id = statsig.subscribe(SdkEvent::ALL, |event| { println!("Event received: {}", event.get_name()); }); // Unsubscribe from specific event types statsig.unsubscribe(SdkEvent::GATE_EVALUATED); statsig.unsubscribe(SdkEvent::SPECS_UPDATED); // Unsubscribe using subscription ID statsig.unsubscribe_by_id(&config_sub_id); // Unsubscribe from all events statsig.unsubscribe_all(); ``` ## Local Overrides Local Overrides are a way to override the values of gates, configs, experiments, and layers for testing purposes. This is useful for local development or testing scenarios where you want to force a specific value without having to change the configuration in the Statsig console. ```Rust theme={null} // Overrides the given gate to the specified value statsig.override_gate("test_gate", true, None); // Overrides the given dynamic config to the provided value statsig.override_dynamic_config("test_1", my_map.clone(), None); //my_map is HashMap // Overrides the given experiment to the provided value statsig.override_experiment("test_xp_1", my_map.clone(), None); //my_map is HashMap // Overrides the given experiment to a particular groupname, available for experiments only statsig.override_experiment_by_group_name("test_xp_1", "a_group_name", None); // Overrides the given layer to the provided value statsig.override_layer("user_promo_experiments", my_map.clone(), None); //my_map is HashMap //Alternatively, get the Experiment object for a given groupName let group_exp = statsig.get_experiment_by_group_name("pricing_experiment", "premium_group"); let premium_price = group_exp.get_double("price", 9.99); ``` ## Persistent Storage The Persistent Storage interface allows you to implement custom storage for user-specific configurations. This enables you to persist user assignments across sessions, ensuring consistent experiment groups even when the user returns later. This is particularly useful for client-side A/B testing where you want to ensure users always see the same variant. ```rust theme={null} pub trait PersistentStorageTrait: Send + Sync { fn load(&self, key: &str) -> Result, PersistentStorageErrorEnum>; fn save(&self, key: &str, config_name: &str, data: &StickyValues) -> Result<(), PersistentStorageErrorEnum>; fn delete(&self, key: &str, config_name: &str) -> Result<(), PersistentStorageErrorEnum>; } ``` ## Data Store The Data Store interface allows you to implement custom storage for Statsig configurations. This enables advanced caching strategies and integration with your preferred storage systems. ```rust theme={null} pub trait DataStoreTrait: Send + Sync { fn initialize(&self) -> Result<(), DataStoreErrorEnum>; fn shutdown(&self) -> Result<(), DataStoreErrorEnum>; fn get(&self, key: &str) -> Result, DataStoreErrorEnum>; fn set(&self, key: &str, value: &str, time: Option) -> Result<(), DataStoreErrorEnum>; fn support_polling_updates_for(&self, key: &str) -> Result; } pub struct DataStoreResponse { pub result: String, pub time: Option, } ``` ## Custom Output Logger The Output Logger interface allows you to customize how the SDK logs messages. This enables integration with your own logging system and control over log verbosity. ```rust theme={null} pub trait OutputLogProvider: Send + Sync { fn initialize(&self); fn debug(&self, tag: &str, msg: String); fn info(&self, tag: &str, msg: String); fn warn(&self, tag: &str, msg: String); fn error(&self, tag: &str, msg: String); fn shutdown(&self); } ``` ## Observability Client The Observability Client interface allows you to monitor the health of the SDK by integrating with your own observability systems. This enables tracking metrics, errors, and performance data. For more information on the metrics emitted by Statsig SDKs, see the [Monitoring documentation](/sdk_monitoring). ```rust theme={null} pub trait ObservabilityClient: Send + Sync { fn init(&self); fn increment(&self, metric_name: &str, value: f64, tags: &HashMap); fn gauge(&self, metric_name: &str, value: f64, tags: &HashMap); fn dist(&self, metric_name: &str, value: f64, tags: &HashMap); fn error(&self, tag: &str, error: &str); } ``` ## Fields Needed Methods (Enterprise Only) This is available for Enterprise contracts. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. These methods allow you to retrieve a list of user fields that are used in the targeting rules for gates, configs, experiments, and layers. ### Description These methods return an array of strings representing the user fields that are referenced in the targeting rules or conditions of the specified feature. This can be useful for understanding which user properties influence a particular feature's behavior. ```rust theme={null} // Get fields needed for a gate let fields_needed: Vec = statsig.get_fields_needed_for_gate("gate_name"); // Get fields needed for a dynamic config let fields_needed: Vec = statsig.get_fields_needed_for_dynamic_config("config_name"); // Get fields needed for an experiment let fields_needed: Vec = statsig.get_fields_needed_for_experiment("experiment_name"); // Get fields needed for a layer let fields_needed: Vec = statsig.get_fields_needed_for_layer("layer_name"); ``` ### Field Mapping The fields returned by these methods correspond to the following user properties: ```rust theme={null} // Field mapping between user properties and internal field names let field_mapping = std::collections::HashMap::from([ ("userID", "u"), ("email", "e"), ("ip", "i"), ("userAgent", "ua"), ("country", "c"), ("locale", "l"), ("appVersion", "a"), ("time", "t"), ("stableID", "s"), ("environment", "en"), ("targetApp", "ta"), ]); // Custom fields are prefixed with "cf:" // Example: fields.add("cf:" + field_name); ``` ## FAQ See the guide on [device level experiments](/guides/device-level-experiments) ## Reference ### Fields Needed Methods (Enterprise Only) ```rust theme={null} // Get user fields needed for a gate evaluation pub fn get_fields_needed_for_gate(&self, gate_name: &str) -> Vec // Get user fields needed for a dynamic config evaluation pub fn get_fields_needed_for_dynamic_config(&self, config_name: &str) -> Vec // Get user fields needed for an experiment evaluation pub fn get_fields_needed_for_experiment(&self, experiment_name: &str) -> Vec // Get user fields needed for a layer evaluation pub fn get_fields_needed_for_layer(&self, layer_name: &str) -> Vec ``` This is available for Enterprise contracts. Please reach out to our support team, your sales contact, or via our [Slack community](https://statsig.com/slack) if you want this enabled. These methods allow you to retrieve a list of user fields that are used in the targeting rules for gates, configs, experiments, and layers. #### Description These methods return an array of strings representing the user fields that are referenced in the targeting rules or conditions of the specified feature. This can be useful for understanding which user properties influence a particular feature's behavior. #### Field Mapping The fields returned by these methods correspond to the following user properties: ``` // Field mapping between user properties and internal field names const fieldMapping = { userID: 'u', email: 'e', ip: 'i', userAgent: 'ua', country: 'c', locale: 'l', appVersion: 'a', time: 't', stableID: 's', environment: 'en', targetApp: 'ta', }; // Custom fields are prefixed with "cf:" // Example: fields.add('cf:' + field); ``` # Ingesting Cloudflare Logs and Metrics into Statsig Source: https://docs.statsig.com/server/concepts/cloudflare Run Statsig server SDK evaluations inside Cloudflare Workers, including configuration, request lifecycle, and how to log exposures from the edge. ## Overview This guide walks you through setting up the Cloudflare Logpush worker in your Cloudflare account and configuring it to send logs and metrics to Statsig using cURL and the Logpush API *** ## ✅ Prerequisites * [Cloudflare Logpush enabled for your account](https://developers.cloudflare.com/logs/logpush/) * [Statsig Server SDK Key](/server-core/) *** ## Configuring Logpush Worker 1. [Create api token](https://developers.cloudflare.com/fundamentals/api/get-started/create-token/) 2. Make sure token has logs edit permissions at the account level 3. [Locate cloudflare account ID](https://developers.cloudflare.com/fundamentals/account/find-account-and-zone-ids/) 4. Run the following command and get the job id ``` ACCOUNT_ID= CLOUDFLARE_API_TOKEN= curl "https://api.cloudflare.com/client/v4/accounts/$ACCOUNT_ID/logpush/jobs" \ --request POST \ --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \ --json '{ "name": "statsig-logpush-job", "destination_conf": "https://api.statsig.com/v1/log_event/cf_log_drain?header_statsig-api-key=&header_Content-Type=application%2Fjson", "dataset": "workers_trace_events", "output_options": { "field_names": [], "output_type": "ndjson", "batch_prefix": "{\"events\":[", "batch_suffix": "\n]}\n", "record_prefix": "\n {\"info\":{", "record_suffix": "}}", "record_delimiter": ",", "timestamp_format": "rfc3339" } }' ``` * In Cloudflare dashboard, navigate to "Analytics & Logs" -> "Logpush", and enable the logpush job ## Verify Logs on Statsig Console Explore your Logs and Metrics at [console.statsig.com](http://console.statsig.com) with the Logs Explorer and Metrics Explorer products under Analytics on the sidebar menu 🙂 *** ## 🔗 Resources * [Manage Logpush with cURL](https://developers.cloudflare.com/logs/logpush/examples/example-logpush-curl/) * [Logpush output types](https://developers.cloudflare.com/logs/logpush/logpush-job/log-output-options/#output-types) * [Create logpush API docs](https://developers.cloudflare.com/api/resources/logpush/subresources/jobs/methods/create/) # Server Data Stores / Data Adapter Source: https://docs.statsig.com/server/concepts/data_store Configure a custom data store adapter in Statsig server SDKs to cache rule configurations in Redis, DynamoDB, or another store you control. One common question when configuring Statsig is how to design your integration around handling potential points of failure. For example - in case of a Statsig API outage, can my integration continue to function? The short answer is yes. Your server SDK will continue to operate normally, serving the most recent set of known values in response to calls like `checkGate`, `getConfig`, `getExperiment`, and `getLayer`. Once the API is back up and operating normally, your SDK will automatically refetch the most up-to-date version of your project. But what about spinning up a new server, or a new SDK instance, while Statsig is down? This is what we built the `DataAdapter`/`DataStore` for. ## DataAdapter (or DataStore) DataAdapters allow you to plug in your own storage solution as a cache that the Statsig Server SDKs use to load your project configurations (ie; all experiments, configs, gates and their targeting/allocation rules). Some of the key use-cases for Data Adapter are: reducing dependency on Statsig servers for initialization, improving initialization time by loading config from a local data store, and providing customers with controls for minimizing network I/O. Data Adapters define a simple API that you have to implement: `initialize`, `get`, `set`, and `shutdown` for setup, reading, writing, and closing. ### Recommended Implementation In most cases, **your webservers should only implement the read path (`get`)**.\ Leaving the write path (`set`) empty is the best practice. Otherwise, every SDK instance across all of your servers will attempt to write to the store whenever it sees an update, which is inefficient and can lead to unnecessary contention or duplication. Instead, you should have a single source of truth that keeps your datastore up-to-date: * Run a **separate out-of-band service** that implements the SDK and is responsible for writing updates into the datastore via `set`. * Or, use a **cron job / periodic job** to fetch the config from the Statsig CDN endpoint and update your datastore under the correct cache keys (`statsig.cache`, `statsig.id_lists`, and `statsig.id_lists::{list_name}`). The Statsig SDK already handles this refresh logic out of the box, so separating the read and write responsibilities is the cleanest and most reliable pattern. ### Data Adapter Logic * When the Statsig SDK is initialized with a DataAdapter, it will first attempt to load config via `DataAdapter.get`. * If the entry exists, it will use it. * If the entry does not exist, and `localMode` is not enabled, it will fetch the config from Statsig servers and (if you have a writer service) persist it via `DataAdapter.set`. * Post-initialization, the SDK will continue to poll Statsig servers for updates and, when available, save them back to your data store (if you’ve implemented `set` and designated a writer service). * If the SDK client is initialized with `localMode=true`, this will disable all network fetches from Statsig. * The cache keys used to be: * `statsig.cache` → config specs * `statsig.id_lists` → lookup of id lists * `statsig.id_lists::{list_name}` → actual id list values * but in the latest node and server core sdks, the format has changed: * `statsig|{path}|{format}|{hashedSDKKey}` where `path` is `/v1/download_config_specs` or `/v1/get_id_lists`, format is `plain_text`, and the `djb2` has of the sdk key is the last bit * the SDK will handle this for you, reach out if you are trying to recreate this path yourself to double check Most DataAdapters are currently only used for the `initialize` path for getting a project definition. At the time of this writing, only `Node.js`, `Ruby`, `Go`, `Java` and `.NET` support polling for updates.\ If you're interested in using a DataAdapter as the source of truth indefinitely, please reach out in our [Slack community](https://statsig.com/slack) and let us know which language this would be useful for! For information on your specific SDK language, see the language-specific docs in the left-hand column. # Forward Proxy Source: https://docs.statsig.com/server/concepts/forward_proxy Use a forward proxy with Statsig server SDKs to route outbound traffic through your network so SDKs can run in restricted environments. ## Statsig Forward Proxy The Statsig Forward Proxy is a service that we developed to be hosted and run in your own infrastructure. If SDKs are configured to use the forward proxy, it provides a closer, but also more reliable hop for retrieving Statsig dependent configurations from within your infrastructure instead of reaching out to our networks. The expected benefits of using the proxy are: 1. Reduced dependency on Statsig Infrastructure Availability 2. Improved performance through localization of download\_config\_spec to your cluster 3. Improved cost savings and reduced network overhead through minimizing requests to Statsig Infrastructure 4. Improved consistency of configurations, as the forward proxy becomes an articulation point for serving configs 5. Support for cool features such as a streaming API through GRPC If you have any questions about how to deploy, or need assistance, feel free to reach out to us on our slack channel or create a new github issue. ### Installation #### Helm Installation The recommended way to install Statsig Forward Proxy in a Kubernetes environment is using the official Helm chart. Follow these steps to deploy: ```bash theme={null} # Add the Statsig Helm repository helm repo add statsig https://statsig-helm.storage.googleapis.com helm repo update # Install the chart helm install statsig-forward-proxy statsig/statsig-forward-proxy ``` The Helm chart provides extensive configuration options for customizing your deployment. For detailed configuration options, refer to the [Statsig Forward Proxy Helm chart documentation](https://github.com/statsig-io/statsig-forward-proxy/blob/main/chart/README.md). #### Manual Deployment For environments where Helm is not available or for more customized deployments, Statsig Forward Proxy can be deployed manually. The proxy is available as a pre-built Docker image, and you can also build your own binary from source. For detailed instructions on manual deployment options and available configuration parameters, refer to the [Manual Deployment section in the GitHub repository](https://github.com/statsig-io/statsig-forward-proxy/tree/main?tab=readme-ov-file#manual-deployment). ### How Forward Proxy works The forward proxy works by setting up a local http or grpc server. When requests are made to the forward proxy, only the initial request is blocking, after that, a background loop is spawned which keeps configurations up to date without impacting the hot path for serving configurations. While providing this capability, the forward proxy also provides other benefits such the ability to enable backup caches and monitoring through technologies such as Redis and Statsd. ## Integration with SDK `Legacy Java/Kotlin`, `Legacy Python`, `Python Server Core` and `Node Server Core` SDKs provide the option for Forward Proxy integration with GRPC streaming. For other server SDKs, you can integrate with the Forward Proxy via HTTP by overriding the initialize endpoints in StatsigOptions. Feel free to consult us in [Slack](https://statsig.com/slack) on this approach. You can now configure SDK network using different network protocol to integrate with Statsig Forward Proxy for different endpoints. There are 3 network endpoints SDK make requests to download\_config\_specs,get\_id\_lists, and log\_events, and you can currently configure the download\_config\_specs to use the proxy. We are actively developing the latter two. #### Network protocol for different endpoints For `download_config_specs` endpoint, where we get specs on evaluating gates/layers/experiments/configs 1. `http`: the default protocol. If SDK is initialized with http, it will poll config updates in background thread. 2. `grpc_websocket`: Establish a grpc streaming from SDK to statsig forward proxy, and listen to updates being pushed from proxy servers. You have to use statsig forward proxy or have a similar proxy server in order to use grpc streaming. Note: Not all SDKs support GRPC yet, if you have an SDK you want support for please reach out to our support channel on slack. 3. `grpc`: Unary RPC, behave similarly to HTTP protocol, after initialization, SDK poll changes from forward proxy server in background thread. #### Listen to Config Change Example ```typescript theme={null} const statsigOptions = { proxyConfigs: { download_config_specs: { proxyAddress: proxyAddress // Your proxy address protocol: "grpc_websocket" } } } Statsig.initialize(server_key, statsigOptions) // Statsig will use listen for config updates from statsig forward proxy using grpc_websocket protocol. And use http to get idlists and post log events from statsig servers. ``` For information on your specific SDK language, see the language specific docs in the left hand column. #### Failover behavior **Initialization** SDK remains the same behavior when forward proxy is being used. But there are several configurations you can use to improve the data availability during initialization. Default behavior is SDK will get from forward proxy. If DataAdapter is presented, get from DataAdapter first, and fallback to Forward Proxy if there is no value served from data adapter. There are several ways you can configure the behavior here. 1. set fallbackToStatsigAPI = true. If you want to fallback to statsig api when initialization or config sync from original sources is false 2. If you want to customize your own order of initialization set initializeSources to have multiple sources,for example initializeSources=\[DataSource.Network, DataSource.DataAdapter, DataSource.StatsigNetwork] will fetch from 3 sources sequentially until it successfully find one **Config Sync (Post initialization)** If sdk is using pulling model, `grpc` or `http` 1. Set fallbackToStatsigAPI=true 2. Set configSyncSources to have multiple sources,for example initializeSources=\[DataSource.Network, DataSource.StatsigNetwork] will fetch from 3 sources sequentially until it successfully find one (Note if you are using streaming, config sync sources does not matter here) If sdk is using `grpc_websocket` protocol, meaning sdk is in a listening mode for any config changes, by default it will 1. Retry connecting to forward proxy, with exponential backoff, 10s, 50s, 250s, 1250s ... until it reconnects, limit for retry is 10 times. 2. If 4th retry still fails, sdk will start polling from Statsig endpoint. Use StatsigOptions.pollingInterval to control how often it 3. While sdk polling from Statsig endpoint, sdk will keep retrying until it connected, Everything mentioned above is configurable with StatsigOptions #### Streaming failover in depth Within SDK, when exceptions being thrown, for example, connection is dropped or forward proxy server is down, grpc will return error to SDK and we start falling back behavior. Python will check if the channel is idle every 2 hours to ensure channel is not idle, this is to prevent some unknown corner case behavior within python grpc library. #### TLS -- Advanced network authentication Forward proxy support TLS and mTLS when use GRPC server, so all grpc network (streaming and unary call) are encrypted To enable it: 1. [Setup forward proxy server with valid certifications](https://github.com/statsig-io/statsig-forward-proxy?tab=readme-ov-file#deploying) 2. [Setup SDK with valid certifications (python example)](/server/pythonSDK). If certifications are misconfigured, SDK will treat as exception and started failing over behavior that you configured. For example: fallback to Statsig API, retry connection, and start fall back behavior. #### Coordinate SDK DataAdapter / DataStore with Forward Proxy Cache service In order to increase reliability of SDK during initialization, setup DataStore will help in case Forward Proxy is not available when initialized. If DataStore is set, the SDK will try to initialize with values from dataStore, and sync (listen or polling depends on setup) from forward proxy in background after initialization. It's recommended for SDK and Forward Proxy to share the same cache service for consistency and reduce cache I/O. Example on DataStore setup in python: ```py theme={null} class DataAdapter(IDataStore): def __init__(self): self.redis_cache = ExampleCache() def get(self, key: str): # IDlist isn't currently supported by proxy, so do normal lookup if "statsig.id_lists" in key: return self.cache.get(key) # This logic must stay in sync with statsig-forward-proxy else: hashed_key = "statsig::" + hashlib.sha256(key.encode()).hexdigest() return self.cache.hget(hashed_key, 'config') def set(self, key: str): # Don't implement set method if you are share the same cache between forward proxy and sdk, forward proxy will write to cache pass def shutdown(self): self.cache.shutdown() ``` #### Deployment Notes Given that every use case for the proxy is different, we don't have strict recommendations. However, we do have general recommendations: 1. Understand what your general QPS to Statsig is. If you need help understanding this, feel free to reach out. 2. If possibly using your payload, get a sense of what throughput can be supported by a single pod and then calculate how much you need to scale out by to support your traffic. 3. Pre-scale the proxy for your max QPS and then begin to gradually rollout the proxy to services. 4. Configure your auto-scaling, for most folks, scaling up at 70% utilization for CPU and memory is typically sufficient, but depending on your usage this might need to be tweaked. If you are seeing scaling issues that can't be resolved by scaling out horizontally, we recommend fronting it with nginx to help with flow control. If you need any help, we are happy to collaborate and work closely, just send a message to our [slack channel](https://statsig.com/slack) to get further assistance. # Ingesting Open Telemetry Data Source: https://docs.statsig.com/server/concepts/open_telemetry Instrument Statsig server SDKs with OpenTelemetry to capture spans, metrics, and logs for SDK initialization, evaluation, and event logging. Setting up OTEL can be tricky, if you have any questions, feel free to reach out to us on slack! ## Overview This guide walks you through setting up the OpenTelemetry Collector in your Kubernetes environment and configuring it to send telemetry data to Statsig using the official `otlphttp` exporter *** ## ✅ Prerequisites * A running Kubernetes cluster (e.g., GKE, EKS, Minikube, etc.) * Helm installed, or another mechanism to apply helm charts to your cluster, like ArgoCD * API Access to a Statsig project with a Server SDK Secret Key * Optional: `kubectl` configured and connected to the cluster, for validating and debugging setup *** ## 1. Setup OpenTelemetry Collector on Kubernetes Our recommend setup follows the official OpenTelemetry Kubernetes guide to install the OpenTelemetry Collector in both **DaemonSet** and **Deployment** modes, using the official Helm Chart. **Note:** You'll find well-tested and ready-to-use `values.yaml` files for both deployments modes in Step 3! **📘 Official Guide:** [OpenTelemetry Collector for Kubernetes – Getting Started](https://opentelemetry.io/docs/platforms/kubernetes/getting-started/) ### Why both? * **DaemonSet Collector**: Runs one instance per node to collect host-level telemetry (e.g., logs, pod and node metrics). * **Deployment Collector**: Runs one instance per cluster to gather telemetry related to the cluster as a whole. We recommend configuring these receivers at the minimum, for a useful observability platform for your kubernetes workloads powered by Statsig. We make use of the powerful [Presets](https://opentelemetry.io/docs/platforms/kubernetes/helm/collector/#presets) provided by the official chart, but also customize some components as can be seen in the sample values files in Step 3. * Minimum for scraping logs: * [Filelog Receiver](https://opentelemetry.io/docs/platforms/kubernetes/getting-started/#filelog-receiver) * [Kubernetes Attributes Processor](https://opentelemetry.io/docs/platforms/kubernetes/getting-started/#kubernetes-attributes-processor) * Minimum for scraping kubernetes metrics * [KubeletStats Receiver](https://opentelemetry.io/docs/platforms/kubernetes/getting-started/#kubeletstats-receiver) * [Kubernetes Cluster Receiver](https://opentelemetry.io/docs/platforms/kubernetes/getting-started/#kubernetes-cluster-receiver) * Minimum for metrics about Otel Collector itself * [OTLP Receiver](https://opentelemetry.io/docs/platforms/kubernetes/getting-started/#otlp-receiver) for [scraping internal metrics](https://opentelemetry.io/docs/collector/internal-telemetry/#configure-internal-metrics) (alternatively scraped through prometheus receiver) *** ## 2. Configure exporting telemetry to Statsig with OTLP HTTP Exporter Statsig supports receiving OTLP formatted, JSON encoded telemetry at the endpoint `https://api.statsig.com/otlp`. The OpenTelemetry Collector supports sending the scraped telemetry to the Statsig endpoint via the official `otlphttp` exporter. ### Authentication Requests are authenticated using request header `statsig-api-key` and a valid SDK Server Secret key generated from [console.statsig.com](http://console.statsig.com) (under Settings > Keys & Environments). It is recommended to keep your secret key secure with a Kubernetes Secret management provider and make it available to access securely from the open telemetry collector pod’s environment. ### Example Exporter Config ```yaml theme={null} exporters: otlphttp: endpoint: https://api.statsig.com/otlp encoding: json headers: statsig-api-key: ${env:STATSIG_SERVER_SDK_SECRET} ``` ## 3. Example Helm Chart Values for a quick and correct setup We provide two complete and tested `values.yaml` configurations for use with the OpenTelemetry Helm charts: ### 🔗 Deployment Collector 👉 [values-gateway.yaml](https://gist.github.com/karan-statsig/d5c70b7fd100ab445985b620dcb7e8c6) ### 🔗 DaemonSet Collector 👉 [values-agent.yaml](https://gist.github.com/karan-statsig/f02ee95fef00aecde402829b5a4c60e8) Use them with the Helm chart version `0.75.1` like this: ```yaml theme={null} helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm repo update # Install Deployment Collector helm install otel-deployment open-telemetry/opentelemetry-collector \ --version 0.75.1 \ -f values-gateway.yaml -n otel --create-namespace # Install DaemonSet Collector helm install otel-daemonset open-telemetry/opentelemetry-collector \ --version 0.75.1 \ -f values-agent.yaml -n otel ``` ## 4. Verify the Setup Check that all pods are running: ```bash theme={null} kubectl get pods -n otel ``` Check logs for a specific collector pod and confirm there are errors reported: ```bash theme={null} kubectl logs -n otel $pod_name ``` ## 5. Explore Explore your Logs and Metrics at [console.statsig.com](http://console.statsig.com) with the Logs Explorer and Metrics Explorer products under Analytics on the sidebar menu 🙂 *** ## 🔗 Resources * [OpenTelemetry Collector Documentation](https://opentelemetry.io/docs/collector/) * [Important Concepts for Kubernetes](https://opentelemetry.io/docs/platforms/kubernetes/collector/components/) * [Helm Chart Reference](https://github.com/open-telemetry/opentelemetry-helm-charts) * [Collector Configuration Reference](https://opentelemetry.io/docs/collector/configuration) * [OTLP Protocol Specification](https://opentelemetry.io/docs/specs/otlp/) # Server Persistent Assignment Source: https://docs.statsig.com/server/concepts/persistent_assignment Configure persistent assignment in Statsig server SDKs so users stay in the same experiment group across requests, sessions, and devices over time. Persistent assignment allows you to ensure that a user's variant stays consistent while an experiment is running, regardless of changes to allocation or targeting. ## Persistent Storage Adapter The persistent storage adapter allows you to plug in your own storage solution that Statsig SDK uses to persist user assignments. The storage interface consists of just a `load` and `save` API for read/write operations. Currently only supported in `Go`, `Ruby`, `Legacy Node`, `Node Core`, `Java Core`, `Kotlin`, `.Net`, `Python Core`, `PHP Core`, `Rust Core` ### Persistent Storage Logic * Providing a storage adapter on Statsig initialization will give the SDK access to read & write on your custom storage * Providing user persisted values to `get_experiment` will inform the SDK to * **save** the evaluation of the current user **on first evaluation** * Will only save when experiment / layer **is active** * **load** the previously saved evaluation of a persisted user **on subsequent evaluations** * **CAVEAT** Persisted Value will be deleted when: * When you provided call `getExperiment` with `user_persisted_values=None` * or When experiment is not active ### Persistent Assignment Options (Limited SDK Support) * **Enforce Targeting**: `boolean`, default: `false` * Whether or not to enforce targeting rules before assigning persisted values ```kotlin theme={null} val options = GetExperimentOptions( ... persistentAssignmentOptions = PersistentAssignmentOptions( enforceTargeting = true, ) ) ``` ```ts theme={null} const options: GetExperimentOptions = { ... persistentAssignmentOptions: { enforceTargeting: true, } } ``` ### Example usage ```ruby theme={null} Statsig.initialize( 'secret-key', StatsigOptions.new( user_persistent_storage: DummyPersistentStorageAdapter.new ) ) persisted_user = StatsigUser.new({ 'userID' => 'test-123' }) exp = Statsig.get_experiment( # User gets saved to persisted storage persisted_user, 'active_experiment', Statsig::GetExperimentOptions.new( user_persisted_values: Statsig.get_user_persisted_values(persisted_user, 'userID') ) ) puts exp.group_name # 'Control' exp = Statsig.get_experiment( # User evaluates using values from persisted storage StatsigUser.new({'userID' => 'unknown'}), 'active_experiment', Statsig::GetExperimentOptions.new( user_persisted_values: Statsig.get_user_persisted_values(persisted_user, 'userID') ) ) puts exp.group_name # 'Control' ``` ```python theme={null} from statsig_python_core import Statsig, StatsigUser, StatsigOptions, ExperimentEvaluationOptions, PersistentStorage options = StatsigOptions(persistent_storage = MyPersistentStorage()) statsig = Statsig.initialize(options).wait user = StatsigUser("a-user") exp = statsig.get_experiment(StatsigUser("a-user"), ExperimentEvaluationOptions(user_persisted_values= PersistentStorage.get_user_persisted_value(user, "user_id"))) print(f"{exp.group_name}") # control ``` ```java theme={null} PersistentStorage persistentStorage = new MyPersistentStorage(); // See https://docs.statsig.com/server-core/java-core/#persistent-storage on how to implement it StatsigOptions options = new StatsigOptions.Builder() .setPersistentStorage(persistentStorage) .build(); Statsig statsig = new Statsig("secret-key", options); statsig.initialize().get(); StatsigUser persistedUser = new StatsigUser.Builder().setUserID("test-123").build(); Map persistedValues = persistentStorage.getValuesForUser(persistedUser, "userID"); if (persistedValues == null) { persistedValues = new HashMap<>(); } Experiment exp = statsig.getExperiment( persistedUser, "active_experiment", new GetExperimentOptions(persistedValues) ); System.out.println(exp.getGroupName()); // "Control" StatsigUser unknownUser = new StatsigUser.Builder().setUserID("unknown").build(); Map valuesForPersistedUser = persistentStorage.getValuesForUser(persistedUser, "userID"); Experiment persistedExp = statsig.getExperiment( unknownUser, "active_experiment", new GetExperimentOptions(valuesForPersistedUser) ); System.out.println(persistedExp.getGroupName()); // "Control" ``` ```typescript theme={null} let persistedStorage = new MyPersistentStorage() // See https://docs.statsig.com/server-core/node-core/#persistent-storage on how to implement it let options = new StatsigOptions(persistentStorage = persistedStorage) let statsig = new Statsig(secretKye, options) let user = new StatsigUser("a-user") let exp = statsig.getExperiment(user, ExperimentEvaluationOptions(userPersistentValues= persistedStorage.getUserPersistedValues(user, "user_id"))) ``` ```php theme={null} $persistent_storage = new MyPersistentStorage(); // See https://docs.statsig.com/server-core/php-core/#persistent-storage on how to implement it $options = new StatsigOptions(persistent_storage: $persistent_storage); $statsig = new Statsig("secret-key", $options); $statsig->initialize(); $persisted_user = new StatsigUser("test-123"); $persisted_values = $persistent_storage->getValuesForUser($persisted_user, "userID") ?? []; $exp = $statsig->getExperiment( $persisted_user, "active_experiment", ["user_persisted_values" => $persisted_values], ); echo $exp->groupName; // "Control" $unknown_user = new StatsigUser("unknown"); $values_for_persisted_user = $persistent_storage->getValuesForUser($persisted_user, "userID"); $persisted_exp = $statsig->getExperiment( $unknown_user, "active_experiment", ["user_persisted_values" => $values_for_persisted_user], ); echo $persisted_exp->groupName; // "Control" ``` ```rust theme={null} let persistent_storage = Arc::new(MyPersistentStorage::new()); // See https://docs.statsig.com/server-core/rust-core/#persistent-storage on how to implement it let options = StatsigOptions::builder() .persistent_storage(Some(persistent_storage.clone())) .build(); let statsig = Statsig::new("secret-key", Some(Arc::new(options))); statsig.initialize().await; let persisted_user = StatsigUser::with_user_id("test-123"); let persisted_values = persistent_storage .get_values_for_user(&persisted_user, "userID") .unwrap_or_default(); let exp = statsig.get_experiment_with_options( &persisted_user, "active_experiment", ExperimentEvaluationOptions { user_persisted_values: Some(persisted_values), ..Default::default() }, ); println!("{}", exp.group_name.as_deref().unwrap_or("")); // "Control" let unknown_user = StatsigUser::with_user_id("unknown"); let values_for_persisted_user = persistent_storage.get_values_for_user(&persisted_user, "userID"); let persisted_exp = statsig.get_experiment_with_options( &unknown_user, "active_experiment", ExperimentEvaluationOptions { user_persisted_values: values_for_persisted_user, ..Default::default() }, ); println!("{}", persisted_exp.group_name.as_deref().unwrap_or("")); // "Control" ``` ```kotlin theme={null} runBlocking { Statsig.initialize( "secret-key", StatsigOptions(userPersistentStorage = MyPersistentStorageAdapter()) ) } val persistedUser = StatsigUser("test-123") var exp = Statsig.getExperimentSync( persistedUser, "active_experiment", GetExperimentOptions( userPersistedValues = Statsig.getUserPersistedValues(persistedUser, "userID"), ), ) println(exp.groupName) // "Control" exp = Statsig.getExperimentSync( StatsigUser("unknown"), "active_experiment", GetExperimentOptions( userPersistedValues = Statsig.getUserPersistedValues(persistedUser, "userID"), ), ) println(exp.groupName) // "Control" ``` ```ts theme={null} await Statsig.initialize( "secret-key", { userPersistentStorage: new MyPersistentStorageAdapter() } ) const persistedUser: StatsigUser = { userID: "123" } let exp = Statsig.getExperimentSync( persistedUser, "active_experiment", { userPersistedValues: Statsig.getUserPersistedValues(user, "userID") }, ) console.log(exp.getGroupName()) // "Control" exp = Statsig.getExperimentSync( { userID: "unknown" }, "active_experiment", { userPersistedValues: Statsig.getUserPersistedValues(user, "userID") }, ) console.log(exp.getGroupName()) // "Control" ``` ```go theme={null} InitializeWithOptions( "secret-key", &Options{ UserPersistentStorage: persistentStorage, } ) persistedUser := User{UserID: "123"} exp := GetExperimentWithOptions( persistedUser, "active_experiment", &GetExperimentOptions{ PersistedValues: GetUserPersistedValues(persistedUser, "userID") } ) fmt.Println(exp.GroupName) // "Control" exp = GetExperimentWithOptions( User{UserID: "unknown"}, "active_experiment", &GetExperimentOptions{ PersistedValues: GetUserPersistedValues(persistedUser, "userID") } ) fmt.Println(exp.GroupName) // "Control" ``` ```csharp theme={null} var options = new StatsigServerOptions(); options.UserPersistentStorage = new MyPersistentStorageAdapter() await StatsigServer.Initialize("server-secret-key", options); var persistedUser = new StatsigUser { UserID = "123" }; var values = await StatsigServer.GetUserPersistedValues(persistedUser, "userID"); var getExpOptions = new StatsigGetExperimentOptions(values); var exp = StatsigServer.GetExperimentSync(persistedUser, "active_experiment", getExpOptions); Console.WriteLine(exp.GroupName); // "Control" var newValues = await StatsigServer.GetUserPersistedValues(persistedUser, "userID"); var newGetExpOptions = StatsigGetExperimentOptions(newValues); var newExp = StatsigServer.GetExperimentSync(new StatsigUser {UserID = "unknown"}, "active_experiment", newGetExpOptions); Console.WriteLine(newExp.GroupName); // "Control" ``` # C++ Server SDK Source: https://docs.statsig.com/server/cpp Use the Statsig C++ Server SDK to evaluate feature gates, run experiments, and log events from native C++ backend services with low-latency evaluation. View the C++ SDK source code and releases ## Setup the SDK If you are using CMake, add the following to a `.cmake` file ```cmake theme={null} FetchContent_Declare(statsig GIT_REPOSITORY https://github.com/statsig-io/cpp-server-sdk.git GIT_TAG v0.1.0 ) FetchContent_MakeAvailable(statsig) ``` And include the following in your `CMakeLists.txt` file ```cmake theme={null} cmake_minimum_required(VERSION 3.11) include(FetchContent) include(${CMAKE_CURRENT_SOURCE_DIR}/cmake/statsig.cmake) ``` Check out the latest versions on [https://github.com/statsig-io/cpp-server-sdk/releases/latest](https://github.com/statsig-io/cpp-server-sdk/releases/latest) After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```cpp theme={null} #include statsig::initialize('server-secret-key'); // Or, if you want to initialize with certain options statsig::Options options; options.localMode = true statsig::initialize('server-secret-key', options) ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```cpp theme={null} statsig::User user; user.userID = "some_user_id" if (statsig::checkGate(user, 'use_new_feature')) { // Gate is on, enable new feature } else { // Gate is off } ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```cpp theme={null} statsig::DynamicConfig config = statsig::get_config(user, 'awesome_product_details') auto item_name = config.value['product_name']; auto price = config.value['price']; auto shouldDiscount = config.value['discount']; ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```cpp theme={null} // Values via getLayer statsig::Layer layer = statsig::getLayer(user, "user_promo_experiments") auto title = layer.get("title", "Welcome to Statsig!") auto discount = layer.get("discount") // or, via getExperiment statsig::DynamicConfig title_exp = statsig::getExperiment(user, "new_user_promo_title") statsig::DynamicConfig price_exp = statsig::getExperiment(user, "new_user_promo_price") title = title_exp.value["title"] discount = price_exp.value["discount"] ... price = msrp * (1 - discount) ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```cpp theme={null} statsig::logEvent(user, 'add_to_cart') ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options `initialize()` takes an optional parameter `options` in addition to the secret key that you can provide to customize the Statsig client. Here are the current options and we are always adding more to the list: You can specify optional parameters with `options` when initializing. * **api** string, default `"https://statsigapi.net/v1"` * The base url to use for network requests from the SDK * **rulesetsSyncIntervalMs**: int, default `10000` * The interval to poll for changes to your gate and config definition changes * **loggingIntervalMs**: int, default `60000` * The default interval to flush logs to Statsig servers * **loggingMaxBufferSize**: int, default `1000`, can be set lower but anything over 1000 will be dropped on the server * The maximum number of events to batch before flushing logs to the server * **localMode**: bool, default `false` * Restricts the SDK to not issue any network requests and only respond with default values (or local overrides) ID Lists are currently not supported in the C++ server SDK ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```cpp theme={null} statsig::shutdown() ``` ## Local Overrides You can override the values returned by the SDK for testing purposes. This can be useful for local development when you want to test specific scenarios. ```cpp theme={null} // Adding gate overrides statsig::overrideGate("a_gate_name", true) // Adding config overrides std::unordered_map overrideValue = { {"overridden key", "overridden field"}, }; statsig::overrideConfig("a_config_name", overrideValue) ``` ## FAQ #### How do I run experiments for logged out users? See the guide on [device level experiments](/guides/first-device-level-experiment) ## Reference ### User ```cpp theme={null} struct User { std::string userID; std::string email; std::string ipAddress; std::string userAgent; std::string country; std::string locale; std::string appVersion; std::unordered_map custom; std::unordered_map privateAttribute; std::unordered_map statsigEnvironment; std::unordered_map customIDs; }; inline bool operator==(User const &a, User const &b) { return a.userID == b.userID && a.email == b.email && a.ipAddress == b.ipAddress && a.userAgent == b.userAgent && a.country == b.country && a.locale == b.locale && a.appVersion == b.appVersion && a.custom == b.custom && a.privateAttribute == b.privateAttribute && a.statsigEnvironment == b.statsigEnvironment && a.customIDs == b.customIDs; }; ``` ### Options ```cpp theme={null} struct Options { std::string api; bool localMode; int rulesetsSyncIntervalMs; int loggingIntervalMs; int loggingMaxBufferSize; Options() : api("https://statsigapi.net"), localMode(false), rulesetsSyncIntervalMs(10 * 1000), loggingIntervalMs(60 * 1000), loggingMaxBufferSize(1000){}; }; ``` # Deprecation Notices Source: https://docs.statsig.com/server/deprecation-notices Deprecation notices for Statsig server SDKs, including end-of-life versions, breaking changes, and recommended upgrade paths to Server Core SDKs. ## Direct Initialization API Access for Server SDKs, October 31, 2025 #### What's changing? Until mid-2024, Statsig's Server SDKs downloaded configuration values directly from Statsig servers upon startup. Since then, we've transitioned to hosting these configuration files on a secure CDN, which greatly reduces initialization time and increases reliability (we use Cloudflare, which has near-zero downtime.) As we've transitioned to this approach, we've made the decision to deprecate direct API access, which is only available in long outdated SDK versions. We're asking all customers to discontinue using these SDK versions, and upgrade to a newer version. #### Change required The only change required is a minor SDK version bump, no code changes are needed. You'll need to bump your SDK version to at least: * **go-sdk** 1.30.3 * **java-server** 1.9.0 * **py-server** 0.28.0 * **ruby-server** 1.29.0 * **statsig-node** 5.25.0 * **.NET** 2.4.1 * **Node-lite** 0.5.0 * **PHP** 3.7.2 We're asking all customers to make this change by October 31st, 2025. CDN access is only available in a more recent version of the PHP and .NET SDKs, so we won't strictly enforce an upgrade for those 2 SDKs. But we still recommend you upgrade as soon as possible. ## New Method for Country Resolution Starting December 9th, 2024, Statsig will **switch to a method of resolving IP Addresses to country codes in our client SDKs with greatly improved accuracy**, which may result in different gate behavior for some users. Statsig currently depends on a [homegrown package](https://github.com/statsig-io/ip3country) for country resolution on both Client and Server SDKs, which lacks IPV6 support. To resolve countries without bloating our SDK sizes, we'll begin to rely on our cloud provider's country resolution when serving requests to the /initialize endpoint. ### Potential Changes in Client Evaluations This may result in different evaluations for some client-side checks, which will be significantly more accurate and complete than the previous method. You may see a greater number of users passing country rules, as this new method will begin to resolve countries for IPV6 users, which now represent a large share of many customers' user base. ### Conflicts with Server-Side Checks If you check some configs on both the Client and Server side, it is possible that a small number of users may pass targeting on the client side and fail on the server, or vice versa, as we'll continue to rely on [IP3Country](https://github.com/statsig-io/ip3country) for server-side country resolution when a country is not explicitly provided on the `StatsigUser` object. We recommend using your cloud provider's country resolution (often available in a load balancer) and setting that as the "country" field on the `StatsigUser` object. This applies to any evaluations for gates/configs/experiments/layers, and also for `getClientInitializeResponse` if you are generating the payload for your client sdk. ## Async Evaluation Functions #### Reason Server SDKs were originally designed for maximum backwards compatibility. This meant that if a Server SDK did not contain support for an operator or configuration, it would fallback to hitting Statsig's servers, ensuring a valid result would be returned. This is why all top level evaluation related functions were asynchronous. In practice however, this is seldom required, so we no longer feel the trade off for sync vs async functions is worth it and will be removing the asynchronous functions in newer releases. #### Example ```java theme={null} var result = await Statsig.checkGate("my_gate"); // Bad // For node v6.0.0 var result = Statsig.checkGate("my_gate") // Good // For .Net and Java var result = Statsig.checkGateSync("my_gate"); // Good ``` #### SDKs .NET Server - [v1.20.0](https://github.com/statsig-io/dotnet-sdk/releases/tag/v1.20.0) NodeJS Server - [v5.10.0](https://github.com/statsig-io/node-js-server-sdk/releases/tag/v5.10.0) Java/Kotlin Server - [v1.12.0](https://github.com/statsig-io/java-server-sdk/releases/tag/v1.12.0) # Legacy .NET Server SDK Source: https://docs.statsig.com/server/dotnet Statsig's legacy .NET Server SDK for evaluating feature gates, experiments, and dynamic configs in C# backend services. New projects should use .NET Core SDK. [Github Repository](https://github.com/statsig-io/dotnet-sdk) ## Setup the SDK The package is hosted on [Nuget](https://www.nuget.org/packages/Statsig/). You can either install it from your Visual Studio's Nuget package manager, or through the NuGet CLI: ```bash theme={null} nuget install Statsig ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```csharp theme={null} using Statsig; using Statsig.Server; await StatsigServer.Initialize( "server-secret-key", // optionally customize the SDKs configuration via StatsigOptions new StatsigOptions( environment: new StatsigEnvironment(EnvironmentTier.Development) ) ); ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```csharp theme={null} var user = new StatsigUser { UserID = "some_user_id", Email = "user@email.com" }; var useNewFeature = await StatsigServer.CheckGate(user, "use_new_feature"); if (useNewFeature) { // Gate is on, enable new feature } else { // Gate is off } ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```csharp theme={null} var config = await StatsigServer.GetConfig(user, "awesome_product_details"); var itemName = config.Get("product_name", "Awesome Product v1"); var price = config.Get("price", 10.0); ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```csharp theme={null} // Values via GetLayer var layer = await StatsigServer.GetLayer(user, "user_promo_experiments"); var title = layer.Get("title", "Welcome to Statsig!"); var discount = layer.Get("discount", 0.1); // or, via GetExperiment var experiment = await StatsigServer.GetExperiment(user, "new_user_promo"); var expTitle = experiment.Get("title", "Welcome to Statsig!"); var expDiscount = experiment.Get("discount", 0.1); var price = msrp * (1 - discount); ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```csharp theme={null} StatsigServer.LogEvent(user, "add_to_cart", "SKU_12345", new Dictionary { { "price", "9.99" }, { "item_name", "diet_coke_48_pack" } }); ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```csharp theme={null} await StatsigServer.Shutdown(); ``` # Legacy Erlang/Elixir Server SDK Source: https://docs.statsig.com/server/erlang Statsig's legacy Erlang and Elixir Server SDK for evaluating feature gates, experiments, and dynamic configs in BEAM-based backend services and apps. Support for the Legacy Erlang SDK ends April 30, 2026. Migrate to the [new Elixir SDK](/server-core/elixir-core) soon. The erlang SDK repository, and this docs site, are a work in progress. If you are trying to use Statsig in erlang or elixir, please reach out to our support team, your sales contact, or in our [Slack community](https://statsig.com/slack). [Github Repository](https://github.com/statsig-io/erlang-sdk) ## Setup the SDK Add a dependency on statsig to your: **mix.exs**: ```elixir theme={null} {:statsig, "~> 0.0.1"} ``` **rebar.config**: ```erlang theme={null} {statsig, "0.0.1"}. ``` **erlang.mk**: ```makefile theme={null} dep_statsig = hex 0.0.1 ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```erlang Erlang theme={null} statsig:initialize(<<"secret-key">>). % or with options Options = #{environment => #{tier => <<"staging">>}}, statsig:initialize(<<"secret-key">>, Options). ``` ```elixir Elixir theme={null} Statsig.initialize("secret-key") # or with options options = %{environment: %{tier: "staging"}} Statsig.initialize("secret-key", options) ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```erlang Erlang theme={null} User = #{<<"userID">> => <<"user-id">>}, GateValue = statsig:check_gate(User, <<"gate_name">>). ``` ```elixir Elixir theme={null} user = %{"userID" => "user-id"} gate_value = Statsig.check_gate(user, "gate_name") ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```erlang Erlang theme={null} Config = statsig:get_config(User, <<"config_name">>), Value = statsig_config:get(Config, <<"param">>, <<"default">>). ``` ```elixir Elixir theme={null} config = Statsig.get_config(user, "config_name") value = StatsigConfig.get(config, "param", "default") ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```erlang Erlang theme={null} Layer = statsig:get_layer(User, <<"layer_name">>), ParamValue = statsig_layer:get(Layer, <<"param">>, <<"default">>). ``` ```elixir Elixir theme={null} layer = Statsig.get_layer(user, "layer_name") param_value = StatsigLayer.get(layer, "param", "default") ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```erlang Erlang theme={null} statsig:log_event(User, <<"event_name">>). ``` ```elixir Elixir theme={null} Statsig.log_event(user, "event_name") ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```erlang Erlang theme={null} statsig:shutdown(). ``` ```elixir Elixir theme={null} Statsig.shutdown() ``` # Legacy Go Server SDK Source: https://docs.statsig.com/server/go Statsig's legacy Go Server SDK for evaluating feature gates, experiments, and dynamic configs. New Go projects should use the Go Core SDK instead. This page covers the legacy Go SDK. For new implementations, use the [Go Core SDK](/server-core/go-core) built on the Server Core framework. View the Go SDK source code and releases ## Setup the SDK via the `go get` CLI: ```bash theme={null} go get github.com/statsig-io/go-sdk ``` Or, add a dependency on the most recent version of the SDK in go.mod: ```go theme={null} require ( github.com/statsig-io/go-sdk v1.26.0 ) ``` See the [Releases tab in GitHub](https://github.com/statsig-io/go-sdk/releases) for the latest versions. After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```go theme={null} import ( statsig "github.com/statsig-io/go-sdk" ) statsig.Initialize("server-secret-key") // Or, if you want to initialize with certain options statsig.InitializeWithOptions("server-secret-key", &Options{Environment: Environment{Tier: "staging"}}) ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```go theme={null} user := statsig.User{UserID: "some_user_id"} feature := statsig.CheckGate(user, "use_new_feature") if feature { // Gate is on, enable new feature } else { // Gate is off } ``` ## Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```go theme={null} user := statsig.User{UserID: "some_user_id"} feature := statsig.GetGate(user, "use_new_feature") if feature.Value { // Gate is on, enable new feature fmt.Print(feature.EvaluationDetails.Reason) } ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```go theme={null} config := statsig.GetConfig(user, "awesome_product_details") // The 2nd parameter is the default value to be used in case the given parameter name does not exist on // the Dynamic Config object. This can happen when there is a typo, or when the user is offline and the // value has not been cached on the client. itemName := config.GetString("product_name", "Awesome Product v1"); double price = config.GetNumber("price", 10.0); bool shouldDiscount = config.GetBool("discount", false); // Or just get the whole json object backing this config if you prefer json := config.Value ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```go theme={null} // Values via getLayer layer := Statsig.GetLayer(user, "user_promo_experiments"); promoTitle := layer.GetString("title", "Welcome to Statsig!"); discount := layer.GetDouble("discount", 0.1); // or, via getExperiment titleExperiment := Statsig.GetExperiment(user, "new_user_promo_title"); priceExperiment := Statsig.GetExperiment(user, "new_user_promo_price"); promoTitle := titleExperiment.GetString("title", "Welcome to Statsig!"); discount := priceExperiment.GetNumber("discount", 0.1); ... price := msrp * (1 - discount); // getting the layer name that an experiment belongs to userPromoLayer := Statsig.GetExperimentLayer("new_user_promo_title"); ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```go theme={null} statsig.LogEvent(Event{ User: user, EventName: "add_to_cart", Value: "SKU_12345", Metadata: map[string]string{"price": "9.99","item_name": "diet_coke_48_pack"}, }) ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options `initialize()` takes an optional parameter `options` in addition to the secret key that you can provide to customize the Statsig client. Here are the current options and we are always adding more to the list: You can specify optional parameters with `options` when initializing using `InitializeWithOptions()` ```go theme={null} type Options struct { API string `json:"api"` Environment Environment `json:"environment"` LocalMode bool `json:"localMode"` ConfigSyncInterval time.Duration IDListSyncInterval time.Duration BootstrapValues string RulesUpdatedCallback func(rules string, time int64) } ``` * **Environment**: default nil * An object you can use to set environment variables that apply to all of your users in the same session and will be used for targeting purposes. * The most common usage is to set the environment tier (string), and have feature gates pass/fail for specific environments. The accepted values are "production", "staging" and "development". * **LocalMode**: default false * Restricts the SDK to not issue any network requests and only respond with default values (or local overrides) * **ConfigSyncInterval**: default 10 seconds * The interval to poll for gate/experiment/config changes. * **IDListSyncInterval**: default 1 minute * The interval to poll for ID list changes. * **BootstrapValues**: default nil * A string that represents all rules for all feature gates, dynamic configs and experiments. It can be provided to bootstrap the Statsig server SDK at initialization in case your server runs into network issue or Statsig server is down temporarily. * **RulesUpdatedCallback**: default nil * The callback that gets invoked whenever the rulesets are updated. It's called with a JSON string that represents the rulesets, and a timestamp for when the rules were updated. * **UserPersistentStorage**: IUserPersistentStorage default nil * A persistent storage adapter for running sticky experiments. * **DisableIdList**: default false * A flag to disable fetching the id list during initialization and background polling for both network and data adapter. ### Client Initialize Response Options When using `getClientInitializeResponse()`, you can specify additional options through the `GCIROptions` struct: ```go theme={null} type GCIROptions struct { IncludeLocalOverrides bool ClientKey string TargetAppID string HashAlgorithm string IncludeConfigType bool ConfigTypesToInclude []ConfigType } ``` * **IncludeLocalOverrides**: default false * When set to true, local overrides will be included in the client initialize response. * This allows you to test local changes to configurations without affecting other users. * Useful for development and testing environments. * **ClientKey**: default empty string * The client SDK key to use for the initialize response. * This key is used to identify the client application and determine which configurations it should receive. * Required when generating a client initialize response for client SDKs. * **TargetAppID**: default empty string * Specifies a target application ID to filter configurations (feature gates, dynamic configs, experiments, and layers). * When specified, the SDK will only return configurations that are targeted to this application ID. * This is useful in multi-tenant or multi-application environments where you want to ensure that only configurations relevant to a specific application are evaluated and returned. * If not specified, the SDK will attempt to determine the target app ID from the provided client key. * **HashAlgorithm**: default empty string * Specifies the hashing algorithm to use for generating stable IDs in the client. * Common values include "djb2" (default if not specified) and "sha256". * This should match the hashing algorithm used by the client SDK. * **IncludeConfigType**: default false (deprecated) * When set to true, the type of each configuration will be included in the response. * This allows clients to differentiate between different types of configurations (e.g., feature gates, dynamic configs, experiments). * Note: This option is deprecated and may be removed in future versions. * **ConfigTypesToInclude**: default empty array * An array of configuration types to include in the response. * If specified, only configurations of the specified types will be included. * Possible values include FeatureGateType, DynamicConfigType, ExperimentType, and LayerType. * If empty, all configuration types will be included (subject to other filtering options). ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```go theme={null} statsig.Shutdown() ``` ## Local Overrides You can override the values returned by the SDK for testing purposes. This can be useful for local development when you want to test specific scenarios. ```go theme={null} func OverrideGate(gate string, val bool) func OverrideConfig(config string, val map[string]interface{}) ``` ## Client SDK Bootstrapping The Statsig server SDK can be used to generate the initialization values for a client SDK. This is useful for server-side rendering (SSR) or when you want to pre-fetch values for a client. ```go theme={null} user := statsig.User{UserID: "some_user_id"} options := statsig.GCIROptions{} options.ClientKey = "client-YOUR_CLIENT_KEY" result := statsig.GetClientInitializeResponseWithOptions(user, &options) // You can then pass 'result' into a Statsig Client SDK ``` ## Data Store A data store can be used to synchronize the configuration/value downloads across multiple SDK instances, and to bootstrap the SDK in offline environments. ### Interface ```go theme={null} type IDataAdapter interface { /** * Returns the data stored for a specific key */ Get(key string) string /** * Updates data stored for each key */ Set(key string, value string) /** * Startup tasks to run before any get/set calls can be made */ Initialize() /** * Cleanup tasks to run when statsig is shutdown */ Shutdown() /** * Determines whether the SDK should poll for updates from * the data adapter (instead of Statsig network) for the given key */ ShouldBeUsedForQueryingUpdates(key string) bool } ``` ### Example Implementation ```go theme={null} type dataAdapterExample struct { store map[string]string mu sync.RWMutex } func (d *dataAdapterExample) Get(key string) string { d.mu.RLock() defer d.mu.RUnlock() return d.store[key] } func (d *dataAdapterExample) Set(key string, value string) { d.mu.Lock() defer d.mu.Unlock() d.store[key] = value } func (d *dataAdapterExample) Initialize() {} func (d *dataAdapterExample) Shutdown() {} func (d *dataAdapterExample) ShouldBeUsedForQueryingUpdates(key string) bool { return false } ``` ## User Persistent Storage User Persistent Storage is a storage adapter for running sticky experiments. It allows you to persist user assignments across sessions. ### Interface ```go theme={null} type IUserPersistentStorage interface { /** * Returns the data stored for a specific key */ Load(key string) (string, bool) /** * Updates data stored for a specific key */ Save(key string, data string) } ``` ### Example Implementation ```go theme={null} type userPersistentStorageExample struct { store map[string]string loadCalled int saveCalled int } func (d *userPersistentStorageExample) Load(key string) (string, bool) { d.loadCalled++ val, ok := d.store[key] return val, ok } func (d *userPersistentStorageExample) Save(key string, value string) { d.saveCalled++ d.store[key] = value } ``` ## Multi-Instance Usage If you need to create multiple independent instances of the Statsig SDK (for example, to use different API keys or configurations), you can use the instance-based approach: ```go theme={null} sdkInstance := NewClientWithOptions(sdkKey, &Options{}) ``` ## FAQ #### How do I run experiments for logged out users? See the guide on [device level experiments](/guides/first-device-level-experiment) #### How can I mock Statsig in tests We recommend using the [Local Override](#local-overrides) APIs in v1.3.0+, in combination with the `LocalMode` option in `StatsigOptions` to force gate/config values in test environments and remove network access to statsig servers. For example: ```go theme={null} c := NewClientWithOptions(secret, &Options{LocalMode: true}) user := User{ UserID: "123", } gateDefault := c.CheckGate(user, "any_gate") // "any_gate" is false by default c.OverrideGate("any_gate", true) // "any_gate" is now true ``` See also [https://github.com/statsig-io/go-sdk/blob/main/overrides\_test.go](https://github.com/statsig-io/go-sdk/blob/main/overrides_test.go) ## Reference ### StatsigUser ```go theme={null} // User specific attributes for evaluating Feature Gates, Experiments, and DynamicConfigs // // Learn more why a UserID or a customID is required: https://docs.statsig.com/sdks/user#why-is-an-id-always-required-for-server-sdks // PrivateAttributes are only used for user targeting/grouping in feature gates, dynamic configs, // experiments and etc; they are omitted in logs. type User struct { UserID string `json:"userID"` Email string `json:"email,omitempty"` IpAddress string `json:"ip,omitempty"` // Many jurisdictions categorize this as PII; verify whether you should log this. UserAgent string `json:"userAgent,omitempty"` Country string `json:"country,omitempty"` Locale string `json:"locale,omitempty"` AppVersion string `json:"appVersion,omitempty"` Custom map[string]interface{} `json:"custom,omitempty"` PrivateAttributes map[string]interface{} `json:"privateAttributes,omitempty"` StatsigEnvironment map[string]string `json:"statsigEnvironment,omitempty"` CustomIDs map[string]string `json:"customIDs"` } ``` ### StatsigOptions ```go theme={null} // Advanced options for configuring the Statsig SDK type Options struct { API string `json:"api"` APIOverrides APIOverrides `json:"api_overrides"` Environment Environment `json:"environment"` LocalMode bool `json:"localMode"` ConfigSyncInterval time.Duration IDListSyncInterval time.Duration LoggingInterval time.Duration LoggingMaxBufferSize int BootstrapValues string RulesUpdatedCallback func(rules string, time int64) InitTimeout time.Duration DataAdapter IDataAdapter OutputLoggerOptions OutputLoggerOptions StatsigLoggerOptions StatsigLoggerOptions EvaluationCallbacks EvaluationCallbacks DisableCDN bool // Disables use of CDN for downloading config specs UserPersistentStorage IUserPersistentStorage IPCountryOptions IPCountryOptions UAParserOptions UAParserOptions } type APIOverrides struct { DownloadConfigSpecs string `json:"download_config_specs"` GetIDLists string `json:"get_id_lists"` LogEvent string `json:"log_event"` } type EvaluationCallbacks struct { GateEvaluationCallback func(name string, result bool, exposure *ExposureEvent) ConfigEvaluationCallback func(name string, result DynamicConfig, exposure *ExposureEvent) ExperimentEvaluationCallback func(name string, result DynamicConfig, exposure *ExposureEvent) LayerEvaluationCallback func(name string, param string, result DynamicConfig, exposure *ExposureEvent) ExposureCallback func(name string, exposure *ExposureEvent) IncludeDisabledExposures bool } type OutputLoggerOptions struct { LogCallback func(message string, err error) EnableDebug bool DisableInitDiagnostics bool DisableSyncDiagnostics bool } type StatsigLoggerOptions struct { DisableInitDiagnostics bool DisableSyncDiagnostics bool DisableApiDiagnostics bool DisableAllLogging bool } type IPCountryOptions struct { Disabled bool // Fully disable IP to country lookup LazyLoad bool // Load in background EnsureLoaded bool // Wait until loaded when needed } type UAParserOptions struct { Disabled bool // Fully disable UA parser LazyLoad bool // Load in background EnsureLoaded bool // Wait until loaded when needed } // See https://docs.statsig.com/guides/usingEnvironments type Environment struct { Tier string `json:"tier"` Params map[string]string `json:"params"` } // options for getClientInitializeResponse type GCIROptions struct { IncludeLocalOverrides bool ClientKey string HashAlgorithm string //supports "sha256", "djb2", "none", default "sha256" } ``` ### Event ```go theme={null} type Event struct { EventName string `json:"eventName"` User User `json:"user"` Value string `json:"value"` Metadata map[string]string `json:"metadata"` Time int64 `json:"time"` } ``` ### FeatureGate ```go theme={null} type FeatureGate struct { Name string `json:"name"` Value bool `json:"value"` RuleID string `json:"rule_id"` IDType string `json:"id_type"` GroupName string `json:"group_name"` EvaluationDetails *EvaluationDetails `json:"evaluation_details"` } ``` ### DynamicConfig ```go theme={null} type DynamicConfig struct { Name string `json:"name"` Value map[string]interface{} `json:"value"` RuleID string `json:"rule_id"` IDType string `json:"id_type"` GroupName string `json:"group_name"` EvaluationDetails *EvaluationDetails `json:"evaluation_details"` AllocatedExperimentName string `json:"allocated_experiment_name"` GetString(key string, fallback string) string GetNumber(key string, fallback float64) float64 GetBool(key string, fallback bool) bool GetSlice(key string, fallback []interface{}) []interface{} GetMap(key string, fallback map[string]interface{}) map[string]interface{} } ``` # Legacy Java/Kotlin Server SDK Source: https://docs.statsig.com/server/java Statsig's legacy Java and Kotlin Server SDK for evaluating feature gates, experiments, and dynamic configs in JVM backend services. New projects use Java Core. These docs are for using our Java/Kotlin SDK in a multi-user, server side context. For client side android applications, check out our [Android SDK](/client/Android) or one of the other client SDKs for your client side applications. This SDK is written in Kotlin, but exposes methods and overrides to Java based applications. [Github Repository](https://github.com/statsig-io/java-server-sdk) ## Setup the SDK `v1.X.X+` of the SDK is now published only to Maven Central. To install the SDK, set the Maven Central repository in your `build.gradle`. You probably already have this for other dependencies. ```groovy theme={null} repositories { mavenCentral() } ``` Then add the dependency: ```groovy theme={null} implementation 'com.statsig:serversdk:1.X.X' // replace with the most up to date version // For >v1.24.0 If you are not using streaming and want to reduce the package size you can: implementation 'com.statsig:serversdk:1.X.X' { exclude(group = "io.grpc", module = "*") } ``` You can find the versions in the github releases of the [open source sdk repository](https://github.com/statsig-io/java-server-sdk/releases), or from the [maven central repository](https://mvnrepository.com/artifact/com.statsig/serversdk). ### Jitpack Deprecation `v0.X.X` versions of the SDK are available from jitpack, but newer versions will not be published to jitpack. If you update Statsig to be pulled from Maven Central instead of jitpack, you can remove `maven { url 'https://jitpack.io' }` if Statsig was the only library you got from jitpack and you previously relied on v0.X.X of the SDK. After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```java Java theme={null} import com.statsig.sdk.Statsig; StatsigOptions options = new StatsigOptions(); // Customize options as needed. For example: // options.initTimeoutMs = 9999; Future initFuture = Statsig.initializeAsync("server-secret-key", options); initFuture.get(); ``` ```kotlin Kotlin theme={null} import com.statsig.sdk.Statsig val options = StatsigOptions().apply { // Customize options as needed. For example: initTimeoutMs = 9999 } async { Statsig.initialize("server-secret-key", options) }.await() ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```java Java theme={null} StatsigUser user = new StatsigUser("user_id"); Boolean isFeatureOn = Statsig.checkGateSync(user, "use_new_feature"); if (isFeatureOn) { // Gate is on, use new feature } else { // Gate is off } ``` ```kotlin Kotlin theme={null} val user = StatsigUser("user_id"); val featureOn = Statsig.checkGateSync(user, "use_new_feature") if (featureOn) { // Gate is on, use new feature } else { // Gate is off } ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```java Java theme={null} DynamicConfig config = Statsig.getConfigSync(user, "awesome_product_details"); String itemName = config.getString("product_name", "Awesome Product v1"); double price = config.getDouble("price", 10.0); ``` ```kotlin Kotlin theme={null} val config = Statsig.getConfigSync(user, "awesome_product_details") val itemName = config.getString("product_name", "Awesome Product v1") val price = config.getDouble("price", 10.0) ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```java Java theme={null} // Values via getLayer Layer layer = Statsig.getLayerSync(user, "user_promo_experiments"); String title = layer.getString("title", "Welcome to Statsig!"); double discount = layer.getDouble("discount", 0.1); // or, via getExperiment DynamicConfig experiment = Statsig.getExperimentSync(user, "new_user_promo"); String expTitle = experiment.getString("title", "Welcome to Statsig!"); double expDiscount = experiment.getDouble("discount", 0.1); ``` ```kotlin Kotlin theme={null} // Values via getLayer val layer = Statsig.getLayerSync(user, "user_promo_experiments") val title = layer.getString("title", "Welcome to Statsig!") val discount = layer.getDouble("discount", 0.1) // or, via getExperiment val experiment = Statsig.getExperimentSync(user, "new_user_promo") val expTitle = experiment.getString("title", "Welcome to Statsig!") val expDiscount = experiment.getDouble("discount", 0.1) ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```java Java theme={null} Statsig.logEvent(user, "add_to_cart", "SKU_12345", Map.of("price", "9.99", "item_name", "diet_coke_48_pack")); ``` ```kotlin Kotlin theme={null} Statsig.logEvent(user, "add_to_cart", "SKU_12345", mapOf("price" to "9.99", "item_name" to "diet_coke_48_pack")) ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```java Java theme={null} FeatureGate gate = Statsig.getFeatureGateSync(user, "use_new_feature"); boolean value = gate.getValue(); String ruleId = gate.getRuleID(); EvaluationDetails details = gate.getEvaluationDetails(); ``` ```kotlin Kotlin theme={null} val gate = Statsig.getFeatureGateSync(user, "use_new_feature") val value = gate.getValue() val ruleId = gate.getRuleID() val details = gate.getEvaluationDetails() ``` ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```java Java theme={null} Statsig.shutdown(); ``` ```kotlin Kotlin theme={null} Statsig.shutdown() ``` # Legacy Node.js Server SDK Source: https://docs.statsig.com/server/nodejsServerSDK Statsig's legacy Node.js Server SDK for evaluating feature gates, experiments, and dynamic configs in Node backend services. New projects use Node Core. [Github Repository](https://github.com/statsig-io/node-js-server-sdk) ## Setup the SDK The Node.js SDK is hosted [here](https://www.npmjs.com/package/statsig-node). You can install the SDK using NPM or Yarn: ```bash npm theme={null} npm install statsig-node ``` ```bash yarn theme={null} yarn add statsig-node ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```javascript theme={null} const Statsig = require("statsig-node"); await Statsig.initialize( "server-secret-key", { environment: { tier: "staging" } } // optional, if not set, for >v6.0.0, sdk will default to be production ); ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```javascript theme={null} const user = { userID: '12345', email: '12345@gmail.com', ... }; const showNewDesign = Statsig.checkGate(user, 'new_homepage_design'); if (showNewDesign) { // show new design here } else { // show old design here } ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```javascript theme={null} const config = Statsig.getConfig(user, "awesome_product_details"); // The 2nd parameter is the default value to be used in case the given parameter name does not exist on // the Dynamic Config object. This can happen when there is a typo, or when the user is offline and the // value has not been cached on the client. const itemName = config.get("product_name", "Awesome Product v1"); const price = config.get("price", 10.0); const shouldDiscount = config.get("discount", false); ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```javascript theme={null} // Values via getLayer const layer = Statsig.getLayer(user, "user_promo_experiments"); const promoTitle = layer.get("title", "Welcome to Statsig!"); const discount = layer.get("discount", 0.1); // or, via getExperiment const promoExperiment = Statsig.getExperiment(user, "new_user_promo"); const promoTitle = promoExperiment.get("title", "Welcome to Statsig!"); const discount = promoExperiment.get("discount", 0.1); ... const price = msrp * (1 - discount); ``` ## Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```javascript theme={null} const gate = Statsig.getFeatureGate(user, 'new_homepage_design'); console.log(gate.name); // 'new_homepage_design' console.log(gate.value); // true or false console.log(gate.ruleID); // rule ID that was evaluated console.log(gate.evaluationDetails); // evaluation metadata ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```javascript theme={null} Statsig.logEvent(user, "add_to_cart", "SKU_12345", { price: "9.99", item_name: "diet_coke_48_pack", }); ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options `initialize()` takes an optional parameter `options` in addition to the secret key that you can provide to customize the Statsig client. Here are the current options and we are always adding more to the list: `initialize()` takes an optional parameter `options` in addition to the secret key that you can provide to customize the Statsig client. Here are the current options and we are always adding more to the list: The base url to use for all network requests. Defaults to the statsig API. An object you can use to set environment variables that apply to all of your users in the same session and will be used for targeting purposes. The most common usage is to set the environment tier ('production', 'staging' or 'development'), e.g. `{ tier: 'staging' }`, and have feature gates pass/fail for specific environments. Since v6.0.0 default environment tier is production A string that represents all rules for all feature gates, dynamic configs and experiments. It can be provided to bootstrap the Statsig server SDK at initialization in case your server runs into network issue or Statsig server is down temporarily. A callback function that's called whenever we have an update for the rules; it's called with a JSON string (used as is for `bootstrapValues` mentioned above) and a timestamp, like below: ``` options.rulesUpdatedCallback(specsString, timeStamp) ``` The logger interface to use for printing to stdout/stderr Disables all network access, so the SDK will only return default (or overridden) values. Useful in testing. Sets a maximum time to wait for the config download network request to resolve before considering the SDK initialized and resolving the call to `initialize()` An adapter with custom storage behavior for config specs. Can be used to bootstrap Statsig server (takes priority over `bootstrapValues`). Can also be used to continuously fetch updates in place of the Statsig network. See [Data Stores](/server/concepts/data_store). For example, see our 1P implementation via Redis [statsig-node-redis](https://github.com/statsig-io/node-js-server-sdk-redis). A persistent storage adapter for running sticky experiments. See [examples](/server/nodejsServerSDK#user-persistent-storage). Sets the polling interval for the SDK to ask Statsig backend for changes on the rulesets. Sets the polling interval for the SDK to ask Statsig backend for changes on the ID Lists. Sets the interval for the SDK to periodically flush all logging events to Statsig backend. Sets the maximum number of events the SDK's logger will batch before flushing them all to Statsig backend. Disables diagnostics events from being logged and sent to Statsig Method of initializing IP to country lookup on `statsig.initialize()`. Method of initializing ID lists on `statsig.initialize()`. The maximum number of retry attempts when sending `/log_event` requests to Statsig server A fixed number or callback on the retry attempt number to configure the time in ms to wait between each `/log_event` retry. If using a fixed number, a 10x multiplier will be applied on each subsequent retry Provides callback functions for handling custom logic during evaluations of gates, dynamic configs, experiments, or layers. You can provide specific callbacks for each evaluation type to perform tasks such as custom logging (if you prefer not to use Statsig's default logging), or side effects. **Note:** if you'd like to turn off Statsig's default logging, set `disableExposureLogging: true` when making checks. Available callbacks: ``` gateCallback?: (gate: FeatureGate, user: StatsigUser, event: LogEvent) => void; dynamicConfigCallback?: (config: DynamicConfig, user: StatsigUser, event: LogEvent) => void; experimentCallback?: (config: DynamicConfig, user: StatsigUser, event: LogEvent) => void; layerCallback?: (layer: Layer, user: StatsigUser) => void; layerParamCallback?: (layer: Layer, paramName: string, user: StatsigUser, event: LogEvent) => void; ``` ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```javascript theme={null} statsig.shutdown(); ``` ## Flush To manually flush logged events: ```javascript theme={null} await statsig.flush(); ``` ## Client SDK Bootstrapping The Statsig server SDK can be used to generate the initialization values for a client SDK. This is useful for server-side rendering (SSR) or when you want to pre-fetch values for a client. ```typescript theme={null} const values = Statsig.getClientInitializeResponse(user); // Record | null if (values != null) { // Bootstrap the Statsig React Client SDK return ; } ``` ## Local Overrides You can override the values returned by the SDK for testing purposes. This can be useful for local development when you want to test specific scenarios. ```typescript TypeScript theme={null} // Overrides the given gate to the specified value Statsig.overrideGate("a_gate_name", true, "a_user_id"); // Overrides the given config (dynamic config or experiment) to the provided value Statsig.overrideConfig("a_config_or_experiment_name", { key: "value" }, "a_user_id"); // Overrides the given layer to the provided value Statsig.overrideLayer("a_layer_name", { key: "value" }, "a_user_id"); ``` ```javascript JavaScript theme={null} // Overrides the given gate to the specified value Statsig.overrideGate("a_gate_name", true, "a_user_id"); // Overrides the given config (dynamic config or experiment) to the provided value Statsig.overrideConfig("a_config_or_experiment_name", { key: "value" }, "a_user_id"); // Overrides the given layer to the provided value Statsig.overrideLayer("a_layer_name", { key: "value" }, "a_user_id"); ``` These can be used to set an override for a specific user, or for all users (by not providing a specific user ID). Experiments/Autotune are overridden with the `overrideConfig` API. ### Overriding in getClientInitializeResponse You can also override feature gates, dynamic configs, experiments, and layers when calling `getClientInitializeResponse`. This is useful when you need to provide specific values to the client SDK. ```typescript TypeScript theme={null} // Get client initialize response with overrides const response = Statsig.getClientInitializeResponse(user, clientSDKKey, { overrides: { // Override feature gates featureGates: { 'my_gate': true, // Override gate value to true }, // Override dynamic configs and experiments dynamicConfigs: { // Override config value directly 'price_config': { value: { price: 9.99 } }, // Override experiment by setting the group assignment 'color_experiment': { groupName: 'Control' // Uses the value for the Control group }, // Override both value and group assignment 'spacing_experiment': { value: { spacing: 64 }, groupName: 'Variant_B' } }, // Override layers layers: { 'my_layer': { value: { param: 123 } } } } }); ``` ```javascript JavaScript theme={null} // Get client initialize response with overrides const response = Statsig.getClientInitializeResponse(user, clientSDKKey, { overrides: { // Override feature gates featureGates: { 'my_gate': true, // Override gate value to true }, // Override dynamic configs and experiments dynamicConfigs: { // Override config value directly 'price_config': { value: { price: 9.99 } }, // Override experiment by setting the group assignment 'color_experiment': { groupName: 'Control' // Uses the value for the Control group }, // Override both value and group assignment 'spacing_experiment': { value: { spacing: 64 }, groupName: 'Variant_B' } }, // Override layers layers: { 'my_layer': { value: { param: 123 } } } } }); ``` For experiments, you can override them in two ways: 1. By setting a `value` override on their dynamic config to directly specify the parameter values 2. By setting the `groupName` assignment to use the value for that group name (e.g., "Control" or "Test") You can also combine both approaches to override both the group assignment and the parameter values. ## Manual Exposures Statsig SDKs automatically log an exposure event every time a gate/experiment/config is checked. In some scenarios, you may want to control when to log an exposure. You can disable the automatic logging like this: ### Gates ```javascript theme={null} const result = Statsig.checkGate(aUser, 'a_gate_name', {disableExposureLogging: true}); ``` Then, to manually log the exposure: ```javascript theme={null} Statsig.manuallyLogGateExposure(aUser, 'a_gate_name'); ``` ### Dynamic Configs ```javascript theme={null} const config = Statsig.getConfigWithExposureLoggingDisabledSync(aUser, 'a_dynamic_config_name'); ``` Then, to manually log the exposure: ```javascript theme={null} Statsig.manuallyLogConfigExposure(aUser, 'a_dynamic_config_name'); ``` ### Experiments ```javascript theme={null} const experiment = Statsig.getExperimentWithExposureLoggingDisabledSync(aUser, 'an_experiment_name'); ``` Then, to manually log the exposure: ```javascript theme={null} Statsig.manuallyLogExperimentExposure(aUser, 'an_experiment_name'); ``` ### Layers ```javascript theme={null} const layer = Statsig.getLayerWithExposureLoggingDisabledSync(aUser, 'a_layer_name'); const paramValue = layer.get('a_param_name', 'fallback_value'); ``` Then, to manually log the layer parameter exposure: ```javascript theme={null} Statsig.manuallyLogLayerParameterExposure(aUser, 'a_layer_name', 'a_param_name'); ``` ## Cloudflare Workers Setup ### Polling for updates The SDK cannot poll for updates across requests since Cloudflare does not allow for timers. To solve for this, a manual sync API is available for independently updating the SDK internal store. ```javascript theme={null} if (env.lastSyncTime < Date.now() - env.syncInterval) { env.lastSyncTime = Date.now(); context.waitUntil(Statsig.syncConfigSpecs()); } ``` ### Flushing events The SDK enqueues logged events and flushes them in batches. In order to ensure events are properly flushed, we recommend calling `flush` using [`context.waitUntil`](https://developers.cloudflare.com/workers/runtime-apis/handlers/fetch/#contextwaituntil). This will keep the request handler alive until events are flushed without blocking the response. ```javascript theme={null} context.waitUntil(Statsig.flush()); ``` ### Node.JS Compatibility Many native JavaScript API and Node standard libraries can be accessed in Cloudflare via the [`nodejs_compat`](https://developers.cloudflare.com/workers/runtime-apis/nodejs/) compatibility flag. The SDK is now compatible with `nodejs_compat` (since v5.16.0). In older versions, manual polyfilling is required. ## User Persistent Storage User Persistent Storage is a storage adapter for running sticky experiments. It allows you to persist user assignments across sessions. ### Interface ```typescript theme={null} export interface IUserPersistentStorage { /** * Returns the full map of persisted values for a specific user key * @param key user key */ load(key: string): UserPersistedValues; /** * Save the persisted values of a config given a specific user key * @param key user key * @param configName Name of the config/experiment * @param data Object representing the persistent assignment to store for the given user-config */ save(key: string, configName: string, data: StickyValues): void; /** * Delete the persisted values of a config given a specific user key * @param key user key * @param configName Name of the config/experiment */ delete(key: string, configName: string): void; } ``` ### Example Implementation ```typescript theme={null} class UserPersistentStorageExample implements IUserPersistentStorage { public store: Record = {}; load(key: string): UserPersistedValues { return this.store[key]; } save(key: string, configName: string, data: StickyValues): void { if (!(key in this.store)) { this.store[key] = {}; } this.store[key][configName] = data; } delete(key: string, configName: string): void { delete this.store[key][configName]; } } ``` ## Multi-Instance Usage If you need to create multiple independent instances of the Statsig SDK (for example, to use different API keys or configurations), you can use the instance-based approach: ```javascript theme={null} // Statsig.initialize becomes: const sdkInstance = new StatsigServer(secretKey, options); await sdkInstance.initializeAsync(); ``` ## Forward Proxy Configuration You can configure the SDK to use a forward proxy for network requests: ```javascript theme={null} const proxyAddress = "0.0.0.0:50051" const options = { proxyConfigs: { 'download_config_specs': { "proxyAddress": proxyAddress, "protocol": "grpc_websocket" as NetworkProtocol } } } await Statsig.initialize(secretKey, options) ``` ## FAQs ### How can I use the node SDK for server side rendering? See [Client SDK Bootstrapping | SSR](#bootstrap) ### How can I mock Statsig for testing? See [LocalOverrides](#local-overrides) ## Reference ### Type StatsigUser ```typescript theme={null} export type StatsigUser = // at least one of userID or customIDs must be provided ({ userID: string } | { customIDs: Record }) & { userID?: string; customIDs?: Record; email?: string; ip?: string; userAgent?: string; country?: string; locale?: string; appVersion?: string; custom?: Record< string, string | number | boolean | Array | undefined >; privateAttributes?: Record< string, string | number | boolean | Array | undefined > | null; statsigEnvironment?: StatsigEnvironment; } ``` ### Type StatsigOptions ```typescript theme={null} export type StatsigOptions = { api: string; apiForDownloadConfigSpecs: string; apiForGetIdLists: string; bootstrapValues: string | null; environment: StatsigEnvironment | null; rulesUpdatedCallback: RulesUpdatedCallback | null; logger: LoggerInterface; localMode: boolean; initTimeoutMs: number; dataAdapter: IDataAdapter | null; rulesetsSyncIntervalMs: number; idListsSyncIntervalMs: number; loggingIntervalMs: number; loggingMaxBufferSize: number; disableDiagnostics: boolean; initStrategyForIP3Country: InitStrategy; initStrategyForIDLists: InitStrategy; postLogsRetryLimit: number; postLogsRetryBackoff: RetryBackoffFunc | number; disableRulesetsSync: boolean; disableIdListsSync: boolean; disableAllLogging: boolean; }; export type RulesUpdatedCallback = (rulesJSON: string, time: number) => void; export type RetryBackoffFunc = (retriesRemaining: number) => number; export type StatsigEnvironment = { tier?: 'production' | 'staging' | 'development' | string; [key: string]: string | undefined; }; export type InitStrategy = 'await' | 'lazy' | 'none'; export interface LoggerInterface { debug?(message?: any, ...optionalParams: any[]): void; info?(message?: any, ...optionalParams: any[]): void; warn(message?: any, ...optionalParams: any[]): void; error(message?: any, ...optionalParams: any[]): void; logLevel: 'none' | 'debug' | 'info' | 'warn' | 'error'; } ``` ### Type FeatureGate ```typescript theme={null} export type FeatureGate = { readonly name: string; readonly ruleID: string; readonly idType: string | null; readonly value: boolean; readonly evaluationDetails: EvaluationDetails | null; readonly groupName: null; // deprecated }; ``` ### Type DynamicConfig ```typescript theme={null} export default class DynamicConfig { name: string; value: Record; get( key: string, defaultValue: T, typeGuard: ((value: unknown) => value is T | null) | null = null, ): T; getValue( key: string, defaultValue?: boolean | number | string | object | Array | null, ): unknown | null; getRuleID(): string; getGroupName(): string | null; getIDType(): string | null; getEvaluationDetails(): EvaluationDetails | null; ``` ### Type Layer ```typescript theme={null} export default class Layer { name: string; public get( key: string, defaultValue: T, typeGuard: ((value: unknown) => value is T) | null = null, ): T; getValue( key: string, defaultValue?: boolean | number | string | object | Array | null, ): unknown | null; getRuleID(): string; getGroupName(): string | null; getAllocatedExperimentName(): string | null; getEvaluationDetails(): EvaluationDetails | null; ``` ### DataAdapter ```typescript theme={null} export interface IDataAdapter { get(key: string): Promise; set(key: string, value: string, time?: number): Promise; initialize(): Promise; shutdown(): Promise; supportsPollingUpdatesFor(key: DataAdapterKey): boolean; } ``` ### EvaluationDetails ```typescript theme={null} export class EvaluationDetails { readonly configSyncTime: number; readonly initTime: number; readonly serverTime: number; readonly reason: EvaluationReason; } ``` ### EvaluationReason ```typescript theme={null} export type EvaluationReason = | 'Network' | 'LocalOverride' | 'Unrecognized' | 'Uninitialized' | 'Bootstrap' | 'DataAdapter' | 'Unsupported'; ``` # Legacy PHP Server SDK Source: https://docs.statsig.com/server/php Statsig's legacy PHP Server SDK for evaluating feature gates, experiments, and dynamic configs in PHP backend applications. New projects use PHP Core. [Github Repository](https://github.com/statsig-io/php-sdk) ## Setup the SDK You can install the PHP SDK using composer. ```bash theme={null} composer require statsig/statsigsdk ``` The SDK is also [open source and hosted on github](https://github.com/statsig-io/php-sdk). The package is published to [packagist](https://packagist.org/packages/statsig/statsigsdk). To successfully use the PHP SDK, you need to: 1. Install it 2. Provide a storage adapter to cache config and event logs (easy default exists) 3. Schedule a Cron job to poll for config changes and flush event logs to Statsig 4. Initialize and use the SDK After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. There is also a parameter `options` that requires you to pass in a storage adapter for storing configurations and event logs. In this below example, we use a local file storage adapter, but you can write your own to plug in Redis or another storage solution. You should create an adapter that implements `Statsig\Adapters\IConfigAdapter` that hooks up to your caching solution. By default, we provide a local file solution that can be useful for first time setup but is challenging to work with in production settings. For help with the interface and implementing an adapter, browse the [adapters directory](https://github.com/statsig-io/php-sdk/tree/main/src/Adapters) in the open source SDK repository. ```php theme={null} require_once __DIR__ . '/vendor/autoload.php'; // path to installation folder use Statsig\StatsigServer; use Statsig\StatsigOptions; use Statsig\Adapters\LocalFileDataAdapter; use Statsig\Adapters\LocalFileLoggingAdapter; $config_adapter = new LocalFileDataAdapter(); $logging_adapter = new LocalFileLoggingAdapter(); $options = new StatsigOptions($config_adapter, $logging_adapter); $this->statsig = new StatsigServer("server-sdk-key", $options); ``` ### 🔥 Warning - You need to schedule a job 🔥 #### V3.0+ If you do not configure a job to update the config values, SDK does not fire a network request to fetch the latest config value anymore, instead, config values fetched earlier will be used. See [Cron Jobs](#cron-jobs) ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```php theme={null} use Statsig\StatsigUser; $user = StatsigUser::withUserID("123"); $user->setEmail("testuser@statsig.com"); $this->statsig->checkGate($user, ""); ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```php theme={null} $this->statsig->getConfig($user, ""); ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```php theme={null} // Values via getLayer $layer = $this->statsig->getLayer($user, "user_promo_experiments"); $title = $layer->get("title", "Welcome to Statsig!"); $discount = $layer->get("discount", 0.1); // or, via getExperiment $title_experiment = $this->statsig->getExperiment($user, "new_user_promo_title"); $price_experiment = $this->statsig->getExperiment($user, "new_user_promo_price"); $title = $title_experiment->get("title", "Welcome to Statsig!") $discount = $price_experiment->get("discount", 0.1) ... $price = $msrp * (1 - $discount) ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```php theme={null} $event = new StatsigEvent("purchase"); $event->setUser($user); $event->setValue("subscription"); $event->setMetadata(array("promotion" => "2022 deals")); $this->statsig->logEvent($event); ``` At the end of the request, you can flush events to the log file using: ```php theme={null} $this->statsig->flush(); ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Cron Jobs To keep your configurations up to date, and send event data to Statsig, you can create two jobs. Here, we document them as cron jobs, but you can use any out of band process to run them. If you are using Laravel, for example, you can use Commands to run them locally and on a schedule. ### Sync The first job runs `sync.php` to download the latest definition of gates/configs/experiments from Statsig, and save it to a config file locally. If you do not update this file, your gate/config/experiment values may be stale and will be refetched during a request, which may lead to slower response times. ```bash theme={null} # Run once php sync.php --secret ``` ```bash theme={null} # Create a cron job that runs as statsigsync every minute echo '*/1 * * * * statsigsync php /my/path/to/statsig/sync.php --secret > /dev/null' | sudo tee /etc/cron.d/statsigsync sudo service cron reload # reload the cron daemon ``` You should provide your own custom adapter that implements Statsig\Adapters\IDataAdapter ```bash theme={null} php send.php --secret \ --adapter-class Namespace\For\MyConfigAdapter \ --adapter-path /path/to/MyConfigAdapter.php \ --adapter-arg an_argument_for_my_adapter \ --adapter-arg another_argument ``` By default, sync.php will use the Statsig LocalFileDataAdapter which writes to /tmp/statsig.configs ### Send The second runs `send.php` to send the exposure data and log events to statsig. Without this data, your events will need to be logged during the lifetime of the request, which may lead to slower response times. ```bash theme={null} # Run once php send.php --secret ``` ```bash theme={null} # Create a cron job that runs as statsigdata every minute echo '*/1 * * * * statsigdata php /my/path/to/statsig/send.php --secret > /dev/null' | sudo tee /etc/cron.d/statsigdata sudo service cron reload # reload the cron daemon ``` You should provide your own custom adapter that implements Statsig\Adapters\ILoggingAdapter ```bash theme={null} php send.php --secret \ --adapter-class Namespace\For\MyLoggingAdapter \ --adapter-path /path/to/MyLoggingAdapter.php \ --adapter-arg an_argument_for_my_adapter \ --adapter-arg another_argument ``` By default, send.php will use the Statsig LocalFileDataAdapter which writes to /tmp/statsig.logs ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! # Legacy Python Server SDK Source: https://docs.statsig.com/server/pythonSDK Statsig's legacy Python Server SDK for evaluating feature gates, experiments, and dynamic configs in Python backend services. New projects use Python Core. [Github Repository](https://github.com/statsig-io/python-sdk) ## Setup the SDK Install the sdk using [pip3](https://pypi.org/project/statsig/): The Statsig SDK is not compatible with python 2. You must be on python 3.7+ to use the Statsig SDK. ```bash theme={null} pip3 install statsig ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. There is also an optional parameter named `options` that allows you to pass in a [StatsigOptions](#statsig-options) to customize the SDK. ```python theme={null} from statsig import statsig statsig.initialize("server-secret-key") # or with StatsigOptions options = StatsigOptions(tier=StatsigEnvironmentTier.development) statsig.initialize("server-secret-key", options) # check if sdk is initialized initialized = statsig.is_initialized() ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```python theme={null} from statsig.statsig_user import StatsigUser ... statsig.check_gate(StatsigUser("user-id"), "gate-name") ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```python theme={null} config = statsig.get_config(StatsigUser("user-id"), "config-name") config_json = config.get_value() ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```python theme={null} # Values via getLayer layer = statsig.get_layer(user, "user_promo_experiments") title = layer.get("title", "Welcome to Statsig!") discount = layer.get("discount", 0.1) # or, via getExperiment title_exp = statsig.get_experiment(user, "new_user_promo_title") price_exp = statsig.get_experiment(user, "new_user_promo_price") title = title_exp.get("title", "Welcome to Statsig!") discount = price_exp.get("discount", 0.1) ... price = msrp * (1 - discount) ``` ## Retrieving Feature Gate Metadata In certain scenarios, you may need more information about a gate evaluation than just a boolean value. For additional metadata about the evaluation, use the Get Feature Gate API, which returns a FeatureGate object: ```python theme={null} gate = statsig.get_feature_gate(StatsigUser("user-id"), "gate-name") print(gate.name) # 'gate-name' print(gate.value) # True or False print(gate.rule_id) # rule ID that was evaluated print(gate.evaluation_details) # evaluation metadata ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```python theme={null} from statsig.statsig_user import StatsigUser from statsig.statsig_event import StatsigEvent statsig.log_event(StatsigEvent(StatsigUser("user-id"), "event-name")) ``` Python supports `retry_queue_size`, which allows you to adjust the memory allocated for handling retries. While service outages are rare, increasing the retry\_queue\_size can help minimize event loss by providing additional memory to buffer events during such occurrences. This option is generally not needed for typical use but offers added flexibility in exceptional situations. ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options `initialize()` takes an optional parameter `options` in addition to the secret key that you can provide to customize the Statsig client. Here are the current options and we are always adding more to the list: Create a `StatsigOptions` class to pass in with the following available parameters: (unit of measure for time related options is seconds) Sets the environment tier (for gates to evaluate differently in development and production) You can set an environment tier with the `StatsigEnvironmentTier` enum or just as a `str` Enforces a minimum timeout on network requests from the SDK Sets the maximum timeout on download config specs and id lists network requests for initialization How often the SDK updates rulesets from Statsig servers How often the SDK updates idlists from Statsig servers Disables all network requests. SDK returns default values and will not log events. Useful in combination with overrides to mock behavior for tests a string that represents all rules for all feature gates, dynamic configs and experiments. It can be provided to bootstrap the Statsig server SDK at initialization in case your server runs into network issue or Statsig server is down temporarily. a callback function that's called whenever we have an update for the rules; it's called with a logical timestamp and a JSON string (used as is for bootstrapValues mentioned above). Note that as of v0.6.0, this will be called from a background thread that the SDK uses to update config values. The number of events to batch before flushing the queue to the network. Default 500. Note that events are also batched every minute by a background thread A data store with custom storage behavior for config specs. Can be used to bootstrap Statsig server (takes priority over `bootstrap_values`). Configuration network for each endpoint, for example, download\_config\_spec, get\_id\_lists Fallback to Statsig CDN for download config specs and get id lists if the overridden api failed. List of sources SDK tries to get download\_config\_specs from when initialize. The list is ordered, SDK tries to get source from first element, and stops when getting dcs successfully List of sources SDK tries to get download\_config\_specs from when downloading. The list is ordered, SDK tries to get source from first element, and stops when getting dcs successfully Example: ```python theme={null} from statsig import statsig, StatsigEnvironmentTier, StatsigOptions options = StatsigOptions(None, StatsigEnvironmentTier.development) statsig.initialize("secret-key", options).wait() ``` You can also use the `set_environment_parameter` function, but that takes in string values only: ```python theme={null} from statsig import statsig, StatsigEnvironmentTier, StatsigOptions options = StatsigOptions() options.set_environment_parameter("tier", StatsigEnvironmentTier.development.value) statsig.initialize("secret-key", options).wait() ``` ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```python theme={null} statsig.shutdown() ``` ## Client SDK Bootstrapping The Statsig server SDK can be used to generate the initialization values for a client SDK. This is useful for server-side rendering (SSR) or when you want to pre-fetch values for a client. ```python theme={null} values = statsig.get_client_initialize_response(user); # dict() | None # To apply local overrides, set include_local_overrides = True (python sdk v0.32.0+) values = statsig.get_client_initialize_response(user=user, include_local_overrides=True); # dict() | None ``` ## Local Overrides You can override the values returned by the SDK for testing purposes. This can be useful for local development when you want to test specific scenarios. ```python theme={null} # Adding/Removing gate overrides statsig.override_gate("a_gate_name", true, "a_user_id") statsig.remove_gate_override("a_gate_name", "a_user_id") # Adding/Removing config overrides statsig.override_config("a_config_name", {"key": "value"}, "a_user_id") statsig.remove_config_override("a_config_name", "a_user_id") # Adding/Removing experiment overrides statsig.override_experiment("an_experiment_name", {"key": "value"}, "a_user_id") statsig.remove_experiment_override("an_experiment_name", "a_user_id") # Remove All Overrides statsig.remove_all_overrides() # You can also override with custom ids custom_id_user = StatsigUser("a_user_id", custom_ids={"statsigId": "a_statsig_id"}) statsig.override_gate("a_gate_name", true, "a_statsig_id") # Local overrides will prioritize override with userId, then look up the custom id to override. # To prevent clashing overrides, it is recommended to not use the same value for userId and customIds for different users. ``` ## Multi-Instance Usage If you need to create multiple independent instances of the Statsig SDK (for example, to use different API keys or configurations), you can use the instance-based approach: ```python theme={null} sdk_instance = StatsigServer() sdk_instance.initialize(secret_key, options); ``` ## Forward Proxy Configuration You can configure the SDK to use a forward proxy for network requests: Basic setup to stream download config spec from forward proxy: ```python theme={null} proxyAddress = "0.0.0.0:50051" // local address update to your address Statsig.initialize(secret_key, StatsigOptions(proxy_configs={ NetworkEndpoint.DOWNLOAD_CONFIG_SPECS: ProxyConfig(NetworkProtocol.GRPC_WEBSOCKET, proxyAddress)})) ``` When the SDK is disconnected from forward proxy when use grpc\_websocket, the sdk will retry connection with exponential backoff, after `push_worker_failover_threshold` retries, the sdk will start polling from Statsig until reconnecting to the forward proxy. You can customize Streaming Failover Behavior. You can also define the sources/endpoints SDK poll from, SDK will try from source at index 0, and stops trying if get a response. ```python theme={null} statsigOptions = StatsigOptions( proxy_configs={ NetworkEndpoint.DOWNLOAD_CONFIG_SPECS: ProxyConfig( protocol=NetworkProtocol.GRPC_WEBSOCKET, proxy_address=address, push_worker_failover_threshold=1, # start polling from Statsig endpoint after 1 retry failed # 1st retry 5000 ms later, 2nd retry 2 * 5000ms = 10 seconds .... retry_backoff_multiplier=2, max_retry_attempt=8, retry_backoff_base_ms=5000 ) }, # Get from network first, which is forward proxy here, if fails, try datastore, if fails try poll from Statsig endpoint initialize_sources=[ DataSource.NETWORK, DataSource.DATASTORE, DataSource.STATSIG_NETWORK, ], ) ``` ## FAQs ### How can I mock Statsig for testing? The python server SDK, starting in version 0.5.1+, supports a few features to make testing easier. First, there is a `StatsigOption` parameter called `localMode`. Setting `localMode` to true will cause the SDK to never hit the network, and only return default values. This is perfect for dummy environments or test environments that should not access the network. Next, there are the `overrideGate` and `overrideConfig` APIs on the global `statsig` interface, see [Local Overrides](#local-overrides) These can be used to set a gate or config override for a specific user, or for all users (by not providing a specific user ID). We suggest you enable `localMode` and then override gates/configs/experiments to specific values to test the various code flows you are building. ### Can I generate the initialize response for a client SDK using the Python server SDK? Yes. See [Client Initialize Response](#bootstrap). ## Reference ### StatsigUser ```python theme={null} @dataclass class StatsigUser: """An object of properties relating to the current user user_id or customID is required: https://docs.statsig.com/sdks/user#why-is-an-id-always-required-for-server-sdks Provide as many as possible to take advantage of advanced conditions in the statsig console A dictionary of additional fields can be provided under the custom field Set private_attributes for any user property you need for gate evaluation but prefer stripped from logs/metrics """ user_id: Optional[str] = None email: Optional[str] = None ip: Optional[str] = None user_agent: Optional[str] = None country: Optional[str] = None locale: Optional[str] = None app_version: Optional[str] = None custom: Optional[dict] = None # key: string, value: string private_attributes: Optional[dict] = None # key: string, value: string custom_ids: Optional[dict] = None # key: string, value: string ``` ### StatsigOptions ```python theme={null} class StatsigOptions: """An object of properties for initializing the sdk with additional parameters""" def __init__( self, api: Optional[str] = None, api_for_download_config_specs: Optional[str] = None, api_for_get_id_lists: Optional[str] = None, api_for_log_event: Optional[str] = None, tier: Union[str, StatsigEnvironmentTier, None] = None, init_timeout: Optional[int] = None, timeout: Optional[int] = None, rulesets_sync_interval: int = DEFAULT_RULESET_SYNC_INTERVAL, idlists_sync_interval: int = DEFAULT_IDLIST_SYNC_INTERVAL, local_mode: bool = False, bootstrap_values: Optional[str] = None, rules_updated_callback: Optional[Callable] = None, event_queue_size: Optional[int] = DEFAULT_EVENT_QUEUE_SIZE, data_store: Optional[IDataStore] = None, idlists_thread_limit: int = DEFAULT_IDLISTS_THREAD_LIMIT, logging_interval: int = DEFAULT_LOGGING_INTERVAL, #deprecated disable_diagnostics: bool = False, custom_logger: Optional[OutputLogger] = None, enable_debug_logs = False, disable_all_logging = False, evaluation_callback: Optional[Callable[[Union[Layer, DynamicConfig, FeatureGate]], None]] = None, retry_queue_size: int = DEFAULT_RETRY_QUEUE_SIZE, proxy_configs: Optional[Dict[NetworkEndpoint, ProxyConfig]] = None, fallback_to_statsig_api: Optional[bool] = False, initialize_sources: Optional[List[DataSource]] = None, config_sync_sources: Optional[List[DataSource]] = None, ): ``` ### FeatureGate ```python theme={null} class FeatureGate: def get_value(self): """Returns the underlying value of this FeatureGate""" def get_name(self): """Returns the name of this FeatureGate""" def get_evaluation_details(self): """Returns the evaluation detail of this FeatureGate""" ``` ### DynamicConfig ```python theme={null} class DynamicConfig: def get(self, key, default=None): """Returns the value of the config at the given key or the provided default if the key is not found """ def get_typed(self, key, default=None): """Returns the value of the config at the given key iff the type matches the type of the provided default. Otherwise, returns the default value """ def get_value(self): """Returns the underlying value of this DynamicConfig""" def get_name(self): """Returns the name of this DynamicConfig""" def get_evaluation_details(self): """Returns the evaluation detail of this DynamicConfig""" ``` ### Layer ```python theme={null} class Layer: def get(self, key, default=None): """Returns the value of the layer at the given key or the provided default if the key is not found """ def get_typed(self, key, default=None): """Returns the value of the layer at the given key iff the type matches the type of the provided default. Otherwise, returns the default value """ def get_name(self): """Returns the name of this Layer""" def get_values(self): """Returns all the values in this Layer but does not trigger an exposure log""" def get_evaluation_details(self): """Returns the evaluation detail of this Layer""" ``` ### EvaluationDetails ```python theme={null} class EvaluationDetails: reason: EvaluationReason config_sync_time: int init_time: int server_time: int class EvaluationReason(str, Enum): network = "Network" local_override = "LocalOverride" unrecognized = "Unrecognized" uninitialized = "Uninitialized" bootstrap = "Bootstrap" data_adapter = "DataAdapter" unsupported = "Unsupported" error = "error" ``` ### DataStore ```python theme={null} class IDataStore: def get(self, key: str) -> Optional[str]: return None def set(self, key: str, value: str): pass def shutdown(self): pass def should_be_used_for_querying_updates(self, key: str) -> bool: return False ``` ### ForwardProxy - ProxyConfig ```python theme={null} class NetworkProtocol(Enum): HTTP = "http" GRPC = "grpc" GRPC_WEBSOCKET = "grpc_websocket" class NetworkEndpoint(Enum): LOG_EVENT = "log_event" DOWNLOAD_CONFIG_SPECS = "download_config_specs" GET_ID_LISTS = "get_id_lists" ALL = "all" class ProxyConfig: def __init__( self, protocol: NetworkProtocol, proxy_address: str, # Websocket worker failover config max_retry_attempt: Optional[int] = None, # default is 10 retry_backoff_multiplier: Optional[int] = None, # default is # default is 5 retry_backoff_base_ms: Optional[int] = None, # default is 10,000 ms # Push worker failback to polling threshold, fallback immediate set 0, # n means fallback after n retry failed push_worker_failover_threshold: Optional[int] = None, # default is 4, about 30 minutes ): self.proxy_address = proxy_address self.protocol = protocol self.max_retry_attempt = max_retry_attempt self.retry_backoff_multiplier = retry_backoff_multiplier self.retry_backoff_base_ms = retry_backoff_base_ms self.push_worker_failover_threshold = push_worker_failover_threshold ``` # Ruby Server SDK Source: https://docs.statsig.com/server/ruby Use the Statsig Ruby Server SDK to evaluate feature gates, experiments, and dynamic configs from Ruby and Ruby on Rails backend applications. View the Ruby SDK source code and releases ## Setup the SDK If you are using Bundler, add the [gem](https://rubygems.org/gems/statsig) to your Gemfile from command line: ```shell theme={null} bundle add statsig ``` or directly include it in your Gemfile and run `bundle install`: ```shell theme={null} gem "statsig", ">= X.Y.Z" ``` Check out the latest versions on [https://rubygems.org/gems/statsig](https://rubygems.org/gems/statsig) After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```ruby theme={null} require 'statsig' Statsig.initialize('server-secret-key') ``` ```ruby theme={null} # Or, if you want to initialize with certain options options = StatsigOptions.new({'tier' => 'staging'}, network_timeout: 5) # And a callback when the initialization network request fails def error_callback(e) puts e end ... Statsig.initialize('server-secret-key', options, method(:error_callback)) ``` ### Initializing Statsig in a Rails Application If your application is using Rails, you should initialize Statsig in `config/initializers/statsig.rb`: ```ruby theme={null} Statsig.initialize('server-secret-key', options) ``` ### Initializing Statsig when using Unicorn, Puma, Passenger, or Sidekiq For **Unicorn**, you should initialize Statsig within an `after_fork` hook in your `unicorn.rb` config file: ```ruby theme={null} after_fork do |server,worker| Statsig.initialize('server-secret-key', options) end ``` For **Puma**, you should initialize Statsig within an `on_worker_boot` hook in your `puma.rb` config file: ```ruby theme={null} on_worker_boot do Statsig.initialize('server-secret-key', options) end ``` For **Passenger**, you should initialize Statsig in your `config.ru` config file: ```ruby theme={null} if defined?(PhusionPassenger) PhusionPassenger.on_event(:starting_worker_process) do |forked| Statsig.initialize('server-secret-key', options) end end ``` For **Sidekiq**, you should initialize Statsig in your `sidekiq.rb`/server configuration file: ```ruby theme={null} Sidekiq.configure_server do |config| config.on(:startup) do Statsig.initialize end config.on(:shutdown) do Statsig.shutdown end end ``` If you are using Rails in combination with any of the above, you should be sure to initialize using the specific process lifecycle hooks exposed by the respective tool. You can initialize in multiple places, which should ensure the SDK is fully usable including all background processing. `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```ruby theme={null} user = StatsigUser.new({'userID' => 'some_user_id'}) if Statsig.check_gate(user, 'use_new_feature') # Gate is on, enable new feature else # Gate is off end ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```ruby theme={null} config = Statsig.get_config(user, 'awesome_product_details') # The 2nd parameter is the default value to be used in case the given parameter name does not exist on # the Dynamic Config object. This can happen when there is a typo, or when the user is offline and the # value has not been cached on the client. item_name = config.get('product_name', 'Awesome Product v1'); price = config.get('price', 10.0); shouldDiscount = config.get('discount', false); # Or just get the whole json object backing this config if you prefer json = config.value ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```ruby theme={null} # Values via getLayer layer = Statsig.get_layer(user, "user_promo_experiments") title = layer.get("title", "Welcome to Statsig!") discount = layer.get("discount", 0.1) # or, via getExperiment title_exp = Statsig.get_experiment(user, "new_user_promo_title") price_exp = Statsig.get_experiment(user, "new_user_promo_price") title = title_exp.get("title", "Welcome to Statsig!") discount = price_exp.get("discount", 0.1) ... price = msrp * (1 - discount) ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```ruby theme={null} Statsig.log_event( user, 'add_to_cart', 'SKU_12345', { 'price' => '9.99', 'item_name' => 'diet_coke_48_pack' } ) ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Statsig Options `initialize()` takes an optional parameter `options` in addition to the secret key that you can provide to customize the Statsig client. Here are the current options and we are always adding more to the list: You can specify optional parameters with `options` when initializing. * **environment**: Hash, default `nil` * a Hash you can use to set environment variables that apply to all of your users in the same session and will be used for targeting purposes. * The most common usage is to set the "tier" (string), and have feature gates pass/fail for specific environments. The accepted values are "production", "staging" and "development", e.g. `StatsigOptions.New({ 'tier' => 'staging' })`. * **download\_config\_specs\_url**: String, default `"https://api.statsigcdn.com/v2/download_config_specs/"` * The url used specifically to call download\_config\_specs * **log\_event\_url**: String, default `"https://statsigapi.net/v1/log_event"` * The url used specifically to call log\_event * **get\_id\_lists\_url**: String, default `"https://statsigapi.net/v1/get_id_lists"` * The url used specifically to call get\_id\_lists * **rulesets\_sync\_interval**: Number, default `10` * The interval (in seconds) to poll for changes to your Statsig configuration * **idlists\_sync\_interval**: Number, default `60` * The interval (in seconds) to poll for changes to id lists * **disable\_rulesets\_sync**: Boolean, default `false` * Disable background syncing for rulesets * **disable\_idlists\_sync**: Boolean, default `false` * Disable background syncing for id lists * **logging\_interval\_seconds**: Number, default `60` * How often to flush logs to Statsig * **logging\_max\_buffer\_size**: Number, default `1000`, can be set lower but anything over 1000 will be dropped on the server * The maximum number of events to batch before flushing logs to the server * **local\_mode**: Boolean, default `false` * Restricts the SDK to not issue any network requests and only respond with default values (or local overrides) * **bootstrap\_values**: String, default `nil` * A string that represents all rules for all feature gates, dynamic configs and experiments. It can be provided to bootstrap the Statsig server SDK at initialization in case your server runs into network issue or Statsig server is down temporarily. * **rules\_updated\_callback**: function, default `nil` * A callback function that will be called anytime the rulesets are updated * **data\_store**: IDataStore, default `nil` * A class that extends IDataStore. Can be used to provide values from a common data store (like Redis) to initialize the Statsig SDK. * **idlist\_threadpool\_size**: Number, default `3` * The number of threads allocated to syncing IDLists * **logger\_threadpool\_size**: Number, default `3` * The number of threads allocated to posting event logs * **disable\_diagnostics\_logging**: Boolean, default `false` * Should diagnostics be logged. These include performance metrics for initialize * **disable\_sorbet\_logging\_handlers**: Boolean, default `false` * Statsig utilizes Sorbet ([https://sorbet.org](https://sorbet.org)) to ensure type safety of the SDK. This includes logging to console when errors are detected. You can disable this logging by setting this flag to true. * **network\_timeout**: Number, default `nil` * Maximum number of seconds to wait for a network call before timing out * **post\_logs\_retry\_limit**: Number, default `3` * Number of times to retry sending a batch of failed log events * **post\_logs\_retry\_backoff**: Number/Function, default `nil` * The number of seconds, or a function that returns the number of seconds based on the number of retries remaining which overrides the default backoff time between retries * **user\_persistent\_storage**: IUserPersistentStorage, default `nil` * A storage adapter for persisted values. Can be used for sticky bucketing users in experiments. Implements Statsig::Interfaces::IUserPersistentStorage. ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```ruby theme={null} Statsig.shutdown ``` ## Client SDK Bootstrapping The Statsig server SDK can be used to generate the initialization values for a client SDK. This is useful for server-side rendering (SSR) or when you want to pre-fetch values for a client. ```ruby theme={null} values = Statsig.get_client_initialize_response(user); # Hash[String, Any] | Nil ``` ## Local Overrides You can override the values returned by the SDK for testing purposes. This can be useful for local development when you want to test specific scenarios. ```ruby theme={null} # Adding gate overrides Statsig.override_gate("a_gate_name", true) # Adding config overrides Statsig.override_config("a_config_name", {"key" => "value"}) ``` 1. These only apply locally - they do not update definitions in the Statsig console or elsewhere. 2. The local override API is not designed to be a full mock. They are only a convenient way to override the value of the gate/config/etc. ## Manual Exposures Statsig SDKs automatically log an exposure event every time a gate/experiment/config is checked. In some scenarios, you may want to control when to log an exposure. **Gates** ```ruby theme={null} result = Statsig.check_gate(user, 'a_gate_name', CheckGateOptions.new(disable_log_exposure: true)) ``` ```ruby theme={null} Statsig.manually_log_gate_exposure(user, 'a_gate_name') ``` **Configs** ```ruby theme={null} config = Statsig.get_config(user, 'a_dynamic_config_name', GetConfigOptions.new(disable_log_exposure: true)) ``` ```ruby theme={null} Statsig.manually_log_config_exposure(user, 'a_dynamic_config_name') ``` **Experiments** ```ruby theme={null} experiment = Statsig.get_experiment(user, 'an_experiment_name', GetExperimentOptions.new(disable_log_exposure: true)) ``` ```ruby theme={null} Statsig.manually_log_experiment_exposure(user, 'an_experiment_name') ``` **Layers** ```ruby theme={null} layer = Statsig.get_layer(user, 'a_layer_name', GetLayerOptions.new(disable_log_exposure: true)) paramValue = layer.get('a_param_name', 'fallback_value') ``` ```ruby theme={null} Statsig.manually_log_layer_parameter_exposure(user, 'a_layer_name', 'a_param_name') ``` ## User Persistent Storage User Persistent Storage is a storage adapter for running sticky experiments. It allows you to persist user assignments across sessions. ### Interface ### Interface ```ruby theme={null} class IUserPersistentStorage def load(key) nil end def save(key, data) end end ``` ### Example Implementation ```ruby theme={null} class DummyPersistentStorageAdapter < Statsig::Interfaces::IUserPersistentStorage attr_accessor :store def initialize @store = {} end def load(key) return nil unless @store&.key?(key) @store[key] end def save(key, data) @store[key] = data end end ``` ## Multi-Instance Usage If you need to create multiple independent instances of the Statsig SDK (for example, to use different API keys or configurations), you can use the instance-based approach: ```ruby theme={null} sdk_instance = StatsigDriver.new(secret_key, options, error_callback) ``` ## FAQ #### How do I run experiments for logged out users? See the guide on [device level experiments](/guides/first-device-level-experiment) #### How can I mock or override the SDK for testing? Starting in `v1.12.0+`, the Ruby SDK supports `localMode` and `overrides`, see [Local Overrides](#local-overrides) * `localMode` is a boolean parameter in `StatsigOptions` when initializing the SDK. It restricts all network traffic, so the SDK operates offline and only returns default or override values. #### Can I generate the initialize response for a client SDK using the Ruby server SDK? Yes. See [Client SDK Bootstrapping](#client-sdk-bootstrapping). ## Reference ### Type StatsigUser ```ruby theme={null} export type StatsigUser = { class StatsigUser attr_accessor :user_id attr_accessor :email attr_accessor :ip attr_accessor :user_agent attr_accessor :country attr_accessor :locale attr_accessor :app_version attr_accessor :statsig_environment attr_accessor :custom_ids # Hash of key:string value:string attr_accessor :private_attributes # Hash of key:string value:string @custom # Hash of key:string value:string def initialize(user_hash) @statsig_environment = Hash.new if user_hash.is_a?(Hash) @user_id = user_hash['userID'] || user_hash['user_id'] @user_id = @user_id.to_s unless @user_id.nil? @email = user_hash['email'] @ip = user_hash['ip'] @user_agent = user_hash['userAgent'] || user_hash['user_agent'] @country = user_hash['country'] @locale = user_hash['locale'] @app_version = user_hash['appVersion'] || user_hash['app_version'] @custom = user_hash['custom'] if user_hash['custom'].is_a? Hash @statsig_environment = user_hash['statsigEnvironment'] @private_attributes = user_hash['privateAttributes'] if user_hash['privateAttributes'].is_a? Hash custom_ids = user_hash['customIDs'] || user_hash['custom_ids'] @custom_ids = custom_ids if custom_ids.is_a? Hash end end end ``` ### Type StatsigOptions ```ruby theme={null} class StatsigOptions attr_accessor :environment attr_accessor :download_config_specs_url attr_accessor :log_event_url attr_accessor :get_id_lists_url attr_accessor :rulesets_sync_interval attr_accessor :idlists_sync_interval attr_accessor :disable_rulesets_sync attr_accessor :disable_idlists_sync attr_accessor :logging_interval_seconds attr_accessor :logging_max_buffer_size attr_accessor :local_mode attr_accessor :bootstrap_values attr_accessor :rules_updated_callback attr_accessor :data_store attr_accessor :idlist_threadpool_size attr_accessor :logger_threadpool_size attr_accessor :disable_diagnostics_logging attr_accessor :disable_sorbet_logging_handlers attr_accessor :network_timeout attr_accessor :post_logs_retry_limit attr_accessor :post_logs_retry_backoff attr_accessor :user_persistent_storage def initialize( environment = nil, download_config_specs_url: nil, log_event_url: nil, get_id_lists_url: nil, rulesets_sync_interval: 10, idlists_sync_interval: 60, disable_rulesets_sync: false, disable_idlists_sync: false, logging_interval_seconds: 60, logging_max_buffer_size: 1000, local_mode: false, bootstrap_values: nil, rules_updated_callback: nil, data_store: nil, idlist_threadpool_size: 3, logger_threadpool_size: 3, disable_diagnostics_logging: false, disable_sorbet_logging_handlers: false, network_timeout: nil, post_logs_retry_limit: 3, post_logs_retry_backoff: nil, user_persistent_storage: nil ) end end ``` ### DataStore ```ruby theme={null} module Statsig module Interfaces class IDataStore def init end def get(key) nil end def set(key, value) end def shutdown end end end end ``` # Legacy Rust Server SDK Source: https://docs.statsig.com/server/rust Statsig's legacy Rust Server SDK for evaluating feature gates, experiments, and dynamic configs in Rust backend services. New projects use Rust Core SDK. Support for the Legacy Rust SDK ends April 30, 2026. Migrate to the [new Rust SDK](/server-core/rust-core) soon. Rust SDK on Github ## Setup the SDK To use the SDK, add `statsig` as a dependency in your `Cargo.toml`. The latest version can be found at [crates.io/crates/statsig](https://crates.io/crates/statsig). ```toml theme={null} [dependencies] statsig = "X.Y.Z" # <- update version ``` After installation, you will need to initialize the SDK using a [Server Secret Key from the Statsig console](https://console.statsig.com/api_keys). Do NOT embed your Server Secret Key in client-side applications, or expose it in any external-facing documents. However, if you accidentally expose it, you can create a new one in the Statsig console. ```rust theme={null} use statsig::{Statsig}; Statsig::initialize("secret-key").await; // or with StatsigOptions use statsig::{Statsig, StatsigOptions}; let env = HashMap::from([("tier".to_string(), "staging".to_string())]); let opts = StatsigOptions { environment: Some(env), ..StatsigOptions::default() }; Statsig::initialize_with_options("secret-key", opts).await; ``` `initialize` will perform a network request. After `initialize` completes, virtually all SDK operations will be synchronous (See [Evaluating Feature Gates in the Statsig SDK](https://blog.statsig.com/evaluating-feature-gates-in-the-statsig-sdk-a6f8881a1ad8)). The SDK will fetch updates from Statsig in the background, independently of your API calls. ## Working with the SDK ## Checking a Feature Flag/Gate Now that your SDK is initialized, let's fetch a [**Feature Gate**](/feature-flags/overview). Feature Gates can be used to create logic branches in code that can be rolled out to different users from the Statsig Console. Gates are always **CLOSED** or **OFF** (think `return false;`) by default. From this point on, all APIs will require you to specify the user (see [Statsig user](#statsig-user)) associated with the request. For example, check a gate for a certain user like this: ```rust theme={null} let user = StatsigUser::with_user_id("a-user".to_string()); if Statsig::check_gate(&user, "a_gate").ok().unwrap_or(false) { // Gate is on, enable new feature } else { // Gate is off } ``` ## Reading a Dynamic Config Feature Gates can be very useful for simple on/off switches, with optional but advanced user targeting. However, if you want to be able send a different set of values (strings, numbers, and etc.) to your clients based on specific user attributes, e.g. country, [**Dynamic Configs**](/dynamic-config) can help you with that. The API is very similar to Feature Gates, but you get an entire json object you can configure on the server and you can fetch typed parameters from it. ```rust theme={null} let config = Statsig::get_config(&user, "a_config").ok().unwrap(); let value = config.get_string("a_key", "default_value"); ``` ## Getting a Layer/Experiment Then we have **Layers/Experiments**, which you can use to run A/B/n experiments. We offer two APIs, but we recommend the use of [layers](/layers) to enable quicker iterations with parameter reuse. ```rust theme={null} let layer = Statsig::get_layer(&user, "a_layer").ok().unwrap(); let param_value = layer.get_string("a_parameter", "default_value"); // or via get_experiment let experiment = Statsig::get_experiment(&user, "an_experiment").ok().unwrap(); let exp_value = experiment.get_string("a_parameter", "default_value"); ``` ## Logging an Event Now that you have a Feature Gate or an Experiment set up, you may want to track some custom events and see how your new features or different experiment groups affect these events. This is super easy with Statsig - simply call the Log Event API and specify the user and event name to log; you additionally provide some value and/or an object of metadata to be logged together with the event: ```rust theme={null} let event = StatsigEvent::new("event_name".to_string()); Statsig::log_event(&user, event); ``` Learn more about identifying users, group analytics, and best practices for logging events in the [logging events guide](/guides/logging-events). ## Statsig User When calling APIs that require a user, you should pass as much information as possible in order to take advantage of advanced gate and config conditions (like country or OS/browser level checks), and correctly measure impact of your experiments on your metrics/events. As explained [here](/sdks/user#why-is-an-id-always-required-for-server-sdks), at least one identifier (userID or customID) is required to provide a consistent experience for a given user. Besides `userID`, we also have `email`, `ip`, `userAgent`, `country`, `locale` and `appVersion` as top-level fields on StatsigUser. In addition, you can pass any key-value pairs in an object/dictionary to the `custom` field and be able to create targeting based on them. Note that while typing is lenient on the `StatsigUser` object to allow you to pass in numbers, strings, arrays, objects, and potentially even enums or classes, the evaluation operators will only be able to operate on primitive types - mostly strings and numbers. While we attempt to smartly cast custom field types to match the operator, we cannot guarantee evaluation results for other types. For example, setting an array as a custom field will only ever be compared as a string - there is no operator to match a value in that array. ### Private Attributes Have sensitive user PII data that should not be logged? No problem, we have a solution for it! On the StatsigUser object we also have a field called `privateAttributes`, which is a simple object/dictionary that you can use to set private user attributes. Any attribute set in `privateAttributes` will only be used for evaluation/targeting, and removed from any logs before they are sent to Statsig server. For example, if you have feature gates that should only pass for users with emails ending in "@statsig.com", but do not want to log your users' email addresses to Statsig, you can simply add the key-value pair `{ email: "my_user@statsig.com" }` to `privateAttributes` on the user and that's it! ## Shutdown To gracefully shutdown the SDK and ensure all events are flushed: ```rust theme={null} Statsig::shutdown().await; ``` # CLI Session Replay Source: https://docs.statsig.com/session-replay/cli-session-replay Use the Statsig CLI to inspect, export, and manage session replay recordings, including filtering by user, session, and time range. CLI Session Replay allows you to record terminal sessions in your Node.js CLI applications and replay them in the Statsig Console. This enables you to understand how users interact with your command-line tools, diagnose issues, and improve the user experience. The plugin records terminal output and resize events. User input is planned for a future version, but given that it's likely to contain sensitive information, we'll release it after introducing methods to pause, filter or redact recordings. ## Installation Install the CLI session replay package for Node.js: ```bash theme={null} npm install @statsig/js-client @statsig/cli-session-replay-node ``` ```bash theme={null} yarn add @statsig/js-client @statsig/cli-session-replay-node ``` ```bash theme={null} pnpm add @statsig/js-client @statsig/cli-session-replay-node ``` ## Basic Usage ```javascript theme={null} import { StatsigClient } from '@statsig/js-client'; import { StatsigCliSessionReplayPlugin } from '@statsig/cli-session-replay-node'; const client = new StatsigClient( 'your-client-key', { userID: 'user-123' }, { loggingEnabled: 'always', // Required for CLI environments plugins: [new StatsigCliSessionReplayPlugin()], } ); // Recording starts here console.log('Hello from CLI!'); await client.initializeAsync(); // Your CLI application logic here console.log('Continue'); ``` **CLI Logging Requirement**: CLI applications must set `loggingEnabled: 'always'` when initializing the StatsigClient. By default, Statsig only enables logging in browser environments, but CLI session replay requires logging to be enabled in all environments to capture and send session data. ## Configuration Options The `StatsigCliSessionReplayPlugin` accepts optional configuration: ```javascript theme={null} import { StatsigCliSessionReplayPlugin } from '@statsig/cli-session-replay-node'; const plugin = new StatsigCliSessionReplayPlugin({ // Override the start timestamp (in milliseconds) startTimestamp: Date.now(), // Custom Asciicast header properties asciicastHeader: { title: 'My CLI App Session', command: 'my-cli-tool --verbose', env: { TERM: 'xterm-256color', SHELL: '/bin/bash' }, } }); ``` ### Configuration Properties * **`startTimestamp`** (optional): Override the recording start time in milliseconds. Defaults to `Date.now()`. * **`asciicastHeader`** (optional): Custom properties for the Asciicast header. For detailed information, visit the [Asciicast v2 File Format page](https://docs.asciinema.org/manual/asciicast/v2/#header). Common fields include: * `title`: Human-readable title for the recording * `command`: The command that was executed * `env`: Environment variables relevant to the session * `theme`: Terminal color theme object. ## Recording Limits * **Duration**: Sessions automatically end after 4 hours * **Size**: Recording stops if the session data exceeds 1MB ## Viewing Recordings CLI session recordings appear in the Statsig Console alongside web session replays. The recordings can be played back to see exactly what happened in the terminal, including: * All terminal output * Terminal resize events * Timing information for each interaction * Session metadata and environment details ## Manual Recording Control You can access the recording instance for manual control: ```javascript theme={null} import { CliRecording } from '@statsig/cli-session-replay-node'; // Check if currently recording if (CliRecording.isRecording()) { console.log('Session is being recorded'); } // Get current recording instance const recording = CliRecording.currentRecording; // Manually finish recording CliRecording.finish(); ``` ## Platform Support CLI Session Replay is currently supported on: * Node.js applications * Linux, macOS, and Windows terminals * Any terminal that supports standard input/output streams # Configure Statsig Session Replay Source: https://docs.statsig.com/session-replay/configure Configure Statsig Session Replay sampling rates, privacy masking rules, network capture settings, and event triggers across your applications. ## Conditional Recording In the Statsig Console, you can configure your Session Replay settings under **Project Settings → Analytics & Session Replay**. You must be a project admin to modify these settings. ### Global Targeting Gate The Global Targeting Gate controls who is *eligible* for session recording. If a user does not pass this gate, their sessions will never be recorded. By default, this is set to Everyone, meaning there are no restrictions—anyone can be recorded. You can think of this as defining the "top of the funnel" for session recording eligibility. ### Global Sampling Rate The Global Sampling Rate determines what percentage of eligible sessions are recorded from the start. By default, this is set to **100%**, meaning all eligible sessions are recorded automatically. You can lower this if you want to limit session recordings but still ensure a consistent percentage of sessions are always captured. This rate applies only to sessions that begin at the start and does not affect conditional triggers. ### Conditional Triggers: Events and Exposures Conditional triggers can start a session recording mid-session, even if it wasn’t recorded from the beginning. These triggers respect the Global Targeting Gate but operate independently of the Global Sampling Rate. When triggered, the recording includes the last 30 seconds leading up to the event (if rolling window is enabled). Types of conditional triggers: * **Individual Gate Exposures** — Trigger based on exposure to a specific gate, optionally filtered by group (e.g., Pass/Fail). * **All Gate Exposures** — Trigger based on exposure to any gate, optionally filtered by group (e.g., Pass/Fail). Individual Gate Exposure triggers override All Gate configuration * **Individual Experiment Exposures** — Trigger based on exposure to a specific experiment, optionally filtered by group (e.g., Test/Control). * **All Experiment Exposures** — Trigger based on exposure to any experiment, filtered by groups Test/Control. No other group names are supported, if an experiment includes additional groups, individual triggers must be configured for each one. Individual Experiment Exposure triggers override All Experiment configuration * **Events** — Trigger based on specific logged event, optionally filtered by event values (e.g., "purchase\_event" with value "book"). For each trigger, you can define an individual sampling rate. This rate is evaluated based on session\_id, meaning the result (pass or fail) will remain consistent for the same session, even if the trigger occurs multiple times. **All Gates** and **All Experiments** conditional triggers are only available in `3.30.1` or higher Important: If a conditional trigger occurs while a session recording is already in progress, the recording simply continues uninterrupted. Session replay settings panel with targeting and sampling controls Flowchart outlining conditional recording logic for events and exposures ### Example Walkthrough Suppose you have the following setup (See image above): * The Global Targeting Gate `session_replay_global_targeting_gate` allows all US users and excludes everyone else. * The Global Sampling Rate is set to 25%, so only 25% of eligible US user sessions are recorded from the start. * For the remaining 75% of eligible users, session recording can still begin mid-session if a conditional trigger occurs. Example Scenario: 1. A US user starts a session. They do not pass the 25% Global Sampling Rate, so their session is not recorded from the beginning. 2. Later, a `purchase_event` occurs with value `book`. This event is set up as a conditional trigger with a 50% sampling rate. If this session fails the sampling rate check, recording does not start. 3. A minute later, the user is exposed to the `cool_new_feature` gate, and the recording begins A trigger's sampling rate is consistent for the entire session based on session\_id. So if `purchase_event` fails the sampling rate once, future occurrences of the same event in that session will also fail. ### Initialization - StatsigTriggeredSessionReplay ```jsx theme={null} import { StatsigClient } from "@statsig/js-client"; import { runStatsigTriggeredSessionReplay } from "@statsig/session-replay"; import { runStatsigAutoCapture } from "@statsig/web-analytics"; const client = new StatsigClient( sdkKey, { userID: "some_user_id" }, { environment: { tier: "production" } } // optional, pass options here if needed ); runStatsigTriggeredSessionReplay(client, { autoStartRecording: true, keepRollingWindow: true, }); runStatsigAutoCapture(client); await client.initializeAsync(); ``` ```jsx theme={null} import { StatsigProvider, useClientAsyncInit } from "@statsig/react-bindings"; import { StatsigTriggeredSessionReplayPlugin } from "@statsig/session-replay"; import { StatsigAutoCapturePlugin } from "@statsig/web-analytics"; function App() { return ( Loading...} options={{ plugins: [ new StatsigTriggeredSessionReplayPlugin({ autoStartRecording: true, keepRollingWindow: true, }), new StatsigAutoCapturePlugin(), ], }} > ); } ``` #### Initialization Options * `autoStartRecording` * `true`: Recording *can* start automatically after initialization. Global targeting gate and sample rate are respected * `false`: You *must* manually start recording using startRecording(). This is helpful if you want to start the recording after a set point and block any auto-recording before then * `keepRollingWindow` * `true`: Statsig maintains a local rolling window of the last 30 seconds of the session, allowing recordings to include context leading up to a trigger. * `false`: If a conditional trigger occurs, recording begins from that moment onward, with no historical context. If you are utilizing bootstrapping, reach out to the Statsig team to confirm your server sdk is supported for conditional recording ## Advanced: Forcing a Recording on Demand You may have a use case where you want to manually start a recording. To do this, we offer the startRecording API which will begin recording as soon as you call it. * `startRecording`: Respects both the Global Targeting Gate and Global Sampling Rate. This is useful if you still want to start your recordings after a certain point (e.g. after login) but still take advantage of the Global Sampling Rate * `forceStartRecording`: Respects the Global Targeting Gate but is not subject to the Global Sampling Rate. Useful for debugging or when you don't want to be subjected to the Global Sampling Rate * `stopRecording`: Stops the current recording, if one is in progress. Calling this method when no recording is active has no adverse effects. After stopRecording is called, conditional recording triggers will **not** automatically restart the recording. Only an explicit call to `startRecording` or `forceStartRecording` will resume recording. If you have access to your Session Replay client, you can call these functions directly on the client instance. ``` const sessionReplayClient = new SessionReplay(client); … if (someCondition) { sessionReplayClient.startRecording(); } ``` If not, you can import the function from `@Statsig/session-replay` and call it using your SDK key ``` import { startRecording } from '@Statsig/session-replay'; … startRecording(CLIENT_SDK_KEY) ``` ## Additional Options These are options offered by the rrweb recorder (the open source recording tool we use) | key | default | description | | ------------------------ | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | blockClass | 'rr-block' | Use a string or RegExp to configure which elements should be blocked | | blockSelector | null | Use a string to configure which selector should be blocked | | ignoreClass | 'rr-ignore' | Use a string or RegExp to configure which elements should be ignored | | ignoreSelector | null | Use a string to configure which selector should be ignored | | ignoreCSSAttributes | null | array of CSS attributes that should be ignored | | maskTextClass | 'rr-mask' | Use a string or RegExp to configure which elements should be masked | | maskTextSelector | null | Use a string to configure which selector should be masked | | maskAllInputs | false | mask all input content as \* | | maskInputOptions | `{ password: true }` | mask some kinds of input \*
    refer to the [list](https://github.com/rrweb-io/rrweb/blob/588164aa12f1d94576f89ae0210b98f6e971c895/packages/rrweb-snapshot/src/types.ts#L77-L95) | | maskInputFn | - | customize mask input content recording logic | | maskTextFn | - | customize mask text content recording logic | | slimDOMOptions | `{}` | remove unnecessary parts of the DOM
    refer to the [list](https://github.com/rrweb-io/rrweb/blob/588164aa12f1d94576f89ae0210b98f6e971c895/packages/rrweb-snapshot/src/types.ts#L97-L108) | | dataURLOptions | `{}` | Canvas image format and quality ,This parameter will be passed to the OffscreenCanvas.convertToBlob(),Using this parameter effectively reduces the size of the recorded data | | inlineStylesheet | true | whether to inline the stylesheet in the events | | hooks | `{}` | hooks for events
    refer to the [list](https://github.com/rrweb-io/rrweb/blob/9488deb6d54a5f04350c063d942da5e96ab74075/src/types.ts#L207) | | packFn | - | refer to the [storage optimization recipe](https://github.com/rrweb-io/rrweb/blob/master/docs/recipes/optimize-storage.md) | | sampling | - | refer to the [storage optimization recipe](https://github.com/rrweb-io/rrweb/blob/master/docs/recipes/optimize-storage.md) | | recordCanvas | false | Whether to record the canvas element. Available options:
    `false`,
    `true` | | recordCrossOriginIframes | false | Whether to record cross origin iframes. rrweb has to be injected in each child iframe for this to work. Available options:
    `false`,
    `true` | | recordAfter | 'load' | If the document is not ready, then the recorder will start recording after the specified event is fired. Available options: `DOMContentLoaded`, `load` | | inlineImages | false | whether to record the image content | | collectFonts | false | whether to collect fonts in the website | | userTriggeredOnInput | false | whether to add `userTriggered` on input events that indicates if this event was triggered directly by the user or not. [What is `userTriggered`?](https://github.com/rrweb-io/rrweb/pull/495) | | plugins | \[] | load plugins to provide extended record functions. [What is plugins?](https://github.com/rrweb-io/rrweb/blob/master/docs/recipes/plugin.md) | | errorHandler | - | A callback that is called if something inside of rrweb throws an error. The callback receives the error as argument. | ## Limits ### 4 Hours Per Session or 30 Min Inactive Time Sessions will end after four hours total or if the user returns from inactive time greater than 30 minutes later. ### Recording Limits | Tier | Monthly Limit | Daily Limit | Hourly Limit | | :--------- | ------------: | ----------: | -----------: | | Free | 50,000 | 3,500 | 3,500 | | Pro | 100,000 | 7,000 | 7,000 | | Enterprise | 100,000 | 7,000 | 7,000 | Once these limit is reached, the SDK will automatically prevent new recordings from starting. You can monitor your session replay usage in your project settings. [Contact us](https://statsig.com/contact/demo) for custom contracts ### Replay Availability Time It can currently take about 1 hour from when the session is recorded to seeing it in your Statsig console. ### Default 30 Day Retention Sessions have a default retention period of 30 days and are automatically deleted after that time. You can configure a shorter retention period in settings if needed. Reducing your retention period does not affect your monthly session replay limit and is typically done for privacy and compliance purposes. # Debug Statsig Session Replay Source: https://docs.statsig.com/session-replay/debug Use Statsig Session Replay to debug user-reported issues by replaying real sessions alongside console logs, network requests, and feature gate values. ## Large Session Recording Warning In the Console, you may encounter the warning: > "This session recording is too large to load." This occurs when a session exceeds 50 MB in size. Unfortunately, once a session has been recorded at this size, it cannot be displayed or fixed retroactively. If you are consistently seeing this warning, here are a few steps you can take to reduce session sizes going forward: 1. **Disable Inline Stylesheets**\ When initializing the `SessionReplayClient`, set `inlineStylesheet` to `false`. We generally recommend keeping `inlineStylesheet` set to `true`, because it ensures recordings accurately reflect your original CSS even if you later update your styles. However, this setting often causes sessions to grow very large and is the most common source of bloat. ```jsx theme={null} runStatsigSessionReplay(client, { inlineStylesheet: false, }); ``` ```jsx theme={null} ... options={{ plugins: [ new StatsigSessionReplayPlugin({ inlineStylesheet: false, }), ], }} ... ``` 2. **Exclude Large Static Elements**\ If certain elements on your page are large but static (e.g., background images or videos), you can exclude them from session capture by adding the `rr-block` class to their `className`. This prevents those elements from being recorded and can significantly reduce session size. 3. **Reach Out for Assistance**\ If you continue to experience large sessions, feel free to reach out in [Slack Community](https://statsig.com/slack). A Statsig team member can review your session and provide tailored recommendations for your setup. # Install Statsig Session Replay Source: https://docs.statsig.com/session-replay/install Install Statsig Session Replay by adding the plugin to your JavaScript or React client SDK and configuring sampling, privacy, and capture options. Session Replay is supported on the Javascript or React SDKs for both desktop and mobile web users on your web application. See the instructions below to install our SDK and record user sessions. ## Option 1 - No code - Add Javascript Snippet to your website ```html theme={null} ``` Get YOUR\_CLIENT\_KEY from Project Settings -> Keys & Environments. Reveal the Client API Key, copy, and paste it over the \[YOUR-API-KEY] in the snippet above. And you're done! This will auto initialize the sdk and start recording sessions, no code required. If you'd like to use your existing Statsig integration, or customize the integration further, see option 2 below. If you still want to use the script tag but also customize the integration, remove your key from the script url and initialize using the javascript code below. ## Option 2 - Custom code - Install via Package Manager ```bash npm theme={null} npm install @statsig/js-client @statsig/session-replay @statsig/web-analytics ``` ```bash yarn theme={null} yarn add @statsig/js-client @statsig/session-replay @statsig/web-analytics ``` ```bash npm theme={null} npm install @statsig/session-replay @statsig/web-analytics @statsig/react-bindings ``` ```bash yarn theme={null} yarn add @statsig/session-replay @statsig/web-analytics @statsig/react-bindings ``` We recommend using autocapture as a great way to get started, but if you don’t want to automatically log and send events, you can remove the runStatsigAutoCapture option from the Javascript snippet or skip the `@statsig/web-analytics` package installation. Next, following the [instructions for the Statsig Javascript SDK](/client/javascript-sdk), initialize Statsig with your SDK key, [user](/concepts/user) and options: ```jsx theme={null} import { StatsigClient } from "@statsig/js-client"; import { runStatsigSessionReplay } from "@statsig/session-replay"; import { runStatsigAutoCapture } from "@statsig/web-analytics"; const client = new StatsigClient( sdkKey, { userID: "some_user_id" }, { environment: { tier: "production" } } // optional, pass options here if needed ); runStatsigSessionReplay(client); runStatsigAutoCapture(client); await client.initializeAsync(); ``` ```jsx theme={null} import { StatsigProvider, useClientAsyncInit } from "@statsig/react-bindings"; import { StatsigSessionReplayPlugin } from "@statsig/session-replay"; import { StatsigAutoCapturePlugin } from "@statsig/web-analytics"; function App() { return ( Loading...} options={{ plugins: [ new StatsigSessionReplayPlugin(), new StatsigAutoCapturePlugin(), ], }} > ); } ``` If you'd like to use Conditional Triggers you must use StatsigTriggeredSessionReplay. See Configure (Next page) for more information That's it! Continue reading Configure to learn more about controlling who, what, and when you record sessions. # Session Replay Overview Source: https://docs.statsig.com/session-replay/overview Overview of Statsig Session Replay for capturing and replaying real user sessions to debug bugs, study UX, and investigate experiment anomalies. Session Replay allows you to record users using your website or product, and play back those recorded sessions. This allows you to better understand how users use your service or website, diagnose problems, and uncover insights that help improve conversion and the overall user experience. A session recording plays back like a video in the Statsig Console, but is actually a serialized representation of your website and all of the events and interactions that occurred while the user was interacting with it. The recordings are captured using the [rrweb open source recording library](https://github.com/rrweb-io/rrweb). Recordings using this strategy are performant and space efficient, with options to apply user privacy filters to what is happening on screen. Session replay interface in Statsig console # Privacy Options for Session Replay Source: https://docs.statsig.com/session-replay/privacy Configure privacy and PII masking in Statsig Session Replay, including input masking, element-level redaction, and recording exclusion rules. To support your app’s privacy requirements and align with your organization’s policies, we provide multiple ways to control replay privacy: * **Baseline privacy options** - Select from three preset privacy configurations. Each option applies a different level of text masking, helping you protect PII and sensitive data according to your app’s and users’ needs * **Fine-grained privacy controls** - After choosing a baseline configuration, use CSS selector rules to mask, reveal, or block specific elements * **Global Targeting Gate** - Use a feature gate to define which users are eligible for replays, ensuring that recordings are limited to specific users or cohorts. In the Statsig Console, you can configure your privacy settings under **Project Settings → Analytics & Session Replay**. You must be a project admin to modify these settings. ### Baseline Privacy Options * **Passwords (Default)** — Only password inputs are replaced with asterisks (\*). All other text and inputs are shown as is * **Inputs** — All text in inputs are replaced with asterisks (\*). All other text is shown as is * **Maximum** — All text and inputs are replaced with asterisks (\*) ### Selector Rules Use CSS selectors to precisely control how individual elements are handled during session replay—whether they are masked, unmasked, or blocked. * **Masking** and **unmasking** apply only to text content. Masked text is replaced with asterisks (\*). * **Blocking** removes the element entirely from the replay and replaces it with a black placeholder of the same size. * **Password inputs cannot be unmasked**, regardless of selector rules. Selector rules override the baseline privacy settings. When multiple selector rules apply to the same element, the following precedence is enforced: **Block → Mask → Unmask** See the examples below for examples of how precedence is enforced. ```js theme={null} // Everything within the blocked class will // appear as a single black placeholder
    I will be part of the black placeholder
    ``` ```js theme={null} // The closest rule will apply
    Masked Text
    ``` ```js theme={null} // With conflicting rules applied at the same level, // the higher precedence will apply
    ``` ```js theme={null} // With baseline privacy setting set to Maximum, all text is masked // by default but this can be overwritten by unmasking
    Masked Text
    ``` All selectors must be valid CSS selectors. For details on supported selector syntax, see MDN’s list of CSS selectors Using selector rules or baseline privacy settings besides Passwords, will cause `maskTextFn`, `maskInputFn`, `maskTextSelector`, `maskAllInputs`, `maskInputOptions`, and `blockSelector` options passed in during initialization to be overwritten ### Global Targeting Gate The Global Targeting Gate controls who is *eligible* for session recording. If a user does not pass this gate, their sessions will never be recorded. By default, this is set to Everyone, meaning there are no restrictions—anyone can be recorded. You can think of this as defining the "top of the funnel" for session recording eligibility. If you are utilizing bootstrapping, reach out to the Statsig team to confirm your server sdk is supported # Watch Session Replays Source: https://docs.statsig.com/session-replay/watch Watch and explore captured sessions in Statsig Session Replay, including event timelines, network activity, console logs, and gate exposures. Session Replays can be found under the User’s group in the Statsig console’s navigation panel. Session replay navigation in Statsig console The main interface contains three major sections. The leftmost column shows a list of available replays to watch, and a way to filter among them. The middle playback surface is where can play the replay, pause, and skip ahead and back to events of interest. You can also skip ahead by clicking the next button to jump ahead to the next event. You can also click on an event of interest in the right most “events panel” to skip to that event in the replay timeline. Session replay interface with playback controls and events panel For a more immersive replay experience you can hide the events panel: Session replay interface with hidden events panel You can also enter full screen mode. # Find a Replay Session replay selection interface with replay cards You can select a replay to watch by selecting it from the list of replays on the left. The “Replay Card” contains information such as the URL, Browser, Country, and more, that you can use to quickly scan for interesting replays or as a way to quickly filter to interesting replays. To more narrowly scope the set of replays to watch, you can add filters. Filters allow you to scope to replays that contain a specific event, user, Feature Gate exposure (pass or fail), or Experiment group exposure. As a reminder, session replay is currently in beta - we will be adding more powerful filtering functionality in the coming weeks. # Playlists Playlists allow you to organize and group related session recordings for easier analysis and collaboration. Instead of searching through individual sessions, you can create curated collections of recordings that focus on specific user behaviors, issues, or research questions. ## Key Features * Organized Collections: Create named playlists to group related session recordings together. This makes it easier to focus on specific user journeys, bug reports, or research topics without getting lost in a sea of individual sessions. * Collaborative Analysis: Share playlists with your team members to collaborate on user experience analysis. Team members can view the same curated set of recordings to discuss findings and insights together. * Persistent Organization: Unlike temporary filters, playlists persist your organizational structure, making it easy to return to specific sets of recordings for ongoing analysis or follow-up research. * Seamless Integration: Playlists integrate seamlessly with the existing session replay interface. When you open a playlist, you can view all the recordings within it using the same powerful replay player and analysis tools. # SRM Checks Source: https://docs.statsig.com/stats-engine/methodologies/srm-checks How Statsig detects sample ratio mismatch (SRM) in experiments and how to debug skewed traffic splits caused by targeting, exposure, or logging issues. ## SRM - Sample Ratio Mismatch Sample ratio mismatch (SRM for short) is when the observed allocation of **unique** users between test groups differs from the expected allocation or "split" of the test. We have a brief [rundown on this topic here](https://www.statsig.com/blog/sample-ratio-mismatch) on our blog. This is a signal that there could be some unknown bias in the test. This is a major problem because unless you can clearly diagnose the reason for the imbalance, there's not an easy way to know how much this bias impacts your results. ## SRM Checks Statsig runs SRM checks on all experiments and feature gates as part of our Health Checks (described [here](/experiments-plus/monitor)). We use a Chi-squared test to identify if the split of users between groups is indicative of a Sample Ratio Mismatch. SRM health check results interface We automatically analyze data by common dimensions logged by the Statsig SDK to identify potential drivers of SRM. These include sdk\_type, sdk\_version, reason, is\_bot, browser\_name, browser\_version, os, os\_version, and region to identify potential causes. SRM dimension analysis breakdown # Egress, Privacy, & Storage Source: https://docs.statsig.com/statsig-warehouse-native/analysis-tools/data-privacy Understand how Statsig Warehouse Native handles data privacy in your warehouse, including what data is read, where computations run, and what is exported. One advantage of using Statsig Warehouse Native is that user-level data comes directly from your source of truth without needing to be copied or leave your warehouse. This page walks through how Statsig interacts with your warehouse. ## Permissions The permissions Statsig requires are: * Job Access (where applicable) to run queries as a service user * Read Access on event, metric, and exposure data you want to use for experiment analysis * Statsig only selects from these tables * Read, Write, Delete access in a Statsig Staging environment you specify * In Bigquery this is a dataset * In Snowflake and Redshift this is a schema * In Databricks this is a database, or a database within a separate workspace with a scoped delta share to your production datasets Best practice is to create a new dataset for Statsig Staging to make sure this environment is isolated. Statsig only modifies tables it creates as part of analysis. ## What Statsig Reads Statsig only reads two forms of data, both of which are very small (generally in Kilobytes) * Small samples used for validating setup in the following surfaces: * Metrics Tab * Metric Source Tab * Assignment Source Tab * (If using Statsig exposures) Users Tab * Aggregated results at the group or experiment level. This data doesn't contain user IDs and is rolled up at the group, metric, group/metric or experiment level. This includes: * Existing groups and group sizes for experiment setup * Pulse Results * Total exposure counts for power analysis * Daily total metric values * Experiment-level health checks (e.g. distinct metrics, count of exposures) ## During analysis, data stays in your warehouse. Intermediate tables and results are written to the Statsig data staging set. When the results of Health Checks or Pulse become available, Statsig consumes those result sets and stores them on its servers as well (usually \< 1000 rows): Analysis Flow ## Data Retention in Statsig Customer data contained within exposure events when using the SDK for assignment is retained for a maximum of 30 days purely for diagnostics and debugging purposes. Statsig will automatically remove any customer data no longer than 30 days after the events are sent to our system. ## Storage Management Experimentation staging datasets can generate a lot of data, since they can potentially blow up your data by the number of experiments you run; for example, if you have user-day data and run 100 experiments that all expose every user, you'd end up with 100 copies of the data with slight differences based on when users were exposed to the various experiments. To help manage this, Statsig has a table management system to help manage storage costs and visibility: * Temporary artifacts (tables generated as part of explore queries or pulse results) are dropped 2-7 days after creation. This gives some buffer to debug, but they won't maintain long-term copies of data * When you finish an experiment by making a decision, you're prompted to delete the tables. By default, Statsig leaves the result sets (on the order of kilobytes) in your warehouse for reference, but you can override this setting. * At any time, you can drop tables for your experiment from the three-dot menu in the experiment. # Pipeline Overview Source: https://docs.statsig.com/statsig-warehouse-native/analysis-tools/pipeline-overview Understand the queries and computations Statsig Warehouse Native runs on your data warehouse for experiment analysis, including pipeline stages and outputs. This page is intended to give you a high level overview of the pipeline Statsig Warehouse Native will run on your warehouse. ## Main Steps The main steps in the pipeline are: * Identifying users' first exposures * Annotating Metric Sources with exposure data * Creating metric-user-day level staging data * Running intermediate rollups for better performance * Calculating group-level summary Statistics Pipeline View ## Types of DAGs Statsig lets you run your pipeline in a few different ways: * A **Full Refresh** totally restates the experiment's data and calculates it from scratch. This is useful for starting an experiment, or if underlying data has changed * An **Incremental Refresh** appends new data to your experiment data. This reduces the cost of running scheduled updates to your results * A **Metric** refresh allows you to update a specific metric in case you changed a definition, or want to add new metrics to your analysis ## Artifacts and Entity Relationships The following tables will be generated and stored in your warehouse per-experiment. You have full access to these data sources for your own analysis, models, or visualizations. For experiments, `experiment_id` will be the name of the experiment; for Feature Gates, `experiment_id` will be the name of the gate along with the specific rule ID (e.g. `chatbot_llm_model_switch_31e9jwlgO1bSSznKntb2gp_exposures_summary`) This is not an exhaustive list, but includes most of the core result/staging tables that you might be interested in using for your own analysis. Note - These are internal tables and will change as the product evolves. Changes will be documented here. | Table | Description | Notes | | ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------- | | `first_exposures_` | Deduplicated and stitched (for experiments with ID resolution) first exposure events | Useful for ad-hoc analysis | | `exposures_summary_` | Timeseries of exposures per group for display in Pulse | | | `unit_day_metrics_` | User-day level metric aggregations table | Useful for ad-hoc analysis | | `unit_covariate_metrics_` | User-level pre-experiment aggregations for regression adjustment/CUPED | | | `funnel_events_` | Staging table for running funnel analysis | | | `percentile_values_` | Staging table for running percentile analysis | | | `distinct_values_` | Staging table for running count distinct analysis | | | `windowed_metrics_` | Staging table for generating running totals when restating Pulse | | | `ratio_aggregations_` | Staging table for generating running totals when restating Pulse | | | `results__` | Outputs of Statistical Analysis for different rollups (e.g. daily, days-since-exposure, cumulative, 7-day). Exported to Statsig | Pulse inputs - useful for replicating Statistical analysis | | `ratio_results__` | Outputs of Statistical Analysis for ratio metrics in different rollups (e.g. daily, days-since-exposure, cumulative, 7-day). Exported to Statsig | Pulse inputs - useful for replicating Statistical analysis | The high level relationships/contents of these tables are represented below - refer to the Main Steps image below for scheduling details. WHN ER Diagram ## Other Jobs Alongside and inside this main flow, Statsig will also: * Run Health Checks and a Summary View for exposures * Calculate top dimensions for dimensional metrics * Calculate funnel steps * Run CUPED and Winsorization procedures during the group-level summaries to reduce variance and outlier influence * Calculate inputs to the Delta Method to avoid bias on Ratio and Mean metrics Statsig generates experiment-level tables - this makes it easy to do your own follow-up analyses on specific experiments. ## Visibility Clicking into the history icon on your pulse results, you'll be able to see the Jobs and IDs we ran for each pulse reload, alongside relevant information on compute time and cost. This will also be fully transparent from your own Warehouse's history and usage management, but having the costs in console is helpful knowledge for the cross-functional experimentation teams running the analysis. ## Exposure Export Table Statsig dedupes and records production exposures into the forwarded exposures table configured in your warehouse Data Connection. This table contains each user's first exposure to an experiment. For feature gates, we dedupe and record exposures for partial rollouts (e.g. 5% or 50% rollouts - but not 0% or 100% rollouts). | Column Name | Data Type | Description | | ------------------ | --------- | ------------------------------------------------------ | | experiment\_id | string | The identifier for the gate/experiment | | group\_id | string | groupID for experiments; ruleID+Pass/Fail for gates | | group\_name | string | Name of the experiment group (e.g. Control vs Test) | | user\_id | string | The ID passed in as the Statsig userID | | stable\_id | string | Statsig Client SDK managed stable device identifier | | \[your custom ids] | string | One column for every custom unitID you use on Statsig | | timestamp | timestamp | Timestamp of the first exposure | | user\_dimensions | object | Warehouse specific object with all the user dimensions | `user_dimensions` is populated in the daily deduplicated export. Fast-forwarded exposure rows can omit some fields in this object until the next daily load. ### Common Fields in user\_dimensions `user_dimensions` contains user attributes captured alongside the first exposure. The exact shape can vary by SDK and project configuration, but these are some of the most common fields you may see: | Field | Description | Notes | | ----------------- | -------------------------------- | ------------------------------------------------------- | | `os` | Normalized operating system name | Canonical OS field on the exposure side. | | `os_version` | Operating system version | Derived from SDK metadata or user agent parsing. | | `browser_name` | Browser name | Derived from SDK metadata or user agent parsing. | | `browser_version` | Browser version | Derived from SDK metadata or user agent parsing. | | `device_model` | Device model | Forwarded or inferred when available. | | `ip` | IP address | Present when available from the SDK or request context. | | `country` | Country | Derived from request context or IP lookup. | | `locale` | Locale | Forwarded or inferred when available. | | `language` | Language | Forwarded or inferred when available. | | `appVersion` | Application version | Forwarded when present on the SDK user object. | | `sessionID` | Session identifier | Forwarded when present on the SDK user object. | | `appIdentifier` | Application identifier | Forwarded when present on the SDK user object. | Additional non-null fields from the SDK user object may also appear in `user_dimensions`. Custom IDs are typically exported as dedicated top-level columns in the exposure table rather than being queried from this object. Input fields such as `deviceOS` and `systemName` are used to derive the exported `os` field. If you want to analyze operating system on forwarded exposures, query `user_dimensions.os`. ## Event Export Table If you log custom events through a Statsig SDK, Statsig also forwards those events into a configurable table in your warehouse. This is the table used when Warehouse Native customers rely on Statsig SDK logging for outcome events. * Use `user_object` for user fields associated with the event. * Use `statsig_metadata` for SDK and exposure-processing metadata. * Use `company_metadata` for the event metadata payload you logged. | Column Name | Data Type | Description | | ------------------ | --------- | -------------------------------------------------------------------------- | | user\_id | string | The ID passed in as the Statsig userID | | stable\_id | string | Statsig Client SDK managed stable device identifier | | \[your custom ids] | string | One column for every custom unitID you use on Statsig | | timestamp | timestamp | Event timestamp | | event\_name | string | Name of the logged custom event | | event\_value | string | Optional event value | | user\_object | object | Warehouse specific object containing user fields associated with the event | | statsig\_metadata | object | Warehouse specific object containing Statsig SDK and exposure metadata | | company\_metadata | object | Event metadata payload logged with the event | ### Common Fields in user\_object `user_object` contains user fields associated with the event. It often includes the same common fields as `user_dimensions`, plus any additional non-null fields sent on the SDK user object. | Field | Description | Notes | | ----------------- | -------------------------------- | ------------------------------------------------------- | | `os` | Normalized operating system name | Canonical OS field on the event-side user object. | | `os_version` | Operating system version | Derived from SDK metadata or user agent parsing. | | `browser_name` | Browser name | Derived from SDK metadata or user agent parsing. | | `browser_version` | Browser version | Derived from SDK metadata or user agent parsing. | | `device_model` | Device model | Derived from `deviceModel` when provided. | | `ip` | IP address | Present when available from the SDK or request context. | | `city` | City | Added when geographic inference is available. | | `state` | State or region | Added when geographic inference is available. | | `country` | Country | Derived from request context or IP lookup. | | `locale` | Locale | Forwarded or inferred when available. | | `language` | Language | Forwarded or inferred when available. | | `appVersion` | Application version | Forwarded when present on the SDK user object. | | `sessionID` | Session identifier | Forwarded when present on the SDK user object. | | `appIdentifier` | Application identifier | Forwarded when present on the SDK user object. | ### Common Fields in statsig\_metadata `statsig_metadata` contains SDK-level and exposure-processing metadata associated with the event. These are some of the most common customer-facing fields: | Field | Description | Notes | | ----------------------- | ------------------------------- | --------------------------------------------------------------------- | | `deviceType` | High-level device category | Derived from the normalized OS, for example `Desktop` or `Mobile`. | | `targetAppID` | Target app identifier | SDK target app metadata. | | `statsigTier` | Statsig environment tier | For example `prod` or `staging`. | | `keyEnvironment` | SDK key environment | Environment associated with the SDK key. | | `keyID` | SDK key identifier | Useful for debugging ingestion and environment issues. | | `samplingRate` | Event sampling rate | Present when the event is sampled. | | `is_bot` | Bot classification | Set when Statsig classifies the event as bot traffic. | | `billing_type` | Exposure billing classification | Common on exposure-related events. | | `groupID` | Experiment group identifier | Common on exposure-related events. | | `ruleID` | Rule identifier | Common on exposure-related events. | | `continuous_rollout_id` | Continuous rollout identifier | Present for continuous rollout exposures. | | `configExposureType` | Exposure config type | For example `dynamic_config` or `experiment`. | | `isSwitchback` | Switchback flag | Present when the exposure is associated with a switchback experiment. | | `is_autotune` | Autotune flag | Present when the exposure is associated with an autotune experiment. | Other SDK debugging and exposure-processing metadata may also appear in `statsig_metadata`. ### What Goes in company\_metadata `company_metadata` stores the metadata payload logged with the event. This object does not have a fixed schema and will vary based on the event and your SDK usage. Most event-specific business context, such as `price`, `currency`, `category`, `plan`, `screen`, `route`, or nested objects like `cart`, `items`, and `context`, will appear here. # Assignment Sources Source: https://docs.statsig.com/statsig-warehouse-native/configuration/assignment-sources Configure assignment sources in Statsig Warehouse Native so experiment exposures from your warehouse map to the correct experiment and variant. Assignment Sources are how you schematize your assignment data for Statsig, and they serve as the input data for determining who is in an experiment, and which treatment they got. ## Creating an Assignment Source To create an assignment source, go to the data tab in Statsig and go to the Assignment Sources pane. An Assignment Source is defined as a SQL query and a mapping of the output columns to specific fields Statsig requires (user identifiers, a `timestamp`, an experiment identifier, and a group identifier). Assignment Source ## Scanning Assignment Sources Statsig scans assignment sources on-demand and/or on a schedule to find experiment data. These jobs are very quick and identify unique groups, the ID types present in the experiment, and the estimated of users per group. Once the scan is complete, you can view and create experiments from the Assignment source. The assignment's experience will also populate the Experiment creation flow after the scan completes. Assignment source scan results showing detected experiments ## Manage Assignment Sources In the Assignment Source tab, you can see your Assignment sources and the experiments they're being used in. Assignment Source Tab ## Example Data For experiment assignment sources, Statsig requires information on who was exposed, when, and to what experiment: | Column Type | Description | Format/Rules | | ---------------------- | ----------------------------------------------------------------------------------------- | ------------------------------ | | timestamp | **Required** an identifier of when the experiment exposure occurred | Castable to Timestamp/Date | | unit identifier | **Required** at least one entity to which this metric belongs | Generally a user ID or similar | | experiment identifier | **Required** the experiment the exposure was for | Usually an experiment name | | group identifier | **Required** the experimental variant the user was assigned to | Usually a group name | | additional identifiers | *Optional* Entity identifiers for reuse across identifier types | | | context columns | *Optional* Fields which can be used to group by and filter results in exploratory queries | | For example, you could pull from exposure event logging directly: | timestamp | user\_id | company\_id | experiment\_name | group\_name | country | | ------------------- | --------------- | ----------- | ------------------- | ----------- | ------- | | 2023-10-10 00:01:01 | my\_user\_17503 | c\_22235455 | ranking\_v1\_vs\_v2 | v1 | US | | 2023-10-10 00:02:15 | my\_user\_18821 | c\_22235455 | ranking\_v1\_vs\_v2 | v2 | CA | | 2023-10-10 00:02:22 | my\_user\_18821 | c\_22235455 | search UI revamp | control | CA | # WHN Console API Source: https://docs.statsig.com/statsig-warehouse-native/configuration/console-api Use the Statsig Console API with Warehouse Native to programmatically manage metric sources, assignment sources, experiments, and pipeline configurations. Statsig's console API allows you to programmatically perform CRUD operations on all of the configuration objects above. This means you can do bulk, programmatic edits, or use the API to sync your definitions into Statsig as a triggered or scheduled job. You can also use this API to trigger pulse results and manage workflows in Statsig. [Click here to see the full Console API Docs.](/console-api/introduction) # Data & Semantic Layer Source: https://docs.statsig.com/statsig-warehouse-native/configuration/data-and-semantic-layer Configure the data and semantic layer in Statsig Warehouse Native, including dbt and Looker integrations for reusing existing metric definitions. When using Statsig Warehouse Native, you configure metrics, experiments, and other schematized data objects that allow Statsig to perform analysis on top of your warehouse. This functions as a lightweight Semantic Layer, which can integrate neatly into popular Semantic Layer tools like Cube or in-house YAML-based stores. This section dives into the building blocks of Statsig's configuration layer. Warehouse Native data flow architecture diagram # Dimension Analysis Source: https://docs.statsig.com/statsig-warehouse-native/configuration/dimensional-analysis A powerful way to understand who or what is causing a metric movement, dimension analysis lets you break down an experiment result by unit or action. dimension example You can also configure [Differential Impact Detection](/experiments-plus/differential-impact-detection) to help Statsig automatically flag when different classes of users are responding differently to your experiments. To see the Pulse result breakdowns for all categories within a metric, click on the (+) sign next to the metric. # Unit Dimensions Unit dimensions refer to unit-level attributes that are either part of the user object you log to Statsig, provide as part of your [assignment data](/statsig-warehouse-native/configuration/assignment-sources), or provide via an [Entity Property](/statsig-warehouse-native/configuration/entity-properties). Examples of these attributes are operating system, country, region, or user segments. Using [explore queries](/pulse/custom-queries), you can filter to specific unit dimensions or group results by a dimension. For example, you could "See results for users in the US", or "See results for users using iOS, grouped by their country". The dimension will be chosen based on the last available record at or before exposure. In other words, information from AFTER the unit is exposed to a given experiment will *not* be used in the experiment analysis, since that could potentially lead to data leakage and imbalanced comparisons. # Metric Dimensions Metric Dimensions break down a metric's results based on the values in columns from your [metric source](/statsig-warehouse-native/configuration/metric-sources) for a given metric. You configure these breakdowns per-metric, after which they'll be calculated for that metric across all pulse results. Note that, unlike Unit Dimensions, these dimensions are not mutually exclusive. For example, a with a user dimension a user can only be from one country for the purpose of pulse analysis, but a user on an e-commerce website could buy all of "clothes", "books", and "snacks" within a "total purchases" metric and contribute to each of those dimensions as well as the overall value. # Details By default, dimensional analysis: * only considers dimensions with at least 100 units in the experiment that participated (had a non-zero value) * chooses the top 10 dimensions by total value, and puts all others into an "OTHER" bucket. This is to avoid extreme results and cases where assumptions of centrality do not hold due to low sample on a specific dimension, and to avoid excessive multiple-comparisons on the tail-end of dimensional breakdown. This is configurable, but controlled due to the potential for error - reach out to Statsig support to see if your use case makes sense. ## Loading Timing For precomputed user dimensions that are configured and run on a schedule, dimension data is processed asynchronously and may take a few minutes to become available after the main experiment results load. You may temporarily see "No dimensions available for this time range" messages while the data is being processed, especially after the first reload of the day. This is expected behavior - simply wait a few minutes and refresh the page to see the dimensional breakdowns. This timing behavior only affects precomputed user dimensions. User-triggered dimensional analysis does not experience this delay. # Entity Properties Source: https://docs.statsig.com/statsig-warehouse-native/configuration/entity-properties Use Entity Properties in Statsig Warehouse Native to attach categorical attributes to experiment units for filtering and grouping results in Explore. You can either provide additional detail about an entity that doesn't typically change (e.g. a user's home country), or a property that may change as part of an experiment (e.g. Subscriber Status : True/False). For the latter, you provide a timestamp field which will be used to identify most recent value prior to the user's exposure. This prevents imbalanced groups and biased results from when an experimental treatment impacts the property, for example if it increased the subscription rate. Entity Properties configuration interface Entity Properties setup screen with timestamp configuration ## Example Data For property sources, Statsig only needs a user\_id and property fields. Property sources can define **fixed** properties (e.g. a users Country of origin), but can also define **dynamic** in which case you need to provide a timestamp for Statsig to identify the most recent pre-exposure record. | Column Type | Description | Format/Rules | | ---------------- | ------------------------------------------------------------------------------------------- | ------------------------------ | | timestamp | *Optional* an identifier of when the property was defined. Required for dynamic properties | Castable to Timestamp/Date | | unit identifier | **Required** At least one entity to which this metric belongs | Generally a user ID or similar | | property columns | **Required** Fields which can be used to group by and filter results in exploratory queries | | For example, a static property source could just be: | user\_id | company\_id | country | | --------------- | ----------- | ------- | | my\_user\_17503 | c\_22235455 | US | | my\_user\_18821 | c\_22235455 | CA | Which could be used to filter and group by any experiment that was exposed one either user\_id or company\_id. For a dynamic property, it might look like this: | user\_id | timestamp | company\_id | intent\_segment | spend\_segment | | --------------- | ---------- | ----------- | --------------- | -------------- | | my\_user\_17503 | 2023-10-10 | c\_22235455 | high\_intent | high | | my\_user\_17503 | 2023-10-11 | c\_22235455 | high\_intent | high | | my\_user\_17503 | 2023-10-12 | c\_22235455 | mid\_intent | high | | my\_user\_18821 | 2023-10-10 | c\_22235455 | low\_intent | low | | my\_user\_18821 | 2023-10-11 | c\_22235455 | low\_intent | mid | | my\_user\_18821 | 2023-10-12 | c\_22235455 | low\_intent | mid | The first user in this example has their intent\_segment property change on `2023-10-12`; based on what the intent\_segment was prior to their exposure, they might have different intent\_segment values for different experiment analyses. # Metric Examples Source: https://docs.statsig.com/statsig-warehouse-native/configuration/metric-examples Example metric definitions in Statsig Warehouse Native, including SQL snippets and metric source mappings for common product, revenue, and engagement metrics. For customers transitioning from other Warehouse Native Vendors, the format of metrics should be similar and generally customers have been able to use APIs to fetch remote configurations, translate them, and post them to Statsig without issue. For customers migrating from in-house systems, there may be gaps in translation between how they think about experiment metrics and how Statsig handles them. This page is intended as a collection of common use cases and how they're handled in Statsig. ## Average User Revenue from a Wide Table In many cases, companies will have a primary source-of-truth table about user engagement with one row per user-day and many columns representing actions taken or other values. This is very easy to integrate with Statsig. First, enter the table path and optionally a partition column to use for date partitioning: Metric source setup specifying table path and date partition Then, configure your timestamp field and ID types. Add any custom SQL aliases for other users, e.g. dividing revenue by 100 to convert from cents to dollars. Timestamp and ID type configuration with revenue alias Go to the metrics tab, press create, configure your name/source, and then configure a sum metric on the column with the revenue value. Create metric dialog selecting revenue source Sum metric configuration for revenue column ### How it works in experiments First, Statsig aggregates each unit-level record across the days they are enrolled in the experiment. Then, Statsig will calculate the mean unit-level revenue per experiment group, imputing 0s for all exposed users with no revenue. Statsig provides a description of this in-product for any user who wants to learn more: Inline description explaining aggregation behavior ## Average Current Account Value Often, you will want to understand if your experiment has altered the "state" of users. Let's say you care about the current account value today on users in test vs. control of your experiment - have you helped users grow their account? On your end, you'll just need a table or query that tracks users' account values each day. Then, set up a metric source pointing to that table or query. Go to the metrics tab, press create, configure your metric name & source, and then configure a latest value metric on the column with the account value. Create metric form for latest account value Latest value metric configuration using account column ### How it works in experiments First, at unit level, Statsig calculates each day's latest non-null value within any cohort bounds and takes the latest value from the latest day available. Then, Statsig will calculate the mean unit-level value per experiment group on each day, imputing 0s for all exposed users with no value. Statsig provides a description of this in-product for any user who wants to learn more: Product tooltip describing latest value aggregation logic ## Users' D7 Participation Rate A common metric in experimentation is measuring whether exposed users take specific actions within a defined time window. On your end, you will just need to provide an event table that records user action with essential columns such as user\_id, timestamp and event type. Similarly as above, configure your timestamp field and ID types. Event table metric source definition for participation rate ID type configuration for participation event data Then you can navigate to the metric catalog and create a unit count metric using the defined metric source. You could leverage the 'Add Filter' option to focus on specific events relevant to your designed metric. Unit count metric creation with filters for specific event When defining the metric, you can choose from several rollup modes: * Daily Participation Rate -> it measures the days a unit was active after being exposed to the experiment divided by its total days in the experiments * On-Time Event -> it measures if a unit performed an action any time after being exposed to the experiment * Latest Value -> it measures if a unit passed metric filters on their last observed record * Custom Attribution Window -> to include data for each unit in a specified time window after being exposed to the experiment Metric source selection during creation flow In our example, we want to measure the user participation within 7 days. So you can pick 'Custom Attribution Window' as your rollup mode and set start = 0 end = 6 to define a 7-day window. Option to enable 'Only include units with a completed window' to exclude users who haven't reached the full 7-day period from your analysis. Rollup mode options for participation metrics ### How it works in experiments First, at unit level, Statsig will create a 0/1 flag if the event is triggered during the specified time window. Then, at the group level, the mean is calculated as the SUM of the unit-level flags, divided by the count of UNIQUE UNITS exposed to the experiment. Statsig provides more details about how Unit Count (Window) Metrics are calculated [here](/statsig-warehouse-native/metrics/unit-count-window). ## User Funnel Metric A common analysis in experimentation is understanding how a new feature impacts dropoff rates at each step of a user funnel. To create a funnel metric in Statsig, you need an event table that records each step of the events you want to track. The setup for your metric source follows the same process as described earlier. When you navigate to the metric catalog, select 'Funnel' as your metric type. Choose the unit level for your funnel steps – this can be a distinct count of users or sessions based on what you want to measure. Average session count metric configuration Then, you cam define your funnel steps, specifying the sequence of events users go through. Example showing multi-event participation configuration In the Advanced Settings, you can further customize your funnel metric to fit different use cases. Options include specify calculation window, measure time to convert, treat exposure as initial funnel event, etc. These settings provide full flexibility, allowing you to tailor the funnel metric based on your specific analysis needs. Metric filters applied to isolate specific event values ### How it works in experiments First, at unit level, a 1/0 (or session-count number for session funnels) metric is constructed for each step of the funnel. This flag is 1 if the unit completed that step some time after all previous steps were completed in order. If using a session-level funnel, it's the number of sessions where that is true, e.g. all previous steps were completed in order for that session key. Then, at the group level, the stepwise mean is calculated as the total of each step's metric divided by the total metric from the previous step. The overall mean is calculated as the units/sessions that completed the funnel divided by the unit/sessions that started the funnel. Statsig provides a description of this in-product for any user who wants to learn more: Metric catalog entry summarizing event-based conversion metric ## User Retention Rate A retention metric is a great way to measure changes in user stickiness and product growth with the new feature you've built. To create a retention metric in Statsig, you'll need an event table that captures the key activities indicating user retention. The setup for your metric source follows the same process as described earlier. When you navigate to the metric catalog, select 'Retention' as your metric type. Configure the retention period and look back window. For example, if you set your 'Retention Period End' to be 14 and retention lookback window to be 7, retention is measured as whether the user has triggered the retention event between day 8 and day 14. Configuration UI for retention-style rollup You also have the option to "Use a different start and completion event for retention calculations" if you don’t want to use exposure as the starting event or if you want to define a specific subset of events as your retention event. For example, based on the setup shown in the screenshots, we will be measuring the week 2 retention rate of users who made a purchase in week 1. Metric details view describing rollup logic In the Advanced Settings, you can configure what's the ID type for your retention metric. Metric result example for session count ### How it works in experiments First, for each unit per day, Statsig checks if the retention start event is triggered and assigns a 0/1 flag, which serves as the denominator of the calculation. Next, Statsig checks if the retention completion event occurs within the specified time window and assigns a 0/1 flag, which serves as the numerator of the calculation. Finally, at the group level, retention is calculated as sum(numerator) / sum(denominator) to determine the overall retention rate. Statsig provides a description of this in-product for any user who wants to learn more: Example of metric insight card for participation rate # Metric Sources Source: https://docs.statsig.com/statsig-warehouse-native/configuration/metric-sources Define metric sources in Statsig Warehouse Native that map warehouse tables and SQL queries to metric definitions used across experiments and analytics. Metric Sources are how you schematize your warehouse data for Statsig, and they serve as the input data for metrics. ## What is a Metric Source A metric source is a key part of Statsig's semantic layer (and integrations to other Semantic Layers). A metric source consists of a data source, which is either: * A SQL Query that will be read as a view * A warehouse table And configuration around the source: * **\[Required]** identifier and timestamp columns * **\[Optional]** Aliases, partition information * **\[Beta]** Data quality checks and configuration This is the gateway for your data to be used in parameterized queries for experiment analysis, analytics, and more within the Statsig console. ## Data Sources ### Getting Data Statsig Metric Sources can use a query or a table as the source of their data. A query-based source will read a SQL query and use that as a source of truth. This, plus Statsig's built-in [query macros](/statsig-warehouse-native/configuration/query-tools), provides an efficient and extremely flexible way to create experimental data sources. Query-based metric source configuration For larger or managed datasets, it's recommended to use table sources instead. This will minimize data scan and provides a more 1:1 mapping of "data source" to "metric source". Statsig date macros are automatically applied in each experiment result reload when you use table sources. Table-based metric source configuration ### Configuring Data For any source, you'll be asked to select which field to use for the timestamp of the logs/metrics being provided, as well as 1 to N user identifiers that can be used to join to experiment data. Interface for selecting timestamp and ID fields For table sources, you can optionally provide a partitioning column to reduce data scan, and provide aliases to format data as desired and make your column names more human-readable. Partition and alias settings for table source ### Types of Data Statsig works natively with many different types and granularities of data. Common patterns are: #### 1. Raw event logging (event level data), using the log timestamp as the timestamp, example: | `event_time` | `user_id` | `event_name` | `platform` | `value` | | ------------------- | --------- | -------------- | ---------- | ------- | | 2024-03-01 10:05:12 | `u_123` | `page_view` | `web` | `null` | | 2024-03-01 10:05:45 | `u_123` | `button_click` | `web` | `null` | | 2024-03-01 10:07:02 | `u_456` | `purchase` | `ios` | 29.99 | *Metric examples with this source:* *- Number of users with purchase: You can create a [UNIT COUNT](/statsig-warehouse-native/metrics/unit-count-once) metric, with a filter of event\_name = 'purchase'.* *- % of users with page view who clicked: You can create a [RATIO](/statsig-warehouse-native/metrics/ratio) metric with this metric source in both denominator and numerator, and apply event\_name filter accordingly.* #### 2.Fact tables (one row per entity per day), using the date of the row as the timestamp, example: | `order_date` | `order_id` | `user_id` | `order_status` | `items_count` | `revenue` | | ------------ | ---------- | --------- | -------------- | ------------- | --------- | | 2024-03-01 | `o_10001` | `u_123` | `completed` | 2 | 49.98 | | 2024-03-01 | `o_10002` | `u_456` | `completed` | 1 | 19.99 | | 2024-03-02 | `o_10003` | `u_123` | `refunded` | 1 | 19.99 | *Metric examples with this source:* *- Revenue: You can create a [SUM](/statsig-warehouse-native/metrics/sum) metric on 'revenue' column, with any filters you need.* *- Average order value: You can create a [MEAN](/statsig-warehouse-native/metrics/mean) metric on 'revenue' column, with any filters you need.* #### 3.Aggregated fact tables at unit day granularity, using the date of the row as the timestamp, example: | `date` | `user_id` | `sessions` | `purchases` | `revenue` | | ---------- | --------- | ---------- | ----------- | --------- | | 2024-03-01 | `u_123` | 3 | 1 | 29.99 | | 2024-03-01 | `u_456` | 1 | 0 | 0.00 | | 2024-03-02 | `u_123` | 2 | 0 | 0.00 | *Metric examples with this source:* *- Revenue: You can create a [SUM](/statsig-warehouse-native/metrics/sum) metric on 'revenue' column, with any filters you need.* *- Number of users with purchase: You can create a [UNIT COUNT](/statsig-warehouse-native/metrics/unit-count-once) metric, with a filter of purchase > 0.* ### Types of data that needs some transformation There are times when you might have a table that does not fall into the schema mentioned above. For example, a wide user dimension table contains one row per user with pre-aggregated or derived behavioral attributes. | `user_id` | `signup_date` | `first_active_date` | `first_page_view_date` | `first_purchase_date` | `last_active_date` | `lifetime_revenue` | `is_power_user` | | --------- | ------------- | ------------------- | ---------------------- | --------------------- | ------------------ | ------------------ | --------------- | | `u_123` | 2023-11-12 | 2023-11-12 | 2023-11-12 | 2023-11-20 | 2024-03-02 | 249.85 | `true` | | `u_456` | 2024-01-05 | 2024-01-06 | 2024-01-05 | 2024-02-10 | 2024-02-18 | 19.99 | `false` | | `u_789` | 2024-02-10 | 2024-02-10 | 2024-02-10 | `null` | 2024-03-01 | 89.97 | `false` | This type of table is not compatible with Statsig because it does not have a single timestamp column that can be consistently configured across all events (for example, `signup`, `first_active`, and `last_active`). Statsig’s stats engine relies on a timestamp column to join metric data with exposure timestamps, ensuring that only metric events that occur *after* a user is exposed to an experiment are included in analysis. For example, suppose you configure `signup_date` as the timestamp column for this metric source. If you then attempt to build a funnel metric across multiple events (for example, `signup` → `first_active` → `first_purchase`), Statsig will treat all funnel steps as occurring at the signup date. This happens because the metric source can only use a single configured timestamp column. As a result, the event-specific timestamps stored in other columns (such as `first_active_date` or `first_purchase_date`) are ignored, and the funnel no longer reflects the true timing of each event. To address this, there are two recommended options: * **Use upstream tables as the metric source**: Define metrics directly from event logs or fact tables where each row represents a single event and includes a clear timestamp. * **Collapse the table into a long format**: Reshape the wide user table into a long table (for example, one row per user per event or per day) with a unified timestamp column that can be configured in Statsig. | `user_id` | `event_type` | `event_timestamp` | | --------- | -------------- | ------------------- | | `u_123` | `signup` | 2023-11-12 09:15:00 | | `u_123` | `first_active` | 2023-11-13 10:02:41 | | `u_123` | `last_active` | 2024-03-02 18:45:10 | | `u_456` | `signup` | 2024-01-05 14:22:09 | | `u_456` | `first_active` | 2024-01-06 08:11:54 | ## Managing Metric Sources In the metric source tab, you can see your metric sources and the metrics/experiments they're being used in. This varies; in some cases, it can make sense to have a broad metric source that's reused with many metrics using different filters and aggregations. In others, a metric source might exist for one metric (such as a set of specific events for a funnel). Metric sources list showing usage across metrics ### Programmatic Updates You can create and modify metric sources via API and as part of your release flow for data systems. This is full-service and allows for the creation of read-only artifacts. Refer to the [console API](/statsig-warehouse-native/configuration/console-api) and [Semantic Layer Sync](/statsig-warehouse-native/configuration/semantic-layer-sync) sections. ### Note - Governance If you are concerned about granting Statsig broad warehouse access, our recommended solution is to only give Statsig access to its own staging schema/dataset, and create views or materialize staging tables in that location for the data you want Statsig to see. ## Daily Vs. Realtime Sources When specifying a timestamp, you can also specify if the metric source contains data at a daily or timestamp granularity by toggling the "Treat Timestamp as Date" setting. Timestamp granularity configuration interface When this setting is **not** enabled, the system performs a timestamp-based join. This means that events are attributed to the experiment results based on the exact time they occur in relation to the exposure time. For example, if a user is exposed to an experiment at `2024-01-01T11:00:00` and an event occurs at `2024-01-01T11:01:00` on the same day, the event will be attributed to the experiment results because it happened after the exposure. Conversely, if the event occurs at `2024-01-01T10:59:00`, just before the exposure, it will not be attributed to the experiment results since it happened prior to the exposure. On the other hand, if the "Treat Timestamp as Date" setting is enabled, the system performs a date-based join. In this case, all events occurring on the same calendar day as the exposure, regardless of the time, will be included in the experiment results. This includes data from the first day of exposures, ensuring that day-1 metrics are not omitted from the analysis. All Statsig needs to create metrics is a timestamp or date, and a unit (or user) identifier. Context fields let you pull multiple metrics from the same base query, and select values to sum, mean, or group by. | Column Type | Description | Format/Rules | | ---------------------- | ------------------------------------------------------------------- | ------------------------------ | | timestamp | **Required** an identifier of when the metric data occurred | Castable to Timestamp/Date | | unit identifier | **Required** At least one entity to which this metric belongs | Generally a user ID or similar | | additional identifiers | *Optional* Entity identifiers for reuse across identifier types | | | context columns | *Optional* Fields which will be aggregated, filtered, or grouped on | | For example, you could pull from event logging and aggregate the event-level data to create metrics: | timestamp | user\_id | company\_id | event | time\_to\_load | page\_route | | ------------------- | --------------- | ----------- | ----------- | -------------- | ----------- | | 2023-10-10 00:01:01 | my\_user\_17503 | c\_22235455 | page\_load | 207.22 | / | | 2023-10-10 00:02:15 | my\_user\_18821 | c\_22235455 | page\_load | 522.38 | /search | | 2023-10-10 00:02:22 | my\_user\_18821 | c\_22235455 | serp\_click | null | /search | You could create an average TTL metric by averaging time\_to\_load, and group it by page route or filter to specific routes when creating your metric. As another example, you might pre-calculate some metrics yourself at a user-day grain - either to match your source-of-truth exactly or to add more complex logical fields: | timestamp | user\_id | company\_id | country | page\_loads | satisfaction\_score | revenue\_usd | net\_revenue\_usd | | ---------- | --------------- | ----------- | ------- | ----------- | ------------------- | ------------ | ----------------- | | 2023-10-10 | my\_user\_17503 | c\_22235455 | US | 13 | 9 | 130.21 | 112.33 | | 2023-10-10 | my\_user\_18821 | c\_22235455 | CA | 1 | 2 | 0 | 0 | | 2023-10-10 | my\_user\_18828 | c\_190887 | DE | 0 | null | 22.1 | 0 | You can create different metrics by summing and filtering on those daily fields. ## (Very) Slow Metric Sources Statsig uses techniques like Statsig macros, push-down-filters (predicate filters) and using partition keys to make queries in your warehouse efficient. While Metric Sources can include joins or complex queries, they should be performant. If they are not - using any metrics based off this metric source will become expensive (or cause timeouts and failures). The same is true for assignment sources. Statsig will flag a metric source as slow if it takes more than 30 seconds to retrieve a sample of up to 100 records from the table. If the query is expensive, we recommend considering the following steps in sequence to optimize for your metric source: * Include filters based off partition column * Use [Statsig macros](/statsig-warehouse-native/guides/best-practices#use-statsigs-macros) in SQL * Pre-calculate some of the metrics to avoid joins or complex queries * (Do this cautiously) Upgrade your computing resources if you are on a very small cluster. (Note: if you were flagged for a slow Assignment Source, the same guidance here applies to that too!) # Metrics Overview Source: https://docs.statsig.com/statsig-warehouse-native/configuration/metrics Configure metrics in Statsig Warehouse Native using metric sources, including aggregation types, rollups, filters, and metric directionality. Metrics are measures of user or system behavior that are used as evaluation criteria for experiments and for performing analysis. Metrics are organized within your Metrics Catalog. Metrics catalog dashboard ## Creating Metrics Metrics are a combination a metric source, an aggregation, and optional filters and advanced settings. The metric source provides the raw data, and the aggregation defines how Statsig aggregates data across different granularities like user-level, group-level, or for daily timeseries. Metrics can support multiple units of analysis - for example a revenue metric can be used for a "User Level" and a "Store Level" experiment, as long as the metric source has a mapping for both ID types. Filters are also a core component of metrics. Statsig offers a rich set of filtering options, including SQL-based filters, so you can reuse the same metric source for many use cases. Metric creation flow with aggregation and filters ## Using Metrics Metrics can be used for standalone analysis, as part of your experiment scorecard, or as guardrails for feature releases. They can also be put into collections based on Tags for easy addition on these various surfaces. Metrics can also be configured to fire alerts globally if any experiment or gate causes a regression. Statsig's recommendation is to use tags heavily - a combination of team-level and surface or product-level tags ensures easy discovery of metrics To view details about a metric, you can navigate to the Metrics page where you can the definition, related experiments, and a timeseries of the metric value. Metric detail page showing definition and timeseries In the [insights tab](/aggregated-impact/) and in [meta-analysis](/statsig-warehouse-native/features/meta-analysis), you can perform more detailed analysis of how experiments have impacted a metric, and how the metric relates to other metrics in your catalog. Insights view summarizing aggregated impact ## Loading Metrics If you click the Reload Metric Data button, Statsig will automatically sync your latest data and show on this page. Note that the default loading window is 90 days but you can configure it to longer period of time if needed. Reload metric data button and schedule settings To use metrics to measure topline impact, or for ongoing tracking in the metrics page, you'll want to schedule loads of the metric values. This can be done ad-hoc by clicking load on the metric page, or scheduled from there (3 dot menu -> edit scheduled reload). You can also set this as a project level setting in your settings under Data Connection. ## Metric Management Without a well-managed Metric Catalog, it's hard to trust results as end users don't understand if they can trust a metric, or the nuances of how it's defined. Statsig helps you solve this with a variety of tools: * \[Verified Metrics] and programmatic management allow you to vet your core metrics and make it clear which metrics set the gold standard * \[[RBAC](/statsig-warehouse-native/features/roles-and-access) and Team Ownership] can be enabled to help limit the number of potential editors for Metrics, keeping control of core definitions in the experiment team's hands * \[Local Metrics] in Pulse or Explore queries allow you to make experiment-scoped and clearly-labeled changes to metric definitions without adding a large amount of single-user metrics to your metrics catalog * Scorecard hover-over definitions make it trivial to discover, in-context, what settings were applied to a metric and how the calculation was performed. ## Metric Types Statsig offers the largest coverage of metric types of any enterprise platform. These can be roughly divided into 4 categories: ### Aggregations Aggregations are basic unit-level counts or sums measuring user behavior. These are often used as inputs to ratios, especially for [cluster-based experiments](/metrics/different-id) where you want to normalize measures. Aggregations are aggregated at the unit-level, and are then averaged across all units in the experiment during Pulse analysis. Supported aggregations are: * [Count](/statsig-warehouse-native/metrics/count) * [Sum](/statsig-warehouse-native/metrics/sum) * [Logged Sums and Counts](/statsig-warehouse-native/metrics/log) * [Count Distinct](/statsig-warehouse-native/metrics/count-distinct) * [First Value](/statsig-warehouse-native/metrics/latest-value#first-value) * [Latest Value](/statsig-warehouse-native/metrics/latest-value) * Thresholds * A special cases of sum/count which measure a 1/0 flag for if a user passed a threshold value during the experiment - e.g., "the number of users who spent more than \$100" ### Unit Counts, Retention, and Conversion Statsig offers a large number of ways to "count" units in an experiment. This allows you to measure questions like: * did users sign up more often * are more users are currently a subscriber at the end of my experiment * how many users refunded in the first week of the experiment These all fall under the "Unit Count" type, with rollups specifying details of the calculation. In general, unit counts turn into a 1/0 flag, or sum of user-days at the user-level, and are then averaged across all units in the experiment during pulse analysis. Supported unit count types are: * [One-Time Event](/statsig-warehouse-native/metrics/unit-count-once), measuring if a user performed an action at all during the experiment * [Windowed](/statsig-warehouse-native/metrics/unit-count-window), measuring if a user performed an action within some time window after exposure * [Latest Participation](/statsig-warehouse-native/metrics/unit-count-latest), measuring if a user fulfilled some criteria on their latest recorded record (e.g. is this user currently a subscriber) * [Daily Participation](/statsig-warehouse-native/metrics/unit-count-rate), the rate at which units were daily active users during the experiment * [Retention](/statsig-warehouse-native/metrics/retention), the rolling rate of retention on an action or set of actions with a configurable time window ### Ratios and Funnels Ratios and funnel measures rates or conversion. These are aggregated differently, and measure the total of a numerator metric divided by the total of a denominator metric across each experiment group. These also apply the delta method to correct for covariance between the component metrics. [Ratio Metrics](/statsig-warehouse-native/metrics/ratio) allow you to measure ratios of two different metrics, giving nuance to results and helping you to normalize results by another measure - for example: * B2C: Average purchase revenue (`SUM(Revenue) / COUNT(orders)`) * B2B: Revenue per User (`SUM(Revenue) / COUNT_DISTINCT(user_id)`) [Funnel Metrics](/statsig-warehouse-native/metrics/funnel) allow you to analyze user-conversion through multi-step flows at an overall and stepwise level. This helps you to identify dropoff points in your user journeys, and are a critical measurement tool for increasing user conversion. Statsig offers session-level funnels, going beyond user conversion to conversion within distinct checkout flows, support conversations, or viewing sessions. ### Performance Metrics Teams working on performance problems use Statsig to analyze the impact of their changes at a system level. These metric types can also be useful for user behavior. The types most commonly used here are: * [Percentile Metrics](/statsig-warehouse-native/metrics/percentile) allow you to measure changes in values like the P99.9 - useful for measuring improvements or regressions in latency, TTL, or measuring median change when the mean is skewed * [Mean Metrics](/statsig-warehouse-native/metrics/mean) are an easy shorthand for ratio metrics summing an event-level value and dividing by the total records ## Filters Statsig offers a large variety of ways to filter your data. You can, of course, write filters in SQL, but leaving metric sources broad and allowing users to construct filters in the UI or through semantic layer syncs gives a high degree of 'no-code' flexibility to metric creation and follow-up analysis, especially using [local metrics](/metrics/local-metrics). ### Any of / None of These are the equivalent to a SQL `IN` or `NOT IN` statement, and can also be used for equality/inequality. You can supply 1 to N values to the filter that will be included or excluded from your result set. ### Inequalities You can compare numerical values using `=`, `>`, `<`, `>=`, or `<=`. Equality can also be used to compare non-numerical values, which will be evaluated as a string-casted comparison. ### Null Checks Statsig supports checks for `Is Null` or `Non Null`, which can be useful for identifying if a flag was logged or for filtering out partial data. ### Contains/Does Not Contain/Starts With/Ends With These operators are similar to a SQL `LIKE` operator: * Contains checks for if a substring is in a string, e.g. \`field LIKE '%search\_string%' * Does Not Contain checks for if a substring is in a string, e.g. \`field not LIKE '%search\_string%' * Starts With checks for if a substring starts the field, e.g. \`field LIKE 'search\_string%' * Ends With checks for if a substring ends the field, e.g. \`field LIKE '%search\_string' ### Is After Exposure This filter allows you to specify a **secondary** date/timestamp field that has to come after the user's enrollment to the experiment. By default, Statsig only considers metric data where the primary metric timestamp is after the user first saw the experimental intervention. However, you might have another field like "first\_saw\_content\_at". You can use `Is After Exposure` to enforce that this secondary timestamp also takes place after the user's exposure. ### SQL Filters SQL filters allow you to inject any SQL filter string into your metric definition, which will be validated before being saved. This is flexible and lets you interact with complex objects or do complex logic operations as needed to define your metrics. For example, the sql filter `(weight_lb)/pow(height_inches, 2) > 25` would be added to your metric source query for this metric as: ```sql theme={null} SELECT FROM WHERE AND ((weight_lb)/pow(height_inches, 2) > 25) ``` ## Settings Each metric type's page has specific information on settings relevant to that metric. This section is a brief introduction to common settings on metrics. ### Breakdowns Most aggregation-type metrics (sum, count, count distinct, unit count, means, ratios, percentiles, first/latest, min, max) allow you to generate a breakdown view of any column automatically. You can specify this column in the metric set up, and you will be able to click into the experiment result to see the breakdown. ### Cohorts Cohort settings allow you to specify a window for data collection after a unit's exposure. For example, a 4-6 day cohort window would only count actions from days 4, 5, and 6 after a unit was exposed to an experiment. Please refer to the full documentation on cohorts [here](/statsig-warehouse-native/features/cohort-metrics). ### Baking Many metric types support baking. Statsig will wait to calculate baked metrics, and use "old" data for baked metrics. This is appropriate for cases like credit card chargebacks, where you may adjust your payments dataset to account for chargebacks in a "net revenue" metric. See additional information in the [cohort documentation](/statsig-warehouse-native/features/cohort-metrics). Statsig will: * Not calculate baked metrics until the bake period has elapsed since the user's enrollment * On a given load, Statsig will pull historical data that "just baked" as of that day * Calculate will only calculate results for users whose bake window has elapsed to avoid diluting metrics This way incomplete data is not shown in pulse (either incomplete in the warehouse, or incomplete because of the late-landing data) ### Thresholding Thresholding functionally converts a sum, count, count-distinct, max/min, or first/latest metric into a 1/0 unit metric by converting the user's total value to a specified threshold. For example, you might want to measure the count of users who spent at least \$100 in their first week on the platform. You'd specify this as a 0-6 day cohort metric, summing revenue, with a threshold of \$100. ### CUPED You can enable/disable CUPED per metric, and also specify the lookback window for which users' pre-exposure data will be pulled and used as an input to the CUPED regression. ### Winsorization See [winsorization](/stats-engine/methodologies/winsorization). Winsorization is applicable to aggregate metrics and allows you to specify an upper/lower, percentile-based threshold which data above/below will be clamped to. This reduces the incidence of inflated variance or skewed means from outlier data or buggy logging. ### Capping Capping is an alternate outlier-control method that allows specifying a unit-day cap (specifiable per unit type) that values will be clamped to. For example, you might clamp "total purchase count" to 100 to avoid resellers skewing your metrics on an auction platform, if 100 is a reasonable upper bound for typical users. # Qualifying Events Source: https://docs.statsig.com/statsig-warehouse-native/configuration/qualifying-events Use qualifying events in Statsig Warehouse Native to filter experiment exposures and metric calculations to specific user actions or eligibility windows. ## Using Qualifying Events Qualifying events are used to simulate exposures for power analysis, and to filter exposures to users who triggered a specific event. Setting up a Qualifying Event is identical to setting up an Assignment Source, except they do not require experimental information. Please note that the qualifying event must occur after the exposure as indicated by their respective timestamps. Context columns can be used to filter the qualifying event for power analysis - for example you might have a Qualifying Event for page load, and filter to different page identifiers for power analyses of experiments on different surfaces. ## Filter by Qualifying Events Statsig has best-in-class controls around using qualifying events to filter exposures in an experiment in cases of over-exposure. qualifying event filter In the image above: * We're using forwarded sdk events from Statsig to filter exposures. This could also be a single-event source or any data source * We've chosen to replace the actual exposure timestamp with the first observed qualifying event - this will make our metric timelines and cohort timelines more accurate * We only count qualifying events from within an hour of an exposure event - this helps to avoid issues with late-landing attribution * We've added a filter to only include events from the `www.statsig.com` domain Some customers prefer this flexibility, and many just set up qualifying events as individual events to plug into their analyses. You can often identify over-exposed experiments by observing uniformly low participation rate on all metrics. ## Power Analysis Qualifying events can be used in power analysis as a way to estimate your experimentation population by how many people historically hit your proposed trigger event. power analysis with qualifying event These can be filtered by event attributes, or by entity properties (e.g. users who triggered this event and are also in a segment or country). ## Example Data | Column Type | Description | Format/Rules | | ---------------------- | ----------------------------------------------------------------------------------------- | ------------------------------ | | timestamp | **Required** an identifier of when the qualifying event occurred | Castable to Timestamp/Date | | unit identifier | **Required** At least one entity to which this metric belongs | Generally a user ID or similar | | additional identifiers | *Optional* Entity identifiers for reuse across identifier types | | | context columns | *Optional* Fields which can be used to group by and filter results in exploratory queries | | For example, you could pull from page load event logging directly and save it as a qualifying event called `Page Load`: | timestamp | user\_id | company\_id | page\_route | | ------------------- | --------------- | ----------- | ----------- | | 2023-10-10 00:01:01 | my\_user\_17503 | c\_22235455 | / | | 2023-10-10 00:02:15 | my\_user\_18821 | c\_22235455 | /search | | 2023-10-10 00:03:12 | my\_user\_22251 | c\_9928 | /profile | # Macros Source: https://docs.statsig.com/statsig-warehouse-native/configuration/query-tools Reference for Statsig Warehouse Native query tools that help you inspect generated SQL, debug data sources, and validate metric and experiment configuration. In **Metric** and **Assignment sources**, you can use Statsig Macros to directly inject a DATE() type which will be relative to the experiment period being loaded. * `{statsig_start_date}` * `{statsig_end_date}` For example, in an incremental reload from `2023-09-01` to `2023-09-03`, this query: ```sql theme={null} SELECT user_id, event, ts, dt FROM log_table WHERE dt BETWEEN `{statsig_start_date}` AND `{statsig_end_date}` ``` resolves to ```sql theme={null} SELECT user_id, event, ts, dt FROM log_table WHERE dt BETWEEN DATE('2023-09-01') AND DATE('2023-09-03') ``` This is a powerful tool since you can inject filters into queries with joins or CTEs and be confident that the initial scan will be pruned. We will adjust the range as necessary in some cases. For example, in entity properties we will add some pre-experiment buffer to allow for late-landing property data. We then choose the most recent value as of each unit's exposure. For CUPED, we will adjust the range to include the pre-experiment window for CUPED calculations. ## Timezone Note All timestamps in Warehouse Native from statsig are in UTC, and it is assumed that timestamps in the warehouse will be timezone-less/UTC as well. ## Advanced Macros * `{statsig_start_date_int}` * `{statsig_end_date_int}` Int versions of the above for use with number based partitioning - e.g. '2025-03-01' => 20250301 * `{statsig_experiment_start_timestamp}` Only available for metric sources and entity property sources; this will resolve to the start timestamp of the experiment. In non-experiment contexts, it resolves to `TIMESTAMP('1970-01-01')`. This is useful for generating entity properties such as "30d revenue before the experiment started". # Semantic Layer Sync Source: https://docs.statsig.com/statsig-warehouse-native/configuration/semantic-layer-sync Sync metric definitions from your semantic layer or dbt project into Statsig Warehouse Native to reuse existing business logic in experiments. If you have centrally defined metrics, Statsig offers the ability to sync its data sources and metrics as part of your data version management workflow. Using Statsig's [Console API](/console-api/metrics) you can automatically sync changes you make to the matching definitions on Statsig, and you can optionally make the metrics read-only in the Statsig console. We have a demonstration [GitHub repository](https://github.com/statsig-io/semantic_layer) that utilizes [a script](https://github.com/statsig-io/semantic_layer/blob/main/.github/scripts/statsig_sync.py) executed by [a GitHub Action](https://github.com/statsig-io/semantic_layer/blob/main/.github/workflows/statsig_sync.yml). This setup automatically synchronizes changes to .yml files located in the /metrics or /metric\_sources directories in the repo. This means that whenever you create or update these files, the script either updates existing metrics or metric sources in Statsig or creates new ones accordingly. To use this example template, follow these steps: 1. Fork [this repository](https://github.com/statsig-io/semantic_layer) to get started. 2. In your forked repository, add your Statsig Console API Key to GitHub Secrets. 3. Tailor the metric definitions to align with your data needs. 4. Verify the automation by modifying relevant files and observing the triggered GitHub Action. ## Detailed Guide ### Forking the Repository 1. **Fork this repository** to create a copy in your GitHub account. Untitled ### Adding the Statsig Console API Key 2. Navigate to `Settings > Secrets and variables > Actions` in your repository settings. Create a new secret named `STATSIG_API_KEY` with your Statsig Console API key as its value. This key facilitates authentication with the Statsig Console API for the synchronization process. Untitled ### Customizing Metric Definitions 3. Metric definitions reside within the `./metrics` directory, and metric source definitions are found in the `./metric_sources/` directory. To customize: * Utilize the Statsig Console API to fetch an existing **metric\_source** or **metric** using GET requests for [metric sources](/console-api/metrics#post-/metrics/metric_source/-name-) and [metrics](/console-api/metrics#get-/metrics/-metric_id-). * Remove the provided example metrics and metric sources, and replace them with your definitions in `./metric_sources/*.yml` and `./metrics/*.yml`. *Note:* For enhanced readability, we modified `metric.warehouseNative[]` to `metric.metricDefinition[]` in our examples. You can see this change [here](https://github.com/statsig-io/semantic_layer/blob/1611a68703caf18d7fa32088ff06d568d8b3b03a/.github/scripts/statsig_sync.py#L38). Feel free to adjust the translations or revert to using `metric.warehouseNative[]` in your definitions. ### Verifying Automation 4. To test, edit a metric or metric source description in your repository. This action should trigger the GitHub Action, visible under the `Actions` tab. The process will then either create or update your metrics and metric sources in Statsig based on the repository's semantic definitions. Untitled Untitled Untitled
    This example serves as a basic template. We encourage testing and further development to meet production standards. Please share any feedback or improvements you've made to this workflow with our support team, your sales contact, or in our [Slack community](https://statsig.com/slack). Thank you for any contributions! # Tags & Teams Source: https://docs.statsig.com/statsig-warehouse-native/configuration/tags-and-teams Organize Statsig Warehouse Native metrics, experiments, and other resources by tags and team ownership for easier discovery and access control. Statsig offers tools to enrich data sources, metrics, and entities with team ownership as well as arbitrary tags. These provide a powerful way to organize your data configuration, and also feed into advanced analytics tools like [meta-analysis](/statsig-warehouse-native/features/meta-analysis). ## Tags Experiments, Metrics, and Sources can all be tagged for easy discovery, search, and context. Metrics which share a tag can be bulk-added to scorecard or guardrail metric sets. Statsig also provides a Core tag - this is meant to identify company key metrics that should be used as guardrail metrics across experiments. See more at the [tags page](/metrics/create-metric-tags). ## Teams For larger organizations, the Teams feature enables an organizational and settings/ permissions layer on top of a Project. Teams are configured at the Project (not Organization) level, and are default-editable by all Project Admins. Once teams are configured and a user is assigned to a team, any config (gates/ experiments/ metrics, etc.) they create will be associated with the team they belong to, and will inherit the settings of that team. Users who are members of multiple teams will have the choice of which team to associate their config with at creation time. This is extremely useful for managing ownership and review practices. Learn more [here](/access-management/teams). # Athena Connection Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/athena Connect Amazon Athena to Statsig Warehouse Native, including IAM roles, S3 staging buckets, workgroups, query result locations, and required permissions. ## Athena Warehouse Native Overview To set up connection with Athena, Statsig needs the following * An S3 Bucket * A Glue Database for staging * An S3 Query Result Location * Athena Access Permissions (can be via an AWS Role or an AWS User) ## Setup Statsig Staging Structure 1. Create or choose an S3 Bucket. Statsig will use a subfolder inside this S3 Bucket to store all staging data. Statsig will have write-access to ONLY this scoped subfolder of this S3 Bucket (specifically labeled `Statsig S3 Folder` in your Data Connection settings in the Statsig Console). 2. Create or choose a Glue Database (can be `default`). Statsig will use this as a staging Database to create and manage tables. Statsig will be able to drop/create tables within ONLY this staging Database. 3. Choose an S3 Query Result Location folder within the S3 Bucket. This S3 location will act as the Output Location for `SELECT` queries run in your Athena Warehouse. This Location can be given either: * Explicitly as an S3 location (ex: `s3://my_bucket/my_query_results_folder/`) * OR as part of a setting within an Athena Workgroup * NOTE that your workgroup must have the 'Query result location' field populated accordingly 4. Add this information, along with your AWS Region, to your Data Connection settings in the Statsig Console. ## Grant Permissions to Statsig You need to grant some permissions for Statsig from your AWS console in order for us to access your Athena data. Statsig requires * READ on any tables and data you are using for experimentation * USAGE/WRITE on a Statsig-specific schema we'll use to materialize temp tables and results. This enables us to cache data and perform incremental loads. You will specify which Glue Database and S3 Bucket to use, and we will create a Statsig S3 Subfolder to do our staging operations 1. Create an AWS IAM Policy to house the required access permissions This policy will house the permissions required for Statsig to access your warehouse. You will be able to return to this policy later and edit it as needed. * In your AWS IAM Dashboard, select the Policies page under the Access Management tab * Click 'Create policy' * Switch the Policy Editor type from 'Visual' to 'JSON' * Copy and paste the below JSON template block * Replace the placeholders with your setup information and the Statsig S3 Folder (specified in your Statsig project's Data Connection settings) * Specify all S3 locations and Glue Databases of any read-only assignment/metric data * Remove the descriptor comments ```json expandable theme={null} { "Version": "2012-10-17", "Statement": [ // Allow Statsig to recognize your staging S3 Bucket { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation", "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::__S3_BUCKET__" }, // Allow Statsig to read events/exposures data from your S3 Buckets { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::__PATH_TO_YOUR_READONLY_DATA__/*" }, // Allow Statsig to read events/exposures tables from your Glue Databases { "Effect": "Allow", "Action": [ "glue:GetTable", "glue:GetTables", "glue:GetDatabase", "glue:GetDatabases", "glue:GetPartition", "glue:GetPartitions" ], "Resource": [ "arn:aws:glue:__REGION__:__YOUR_AWS_ACCOUNT_ID__:database/__YOUR_READONLY_DATABASE__", "arn:aws:glue:__REGION__:__YOUR_AWS_ACCOUNT_ID__:table/__YOUR_READONLY_DATABASE__/*" ] }, // Allow Statsig to read/write/use data and tables in an isolated staging S3 subfolder { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "athena:StartQueryExecution", "athena:GetQueryResults", "athena:GetQueryExecution", "athena:StopQueryExecution", "glue:GetTable", "glue:GetTables", "glue:CreateTable", "glue:UpdateTable", "glue:DeleteTable", "glue:GetPartition", "glue:GetPartitions", "glue:CreatePartition", "glue:UpdatePartition", "glue:DeletePartition", "glue:BatchCreatePartition", "glue:BatchDeletePartition", "glue:GetDatabase", "glue:GetDatabases" ], "Resource": [ "arn:aws:s3:::__S3_BUCKET__/__PATH_TO_S3_QUERY_RESULTS_FOLDER__/*", "arn:aws:s3:::__S3_BUCKET__/__STATSIG_S3_FOLDER__/*", "arn:aws:athena:__REGION__:__YOUR_AWS_ACCOUNT_ID__:workgroup/__WORKGROUP_NAME__", "arn:aws:glue:__REGION__:__YOUR_AWS_ACCOUNT_ID__:catalog", "arn:aws:glue:__REGION__:__YOUR_AWS_ACCOUNT_ID__:database/__GLUE_STAGING_DATABASE__", "arn:aws:glue:__REGION__:__YOUR_AWS_ACCOUNT_ID__:table/__GLUE_STAGING_DATABASE__/*" ] } ] } ``` 2. Create an IAM Role or IAM User: With an IAM Role, Statsig will assume your created IAM Role via a Statsig Service Account. The Statsig Account ID for this service account is provided in your Data Connection settings in the Statsig Console. Statsig will run queries directly on behalf of this IAM Role. Optionally, you can add an External ID condition for added security ([AWS External ID Docs](https://aws.amazon.com/blogs/security/how-to-use-external-id-when-granting-access-to-your-aws-resources/)). This External ID will be generated by Statsig, and viewable in your Data Connection settings in the Statsig Console. * In your AWS IAM Dashboard, select the Roles page under the Access Management tab * Click 'Create role' * Choose 'AWS account' as the Trusted entity type * Choose 'Another AWS account' from the options, and copy the Statsig Account ID from your Statsig console * Optionally, require use of an External ID for connections to this role (also specified in your Statsig console) * Continue to next step of setup, and select your IAM Permissions Policy from earlier * Name, review, and create; your Trust Policy JSON should follow the format below: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sts:AssumeRole", "Principal": { "AWS": "__STATSIG_ACCOUNT_ID__" }, "Condition": { "StringEquals": { "sts:ExternalId": "__ROLE_EXTERNAL_ID__" } } } ] } ``` * Add the ARN for this IAM Role to your Data Connection settings in the Statsig Console With an IAM User, Statsig will use AWS Access Keys to gain access to this IAM User. Statsig will run queries directly on behalf of this IAM User. * In your AWS IAM Dashboard, select the Users page under the Access Management tab * Click 'Create user' * Name your user * On the next step of setup, choose 'Attach policies directly' and select your newly created Permissions Policy * Under the Security Credentials tab of this newly created User, find the Access Keys block * Click 'Create access key', and choose 'Third-party service' from the Use Case options * Add the Access Key and Secret Access Key to your Data Connection settings in the Statsig Console ## Setup Reading Data from your Events/Exposures Tables 1. Give Statsig read-access to your Glue Database containing any tables you need to Statsig to read from. Do this by adding the following to your AWS IAM Permissions Policy: ``` { "Effect": "Allow", "Action": [ "glue:GetTable", "glue:GetTables", "glue:GetDatabase", "glue:GetDatabases", "glue:GetPartition", "glue:GetPartitions" ], "Resource": [ "arn:aws:glue:__REGION__:__YOUR_AWS_ACCOUNT_ID__:database/__YOUR_READONLY_DATABASE__", "arn:aws:glue:__REGION__:__YOUR_AWS_ACCOUNT_ID__:table/__YOUR_READONLY_DATABASE__/*" ] } ``` 2. Give Statsig read-access to your S3 Bucket locations of the tables you need Statsig to read from. Add this to your AWS IAM Permissions Policy: ``` { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::__PATH_TO_YOUR_READONLY_DATA__/*" } ``` 3. Read data in Statsig when setting up Metric/Assignment Sources by selecting from these tables using `"database"."table"` format. 4. Repeat for any additional tables, or whenever you need to read a new table from Statsig. If your data warehouse is IP protected, you must include allowlisting of Statsig IP ranges in your setup steps. ## Additional Athena Resources ### S3 Bucket Encryption Guide Statsig supports all accessed S3 Buckets being encrypted. Steps to allow Statsig encrypting S3 Buckets while giving Statsig access are as follows: 1. From the AWS Key Management Service console, create a new KMS Key using the below cryptographic configuration settings: ([AWS SSE KMS Docs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html?icmpid=docs_s3_hp_batch_ops_create_job_encryption_key_type)) * Key Type: Symmetric * Key Usage: Encrypt and Decrypt * Advanced Options * Key Material Origin = KMS * Regionality = Single-Region Key 2. From the Key Policy tab of your newly created KMS Key, find the Key Administrators box. Click Add, and select the AWS IAM Role/User provided to Statsig as an administrator. 3. Navigate to your S3 Bucket. From your S3 Bucket Properties tab, find the Default Encryption box. Click Edit, and select the below default encryption settings: * Encryption Type: SSE-KMS * AWS KMS Key: Enter AWS KMS Key ARN * (enter your newly created KMS Key ARN in the box) * Bucket Key: Enable ### Statsig Staging Architecture Details Statsig will create all of its staging tables within the Glue Staging Database that you create and provide. All of the tables created from Statsig will be of table-type `ICEBERG` (with the exception of the forwarded exposures/events tables, which will be regular `EXTERNAL` Athena tables). Statsig stores all staging data within the `Statsig S3 Folder` of your S3 Bucket. Statsig will handle the creation of the structure within this subfolder. The folder structure is as follows: * S3 Bucket (you create and provide Statsig the name) * Statsig S3 Folder (Statsig creates and names, name provided in the Data Connection settings in your Statsig Console) * Experiment Folder(s) (Statsig will create a subfolder for each unique experiment you run. These will be named `experiment-`) * Staging Tables Folders (Statsig will put data for each created staging table in its own subfolder. These will be named intuitively after the data they contain) * `metadata/` * Iceberg Table Metadata * `data/` * Table Data Files (stored as Parquet) * `statsig_forwarded_exposures/` * Forwarded Exposures Table Subfolder (named in the Data Connection settings in your Statsig Console) * Dates in `YYYY-MM-DD` format (This table is partitioned by date, and there will be a subfolder for each date that has exposure data sent through Statsig) * Forwarded Exposure Data * `statsig_forwarded_events/` * Forwarded Events Table Subfolder (named in the Data Connection settings in your Statsig Console) * Dates in `YYYY-MM-DD` format * Forwarded Event Data ### What IP addresses will Statsig access data warehouses from? [See FAQ](/data-warehouse-ingestion/faq#what-ip-addresses-will-statsig-access-data-warehouses-from) # Bigquery Connection Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/bigquery Connect Google BigQuery to Statsig Warehouse Native, including service account setup, dataset permissions, and required IAM roles. ## Overview To set up a connection with BigQuery, there are two supported methods: * Grant Permissions to a Statsig-owned Service Account * Provide Credentials for Your Own First-Party Service Account In both cases, you'll need: * Your BigQuery Project ID * The dataset Statsig will use to save temporary tables and materialized results Once you’ve chosen your method, start by enabling the BigQuery source in your project settings. ## Grant Permissions to Statsig's Service Account You need to grant some permissions for Statsig from your Google Cloud console in order for us to access your BigQuery data. 1. In your BigQuery's [IAM & Admin settings](https://console.cloud.google.com/iam-admin/), add the Statsig service account you copied in the Statsig Console as a new principal for your project, and give it the following roles: * `BigQuery User` BigQuery IAM permissions configuration
    2. Navigate to your [BigQuery SQL Workspace](https://console.cloud.google.com/bigquery), choose the dataset, click on "+ Sharing" -> "Permissions" -> "Add Principal" to give the same Statsig service account "BigQuery Data Viewer" role on a dataset. Do this for any datasets you want the service user to be able to access. BigQuery dataset permissions interface 3. Following the steps above, give the "BigQuery Data Editor" role on the dataset you want Statsig to use for its staging data. If your data warehouse is IP protected, you must include allowlisting of Statsig IP ranges in your setup steps. Now the service account should have the required permissions to run queries and materialize results. ## Using a First Party Service Account This is not a recommended alternative. Using service accounts is the [preferred default for enhanced security](https://docs.cloud.google.com/iam/docs/migrate-from-service-account-keys). 1. On the BQ service accounts page, click 'Manage Keys' for the service account you want to use BQ 1st party service account 2. Create a new JSON key (which will download a JSON file) BQ 1st party service account BQ 1st party service account 3. In the **Statsig Settings** -> [Data Connection](https://console.statsig.com/data_connection/connection_setup) tab, paste the above key into the **Service Account Private Key** field under the **Advanced** toggle. ## Additional BigQuery Resources ### BigQuery Project ID Find your BigQuery Project ID below 1. Click on your Project Dropdown inside your Cloud Console. Frame 4 2. Copy and paste relevant Project ID from the modal pop-up. Frame 5 If your data warehouse is IP protected, you must include allowlisting of Statsig IP ranges in your setup steps. ### What is Statsig's Customer ID on BigQuery C01d5f80s ### What IP addresses will Statsig access data warehouses from? [See FAQ](/data-warehouse-ingestion/faq#what-ip-addresses-will-statsig-access-data-warehouses-from) ### Additional Setup for Warehouse Explorer Warehouse Explorer makes it easy to find and bring data from any table into Statsig for ad-hoc analysis. Statsig will read from `INFORMATION_SCHEMA.COLUMNS`, which requires explicit permission to view table metadata. You may provide access by repeating steps 1 and 2 from the [Grant Permission](/statsig-warehouse-native/connecting-your-warehouse/bigquery#grant-permissions-to-statsig%E2%80%99s-service-account) section above for the "BigQuery Metadata Viewer" role. # Databricks Connection Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/databricks Connect Databricks to Statsig Warehouse Native, including service principal setup, SQL warehouse selection, and required Unity Catalog permissions. ## Overview To set up a connection with Databricks, you will need the following: * Your Databricks Server Hostname * An HTTP Path to a Cluster or SQL Warehouse * A staging database for writing results and intermediate tables into * An access token with read access on your experiment data and write access to the staging database * Use either Serverless SQL Warehouse or an always-on cluster. Statsig uses interactive queries during setup (and some analysis) which will fail if the cluster takes several minutes to start up. Start by enabling the Databricks source in your project settings. ## Note on Databricks Access For users who use databricks and a dbfs-based deltalake as their primary warehouse, permissions are managed easily for databricks. For customers using databricks as an intermediary to other data sources, you need to make sure that databricks and the Statsig user through databricks has appropriate access to your storage (e.g. S3 for athena tables). Often, permissions are reset at some point which will start to cause errors; to validate this, try creating tables through the Statsig console to verify that the permissions on Statsig's side are sufficient. ## Getting Connection Information 1. Follow the [Databricks documentation](https://docs.databricks.com/integrations/jdbc-odbc-bi.html#get-connection-details-for-a-cluster) to get the hostname and http path of the cluster you'll use to run your experimental analysis. You may want to create a specific cluster for this use case. credentials
    2. Follow [these instructions](https://docs.databricks.com/dev-tools/auth.html#databricks-personal-access-tokens) to get the personal access token that will be used to calculate experiment results on your warehouse. Alternatively, you can follow [these instructions](https://docs.databricks.com/en/administration-guide/users-groups/service-principals.html#manage-personal-access-tokens-for-a-service-principal) to get a personal access token for a service principal. databricks info 3. Create or choose a database to use. For example, you could run this sql in a notebook: ```sql theme={null} staging_database_name = '' spark.sql(f"CREATE DATABASE IF NOT EXISTS {staging_database_name}") ``` If your data warehouse is IP protected, you must include allowlisting of Statsig IP ranges in your setup steps. ### What IP addresses will Statsig access data warehouses from? [See FAQ](/data-warehouse-ingestion/faq#what-ip-addresses-will-statsig-access-data-warehouses-from) ### Additional Setup for Warehouse Explorer Warehouse Explorer makes it easy to find and bring data from any table into Statsig for ad-hoc analysis. When Unity Catalog is enabled, Statsig will read from `.information_schema.columns`. You can provide the catalog name in Data Connection settings, under the Advanced section. Also, you may need to provide read access to the catalog and the `information_schema` schema: ```sql theme={null} GRANT USE CATALOG ON CATALOG TO ; GRANT USE SCHEMA ON SCHEMA .information_schema TO ; ``` `` should be the user associated with the access token you have provided to Statsig above. # Forwarded Data Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/forwarded-data Use forwarded data in Statsig Warehouse Native to send events to Statsig and store them in your warehouse for experiment and analytics use. If you log events or exposures to Statsig's SDK, Statsig will put that data back in your warehouse in near-realtime, on-demand. ## Setting up tables for forwarded data By default, when setting up a data connection to your warehouse, we'll automatically create tables 'exposures' and 'events' where we'll forward SDK data to. If you want to change the name of the table that are used for forwarded data, in the data connection page under the 'advanced' tab and you'll find the option to change the name of said tables. Forwarded data table configuration interface If you've already had data exported and change the table name, future data will be written to the new table. Non-production exposures or log events are not forwarded to external warehouses ## Exposures Logging exposures with Statsig means you'll get real-time diagnostics on the Statsig console, as well as real-time aggregations like exposures by hour. When you run Pulse analysis, raw exposures for the experiment you're loading will be fast-forwarded to catch up with the real-time stream. This means you'll get all of the users in your experiment and see Pulse results as fresh as \~15m (assuming events and metrics come in at the same speed). Under the covers, we perform a just-in-time update of exposures in your warehouse when Pulse is loaded, for the first 1 million exposures that are logged to the experiment. After that, the exposures are batched, deduplicated and written to your warehouse once a day. These fast-forwarded exposures are not deduplicated, and will have some fields (`user_dimensions` in particular) missing. These fields will be provided in the subsequent daily load. Each day, a deduplicated digest will be exported to your warehouse to ensure consistency. This will be deduplicated with the above as part of the standard Pulse Pipeline. ## Note on Duplicates Because Statsig only holds onto exposure data for 30 days, it cannot dedupe beyond that time on its servers. This means that after 30 days, the "deduplicated" data from Statsig will start to re-send exposures from units who were first exposed over 30 days ago, and were re-exposed on that day. These will be correctly deduplicated during analysis in your warehouse, but it means that you should not expect the table to have unique user records, even after filtering out the fast-forwarded exposures (which can also have duplicate records). For gates with 0% or 100% rollout, by default we don't forward exposure to your warehouse. If you need them, please contact our support team, your sales contact, or via our [Slack community](https://statsig.com/slack). ## Events If you're using our SDKs to capture custom events, we'll export these events to your warehouse hourly. You can see Pulse results on metrics derived from those events as fresh as \~1hour. # Other Warehouses Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/other Connect other data warehouses to Statsig Warehouse Native using forwarded data, custom integrations, or supported community connectors. ## Other Warehouses Statsig's architecture is set up to add new warehouses relatively easily. We are selective with which warehouses we choose to support long-term, but generally we will be happy to discuss your needs and consider building a solution that works for you on your cloud provider. In general, Statsig will access your warehouse through API calls and will need corresponding information on the address of your cluster and authentication information; very similar to running queries through datagrip, python, or other services. ## Permissions Statsig will require read permissions on any input tables for your analysis. Statsig also requires full CRUD (create/read/update/delete) permissions on a "sandbox" where staging data and other intermediate data artifacts will be stored as part of experimentation pipelines. ### What IP addresses will Statsig access data warehouses from? If your data warehouse is IP protected, you must include allowlisting of Statsig IP ranges in your setup steps. [See FAQ](/data-warehouse-ingestion/faq#what-ip-addresses-will-statsig-access-data-warehouses-from) # Redshift Connection Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/redshift Connect Amazon Redshift to Statsig Warehouse Native, including IAM roles, network access, user setup, and required schema permissions. ## Overview To set up connection with Redshift, Statsig needs the following information * Cluster Endpoint * A service user Username * A service user Password * A staging schema that Statsig can write results to SHA256 passwords are not currently supported, please utilize MD5 to avoid issues. You can find this information in your aws console within your specific cluster, as shown in the image below. (Open image in new tab for a bigger image). The service user should be able to read necessary experiment data, and be able to write to the Statsig staging schema you specify. Frame 1 When you save the connection, we will run a series of tiny commands to test permissions -- e.g. creating a temp table, running a select/delete statement on that table, and then dropping that table. The provided Service Account will require the following attributes: `enable_case_sensitive_identifier`, `enable_case_sensitive_super_attribute`. If these are not already set, Statsig will set them to TRUE upon setup completion. ## SSH Tunneling For Redshift connections, we also allow users to create an SSH tunnel into their Redshift cluster for a more secure and private access to the database. To enable access, Statsig requires: * SSH Host * SSH Port * SSH User Statsig will use this information to generate an SSH key. Please add this generated key to your `~/.ssh/authorized_keys` file on your SSH proxy machine to enable SSH tunneling. ### What IP addresses will Statsig access data warehouses from? If your data warehouse is IP protected, you must include allowlisting of Statsig IP ranges in your setup steps. [See FAQ](/data-warehouse-ingestion/faq#what-ip-addresses-will-statsig-access-data-warehouses-from) ### Additional Setup for Warehouse Explorer Warehouse Explorer makes it easy to find and bring data from any table into Statsig for ad-hoc analysis. To enable the Warehouse Explorer analytics feature, you may need to provide Statsig with additional permission to query the `pg_table_def` metadata. Only schemas that Statsig has read access to will be included in query results. A superuser or admin can grant access to read additional relevant schemas and table metadata in `pg_table_def` by running ```sql theme={null} GRANT USAGE ON SCHEMA TO ; GRANT SELECT ON ALL TABLES IN SCHEMA TO ; ``` `` should be the service user username that you provided to Statsig above. `` is the name of the schema you'd like to see included in Warehouse Explorer results # Scheduled Reloads Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/scheduled-reloads Configure scheduled reloads in Statsig Warehouse Native to refresh experiment and metric data from your warehouse on a daily or hourly cadence. ## Overview You can control daily reload settings for Metrics and Experiments. While these can be configured on each entity, you can also set defaults that entities can inherit from in your Project Settings. You also have the option of using the Console API to [trigger experiment result loads](/console-api/experiments#post-/experiments/-experiment_id-/load_pulse) (experiment results). This is often used for triggering refreshes when your data pipelines are ready. Project-level scheduled reload settings interface Individual experiment pulse scheduling interface ## Gate/Experiment Pulse Scheduling For feature gates and experiments, individual Pulse scheduling is available separately from the project-level settings. You can schedule daily Pulse metric reloads for individual feature gates and experiments. To access this feature: 1. Navigate to your feature gate/experiment in the Statsig Console 2. Go to the Pulse Results tab 3. Use the scheduling controls to configure daily reloads The scheduling allows you to: * Set a daily reload time in UTC * Choose between Full and Incremental reload types * Save, edit, or cancel scheduled reloads This feature requires gates with partial rollout rules and overrides any project-level settings. # Snowflake Connection Source: https://docs.statsig.com/statsig-warehouse-native/connecting-your-warehouse/snowflake Connect Snowflake to Statsig Warehouse Native, including user and role setup, warehouse selection, and required schema and table permissions. ## Overview To set up Warehouse Native connection with Snowflake, Statsig needs the following * Account Name * Database Name * Schema Name * Service User Name * If authenticating via login credentials: * Service User Password * If authenticating via key-pair authentication: * Private Key * Private Key Passphrase (Optional) The service user needs the following permissions: * READ on any tables you are using for experimentation * USAGE/WRITE on a Statsig-specific Schema we'll use to materialize temp tables and results If your data warehouse is IP protected, you must include allowlisting of Statsig IP ranges in your setup steps. ### Account Name For the Account Name field, please enter in the format of `..`. So, your information may look something like this: `xy12345.us-central1.gcp` To get this information navigate to bottom left in your Snowflake console, as shown in the picture below and copy the link URL: Frame 6 The copied URL will look something like this: `https://xy12345.us-central1.gcp.snowflakecomputing.com` You can extract information from here to get the required fields for Account Name, which for this example would be `xy12345.us-central1.gcp`. ### Using \-\ for Account Name For the Account Name field, you can also enter your Snowflake [account identifier](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html#format-1-preferred-account-name-in-your-organization), which typically takes the form `-`. To find the `` in the Snowflake console, click on your account profile (usually at the bottom left) to view account details as shown below. Snowflake account profile interface ### Database and Schema Name Provide the Schema and corresponding Database where Statsig will be able to materialize results Frame 7 ### Key-Pair Authentication To set up key-pair authentication, first follow the [snowflake documentation](https://docs.snowflake.com/en/user-guide/key-pair-auth) to generate the private and public keys, and then set the public key on the service user. The private key can then be provided here Key-pair authentication configuration interface ### Boilerplate Setup SQL To create a Statsig user with the sufficient privileges, as well as a Statsig staging Schema, run the following code in your Snowflake worksheet that has sysadmin and securityadmin roles. Replace `` and `` with your value, which you will copy over into our console. Depending on the scope of your data, you may want to adjust the size of the warehouse created. ```sql expandable theme={null} BEGIN; -- set up variable values to be used in statements later -- make sure to configure user_name and user_password with your own values SET user_name = ''; -- REPLACE WITH YOUR OWN VALUE SET user_password = ''; -- REPLACE WITH YOUR OWN VALUE SET role_name = 'STATSIG_ROLE'; -- change role to sysadmin for warehouse / database steps USE ROLE sysadmin; -- create a warehouse, database, schema and tables for Statsig CREATE OR REPLACE WAREHOUSE STATSIG WITH warehouse_size='XLARGE'; -- adjust based on your data size CREATE DATABASE IF NOT EXISTS STATSIG_STAGING; CREATE SCHEMA IF NOT EXISTS STATSIG_STAGING.STATSIG_TABLES; -- change current role to securityadmin to create role and user for Statsig's access USE ROLE securityadmin; -- create role for Statsig CREATE ROLE IF NOT EXISTS identifier($role_name); GRANT ROLE identifier($role_name) TO ROLE SYSADMIN; -- create a user for Statsig CREATE USER IF NOT EXISTS identifier($user_name) password = $user_password default_role = $role_name default_namespace = STATSIG_STAGING.STATSIG_TABLES default_warehouse = STATSIG; GRANT ROLE identifier($role_name) TO USER identifier($user_name); -- grant Statsig role access to create warehouse and schema GRANT USAGE ON WAREHOUSE STATSIG TO ROLE identifier($role_name); GRANT USAGE ON SCHEMA STATSIG_STAGING.STATSIG_TABLES TO ROLE identifier($role_name); GRANT CREATE SCHEMA, MONITOR, USAGE ON DATABASE STATSIG_STAGING TO ROLE identifier($role_name); -- ONLY GIVE THIS LEVEL OF ACCESS in the staging schema. GRANT CREATE TABLE, CREATE FUNCTION ON SCHEMA STATSIG_STAGING.STATSIG_TABLES TO ROLE identifier($role_name); GRANT SELECT, UPDATE, INSERT, DELETE ON ALL TABLES IN SCHEMA STATSIG_STAGING.STATSIG_TABLES TO ROLE identifier($role_name); GRANT SELECT, UPDATE, INSERT, DELETE ON FUTURE TABLES IN SCHEMA STATSIG_STAGING.STATSIG_TABLES TO ROLE identifier($role_name); GRANT OWNERSHIP ON FUTURE TABLES IN SCHEMA STATSIG_STAGING.STATSIG_TABLES TO ROLE identifier($role_name); -- grant Statsig role read access to database and schema passed in -- do this at a table level, database level, and/or schema level -- for data Statsig needs to access GRANT USAGE ON DATABASE TO ROLE identifier($role_name); GRANT USAGE ON SCHEMA . TO ROLE identifier($role_name); GRANT SELECT ON ALL TABLES IN DATABASE TO ROLE identifier($role_name); GRANT SELECT ON FUTURE TABLES IN DATABASE TO ROLE identifier($role_name); GRANT SELECT ON ALL VIEWS IN DATABASE TO ROLE identifier($role_name); GRANT SELECT ON FUTURE VIEWS IN DATABASE TO ROLE identifier($role_name); COMMIT; ``` ### What IP addresses will Statsig access data warehouses from? [See FAQ](/data-warehouse-ingestion/faq#what-ip-addresses-will-statsig-access-data-warehouses-from) ### Additional Setup for Warehouse Explorer Warehouse Explorer makes it easy to find and bring data from any table into Statsig for ad-hoc analysis. To enable the Warehouse Explorer analytics feature, you should provide Statsig with additional permission to query the INFORMATION\_SCHEMA metadata tables. These tables let the Statsig user read schema, column, and table definitions. ```sql theme={null} GRANT USAGE ON SCHEMA .INFORMATION_SCHEMA TO ROLE identifier($role_name); ``` # Get Started With CURE Source: https://docs.statsig.com/statsig-warehouse-native/cure/cure-setup Set up Statsig CURE (Causal Uplift Recommendations) for Warehouse Native, including assignment data, metrics, and modeling configuration. ## Defaults By default, CURE will run a univariate regression when enabled - the same as the [standard implementation of CUPED](https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf). To add additional covariates ## Project Defaults In project settings under Experimentation, specify default covariates for your company: Project Settings Covariates can be sourced from the assignment source for the experiment, or from Entity Property Sources. Properties with an Entity Property Source name attached come from a property source and will always be included on the experiment. Properties without an EPS come from an assignment source, and will be used if the column exists on the assignment source of a given experiment. ## Experiment Settings Per-experiment, specify additional covariates or remove covariates specific to your analysis: Experiment Settings Like with the experiment settings, this list is sourced from the experiment assignment source columns as well as relevant Entity Property Sources. ## Metric:Metric One tool our framework enables is using metric:metric covariates; for example, using units' pre-experiment clicks as a covariate for units' in-experiment revenue. We decided against using this for several reasons: * Consistency. It should not be the case that adding a new metric to your analysis significantly alters the results of other metrics * It's not necessary. If you have key metrics that function as covariates, we recommend **explicitly** providing these - for all metrics - as a covariate in an entity property source. This achieves the same result, without "black box" outputs coming from unknown metric covariances Statsig has a strong opinion that throwing in arbitrary covariates based on an experiment's metric selection is a bad practice, and it is better to explicitly include key metric covariates as numerical covariates; please reach out in Slack if you would like to discuss! ## Preventing Adjustments You can turn off CUPED in your pulse results, and can create a project-level setting to enforce this. This will still run CURE, however, which entails some amount of compute cost. To avoid running CURE, you can turn it off on a given metric by un-checking the CUPED option in the metric's setup page. Metric Settings # CURE Source: https://docs.statsig.com/statsig-warehouse-native/cure/introduction Introduction to Statsig CURE (Causal Uplift Recommendations) for Warehouse Native, used to identify which users are most likely to respond to a treatment. CURE (variance Control Using Regression Estimates) is Statsig's extended implementation of [CUPED](/experiments/statistical-methods/methodologies/cuped), a technique which uses pre-experiment data for each experimental unit to control some of the data observed in an online experiment. CUPED can reduce variance moderately to significantly, and can correct for some amount of pre-existing differences between experiment groups at the time of the experiment. CUPED is a powerful technique that provides additional statistical power without needing additional users or time to run an experiment. However, it has one major limitation: it relies on using pre-experiment data for a given metric to reduce the same metric's variance. This means that: * New user experiments cannot leverage the technique * Metrics which are not autocorrelated do not see significant variance reduction ## What is CUPED? Please refer to our documentation [here](/experiments/statistical-methods/methodologies/cuped), or our longer-form blog [here](https://www.statsig.com/blog/cuped). ## How CURE Improves On CUPED CURE solves the no-covariate problem by allowing practitioners to specify additional covariates, which are used in the CUPED regression. The predictions from this regression are used to generate an estimator set that is unbiased, but has lower variance than the original estimator. This can reduce experiment runtimes more than CUPED in isolation and, when combined with Statsig's Entity Properties, allows practitioners to trivially plug in existing feature stores to this variance reduction technique. Similar to CUPED, CURE can modify the point estimates of groups; though the total value across all experimental groups will sum to the same value as in the unadjusted dataset. It is expected that there may be drift in value between the groups (particularly if there's some pre-experiment differences and the correlation is high). Generally this can be seen as the algorithm adjusting for pre-existing deltas in experiment groups, but please reach out in slack if you are concerned. ## How it's different This technique is similar to standard regression estimate used across other experimentation tools, but stands out in several ways: * CURE is applied on all major experimental surfaces that Statsig offers; this solves a common pain point of results not matching between scorecards and explore queries due to incomplete CUPED implementations. Statsig offers CURE across experiment scorecards, drill-down explore queries, filtered queries, and even power analysis * Multivariate regressions allow you to experiment with reduced variance on new users or users without pre-experiment data * CURE manages feature selection, preventing observed overfitting issues seen in similar multivariate approaches, and making the regression adjustments significantly more transparent and less of a vendor black-box # Approach The original version of CUPED specified by Microsoft [(Deng, et. al.)](https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf) is equivalent to running an OLS regression with one independent variables. However, this adjustment does not need to be a single-variable regression (or a regression at all). A regression has the benefit of guaranteeing identical or lowered variance, but so long as an adjustment is unbiased, any adjustment is mathematically valid. Thus by calculating the covariance matrix between covariates and experiment outcomes in SQL, you can cheaply generate the required inputs to generate multi-variate regressions and use these to adjust post-exposure values in an unbiased manner -- so long as this estimator is unbiased (critically, meaning the regression is demeaned and only uses pre-experiment data). Similarly to the original paper's proof, this yields an estimator that has variance less than or equal to the existing estimator. That means that: * if covariates do not exist, nothing happens and the adjusted estimator is identical to the unadjusted * in the worst case where covariates are uncorrelated, the adjusted estimator is identical to the unadjusted * if no additional covariates are applied, CURE is mathematically equivalent to CUPED * if additional covariates are applied that increase the $R^2$ (i.e. portion of explained variance), CURE reduces variance more than CUPED ## Note on CURE for ratio metrics Combining Statsig's CUPED implementation for ratio metrics and CURE covariates, we also enable ratio metrics to use both pre-experimental data and user-provided covariates. With adjusted formula to calculate the covariance of a ratio random variable to a simple variable, we are able to establish a multi-covariate regression just like CURE for simple metrics. However, the way we adjust the variance and mean follow that of ratio CUPED. * As an artifact of the regression, we obtain the shrinkage $R^2$, which we use to scale the post-adjustment variance. * We construct the mean adjustment as the inner product of the coefficient vector $\theta$ and the covariate group means $C$. By calculating the difference in adjustment between the test and control group, we only adjust the test group mean while holding the control group mean the same. # Data Sources When CUPED is enabled, pre-experiment metric data will still be an input to the CURE regression. The extension CURE offers is allowing you to pull data from: * assignment records; if using e2e experiments with Statsig, you can log user attributes using the SDK and they will be exported for your use in CURE, if desired * entity property sources: you can use any timestamp-enabled entity property source to provide covariate data. Note that at launch, non-timestamped entity property sources are disabled due to the potential for data leakage into CURE The latter option is a convenience feature; if desired, you could manage joining user features to your assignment data. In practice, this unlocks scalable use of the platform, unlocking use cases like providing your own table of regression estimates as covariates, e.g. Doordash's [CUPAC](https://careersatdoordash.com/blog/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/) approach. # Configuration CURE can be managed in two places: * Project Settings under Experimentation. The default CURE Covariates setting allows you to specify covariates you want to use across any CUPED-enabled metric in your project * Advanced Settings in the Experiment Settings page. This allows you to specify covariates specific to one experiment # Datasets Statsig will surface information on CURE in experiment diagnostics, giving details on coefficients used and relevant covariate for any given metric. You can also see the adjustments used under the Regression Adjustment job, and check the coefficients table to run your own analysis on the regression model. # Outputs You can view the DAG history to see the exact coefficients used and the table they are stored within. Additionally, Statsig will render an explanation in-console of which variables contributed to reducing variance, and how much variance was reduced -- which has been helpful for users deciding which features to use. # Notes Similar to CUPED, CURE can modify the point estimates of groups; though the total value across all experimental groups will sum to the same value as in the unadjusted dataset, is it expected that there may be drift in value between the groups (particularly if there's some pre-experiment differences and the correlation is high). Generally this can be seen as the algorithm adjusting for pre-existing deltas in experiment groups, but please reach out in slack if you are concerned about a concerning change. ## Feature Selection We use Lasso regression to select important features, reducing the computational cost of adjusting the unit metric level and controlling the multicollinearity. You can view the contribution of each feature on the variance reduction card. The estimated proportion of variance reduction is achieved by $$ \text{Variance Reduction}_{i} = \frac{{Coefficient_{i}}^2*Var(feature_{i})}{\sum {Coefficient_{i}}^2*Var(feature_{i})} $$ by that, $$ \sum \text{Variance Reduction}_{i} = 100\% $$ # References * [From Augmentation to Decomposition: A New Look at CUPED in 2023](https://arxiv.org/html/2312.02935v1) * [Improving Experimental Power through Control Using Predictions as Covariate (CUPAC)](https://careersatdoordash.com/blog/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/) * [Agnostic notes on regression adjustments to experimental data: Reexamining Freedman’s critique](https://projecteuclid.org/journals/annals-of-applied-statistics/volume-7/issue-1/Agnostic-notes-on-regression-adjustments-to-experimental-data--Reexamining/10.1214/12-AOAS583.full) # Autotune (Beta) Source: https://docs.statsig.com/statsig-warehouse-native/features/autotune Use Statsig Autotune with Warehouse Native to automatically allocate traffic to the best-performing variant for a single goal metric using bandits. Autotune Experiments in Warehouse Native have a very similar setup to cloud [Autotunes](/autotune/overview). In Warehouse Native, Autotune success events will be pulled from a Metric Source, and computation of results will happen in your warehouse. Continuous Autotune in WHN is currently in beta, please contact the team to get it enabled in your account # Creating an Autotune To set up an Autotune on Warehouse Native, start by defining your objective. This can either be an *event* or a *value* from a Metric Source. For detailed instructions, see [how to set up a Metric Source](/statsig-warehouse-native/configuration/metric-sources). You can also specify the optimization direction (Maximize or Minimize) to determine how Autotune evaluates variant performance. Autotune configuration interface 1. Navigate to the [Experiments section](https://console.statsig.com/experiments) in the sidebar of the Statsig Console. 2. Click on the [Autotune tab](https://console.statsig.com/autotune) at the top. 3. Click the Create button and enter the name and description of the Autotune Experiment that you want to create. 4. Select an ID Type for your Experiment. 5. Create and name your variants for your Autotune Experiment. The variant that's listed as Control/Default will be returned when the Autotune Experiment is not running. 6. Select your Metric Source that you defined earlier as shown below. Autotune metric source configuration interface There are a few parameters you can specify: * Exploration Window - The initial time period where Autotune will equally split the traffic. This is useful for noisy or temporal metrics where hourly swings in data can bias Autotune's initial measurements. * Attribution Window - The maximum duration between the exposure and success event that counts as a success. We recommend 1 hr for most applications, but adjust accordingly if you expect the success event to lag the exposure event by several hours. * Winner Threshold - The "probability of best" threshold a variant needs to achieve for Autotune to declare it the winner, stop collecting data, and direct all traffic. Setting a lower number will result in faster decisions but increases the probability of making suboptimal decisions (picking the wrong winner). Click "Create" to finalize the setup. 7. Your Autotune is set up and ready to go. Click "Start" when you're ready to launch your Autotune test. # Cohort Metrics Source: https://docs.statsig.com/statsig-warehouse-native/features/cohort-metrics Cohort metrics in Statsig Warehouse Native analyze the impact of an experiment within a specific time frame per experimental unit, like new user cohorts. Cohort metrics are useful for many reasons. Common use cases are: * By ensuring all users have equal periods for data collection, there is an "apples to apples" comparison across user enrolled early/late in the experiment (which often corresponds to power/occasional users), and across different "time periods" that may have extrinsic factors like holidays * If analyzing an unbounded period, experimental units' variance in the population can increase over time - leading to scenario where error bars don't actually converge towards 0 as the experiment is run for longer! * This allows one to skip noisy early metrics, or focus on outcomes after users might churn - e.g. capturing "week-2 engagement" if a product has a 1-week trial period * This can also be used to capture "one-shot retention". [Retention metrics](/statsig-warehouse-native/metrics/retention) are used to capture rolling, ongoing retention. A user metric with a window from day X to day Y is a good way to check if an experiment is causing more users to retain at least X days The downsides of cohort metrics are that: * They do not capture any sort of long-term impact, or how that evolves over time. This is purely a point in time analysis and may not be appropriate for measuring complex, evolving behaviors * They make topline impact estimates lossier and harder to trust Some practitioners have made compelling arguments that cohort metrics are a better "standard" metric for organizations to use in analysis. Statsig tends to believe that the use of cohorts is dependent on business context, but it's worth considering if they should be at least a part of an experiment's measurement (e.g. measuring topline revenue as an overall evaluation criteria, but also measuring 7d revenue alongside it for additional context). Cohort metrics on Statsig are feature-rich. This page explains the different settings available, what they do, and how they interact so that you can be confident that you know what's being measured. ## Basic Cohort Windows Basic cohort windows are fairly simple. They are a filter on metric data with a time range relative to the unit's time of exposure. For example, this cohort metric from 1 to 6 days would filter to events from 24 hours until 168 hours from when they were exposed to the experiment. This is calculated as a timestamp comparison; a unit enrolled at 12pm will have exactly 24 hours until they hit the end of a 0-1 day cohort. For metrics from data sources that are marked as daily data, the cohort comparison is truncated to a date so that day-0 data behaves as expected (e.g. a user exposed on `2025-01-05T09:00` will include the date-based data from `2025-01-05` instead of truncating to times "after 9:00am"). ## Waiting for Maturation By default, cohort metrics can have a mix of maturation in the experimental population. For a 1-week cohort, users enrolled in the last week of the cohort will have a mix of maturities during analysis. This does yield maximal sample, but can "dilute" the analysis with partial cohort windows. To prevent this, mark the metric as needing to "Wait for cohort window to complete". This will drop units' metric data from analysis, and removes them from the experiment analysis population. In the examples below, one metric forces cohorts to complete. It has **less units** in the analysis, since many units do not have a complete window, a **lower total** because of the small unit count, but a **higher mean** since the units it does have have completed their window and have a longer data collection period on average. Metric that did not wait to mature Metric that waited to Mature This can cause confusion since it leads to different populations between metrics, and also filters out the last few days of an experiment's data in the daily timeseries since new cohorts' data is not including. A metric's cohort settings can be seen quickly by hovering over the name of the metric in the experiment scorecard. ## Visual Examples This is what the data collection looks like for a standard cohort metric with a 0-6 day window. Note that this collects data for **7 days** - it's 0-indexed. Basic cohort example If the cohort period goes over the end of the experiment, the default behavior is that the data collection is truncated to the end of the experiment Basic cohort over end example If the metric is configured to only allow completed cohort windows, the unit is completely excluded from the analysis. They are not part of the denominator for the average value of a sum or count metric, and their metric data is filtered from the analysis. Basic cohort example If mature after end is configured in the experiment, it will continue to collect data after the end of the experiment (whether wait for mature is enabled or not). Basic cohort example ## Experiment-based Cohort Settings Cohort controls are also available on experiments. This is because the relevance of these features depends on the kind of experiment being run, as discussed below. ### Allow post-experiment cohort data Checking **Allow Cohort Metrics to Mature After Experiment End** allows metrics to be collected after the end of the experiment. This is recommended for one-time interventions, e.g. a new signup page; getting post-experiment signal from units who did get the intervention means additional statistical power. Allow Cohorts to Mature This is **not** recommended in cases where the intervention is continuous, e.g. a ranking change; post-experiment data will be diluted (e.g. test users might get the control experience in their post-experiment period), diluting results. On the data side, the analysis will be extended past the end of the experiment by the length of the longest cohort window on any metric in the analysis. Non-cohorted metrics will have their date constrained to the analysis period, and cohort metrics will be filtered to the end of the experiment + their cohort window. ### Fixed Duration On the **Experiment Population** section of the experiment setup page, there is an option for fixed-window cohorting under **Configure Analysis Period** with Analysis Type **Fixed Duration**. This only counts metrics for a certain period after **experiment start**. This is useful when there's an experiment with a specific point in time, like email campaigns or other cases of fixed enrollment events. Analysis Period Settings This setting is only available for assign and analyze experiments. ### Cohorted Duration On the **Experiment Population** section of the experiment setup page, there is an option for allocation-based cohorting under **Configure Analysis Period** with Analysis Type **Cohorted Duration**. This is identical to the metric-based cohort, but applies globally to all possible metrics in the experiment analysis. This is a great way to globally add a cohort when appropriate, e.g. new user experiments. When this is used in conjunction with metric cohorts, the minimal end of the cohort window will be used. For example, if a metric cohort is set to end at 7 days and the experiment at 10 days, 7 will be used. If the metric cohort is set to end at 7 days and the experiment at 5 days, 5 will be used. The **only include units with a completed cohort window** setting can also be specified at the experiment level and will apply to all metrics in the experiment if so. If not checked, this will be applied on a metric-by-metric basis. ## Metric Bake Windows In some cases, a metric in your warehouse may not be matured until a certain time period, after which you care about the daily value. Statsig provides the option to specify a bake window for your metrics. Similarly to the option above, metrics who have not reached the bake window's end will be excluded from the numerator and denominator of the metric in the analysis. One example of a metric that may not be immediately mature is daily revenue. If a user makes a purchase but refunds it a few days later, their daily revenue may retroactively change to reflect the refund on a later date. A 28 day long refund period for users might align naturally a 28 day long bake window for revenue metrics. Metric Bake Example In the example above, the user has a revenue metric that considers the last 28 days. Instead of calculating a partial result, they will exclude users from the analysis until they've had 28 days for their data to mature. Partial results can happen because if a metric bakes over a long period, part of the metric values are from before the experimental intervention and will dilute the results. If a user had been exposed 1 day ago, 27/28 days of their revenue metric would be from the pre-intervention period and would dilute the experiment results. # Configuring Experiments Source: https://docs.statsig.com/statsig-warehouse-native/features/configure-an-experiment Configure a Statsig Warehouse Native experiment, including assignment source, variants, metrics, allocation, and analysis settings end to end. Setting up an experiment is a core flow in Statsig. In Warehouse Native, there's two modes to setting up an experiment. In one, you'll connect to existing assignment data from your warehouse. In the other, you'll configure the experiment in Statsig, use the Statsig SDK, and then analyze the resulting exposures in your warehouse. This page goes through creating and configuring an experiment for analysis in Statsig. There are many options and settings; for most A/B/n experiments, defaults will yield a standard, powerful analysis. ## Creating An Experiment To create an experiment, you can go to the experiments tab in your console and press the create button. There's two types of experiments in Statsig Warehouse Native: * Analyze: these are for 3rd-party or in-house exposure sources * Assign & Analyze: these are Statsig-configured experiments. You can set up all of the configuration here, implement it through Statsig SDKs, and track results. Experiment type selection interface Next, you'll give the experiment a name, specify your hypothesis, and pick the experiment from your exposure sources. You can sync sources if your exposure isn't loaded in the dropdown. You'll need to specify the control group as well as the ID type the experiment is using here. Configure Experiment ## Choose Metrics You'll land in the setup page for the experiment. Here, you can further refine your hypothesis and add the Primary and Secondary metrics for your experiment. * Primary metrics should be a short list (1-3) of metrics that specify your overall evaluation criteria for the experiment. Generally this is one target metric, and a first-mile metric (e.g. revenue, and checkout clicks) * Secondary metrics are guardrails and explanatory metrics. These are more observational in nature, and should be treated as less conclusive. Applying bonferroni corrections is a way to formalize this approach, though it can be overly conservative. Choose Metrics ## Choose Groups Statsig infers the groups and group splits in your experiment, but you should check and correct the splits to make sure SRM checks can work as intended. Here, you can also rename groups or delete irrelevant groups. Choose Groups ## Advanced Settings Statsig has many settings you can configure in your experiment. Defaults for these can be set at the org level as well. These include: * Frequentist vs. Bayesian analysis * Target duration of the experiment * Whether to apply [Sequential Testing](/experiments-plus/sequential-testing) adjustments * Allocation Duration * You can choose to stop enrolling new users into the experiment after X days You need to set up [Persistent Assignment](/client/concepts/persistent_assignment) to retain already-enrolled users to stay in the same group * Cohort Duration * You can choose to specify a timeline for collecting metric data * You can choose to only include units with a completed duration, so units with incomplete data will not be included in pulse * Whether to allow cohort metrics to mature after experiment end Once you configure this value, it will overwrite the choice for `only include units with a completed duration` (see above) * [ID stitching](/statsig-warehouse-native/features/id-resolution) * Whether and how to apply [Bonferroni Correction](/stats-engine/methodologies/bonferroni-correction) * Default Confidence Intervals * Default rollup windows for result readouts (cumulative, 1, 7, 14, or 28 days from the analysis date) * Turbo Mode * You can choose to skip the calculations for time series * This is useful if you want to reduce cost and runtime, and you only care about the overall effect * Filter exposures by qualifying event * Only applicable in Analyze-only experiments * Filter assignment source * Add optional, additional filters on the experiment's assignment source. By default, the assignment source is filtered to the experiment and groups being analyzed. This setting can be used to make the primary scorecard ignore known bad actors, or analyze a sub-population in the main scorecard. ## Testing in a lower environment Once experiments are launched, you can't edit the groups without restarting the experiment, as users are already being allocated to each group. We therefore recommend testing each experiment in lower environments before starting. You can do this by clicking the "Test" button in the experiment setup page, then selecting "Enable for Environments". These environments should match your [SDK environment setup](/guides/using-environments/#configuring-environments). Testing in a lower environment and [overrides](/experiments-plus/overrides) can help you manually set your experiment "group" to properly test each variant. Experiment test button interface When an experiment is enabled for lower environments, Warehouse Native customers can check exposure logs to troubleshoot exposures. However, experiment results and cumulative exposures are not available. ## Start The Experiment With all this done, press Save and Analyze to start the experiment. You'll be prompted to finalize the dates, experiment status, and optionally set up a schedule to reload experiment results. Schedule When you press load data, a Pulse analysis will start and you'll be taken to the results page. ## Data Freshness For a Statsig-configured experiment where you're using the Statsig SDKs to generate exposures - the default is that exposures are batched, deduplicated and written to your warehouse once a day. When you launch an experiment, it's helpful to be able to look at early metric deltas (to detect crashes or catch an egregious bug). When Pulse is loaded soon after the experiment starts, we'll update exposures in your warehouse before computing Pulse results. This lets you see Pulse results as fresh as \~15m (assuming events and metrics come in at the same speed). Under the covers, if the # of exposures on an experiment is \< 1 million we perform a just-in-time update of exposures in your warehouse when Pulse is loaded. We only write to your warehouse exposure information that will be used for experiment analysis. We will not write exposures in pre-production environments or from overrides since these are not used in analysis. # Differential Impact Detection Source: https://docs.statsig.com/statsig-warehouse-native/features/differential-impact How Statsig Warehouse Native automatically flags experiments with extreme differential impacts on sub-populations so you can investigate heterogeneous effects. ## What is Differential Impact Detection? Experiments can have interesting effects on sub-populations that are easily missed. They might have a bug that impacts only a certain browser, OS, or country. If the topline impact isn't significant or is canceled out by other changes - these are missed. Statsig will automatically flag experiments when extreme differential impacts are detected for any sub-population you have configured. Once configured, experiments are analyzed for differential impact when Pulse is loaded after Day 1, Day 3 and when the Target Duration is met. Differential impact detection alert in Pulse results ## Enabling this Configure the "Segments of Interest" you want automatically evaluated for Differential Impact Detection. On Statsig Cloud, these are user properties in the [User Object](/concepts/user) you configure when using the Statsig SDK. On Statsig Warehouse Native they can be configured as an [Entity Property](/statsig-warehouse-native/features/entity-properties) too. Segments of Interest configuration interface This feature is also referred to as **Heterogeneous Treatment Effect** or **Segments of Interest**. ## Seeing Differential Impacts If extreme outliers are found for a segment you have configured, Statsig will flag this when you're looking at Pulse results. You will be able to see the data broken out by segments in the Explore section of your Pulse results. Pulse results showing differential impact by segments ## Methodology We use a Welch’s t-test to compare the treatment effect for a particular segment of users to the treatment effect for all other users since we expect to potentially see unequal population variances when comparing user segments. The average treatment effect is calculated as follows: $\overline{TE} = \overline{X_t} - \overline{X_c} $ The variance in treatment effect is calculated as follows: $var(TE) = var(X_t) + var(X_c)$ The n of the treatment effect is calculated as follows: $n_{TE} = min(n_t, n_c)$ With these calculations, we can determine the t-statistic and degrees of freedom as we would for any experiment using [Welch's t-test](/stats-engine/p-value#welchs-t-test). We then use a Bonferroni Correction to adjust our alpha to avoid false positives, setting a threshold where we consider there to be a high likelihood of heterogeneous treatment effect or some likelihood of heterogeneous treatment effect. We consider a specific dimension vs the rest calculation for one metric of one test variant vs the control to be one "comparison" for the sake of the Bonferroni Correction. $\text{high chance of HTE, } \alpha = \frac{0.01}{\text{number of comparisons}}$ $\text{some chance of HTE, } \alpha = \frac{0.05}{\text{number of comparisons}}$ ## Viewing Results You can go to the explore tab of your experiment and filter to the Differential Impact Detection query type to see all historical analyses. ## # Configuring Experiments Source: https://docs.statsig.com/statsig-warehouse-native/features/experiment-options Reference for advanced experiment options in Statsig Warehouse Native, including variance reduction, CUPED, sequential testing, and stratified sampling. # Configuration In Statsig Warehouse Native, there's a large amount of configurability in experiment setup and analysis. Statsig users should be able to run an experiment with "default" options and get powerful, trustworthy statistical analysis of your results, but in many cases an advanced configuration helps users maximize power and helps to measure exactly what is intended to be measured. This page is intended as a broad glossary for advanced experimentation settings - what they do, how to use them, and tradeoffs to be aware of. # Experiment Settings ## Basic Settings ### Hypothesis Hypotheses are required to run experiments in Statsig. Hypotheses should specify what an experiment is aiming to accomplish, and how that will be measured. ### Primary and Secondary Metrics These are the metrics used as the evaluation criteria for your experiment. Generally, Statsig recommends a small number of primary metrics to use as your overall evaluation criteria, and to put guardrails and exploratory metrics into Secondary. There is project-level configuration of the maximum number of primary/secondary metrics available; this behavior varies by company and industry, as well as the complexity of the space being measured. ### Experiment Duration In experiment setup there are fields for *Experiment Measured In* and *Target*. This allows configuration of how long your experiment should be run for, influencing sequential testing as well as notifications/timeline alerts. Statsig's [power analysis tools](/statsig-warehouse-native/features/power-analysis) are the best way to determine what the target duration should be. A power analysis can be attached to an experiment to add context on the duration. ### Experiment Configuration ### Assignment Source and Groups For analysis-only experiments, this section will be pre-filled from the observed data in the data warehouse. Images and descriptions can be configured. Group sizes will be inferred, but should be reviewed and corrected if mismatched with the intended traffic split. This can be updated by resetting the experiment from the decision menu after it has started. ### Groups and Parameters For end-to-end experiments (experiments using Statsig for both assignment and analysis), this section is where targeting, layers, and finally the groups and associated parameters are configured. Exposures generated from the setup will automatically be associated with this experiment for analysis. ### Advanced Settings There are a large number of advanced settings available for customizing analysis. These can have complex interactions with data. ### Pre-computed User Dimensions Dimensions can be configured as default breakdowns in pulse. That is, a user dimension (e.g. country) can be specified here and it will be available in the scorecard results with daily loads. This allows users to skip the process of doing scheduled explore queries for this dimension, and allows the results to be available "inline" in their scorecard. ### Stratified Sampling Stratified Sampling allows users to balance their experiments across behavior or segments. This is achieved by testing random salts, and picking the one that best-balances user attributions. This can be partially achieved during analysis using [CURE](/experiments-plus/cure/) See [Stratified Sampling](/experiments-plus/stratified-sampling) for further documentation. ### ID Type and Secondary ID Type The ID type, which is the unit of randomization for an experiment, is configured when setting up the experiment and is a critical field. This represents the kind of entity being experimented on. For example, if splitting traffic randomly per user, this should be `User ID`. If splitting traffic by company, this should be `Company ID`. Secondary ID types are used to associate metrics on a different ID with the unit of randomization. Concretely, if running an experiment on a logged out cookie ID, specifying `UserID` as a secondary ID allows easy analysis of `UserID` metrics like revenue, while keeping `Cookie ID`s as the unit of analysis (e.g. the denominator in means). This is a powerful way to understand downstream impact of your experiments without introducing survivorship bias or other bias to the analysis. This mapping can come directly from the assignment source. If multiple exposures are logged for a given unit, and at least one has both ID types, Statsig will identify that these two IDs are mapped to each other. Alternately, mappings can be specified in an Entity Property Source - just provide a mapping table of `ID1` to `ID2` and Statsig will connect the data during analysis. Secondary ID mapping can be configured as an enforced 1:1 mapping, or as a first-touch attribution. Refer to the [full documentation](/statsig-warehouse-native/features/id-resolution) for more details. ### Allocation and Cohorting **Configure Allocation Duration** If using a persistent assignment SDK ([docs](/client/concepts/persistent_assignment)), this setting controls when to stop enrolling users into the experiment. This can also be used without a persistent assignment SDK to filter users after the duration period from the analysis - this use case is helpful for enforcing even cohorts across a userbase. For example, if a metric takes 7 days to mature, and stopping the experiment will halt the experiment effect, setting the allocation duration to 14 days and running the experiment for 21 days will analyze the first 14 days of exposed units while analyzing all 7 days of the last cohort's metric behavior. **Configure Analysis Period** This is similar to the allocation duration, but controls the dates from which metric data will be collected. This is useful for truncating the analysis while continuing to run an experiment **Allow cohort metrics to mature after experiment end** This setting allows cohort or baked metrics that take time to mature to continue collecting data after the end of an experiment. This is only recommended if an experiment is a one-time intervention. For example, consider a 14 day experiment on new users that modifies a signup page. Removing the changes to the signup page does not impact users who already saw it, so to maximize data for "first-week revenue" metric, this setting can be checked and data will continue to collect for users enrolled on the 14th day until the 21st day, or for users enrolled on the 10th day until the 17th day, even though the experiment is no longer running after the 14th day. ### Cure Covariates This section allows configuration of covariates for CURE. It is recommended to configure strong defaults in the project settings, and use this to add relevant/domain-specific covariates. See the [CURE](/experiments-plus/cure/) documentation for more details. ### Analysis Settings **Analytics Type** Whether to use Frequentist or Bayesian analysis. This cannot be changed once an experiment starts to avoid cherry-picking methodology. **Apply Sequential Testing \[Frequentist Only]** Controls whether [sequential testing](/experiments-plus/sequential-testing) is applied. This setting is recommended to avoid false positives from peeking. **Bonferroni/Benjamini-Hochberg \[Frequentist Only]** Configures multiple-comparisons corrections - either controlling the false positive rate or false discovery rate. Refer to the more detailed documentation for [Bonferroni](/statsig-warehouse-native/features/statistics/methodologies/bonferroni-correction) and [Benjamini-Hochberg](/statsig-warehouse-native/features/statistics/methodologies/benjamini-hochberg-procedure). **Default Confidence Interval/Chance to Beat Threshold** The confidence level used for this experiment. By default this is 95%, which is a strongly recommended default and industry standard. However, depending on the risk profile of the experiment and the risk tolerance of the experimenter, it might make sense to use a stricter or less strict setting in some cases. **Use Informative Priors \[Bayesian Only]** For bayesian experiments, whether to use informed priors in analysis, and the configuration for them. **Turbo Mode** Whether to use [Turbo Mode](/statsig-warehouse-native/features/turbo) to run experiment reloads more quickly. **Filter Exposures by Qualifying Event** This setting allows filtering exposures to experimental units that did (or did not) trigger a secondary event beyond exposure. This is very useful for analysis-only experiments on web experimentation platforms that over-expose heavily. Generally for end-to-end experiments we recommend using statsig to expose only at the point of intervention. This setting has a few inputs: * A qualifying event, which is joined to exposures to understand if users triggered an event * *Exclude Matching Units* - if toggled, units that triggered this event will be *dropped from the analysis*. If unchecked, the analysis will be filtered to units that triggered the event. * *Use qualifying event timestamp for first exposures* if checked, units' exposure timestamp will be replaced with the qualifying event's timestamp. This is useful for small cohort windows - e.g. wanting to measure if something happened within 10 minutes of the intervention. If the intervention occurred several minutes after the original exposure event, it can be helpful to use the actual time that the user saw the intervention * *Filter events by time window* only consider qualifying events for the inclusion/exclusion filters which occurred within a certain time from the exposure. This is useful if a user may come back and re-trigger the qualifying event when it's no longer relevant to the exposure. **Filter Assignment Source** This setting controls additional filters to be applied to the exposure data for this experiment. This can be useful to filter out bad dates with data that is known to be biased or non-representative, or to filter to a specific subset of interest for the scorecard results. **Default Date Filter** Statsig calculates results across multiple rollups, including Cumulative, 7, and 1 day(s). Cumulative is almost always the correct choice for this setting, as it maximizes statistical power and the other views can be easily switched to in the scorecard. # Explore (Custom) Query Settings Explore queries enable drilldown, filtering, grouping, and more advanced ways to cut metric data by user and metric properties. These are a way to deeply understand experiment results. On Statsig, explore queries run with the same statistics as the scorecard; with no filters or other settings, results should match the scorecard results. ## Metrics Pick the metrics to run the explore/drilldown analysis on - this can include tags and local metrics. ## Group By This setting is the bread and butter of explore/custom queries and allows drilldowns into user segments. This is used to understand differentiate performance and can be combined with [differential impact detection](/experiments-plus/differential-impact-detection) to deeply understand experiment results beyond topline averages. By default only the top 10 dimension levels with over 100 units per segment are shown in results. The rest are grouped into an OTHER category. This is to prevent aggravated false positive rates and keep statistical analysis rigorous; reach out in slack if this needs adjustment. ## Filter Filters can be used to include/exclude certain cohorts of units in an explore analysis by specifying the property and a filter-set for it. ## Time Range for Metric Data This setting filters metric data to a certain date period; e.g., seeing the results for all users on the 3 days from April 10 to April 12. This can be useful to understand blips in data, get a recent cohort if results have changed dramatically, or generally run drill-down analyses. ## Filter by Exposure Date In some cases it is useful to filter to units that entered the experiment on a certain date, or filter those out (e.g. units exposed on Christmas may behave oddly). This option allows users to specify filters based on user exposure date and remove them from analysis. By switching the mode to Days Since First Exposure, users can also drill down into each cohort's behavior during a certain window post-exposure. This is a great way to understand behaviors observed in the days-since-first-exposure timeline view. ## Scheduling After running a query, it can be scheduled. This query will be run daily after any scheduled pulse load to update the results. Scheduled queries can also be put into the experiment Summary page as part of the experiment writeup and downloadable summary. ## Interaction Effect Detection Explore queries can be used to trigger [interaction effect detection](/experiments-plus/interaction-detection) analyses to understand if two experiments are interacting in a synergistic or destructive fashion. # Aggregated Impact Source: https://docs.statsig.com/statsig-warehouse-native/features/exploring-results/aggregated-impact How Statsig Warehouse Native calculates aggregated business impact across metrics for an experiment so you can see the total estimated launch effect. ## Metric Insights and Aggregated Impact Statsig's Insights page provides a clear view of how experiments and feature gates impact a specific metric of interest. It not only helps answer key questions such as "How much impact have I driven?", but also serves as a powerful tool for diagnosing unexpected changes in metrics. Insights presents a reverse perspective of the [Pulse](/pulse/read-pulse) view. While Pulse measures the impact of a new feature on all your metrics, Insights allows you to focus on a single metric and identify which tests are impacting it the most. This makes it particularly useful for assessing your or your team's impact, as well as setting realistic goals for your team or company. ## How to read Insights 1. Navigate to the Insights section on the Statsig console: [https://console.statsig.com/](https://console.statsig.com/) . It is also available in the insight tab for each metric. 2. Select a metric that you want to observe from the selector drop down at the top of the page. 3. Select the ID type, time window and other filters that you want to observe. 4. Based on the filters you choose, you can see the relative impact, topline impact and projected launch impact for any experiment/gate which has this metric. 5. We also sum up the projected launch impacts, adjust based on false positive risk ('winner's curse') and show as the 'Aggregated Impact Estimate'. Insights dashboard showing aggregated impact estimates for a metric ## How the math works Check how the topline and projected launch impact are calculated in this [doc](/stats-engine/topline-impact/#computing-projected-launch-impact). To estimate false positive risk and calculate Aggregated Impact, we use the methodology in this [paper](https://dl.acm.org/doi/10.1145/3534678.3539160) which is widely adopted across the industry. Specifically: $$ Aggregated Impact=\sum_{i}{(1 - FPR_i) \times Projected Launch Impact_i} $$ Where the [projected launch impact](/stats-engine/topline-impact/) is an estimate of the topline impact assuming a decision is made and the test group is launched to all users; the false positive risk is calculated by the following formula: $$ FPR_i = \frac{\alpha_i \times \pi}{\alpha_i \times \pi + (1 - \beta_i) \times (1 - \pi)} $$ In this formula, $ \alpha_i$ is the significance level for experiment i, $ \beta_i$ is the type II error, and 1 - $ \pi$ is the prior success rate based on historical experiment results. # Filter Exposures Source: https://docs.statsig.com/statsig-warehouse-native/features/filtering-exposures Filter experiment exposures in Statsig Warehouse Native by user property, qualifying event, or time window to focus analysis on relevant subpopulations. ## Filter Exposures by Qualifying Event You can use Qualifying Events to filter exposures to units who did or did not trigger an event after being exposed. This is a powerful tool, especially for analysis-only experiments where the assignment tool may have over-exposed units (e.g. assigning units on page load when the intervention was only triggered when a button was clicked). Qualifying event filter configuration interface We recommend caution using this tool, as it's possible to introduce post-assignment data into your assignment data, biasing results. Because of this, qualifying Event filters are disabled on Assign and Analyze experiments by default, since with the Statsig SDK experiments are usually not overexposed. The Statsig team can turn this feature on by request if you have a use case for it! There are a few settings: * Qualifying Event: the event source to qualify exposures with. This can be filtered, so you can pick specific target events within a qualifying event source * Exclude matching units: whether to include, or exclude units who triggered the event * Use qualifying event timestamp for first exposures: if the actual intervention occurred when the unit triggered the qualifying event, check this so that Statsig knows to override the exposure timestamp with the qualifying event timestamp * Filter events by time window: restrict the qualifying event matching to events that happened within X days/minutes of the unit's exposure event ## Filter Assignment Source You can also filter exposures to units based on the columns in assignment source. You can directly use certain columns, as well as apply filters on top of that. Similar to filtering by qualifying event, we recommend using this tool carefully because it can artificially introduce bias to the experiment results. Assignment source filter selection UI # Early Diagnostics Source: https://docs.statsig.com/statsig-warehouse-native/features/freshness Understand data freshness in Statsig Warehouse Native, including how often experiment data is reloaded and how to interpret last-updated timestamps. ## Exposures from Statsig SDKs You'll get just-in-time updates for the first 1 million exposures/experiment when Pulse is loaded; afterwards, updates will be on a daily cadence. After launching an experiment, you may want to check certain results -- such as page load time, impressions on key pages, and business-critical metrics -- as early as possible to detect crashes or catch a bug. If you are using the Statsig SDK to generate exposures, when Pulse is loaded soon after the experiment starts, we'll update exposures in your warehouse before computing Pulse results. This lets you see Pulse results **as fresh as \~15m** (assuming events and metrics come in at the same speed). Under the covers, we perform a just-in-time update of exposures in your warehouse when Pulse is loaded, for the first 1 million exposures. After that, the exposures are batched, deduplicated and written to your warehouse once a day. The daily job does brings in additional fields that the JIT write doesn't (e.g. Group Names). ## Custom Events from Statsig SDKs Custom events logs will be exported to your warehouse hourly, plus a short amount of processing delay. If you're using our SDKs to capture events, we'll export these events to your warehouse hourly. You can see Pulse results on metrics derived from those events **as fresh as \~1hour**. # Full Reloads Source: https://docs.statsig.com/statsig-warehouse-native/features/full-reloads Configure full reloads in Statsig Warehouse Native to recompute experiment and metric data from scratch when source data or definitions change. Full reloads will completely wipe the Staging/Results Datasets Statsig has used for previous pulse calculations, and Statsig will recalculate results from scratch. Generally, this is used for * Initial or historical pulse loads. * Cases where data has been lost or dropped on the customer side, meaning incremental reloads have lost state. * Cases where data changes frequently, e.g. a DBT full reload changes historical data due to chargebacks, model changes, or other reasons. * There's complex data dependencies, and a team wants to ensure that the gap between Statsig and Internal Systems doesn't cause inconsistencies. # ID Resolution (ID Stitching) Source: https://docs.statsig.com/statsig-warehouse-native/features/id-resolution Map cross-platform IDs in Warehouse Native experiment analysis to stitch anonymous and logged-in user data into a single experimental unit for accurate results. Statsig warehouse native natively supports resolving multiple IDs to one identified user, allowing you to easily expose an experiment on one identifier and analyze data coming from one to many mapped identities associated with that experimental unit. Common scenarios where this is used are: * Exposing logged out users and analyzing logged-in metrics like revenue or a funnel going from logged-out marketing page landing -> to a logged-in subscription purchase. * Utilizing one:many relationships, e.g. a single user owns multiple accounts. ID resolution lets you aggregate metric from the user's mapped accounts. Note that you lose power when using this approach but it is statistically sound. ID resolution is a common need in experimentation; generally the responsibility for this mapping is put onto data users or PMs running experiment analysis, which leads to inconsistent results and expensive query logic. Using Advanced ID Resolution streamlines the process, making it consistent and performant and allowing all users to point to trusted identity tables. ## The Challenge: Connecting User Identifiers A common challenge in experimentation is linking user identifiers before and after an event boundary—most often, signups. Experimenters usually have a logged-out ID (e.g., a cookie or Statsig stableID) and, for users who sign up, a userID created afterward. Since business metrics are typically computed at the userID level, teams often want to randomize on logged-out identifiers but measure outcomes on logged-in metrics like revenue or LTV. Most platforms require manual joins or preprocessing to connect these identifiers, leading to complex, error-prone queries that must reconcile exposures across time and mapping tables. Statsig Warehouse Native eliminates this overhead with an automatic, no-code way to connect identifiers across these boundaries—centralized, consistent, and reproducible. Docs_IDresolution_TheProblem_111224b ## Mapping Modes When using ID resolution, you can choose from one of three modes: * Strict 1:1 mapping enforces that identities have a singular mapping. If you have a mapping between two IDs that are always 1:1, this mode enforces that the mapping is singular and warns you if there's data where that's not the case. Users with a single identity can use downstream metrics from the secondary identity, and multi-mapped users are considered corrupted and discarded from the analysis. * First-touch mapping is a way to attribute activities of secondary ID(s) to one primary ID by recognizing the treatment effect comes from the first time the user is exposed to the experiment. * Last-touch mapping is a way to attribute activities of secondary ID(s) to one primary ID by recognizing the treatment effect comes from the most recent time the user is exposed to the experiment. ### Strict 1:1 Mapping Enforced1to1Mapping All potential mappings between identifiers within the experiment date range, on the exposed population, are collected. If the primary ID has multiple secondary IDs, or vice versa, it is considered polluted and dropped from the analysis. Choosing this mode will change the exposures on the primary ID as it disqualifies any records outside of a 1:1 mapping. ### First Touch Mapping (Mixed Population) Enforced1to1Mapping The direction of first-touch mapping will be based on the experiment; all secondary IDs resolve to 1 primary ID, and a single primary ID can have multiple mapped secondary IDs. Data is attributed to the group of the first associated primary ID seen in the exposure. If a secondary ID has multiple associated primary IDs, the group of the first primary ID will be used. Note that this means users that cross groups are not discarded from analysis but instead are assigned based on the first experience they had. Primary ID records that are associated with another Primary ID, but are not the first observed records, are dropped from the analysis. If a user is exposed twice on different primary IDs that resolve to the same secondary IDs, only the primary ID metrics from the first-exposed user will be kept in the analysis. ### Last Touch Mapping (Mixed Population) LastTouchAttribution Same as first touch but data is attributed to the most recent primary ID. ### What does Mixed Population mean? MixedPopulation Both first-touch and last-touch mapping show pulse results based on mixed population. This means that each metric will be based on the corresponding population of the unit type of itself. For example, if you have an experiment that randomizes on Stable ID and the scorecard metrics are a mixture of Stable ID and User ID, for Stable ID metrics, pulse will use the "raw" exposure population of the experiment as it is true to the randomization process. For User ID metrics, pulse will use the resolved population depending on the mapping mode. ### Explanation of Methodology 1. Primary IDs are preferred over secondary IDs if present in the data. As the best practice, the unit type of analysis should match that of randomization. Statsig will always prefer the primary ID if present in the data. If the primary ID is not present, Statsig will use the most recent secondary ID. 2. Secondary IDs are only used to join metrics to exposures, but the unit of analysis is still the primary ID. The unit counts of each metric's results are calculated using the primary ID. 3. Many (primary) to one (secondary) mapping is handled through attributing the secondary ID to **ONE** primary ID. 4. One (primary) to many (secondary) mapping is implicitly handled by treating all secondary IDs as the same unit. e.g. The metric value will be added in a sum metric or counted in a count metric. 5. Statsig supports a mixture of primary and secondary IDs in the same experiment. You can use both primary and secondary IDs in the same experiment. For example, when you run a signup experiment, you can measure the session level metrics for the primary ID and the user level metrics for the secondary ID. To do this, Statsig maintains two populations - one for the primary ID and one for the secondary ID. Th primary ID population is the same as if you had only used the primary ID. ## How to Enable ID Resolution in a Statsig Experiment Setting up identity resolution in Statsig is very simple. You can either log or join data to provide both IDs on your assignment source, or provide one ID in the assignment source along with a mapping table between the IDs in the form of an Entity Property Source. ### Using Property Source To use Identity Resolution across experiments in your project, you will need a lookup table that has both the ID you are exposing on and the selected targeted ID. This table can be configured by setting up an Entity Property Source with both IDs present. Once that's done, you can simply select this source when configuring your secondary ID type, and Statsig handles the join for you. ID resolution source configuration interface If you want to use a Statsig SDK to populate this table, you can log an event like a "Signup" event that has both the logged-out identifier and the user ID on the same event. Events sent via the Statsig SDK are written into your warehouse - and you can configure an Identity Resolution source on top of that using something like this - Identity resolution configuration interface ### Using Assignment Source When creating an assignment source, provide a column for both ID types. It is assumed that your 'Primary ID' will be non-null for exposure records. Your secondary ID can be null. If your secondary ID is sparse (some records are null, and some are not due to logging), Statsig will back-attribute any identified secondary ID to other records from the same Primary ID. ID Resolution Assignment Source When you create an analysis-only experiment or power analysis with this ID type, you can optionally select a Secondary ID. If you do so, you can now use metrics from either ID type in your analysis. For E2E experiments that use the Statsig SDK, this is configurable on the experiment setup page, under Advanced settings. Behind the scenes: * For metric sources with the primary ID, metrics will be joined to exposures based on that primary ID * For metric sources with only the secondary ID, metric will be joined to exposures based on that Secondary ID * If using strict mode, users with a duplicate mapping are dropped from analysis. Using first-touch, units use their first exposure record, and merge data from all mapped secondary IDs. This works natively across Metric Sources, so you can easily set up funnel or ratio metrics across the two ID types. Analysis is done using the primary ID - this process associates metric values that are on an associated secondary ID. ### Mapping Changes If a change is made to the entity property source or assignment source's definition or underlying data, that will be reflected on the next reload. This is **why** a full reload is required, since otherwise historical changes to the mapping can lead to inconsistent data on incremental reloads or explore queries. ### Best Practices We strongly recommend using an [Entity Property Source](/statsig-warehouse-native/configuration/entity-properties) to provide a cleaned unit mapping from your warehouse. However, you can also provide mappings on your exposure source by logging multiple identifiers in the exposure data - Statsig will greedily use this to match across identifiers. For both modes, an experiment can currently only have one mapped ID type - e.g. secondary\_id->user\_id, or secondary\_id->account\_id, but not both. All modes will require a full reload, so that there's not data inconsistency due to historical mappings being changed or new mappings introduced. The property source or assignment source used to provide mappings will be filtered to records within the experiment's date range. If a mapping is "evergreen", or not scoped to a specific time period, you can omit the timestamp on the entity property source. #### Example of a supported schema if your assignment source data contains:
    `{stableID: 'unknown_123', exp_id: 'PDP Test', test_group: 'Control'}` and your metric sources contain data that represents a metric as:
    `{userID: 'known_abc', event: 'page_load'}` Your Entity Source or Assignment source must contain the secondary identity (in this case, `userID`) that will enable Statsig to join your assignment data with your metric data:
    `{stableID: 'unknown_123', userID: 'known_abc', country: 'USA'}` ## Considerations Deduplicating records can lead to biased results, so Statsig preforms two extra health checks on this kind of experiment. * Statsig will check your deduplication rate and warn you if it is unusually high. It's expected that some secondary IDs will have multiple logged-out IDs due to users using different devices or clearing browser history * Statsig will perform a chi-squared test evaluating if the deduplication rate is identical across arms of the experiment. In some cases, an experiment may cause more users to come back (for example an email resurrection campaign), in which case duplicates are expected to be more frequent in that arm and can be a positive outcome. In this case, you can perform first-touch attribution to maintain a common identifier Breakdowns of metric dimensions for experiment results is only supported on properties associated with primary ID types. Secondary ID type dimension breakdown for experiment results and custom queries is currently not supported due to high risk of post-exposure data leaking into the group-by's or filters. # ID Resolution (ID Stitching) Source: https://docs.statsig.com/statsig-warehouse-native/features/id-resolution (legacy) Legacy ID resolution behavior in Statsig Warehouse Native for mapping cross-platform IDs and analyzing anonymous user experiments before the November update. At the earliest, Statsig will update its ID resolution methodology to reflect mixed population on November 15th. ## Mapping Modes When using Advanced ID resolution, you can choose between modes: * Strict 1:1 mapping enforces that identities have a singular mapping. If you have a mapping between two IDs that are always 1:1, this mode enforces that the mapping is singular and warns you if there's data where that's to the case. Users with a single identity can use downstream metrics from the secondary identity, and multi-mapped users are considered corrupted and discarded from the analysis * First-touch mapping is for cases where units might have multiple mappings, in either direction. For example, a single user may have multiple "profiles", or someone may have logged into the same account from several devices or web sessions. In this case, units will use the experiment group of their first exposure for analysis, and aggregate metrics from all of their associated secondary IDs. | Strict 1:1 Mapping | First Touch Mapping | | --------------------------------- | ----------------------------------- | | Enforced1to1Mapping | FirstTouchAttribution | ### Strict 1:1 Mapping All potential mappings between identifiers within the experiment date range, on the exposed population, are collected. If the primary ID has multiple secondary IDs, or vice versa, it is considered polluted and dropped from the analysis. ### First Touch Mapping The direction of first-touch mapping will be based on the experiment; all secondary IDs resolve to 1 primary ID, and a single primary ID can have multiple mapped secondary IDs. If your aim is to only have one secondary ID, you can manage that logic inside the entity property source today, but feel free to reach out to support if there's specific logic you would like to request. Data is attributed to the group of the first associated primary ID seen in the exposure. If a secondary ID has multiple associated primary IDs, the group of the first primary ID will be used. Note that this means users that cross groups are not discarded from analysis but instead are assigned based on the first experience they had. Primary ID records that are associated with another Primary ID, but are not the first observed records, are dropped from the analysis. If a user is exposed twice on different primary IDs that resolve to the same secondary IDs, only the primary ID metrics from the first-exposed user will be kept in the analysis. ### Last Touch Mapping Same as first touch but data is attributed to the most recent primary ID. ### Note on ID stitching Multiple secondary IDs attached to one primary ID still count as "one" experimental primary ID; the metric values will be merged across records from the different secondary IDs - e.g. added in a sum metric or counted in a count metric. We are interested in supporting more complex 1-to-many relationships of identities and are eager to partner with customers to develop these capabilities if a more advanced use-case is required. ## How it Works Setting up identity resolution in Statsig is very simple. You can either log or join data to provide both IDs on your assignment source, or provide one ID in the assignment source along with a mapping table between the IDs in the form of an Entity Property Source. ### Using Property Source To use Identity Resolution across experiments in your project, you will need a lookup table that has both the ID you are exposing on and the selected targeted ID. This table can be configured by setting up an Entity Property Source with both IDs present. Once that's done, you can simply select this source when configuring your secondary ID type, and Statsig handles the join for you. ID resolution source configuration interface If you want to use a Statsig SDK to populate this table, you can log an event (like a "Signup" event that has both the logged-out identifier and the user ID on the same event. Events sent via the Statsig SDK are written into your warehouse - and you can configure an Identity Resolution source on top of that using something like this - Identity resolution configuration interface ### Using Assignment Source When creating an assignment source, provide a column for both ID types. It is assumed that your 'Primary ID' will be non-null for exposure records. Your secondary ID can be null. If your secondary ID is sparse (some records are null, and some are not due to logging), Statsig will back-attribute any identified secondary ID to other records from the same Primary ID. ID Resolution Assignment Source When you create an analysis-only experiment or power analysis with this ID type, you can optionally select a Secondary ID. If you do so, you can now use metrics from either ID type in your analysis. For E2E experiments that use the Statsig SDK, this is configurable on the experiment setup page, under Advanced settings. Behind the scenes: * For metric sources with the primary ID, metrics will be joined to exposures based on that primary ID * For metric sources with only the secondary ID, metric will be joined to exposures based on that Secondary ID * If using strict mode, users with a duplicate mapping are dropped from analysis. Using first-touch, units use their first exposure record, and merge data from all mapped secondary IDs. This works natively across Metric Sources, so you can easily set up funnel or ratio metrics across the two ID types. Analysis is done using the primary ID - this process associates metric values that are on an associated secondary ID. ### Mapping Changes If a change is made to the entity property source or assignment source's definition or underlying data, that will be reflected on the next reload. This is **why** a full reload is required, since otherwise historical changes to the mapping can lead to inconsistent data on incremental reloads or explore queries. ### Best Practices We strongly recommend using an [Entity Property Source](/statsig-warehouse-native/configuration/entity-properties) to provide a cleaned unit mapping from your warehouse. However, you can also provide mappings on your exposure source by logging multiple identifiers in the exposure data - Statsig will greedily use this to match across identifiers. For both modes, an experiment can currently only have one mapped ID type - e.g. secondary\_id->user\_id, or secondary\_id->account\_id, but not both. All modes will require a full reload, so that there's not data inconsistency due to historical mappings being changed or new mappings introduced. The property source or assignment source used to provide mappings will be filtered to records within the experiment's date range. If a mapping is "evergreen", or not scoped to a specific time period, you can omit the timestamp on the entity property source. #### Example of a supported schema if your assignment source data contains:
    `{stableID: 'unknown_123', exp_id: 'PDP Test', test_group: 'Control'}` and your metric sources contain data that represents a metric as:
    `{userID: 'known_abc', event: 'page_load'}` Your Entity Source or Assignment source must contain the secondary identity (in this case, `userID`) that will enable Statsig to join your assignment data with your metric data:
    `{stableID: 'unknown_123', userID: 'known_abc', country: 'USA'}` ## Considerations Deduplicating records can lead to biased results, so Statsig preforms two extra health checks on this kind of experiment. * Statsig will check your deduplication rate and warn you if it is unusually high. It's expected that some secondary IDs will have multiple logged-out IDs due to users using different devices or clearing browser history * Statsig will perform a chi-squared test evaluating if the deduplication rate is identical across arms of the experiment. In some cases, an experiment may cause more users to come back (for example an email resurrection campaign), in which case duplicates are expected to be more frequent in that arm and can be a positive outcome. In this case, you can perform first-touch attribution to maintain a common identifier # Incremental Reloads Source: https://docs.statsig.com/statsig-warehouse-native/features/incremental-reloads Configure incremental reloads in Statsig Warehouse Native to process new data efficiently without recomputing the entire experiment dataset each run. Incremental reloads save state from the last load, and load from the latest data read with a small buffer to ensure completeness. This job wipes data since the last load, plus that buffer, and then appends all new data to the staging datasets before calculating results for changed days. This is the recommended way to load active experiments, and is used for ongoing, daily loads -- especially when datasets are large. # Best Practices and Avoiding False Positives Source: https://docs.statsig.com/statsig-warehouse-native/features/interpreting-results/best-practices Best practices for interpreting Statsig Warehouse Native experiment results, including reading lift, avoiding bias, and trusting statistical significance. We have some suggestions on how to interpret Pulse in a scientifically-sound way: 1. Have a hypothesis in mind before viewing Pulse. What are the metric(s) you expect to shift due to the change you made? What else could have happened? What are signs it has gone wrong? 2. Establish a small set of key metrics that are directly related to your hypothesis and would most directly establish that the experiment worked. Having more than a handful of key metrics is usually a sign of an ill-defined hypothesis or shotgun experimentation. Examining too many metrics will lead to a higher false positive rate (seeing results when only statistical noise exists). 3. Avoid cherry-picking results. For example, don't selectively pick three metrics that look good, but ignore the two that don't. Also avoid picking "good" or "bad" numbers that have no connection to your hypothesis. Context matters a lot, and statistically-significant results should have a plausible explanation (false positive can be a plausible explanation). 4. Seeing multiple (independent) effects that are consistent with a plausible story lends credibility that the observed effects are real, even with borderline p-values. 5. Expect to see false positives and be suspicious of statistically significant results with borderline p-values. For example, a 95% confidence interval (5% significance level) is expected to turn up one statistically significant metric out of twenty due purely to random chance. This number goes up if you start to include borderline metrics (eg. p = 0.06). 6. Look beyond your hypothesis. What other effects can you find? Are there tradeoffs? Are there unexpected behaviors? These can reveal information about your users and how they interact with your product. They are often the source of follow-up experiments and new ideas. # Custom "Explore" Queries Source: https://docs.statsig.com/statsig-warehouse-native/features/interpreting-results/custom-queries Run custom queries on Statsig Warehouse Native experiment results to explore segments, joins, and aggregations beyond the built-in scorecard views. Custom queries are a way to run additional custom experiment analyses on your existing data beyond what is in your main Results tab. You may run them to gain deeper insights from your experiments and feature roll-outs, debug interesting results, or scope down your results to interesting sub-groups. Custom queries allow you to filter or group metrics by event or user dimensions, or filter to a specific set of users to see how an experiment or launch has impacted these users' experience. Custom queries are experimental analyses just like in the main Results tab, and all the same statistical procedures apply. Results are computed as p-values and confidence intervals for your metric deltas, and advanced statistical methods like [CUPED](/stats-engine/methodologies/cuped) and [Sequential Testing](/experiments-plus/sequential-testing) can be used. Be careful when drawing your inferences of Custom Queries, especially when grouping by a dimension with lots of options. This can increase your chance of seeing a false-positive statistically significant result. ### Dimension Loading Timing for Precomputed User Dimensions When viewing results for precomputed user dimensions (which are configured and run on a schedule), be aware that these dimensions are loaded through separate asynchronous explore queries. This means: * The main experiment results will appear first * Precomputed dimensions will continue loading in the background and become available within a few minutes * This timing gap is most noticeable immediately after the first reload of the day * If you see "No dimensions available for this time range" for precomputed dimensions, wait a few minutes and refresh to see if dimensions have completed loading This timing behavior only affects precomputed user dimensions that run on a schedule. User-triggered custom queries do not experience this asynchronous loading delay. ### Running a Custom Query To run a Custom Query, navigate to the **Explore** tab within your experiment. Custom query explore tab interface Custom Query fields: * **Metric(s):** The metric(s) you want to analyze. You can select a single metric, a few metrics, or a Metric Tag. Adding a Tag will include all the metrics within that Tag in your Custom Query. There are three "default" metric selections included as shortcuts: * "Scorecard Metrics", all metrics included in your experiment setup's Primary and Secondary Metrics sections * "Primary Metrics" * "Secondary Metrics" * **Metric Filter:** With metrics selected, you can filter metrics by either Event or User dimensions using the "Add Filter" dropdown. For example, if you wanted to look at your experiment results for Canadian users only, you could filter to "Country = CA". Metric filter dropdown selecting Country equals CA * **Group By:** You can group your Custom Query results by either an Event or User dimension. Whereas Custom Query filters can be applied at the *per-metric* level, the Group By action is at the *query* level (so all included metrics will have whatever Group By you select applied to them). * **Time Range for Metric Data:** The date range you're running your analysis on. By default this will be the "Full date range" of your experiment data. * **(Advanced) ID List Segment filters:** You can choose an ID-list based [Segment](/segments), and your results will only be calculated for users who are in that segment. This can be useful if you forgot to log an important user dimension that you want to filter to, or realized that you only care about a sub-population that you've defined in your own data warehouse. * Careful! This option can easily lead to erroneous and biased results. You will need to make sure the segment is defined based on the user's status *before* they were exposed to the experiment or feature gate. * Similarly, you can choose to *exclude* a certain ID list segment, for example if you want to exclude a set of users who have been retroactively identified as bad actors from your lifts analysis. * **(Advanced) Filter by Exposure Date:** You can also filter the results by Exposure Date which can give you more flexibility. You can choose only include or exclude a date range, or in WHN, you can additionally include/exclude users based on when they were first exposed to the experiment. * This is useful when your metrics have novelty effect, delayed impact, or specific scenarios where you only want to filter your results to certain users. Use it cautiously because it can lead to biased results. User groups in experiment results are based off of first-touch attribution. The filters and grouping applied will be based on the user attributes collected at the time of first exposure in the gate/experiment/layer check. Custom query definition form showing selected metrics and filters ### Viewing a Custom Query in Explore These queries take a few minutes to run (don't worry, we'll send you an email once your results are ready in case you want to hop to another task), but once complete the results will be visible in the **Query History** section of the **Explore** interface. All historical queries (across your team) will be stored here. You can also give your query a display name inline for easier future identification. Explore tab query history list ### Scheduling a Custom Query If you want a daily refresh of a given Custom Query, you can schedule your Custom Query directly from the **Explore** tab. To do this, author the Custom Query you wish to schedule, then tap the "..." menu, then **Schedule**. This Custom Query will now run daily and live in the **Scheduled** tab of your Metric Lifts. Scheduled custom queries tab displaying daily runs Scheduled query configuration interface ### Reviewing Custom Query Results Custom query results look a lot like the main Results tab, because the statistical methods are the same. Statsig uses the same experimental analyses practices on your custom analysis as we do on your main Results. One main point of difference, however, is that your custom query result is a snapshot in time. Once run, the analyses results are saved and will not update if more metric data is collected. If you do want to update your results, you can run a new custom query or schedule custom queries to run at a regular cadence. #### Sequential Testing and Custom Queries If [Sequential Testing](/experiments-plus/sequential-testing) is enabled for your experiment, it can be applied to your custom query results as well. How much and whether to adjust your confidence intervals and p-values will depend on the regular rules of sequential testing: if your custom query doesn't satisfy the experiment's target Days or Unique Exposures from your setup, sequential testing adjustments will be made to your results to account for the under-powered state of the experiment. Since custom queries are computed as a snapshot in time, sequential testing adjustments are computed for that specific analysis only. If you run additional custom queries with more or less data (e.g. more days in the analysis, more unique users in the experiment), the sequential testing adjustments will change accordingly. Some custom queries may have no sequential adjustments applied at all if they meet the configured minimum Days or Unique Exposures. Custom query results table displaying sequential testing adjustments # Pulse FAQs Source: https://docs.statsig.com/statsig-warehouse-native/features/interpreting-results/faq Frequently asked questions about interpreting Statsig Warehouse Native experiment results, including p-values, intervals, exposures, and SRM warnings. Interpreting statistical results can be tricky, and often people will have similar questions as we ramp up. Here's some answers! ## I had a stat sig result, but it turned negative. How should I interpret this? In general, you should trust the current result, as it's incorporating more information about the users in your experiment. There's a number of reasons this can happen: * Random noise, which gets diluted as your sample size gets larger * Within-week seasonality (e.g. an effect is different on Mondays), which gets normalized with more data * The population that saw the experiment early early on is somehow different than the slower adopters. This happens frequently - a daily user will likely see your experiment before someone who users your product once a month. You can look at the time series view to get more insight on this * There was some sort of novelty effect that made the experiment meaningful early on, but fall off. Imagine changing a button - people might click on it early out of curiosity or novelty, but once that effect goes away they'll behave like before. You can use the days-since-exposure view to get more insight on this Best practice for timing is to pick a readout date when you launch your experiment (based on a [power analysis](/experiments-plus/power-analysis)), and to disregard the statistical interpretation of results until then. This is because reading results multiple times before then dramatically increases the rate at which you'll get false positives. ## How should I start with interpreting results? Start by using your scorecard metrics to understand if you've moved the metrics you thought you would. You should come into pulse with a hypothesis on what your experiment should drive, and that hypothesis should be answered by your primary metrics. The delta displayed is based on the observed difference between a test and control population. The error bars are a visualization of a confidence interval. A confidence interval is a range of probable values for the difference between groups. A future sample's 95% confidence interval will have a 95% chance to contain the true value of the difference. This is a bit of a wonkish distinction which means that (unfortunately) you can't say that your observed 95% CI has a 95% chance of containing the true value. In practice, the CI is a representative range of what the true value might be. Keep in mind that these results are statistical interpretations and not facts: * If a result is not stat sig, this means you don't have sufficient evidence to reject the null hypothesis (i.e., based on your experiment design the observed result is reasonably likely to have happened by chance). * Generally, you should treat these results as a lack of evidence for your hypothesis * Underpowered tests may lead to neutral results even if a true effect exists * If a result is stat sig, this means that you have sufficient evidence to reject the null hypothesis (i.e., the probability that you would observe this result, or one more extreme, if the two groups' results were identical is below the pre-determined threshold you set). * Generally, you should treat this result as evidence for your hypothesis * Multiple comparisons (many metrics, rerunning an experiment, or grouping by dimensions) greatly increase the chance of seeing a stat sig result when there's *not* a true effect. Be wary of interpreting results when you see those behaviors! * A test that was extremely unlikely to succeed (moonshots, etc.) that has stat sig results have a high chance of that result being a false positive. In cases like this, it's a strong signal but you should consider trying to reproduce the result or running a back-test, or consider reducing your significance level. Once you've looked at the results on your scorecard, we encourage you to use the all-metrics tab and custom queries to get more information on your experiment, but you shouldn't necessarily trust that if those see a stat-sig movement it is a statistically sound interpretation; as mentioned above if you increase the number of metrics you are looking at, you increase the chance that you will see a false positive. Use this section to look for unexpected large regressions and to generate follow-up hypotheses. ## Results aren't showing up for some metrics This normally happens when your company is both using the SDK or event imports, and also importing precomputed metrics from your data warehouse. Since these can run at different times, the data availability may differ. You can adjust your analysis date range to get a full view of your data. ## Our external source shows more exposure events than Statsig. Are data missing? Exposures on the last day (the day you made a decision) are not counted as exposures. Please filter out that day when you analyze your external data. The exact hours which define a "day" for your project depend on which timezone your project is assigned. ## We log categorical metadata for a custom event, but Pulse doesn't show these breakouts. What's wrong? Pulse is only able to show experimental results for various sub-groups of your metric (e.g. iOS vs. Android) when you've configured your metadata as a Dimension. [Value Dimensions](/pulse/read-pulse#value-dimensions) are the most common dimension type as their metadata get logged directly with your custom events. However, value dimensions must be defined in your [custom event setup](/metrics/metric-dimensions). ## Why do I see "No dimensions available for this time range"? You may see this error when trying to view precomputed user dimensions, particularly after the first reload of the day. This happens because: * Dimensions load asynchronously in separate explore queries after the main scorecard results load * The main experiment results will appear first, while dimensions continue loading in the background * Typically, dimensions will become available within a few minutes after the main scorecard loads If you encounter this error, wait a few minutes and refresh the page to see if the dimensions have completed loading. # Metric Drill-Down Source: https://docs.statsig.com/statsig-warehouse-native/features/interpreting-results/metric-drill-down Drill down into a Statsig Warehouse Native metric to inspect contributions by segment, time period, and user property for deeper experiment analysis. ## Tooltip Overview A tooltip with key statistics and deeper information is shown if you hover over a metric in Pulse. UI for metric hover card in experiments * **Group**: The name of the group of users. For Feature Gates, the "Pass" group is considered the test group while the "Fail" group is the control. In Experiments, these will be the variant names. * **Units**: The number of distinct units included in the metric. E.g.: Distinct users for user\_id experiments, devices for stable\_id experiments, etc. * **Mean**: The average per-unit value of the metric for each group. * **Total**: The total metric value across all units in the group, over the time period of the analysis. ### Calculation details | Metric Type | Total Calculation | Mean | Units | | ----------------------------------------- | ------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------- | | event\_count | Sum of events (99.9% winsorization) | Average events per user (99.9% winsorization) | All users | | event\_user | Sum of event DAU (distinct user-day pairs) | Average event\_dau value per user per day. Note that we call this "Event Participation Rate" as this can be interpreted as the probability a user is DAU for that event. | All users | | ratio | Overall ratio: sum(numerator values)/sum(denominator values) | Overall ratio | Participating users | | sum | Total sum of values (99.9% winsorization) | Average value per user (99.9% winsorization) | All users | | mean | Overall mean value | Overall mean value | Participating users | | user: dau | sum of daily active users | Average metric value per user per day. The probability that a user is DAU | All users | | user: wau, mau\_28day | Not shown | Average metric value per user per day. The probability that a user is xAU | All users | | user: new\_dau, new\_wau, new\_mau\_28day | Count of distinct users that are new xAU at some point in the experiment | Fraction of users that are new xAU | All users | | user: retention metrics | Overall average retention rate | Overall average retention rate | Participating users | | user: L7, L14, L28 | Not shown | Average L-ness value per user per day | All users | ### p-Value In Null Hypothesis Significance Tests, the p-value is the probability that such an extreme difference can arise by random chance when the experiment or test actually has no effect. A low p-value implies the observed difference is unlikely due to random chance. In hypothesis testing, a p-value threshold is used to determine which results are due to a real effect and which are plausibly due to random chance. ([p-value calculation](/stats-engine/p-value)) ### Reverse Power Reverse power is the smallest effect size that an experiment can reliably detect in its current state (some studies refer to this value as ex-post MDE). It is calculated from the sample size and standard error from the control group. Importantly, reverse power does *not* depend on the observed effect size. In practice, reverse power answers such questions like: given how the test actually played out, what is the smallest effect we have sufficient power (typically 80%) to detect? For a two-sided test, the reverse power for a given metric X is computed using the following equation: $$ Reverse Power = \frac{(Z_{1-\beta} + Z_{1-\alpha/2})}{\overline{X}_{\text{control}}}\times \sqrt{\frac{\mathrm{var}(\Delta \overline{X})}{N_{\text{control}}}} \times 100\% $$ For a one-sided test, the reverse power for a given metric X is computed using the following equation: $$ Reverse Power = \frac{(Z_{1-\beta} + Z_{1-\alpha})}{\overline{X}_{\text{control}}}\times \sqrt{\frac{\mathrm{var}(\Delta \overline{X})}{N_{\text{control}}}} \times 100\% $$ * $\overline{X}_{\text{control}}$ is the mean metric value across control users * $var(Δ\overline{X})$ is the population variance of delta * $N_{\text{control}}$ are the observed number of units in the control group * $Z_{1-\beta}$ is the standard Z-score for the selected power. Typically ${1-\beta}$ = 0.8 and $Z_{1-\beta}$ = 0.84 * $Z_{1-\alpha/2}$ and $Z_{1-\alpha}$ are the standard Z-scores for the selected significance level in a two-sided test and in a one-sided test. You can enable reverse power as an optional feature. To manage it, go to Settings -> Product Configuration -> Experimentation -> Organization, where you can toggle it on or off. ## Detailed View Click on **View Details** to access in depth metric information. The detailed view contains three sections: * **Time Series**: How the metrics evolve over time * **Raw Date**: Group level statistics * **Impact**: How the experiment impacts the metric ### Time Series In this view, select and drag as needed to zoom-in on different time ranges. Three types of time series are available in the drop-down: **Daily**: The metric impact on each calendar day without aggregating days together. This is useful for assessing the variability of the metric day-over-day and the impact of specific events. It's the recommended time series view for Holdouts, since it highlights the impact over time as new features are launched. Daily metric impact visualization interface **Cumulative**: Shows the cumulative metric impact from the start of experiment over time. This is a good way to observe trends and see how your confidence interval changes over time. Cumulative metric lift visualization interface **Days Since Exposure**: Shows the metric impact based on how long a user has been in the experiment. Daily data for each user is aligned by the day they entered the experiment (Day 0, Day 1, ...etc), not by calendar date. This allows you to distinguish early (novelty) from long-term effects. This view also shows pre-experiment data which identifies biases between groups before the experiment started. This can happen due to random chance or by some issue in the random assignment process. Days since exposure metric visualization interface ### Raw Data This view shows the group level statistics needed to compute the metric deltas and confidence interval. Includes Units, Mean, and Total (explained above), as well as the Standard Error of the mean (Std Err). Details on the statistical calculations are available [here](/stats-engine). ### Impact Experiment impact metrics interface * **Experiment Delta (absolute)**: The absolute difference of the Mean between test groups i.e. Test Mean - Control Mean. The p-value is shown to indicate whether the observed absolute difference is statistically significant. * **Experiment Delta (relative)**: Relative difference of the Mean i.e. 100% x (Test Mean – Control Mean) / Control Mean. * **Topline Impact**: The measured effect that experiment is having on the overall topline metric each day, on average. Computed on a daily basis and averaged across days in the analysis window. The absolute value is the net daily increase or decrease in the metric, while the relative value is the daily percentage change. * **Projected Launch Impact**: An estimate of the daily topline impact we expect to see if a decision is made and the test group is launched to all users. This takes into account the layer allocation and the size of the test group. Assumes the targeting gate (if there is one) remains the same after launch. See [here](/stats-engine/topline-impact) for details on the exact calculation for topline and projected impact. **FAQs about topline impact** *Why is the projected launch impact smaller than the relative experiment delta?* Often times, an experiment can impact only a subset of the user base that contributes to a topline metric. So the relative experiment delta that we observe is effectively diluted when measured against the topline metric value. For example: Consider a top-of-funnel experiment on the registration page. Among users that hit this page, the treatment is leading to more sign ups and a 10% lift in daily active users (DAU). However, our topline DAU metric includes other user segments outside of the experiment, such as long term users that don't go to the registration page. So what was a 10% lift in the test vs. control comparison, may amount to only a 1% increase in overall DAU. *How can the topline impact be higher than the experiment delta?* It's possible for the topline impact to be higher or lower than the experiment delta. This is because the two values are computed differently and have different meaning. Experiment deltas are based on the unit-level averages: The mean value of the metric is computed for each user across all days, and then averaged to obtain the group mean. The topline impact is computed daily based on the total pooled effect from all users, and we take the average across days to show the daily impact. We chose to compute topline impacts in this way because most metrics are tracked on a daily basis and the topline value tends to be computed as an aggregation across all users, rather than a user-level average. For experiment analysis, on the other hand, best practice is for the analysis unit to match the randomization unit, so metrics are aggregated at the unit level first before computing experiment deltas. # Participating Units Source: https://docs.statsig.com/statsig-warehouse-native/features/interpreting-results/participating-units Understand how Statsig Warehouse Native counts participating units, including exposure rules and how to interpret unit counts in experiment results. ## Definition In Statsig, Participating Units (sometimes referred to as participating users) are a subset of an experiment's total exposed unit count relevant to a particular metric. They count the units with a non-zero denominator value for metric types that require a numerator and denominator value. These units are the ones actually used in statistical calculations for ratio-based metrics. Participating units diagram showing subset of total exposed users Ratio metrics are computed only for users that have a non-zero value in the denominator, e.g. the user must have triggered the denominator event on a given day to be included in the daily ratio. Users that don't trigger the denominator event during an experiment are not included in the test vs. control comparison of a ratio metric. ## Example An example of this type of metric for an e-commerce company could be "Total Items Purchased per Order", measuring the average basket size. The numerator would be defined as the count of total items bought, and the denominator would be the total number of orders submitted. If an experiment were run on the checkout flow, only users who actually had at least one Order event would be included in the "Total Items Purchased per Order" ratio metric. # Read Results Source: https://docs.statsig.com/statsig-warehouse-native/features/interpreting-results/read-results Read and interpret Statsig Warehouse Native experiment results, including scorecards, primary metrics, lift, intervals, and significance indicators. ## Read Experiment Results To read the results of your experiment, go to the **Results** tab, where you will see your experiment hypothesis, **exposures**, and **Scorecard**. ### Exposures Exposures chart showing cumulative users per experiment group At the top of the Results page is the Exposures Chart. Exposures are the unique experimental units enrolled in the experiment. This is typically the number of unique users, and for device-level experimentation, this is the number of devices. The timeline shows you when the experiment was started, and how many exposures were enrolled on any given day. You can see the rate at which users were added into each group of the experiment, how many total users were exposed, and confirm the target ratio matches what you configured in experiment setup. ### Scorecard The experiment **Scorecard** shows the metric lifts for all Primary and Secondary metrics you set up at experiment creation. #### Immediately Post-experiment Start For up to the first 24 hours after starting your experiment (before our daily metric results run), the **Scorecard** section is calculated hourly (this only applies to Statsig Cloud, for WHN projects you will need to reload results on demand or set up a daily schedule). This more real-time scorecard is designed to enable you to confirm that exposures and metrics are being calculated as expected and debug your experiment or gate setup if needed. You should **not** make any experiment decisions based on real-time results data in this first 24 hour window after experiment start. Experiments should only be called once the experiment has hit target duration, as set by your primary metric(s) hitting experimental power. Read more about target duration [here](/experiments-plus/create-new#target-duration). Given data during this early post-experiment start window is designed for diagnostic, not decision-making purposes, you will notice a few key differences between this real-time view and the results that will start showing after daily runs have initiated: * Metric lifts do not have confidence intervals * No time-series view of metric trends * No projected topline impact analysis * No option to apply more advanced statistical tactics, such as CUPED or Sequential Testing All of these are available in daily Results, which will start showing in the next daily run. #### Post-first Day Scorecard Experiment scorecard table displaying metric lifts and confidence intervals The experiment Results daily run calculates the difference between the comparable randomization groups (eg. test and control) across your company's suite of metrics, and applies a statistical test to the results. You can read more about Statsig's stats engine [here](/stats-engine). For every metric, we will show you: * The calculated relative difference (Delta %) * The confidence interval * Whether the result is statistically significant * Positive lifts are green * Negative lifts are red * Non-significant results are grey The formula for calculating lift is: Delta(%) = (Test - Control) / Control Confidence intervals are reported at the selected significance level (95% by default). In a typical two-sided Z-test, we show the confidence interval as +/- 1.96 \* standard error. 99.9% winsorization is automatically applied to event\_count, event\_count\_custom, and sum metrics. This caps extreme outlier values to reduce their impact on experiment results. For metrics added to the **Scorecard** or **Monitoring Metrics** sections of your experiment or gate, you can also apply other optional statistical treatments, such as CUPED (pre-experiment bias reduction) and sequential testing adapted confidence intervals. Read more [here](/stats-engine). * **Experiment results are computed for the first 90 days**: By default, Statsig will compute experiment results for your experiment for only the first 90 days of your experiment. You will be notified via e-mail as you approach the 90 days cap, at which point will be able to extend this compute window for another 30 days at a time. If the experiment runs beyond the compute window, new users will stop getting added into the experiment's result, but analysis for existing users who have been exposed to the experiment will continue to run even if the compute window is not extended, until you make a decision on the experiment. This experiment result calculation window only affects whether a user is included in the experiment's analysis, and does not affect the treatment each user would receive. New users would still receive the experience for the group they get randomized into. ### Experiment Results Views There are a few different views to see your Scorecard metric lifts, namely: * **Cumulative results (default view)**: Displays the aggregate difference between experiment groups and visualizes the corresponding confidence intervals * **Table view**: Displays the same data as the cumulative view but in a table format with additional fields * **Daily results**: Shows the difference between experiment groups aggregated based on days since start of experiment * **Days since exposure**: Shows the difference between experiment groups aggregated based on days since exposure to the experiment Cumulative results includes a detailed view on hover, where you can additionally view the raw statistics used in the metric lift calculations, as well as topline impact. Cumulative results view with hover details ### Dimensions There are two ways in which we can breakdown a given Scorecard metric - one is by a **User Dimension**, the other is by an **Event Dimension**. #### User Dimensions User Dimensions refer to user level attributes that are either part of the user object you log, or additional metadata that Statsig extracts. Examples of these user attributes could be operating system, country, and region. You can create [custom "explore" queries](/pulse/custom-queries) to *filter on* or *group by* available user dimensions. For example, you could "See results for users in the US", or "See results for users using iOS, grouped by their country". Go to the "explore" tab to draft a custom query custom queries #### Event dimensions Events Dimensions refer to the value or metadata logged as part of a custom event that is used to define the metric. If you want to see results for a metric broken down by categories that are specific to that metric, [specify the dimension](/metrics/metric-dimensions) you want to break down by in the **value** or **metadata** attributes when you log the source event. For example, when you log a "click" event on your web or mobile application, you may also log the target category using the **value** attribute as shown below. Statsig will automatically generate results for each category in addition to the top level metric. To see breakdowns for all categories within a metric, click on the (+) sign next to the metric. dimension button dimension results view ### Significance Level Settings These settings can be adjusted at any time to view Scorecard results with different significance levels. * **Apply Benjamini-Hochberg Procedure per Variant**: Select this option to apply the procedure to reduce the probability of false positives by adjusting the significance level for multiple comparisons - [read more here](/stats-engine/methodologies/benjamini-hochberg-procedure). * **Confidence Interval**: Changes the confidence interval displayed with the metric deltas. Choose lower confidence intervals (e.g.: 80%) when there's higher tolerance for false positives and fast iteration with directional results is preferred over longer/larger experiments with increased certainty. * **CUPED**: Toggle CUPED on/ off via the inline settings above the metric lifts. Note that this setting can only be toggled for **Scorecard** metrics, as CUPED is not applied to non-Scorecard metrics. * **Sequential Testing**: Applies a correction to the calculate p-values and confidence intervals to reduce false positive rates when evaluating results before the target completion date of the experiment. This helps mitigate the increased false positive rate associated with the "peeking problem". Toggle Sequential Testing on/ off via the inline settings above the metric lifts. Note that this setting is available only for experiments with a set target duration. analysis settings ### Restarting Results Restart results banner If your experiment has stopped computing results, you can resume updates by clicking the Restart button. There are some important facts to be aware of: * A Restart is not a [Reset](/experiments/ending/ending-experiment#stopping-an-experiment) of your experiment. A Restart will not re-salt (i.e. re-randomize) units in your experiment, and all users will continue to receive the same group assignments. * Statsig will begin computing experiment results anew from the restart point, so your metric results will start over. Old results may still be available in timeseries and explore query views, but they will not be carried forward or updated. * Your Cumulative Exposures chart will update based on new exposures, but the duration of the pause in computations will affect if the chart starts over from zero or your exposure count includes past exposures. It's best to avoid having to Restart Results by actively extending experiments while they're running. Be sure to look out for email alerts from Statsig and check in your experiments regularly. # Meta-Analysis Source: https://docs.statsig.com/statsig-warehouse-native/features/meta-analysis Combine results from multiple Statsig Warehouse Native experiments into a meta-analysis to evaluate the overall impact of a series of related A/B tests. ## The Concept As teams run a number of experiments, it is possible to glean learning across these experiments. This is meta-analysis. Examples of learning people seek to derive include * How hard is a metric to move * Are there more sensitive proxies for the metric we care about? * How are teams doing relative to each other? We've worked with multiple companies to get them to thousands of trustworthy experiments a year. Our inspiration here was looking at what they were trying to learn across these tests. We've built this to be useful whether you're running 50 experiments a year or 5000. Feel free to reach out to help influence our roadmap in [Slack](https://statsig.com/slack). ## Experiment Timeline View This view lets you to filter down to experiments a team has run. At a glance you can answer questions like 1. What experiments are running now? 2. When are they expected to end? 3. What % of experiments ship Control vs Test? 4. What is the typical duration? 5. Do experiments run for their planned duration - or much longer or shorter? 6. Do experiments impact key business metrics - or only shallow or team level metrics? 7. How much do they impact key business metrics? Experiment timeline view dashboard ## Metric Impact (Batting Average) The "batting average" view lets you look at how easy or hard a metric is to move. You can filter to a set of shipped experiments and see how many experiments moved a metric by 1% vs 10%. Like with other meta-analysis views, you can filter down to a team, a tag or even if results were statistically significant. Common ways to use this include * Sniff testing whether the claim that the next experiment will move this metric by 15% is a good idea. * Establishing reasonable goals, based on past ability to move this metric Metric batting average analysis chart ## Metric Correlation View This view lets you visualize two metrics (each data point is an experiment) and visually inspect them for correlation. Often the metric you want to move isn't very sensitive and takes a while to measure. It is helpful to find metrics that are more sensitive and faster to measure - and run experiments on this. This view lets you plot two metrics on the same chart - each data point is an experiment's impact on them. You can quickly get a sense for whether the metrics tend to move together - or not. You can also remove outliers, filter down to a team's experiments or download the underlying dataset. In this hypothetical example - "Checkouts" is the metric you want to move, but it's not very sensitive. "AddToCart" correlates well with "Checkouts", while "ViewItemDetail" doesn't. Metric correlation scatter plot Metric correlation analysis interface ## Metric Insights This view lets you pick a metric and see all experiments and feature rollouts that impact this metric. [Learn more](/aggregated-impact). 417923655-430563dc-4794-4d69-a314-36c76a6fcf74 ## Knowledge Bank The KB acts as a searchable repository of experiment learning across teams. It helps you find shipped, healthy experiments and gain context on past effort and generate ideas on new things to try. Make it easy for new team mates to explore and find experiments a team ran, or where a topic was mentioned. Our meta-analysis tools offer more structured means to discover and look across your experiment corpus, but when you do want free text search, this exists. Knowledge bank search interface # Metric Reloads Source: https://docs.statsig.com/statsig-warehouse-native/features/metric-reloads Configure metric reloads in Statsig Warehouse Native to refresh metric calculations when definitions change without recomputing every experiment. Metric reloads drop all data from staging pipelines associated with a metric, and restate that data from scratch. Where this data is interconnected (e.g. ratios, funnels, etc.), related entities will be updated as well. This is a big time saver in cases where a new metric needs to be added to an experiment, or a metric definition has changed, since you can avoid reloading N unrelated experiment metrics. # MEX on Warehouse Native [Beta] Source: https://docs.statsig.com/statsig-warehouse-native/features/mex-on-warehouse-native Use Metrics Explorer (MEX) on Statsig Warehouse Native to slice metric data by dimensions, time period, and user properties for ad hoc analysis. [Metrics Explorer](/product-analytics/overview) (or MEX) is Statsig's Analytics solution. With Warehouse Native, this integrates directly on top of your tables to provide easy analysis on the same datasets you're using for experimentation. Metrics Explorer in WHN is currently in beta, please contact the team to get it enabled in your account. ## Functionality ### [Drill-Down and Filter](/product-analytics/overview) Filter and group by fields to calculate measures as timeseries, bar-charts, and more. ### [Funnel Analysis](/product-analytics/funnels) Run powerful and complete funnel analysis to understand how users are moving to your product. ### [Group By Experiment](/product-analytics/drilldown#drilling-down) View arbitrary timeseries and rollups grouped by experiment membership, allowing more observational analysis of what happened during your experiment. ### [Retention Chart](/product-analytics/drilldown#retention) Analyze user retention reports to understand how effectively your product or service maintains user interest/engagement over time. ### [Distribution Chart](/product-analytics/drilldown#distribution) View distribution charts to help you visualize the range of user experiences across your product. ### Save, Share, and View SQL Save analyses and share them with coworkers. Viewing the SQL gives you easy access to the queries Statsig ran under the hood, and can be an easy launching-off point for deeper analysis. ## Use it with Replays MEX naturally dovetails with [Session Replay](/session-replay/overview) to make it easier than ever to understand what's driving the user behavior you measure in your experiments. # Other Features Source: https://docs.statsig.com/statsig-warehouse-native/features/other-useful-features Reference for additional useful features in Statsig Warehouse Native, including utility settings, debugging tools, and administrative configuration options. Many features are not listed within the Warehouse Native section since they function near-identically between Cloud and Warehouse Native. These are worth exploring as you integrate with Statsig: * [Feature Gates](/feature-flags/overview) are a best-in-class tool for managing feature release and targeting. Feature gates with partial rollouts can automatically be analyzed as experiments in Statsig Warehouse Native * [Session Replay](/session-replay/overview) is a powerful tool for viewing how users interact with your product, giving richer context to changes in user behavior * [Layers](/layers) allow you to create mutually-exclusive experiments for experiments where changes may collide with other experiments to create a bug or odd result * [Dynamic Configs](/dynamic-config/working-with) provide server-side configuration tools with optional targeting and integrates naturally with experiments. * [Admin](/access-management/introduction) tools allow you to control projects, teams, and user access across Statsig # Power Analysis Source: https://docs.statsig.com/statsig-warehouse-native/features/power-analysis Use power analysis in Statsig Warehouse Native to determine experiment duration with confidence, based on baseline metrics, MDE, and traffic volume. In general it is a good practice to establish a run-time for an experiment * This aligns teams around timelines * For fixed-horizon analysis, this establishes when it is valid to make a conclusion based off of the results The way to do this is by establishing the size of impact you want to be able to consistently measure, if it exists (known as the "MDE" or minimum detectable effect) and run an analysis to determine how many samples - and time - you will need to achieve that MDE. ### Running a Power Analysis To run a power analysis in Statsig, you need to provide 2 inputs: * A population * This step is critical as most experiments will only reach a subset of users, and these users may have different behavior than the overall population * You can base the population on an experiment you already ran, or on a Qualifying Event * A qualifying event is an arbitrary set of historical user-timestamp pairs - for example, if you plan to expose on a button click, you could provide the users who clicked that button in the week before * Metrics * Enter the metrics you plan to use as your evaluation criteria for your experiment. You can add multiple metrics, which can be a useful way to analyze which metrics will be more or less sensitive in your target population Power Analysis UI ### Readout Statsig will simulate an experiment based on your input, calculating population sizes and the relative variance based on historical behavior. The power analysis readout will show you a week by week view of the experiment Stats you can expect to see. In the settings section, you can specify * **Number of Experiment Groups**: The total number of groups in the experiment, including control. * **Control Group %**: What percent of users will be in the control group, e.g. 50% if half of all users will be control. * **Fixed Allocation or Fixed MDE Analysis**: Different types of analyses you want to run. See [Analysis Types](/experiments-plus/power-analysis/#analysis-types) for more details. * **One-sided or Two-sided test**: Toggle this setting to select the type of z-test to use for the analysis. * **Significant Level (α)** * **Power (1-β)** * **Bonferroni Correction Per Variant**: Whether to include α correction for multiple tests in power analysis These will update your results based on the analysis that already ran. # Experiment Quality Score Source: https://docs.statsig.com/statsig-warehouse-native/features/quality-score Assess and improve the quality and trustworthiness of Statsig Warehouse Native experiments using the quality score across allocation and metric health. ## Introduction The Experiment Quality Score is a metric designed to get a quick understanding of the quality - and trustworthiness - of an experiment configured within Statsig. This helps experimenters and their peers across an organization quickly identify potential issues in experiment setup, execution, and data collection, ensuring more confident decision-making. Measuring this score over all experiments can help teams to discover systematic issues in their program, and point out opportunities to mature their experimentation program over time. ## Configuration Experimentation quality score can be enabled in the project settings under Settings -> Experimentation -> Experiment Quality Score. There is a list of pre-defined assessment criteria used. The weights of each criteria can be customized based on an organization's needs, though Statsig provides default values. Experiment quality score configuration interface ## Advanced Configuration For more sophisticated customers there may be additional checks desired, different product teams with different needs, or it may be the case that the default thresholds aren't correct for their use case. For example, maybe hypotheses should be at least 200 characters, AND contain a link to an external planning doc. This can be managed easily through the Statsig console API. By running a POST or PATCH on the `console/v1/experiments` endpoint, one can update the individual scores on any given experiment. Targeting the existing set of scores allows overriding weights (usually to 0), meaning the list can be just the custom set desired. For example, running patch on an experiment with this payload: ``` { "manualQualityScores": [ { "criteriaName": "HYPOTHESIS_LENGTH", "criteriaDescription": "Check passed", "status": "PASSED", "score": 0, "weight": 0 }, { "criteriaName": "MyCompany\'s Hypothesis Check", "criteriaDescription": "Has Internal URL and > 200 Chars", "status": "PASSED", "score": 100, "weight": 100 }, { "criteriaName": "Naming", "criteriaDescription": "Experiment prefixed with team name", "status": "FAILED", "score": 0, "weight": 100 } ] } ``` Would: * Drop the original HYPOTHESIS\_LENGTH check * Keep the other original checks, with their weights * Add a new check, `MyCompany's Hypothesis Check`, for custom logic on the hypothesis * Add a new check, `Naming`, for custom logic on the name Note that the other weights would be normalized. If the original HYPOTHESIS\_LENGTH had a weight of 10, the total weight would now be 290 and scores normalized accordingly. If all of the non-custom checks were passing, the score would be 190/290 or \~66% The general flow for using this approach is to: * Use Console API's `experiments/get` to pull all experiments * For Each Experiment * Run custom logic * Patch results ## Calculation Notes Checks which are in an unready state will be skipped during evaluation, and the other weights will be renormalized to 100%. For example, if the experiment has not started, the `Balanced Exposures` component will be in an unready state and ignored. Checks with a weight of 0 will be omitted entirely from the card. ## Viewing Quality Scores When enabled, Quality Scores will show up in the details tab of an experiment. Applicable checks will be evaluated and contribute to the number shown. The score will be color-coded based on the % threshold it is at. * \>= 85% corresponds to passing/green * \>= 50% corresponds to warning/yellow * \< 50% corresponds to error/red. Experiment quality score display with color-coded status Quality scores are also available through the console API. This allows bulk scrapes of the data for easy analysis. # Loading Pulse Source: https://docs.statsig.com/statsig-warehouse-native/features/reloads Overview of reload options in Statsig Warehouse Native, including incremental, full, and metric-only reloads to refresh experiment data on schedule. ## The Pulse Engine Statsig's experimentation engine runs the necessary setup, diagnostics, and transformations required to generate the data points that power statistical experiment analysis. In Warehouse Native, this consists of a series of query jobs (referred to as a DAG, or Directed Acyclic Graph) that take data from your sources and metric configurations to a final result set. Controls for reloads are available on the pulse results page: Pulse Load Controls ### Types of Reloads Statsig offers a number of ways to reload data: * [Full Reloads](/statsig-warehouse-native/features/full-reloads) completely restate your experiment data. This can be useful if your underlying data changes a lot (e.g. a full DBT reload) day-to-day, and you want to ensure your analysis matches your raw data * [Incremental Reloads](/statsig-warehouse-native/features/incremental-reloads) catch your data up from where it was the last load to where it is today. Running daily incremental reloads is the recommended way to keep your data current without using unnecessary compute resources to recalculate data that hasn't changed. * [Metric Reloads](/statsig-warehouse-native/features/metric-reloads) are a useful feature for when you want to add a metric to an analysis, or when a metric definition has changed. This does an efficient spot replacement of the data for a single metric or set of metrics. Full and incremental loads can be [scheduled](/statsig-warehouse-native/connecting-your-warehouse/scheduled-reloads/) so that you have fresh results each day. ### Transparency For every load, Statsig logs the compute time and jobs, cost, and queries associated with the reload. This is highly visible in your console, and provides a powerful tool for understanding what's taking time or delaying results. ### Efficient Reloads Statsig borrows heavily from its cloud roots to optimize the queries running in your warehouse - generally, in head-to-head evaluations customers report that we use significantly less resources than comparable platforms. To add to this, Statsig offers turbo mode, which skips some enrichment calculations (in particular some time series rollups) in order to very cheaply compute the latest snapshot of your data. With this tool, customers have run experiments on 150+ million users in less than 5 minutes on a snowflake S cluster. ### Cleaning Up Storage Statsig will automatically clean up after itself for explore datasets, power analyses, and stratification artifacts. Once you make a decision on an experiment, you can choose whether or not to delete the staging datasets and/or the result datasets; you can always come back to the experiment to clean up from the experiment menu as well. # Reports Source: https://docs.statsig.com/statsig-warehouse-native/features/reports Create and share reports in Statsig Warehouse Native to summarize experiment results, metric trends, and program-level insights with stakeholders. ## Report Summary Summaries are how experiment results are combined with context and discussion to produce long-term records of what you learned through running the experiment. This lives on the Statsig console, but can be exported as a PDF to your learnings library. Experiment report summary interface This will capture results and in-context discussion, as well as allowing you to embed media, charts,and rich text formatting for deeper discussion and sharing of follow-ups from outside of Statsig. Report editor with rich text content Embedded charts and discussions in report Report export preview showing PDF layout # Roles Source: https://docs.statsig.com/statsig-warehouse-native/features/roles-and-access Configure roles and access controls in Statsig Warehouse Native to manage who can view, edit, and approve experiments, metrics, and pipeline settings. Statsig has custom, configurable RBAC (role-based-access-control) across console. This is especially useful for warehouse native customers, as you can control who can update which metrics as well as visibility into raw data. # WHN Statistics Overview Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics Overview of the statistical methods Statsig Warehouse Native uses to analyze experiments, including testing, intervals, variance reduction, and corrections. Statsig's statistics engine is designed to deliver the best analysis of experiment results in the market. We have four core values around our Stats Engine: Transparency, Trust, Flexibility, and Power Details of the Pulse Statistics Engine are in the pages below and offers a comprehensive view into how results are calculated. ## Transparency All of Statsig's methodology is documented, and the analysis run is totally visible to you in Warehouse Native. Statsig gives you the tools, data, and queries that you need to reproduce the results you see in console, with no black boxes. Most enterprise data science teams have gone through the exercise of hand-validating results as part of their evaluation process. How this manifests is: * Console-based access to the SQL Statsig runs; not only do you have access to the tables and queries on your warehouse, but Statsig will surface SQL snippets in console any time it runs a query so you can be completely sure if what's being calculated * Transparency around costs. Most customers can run pulse analyses for pennies, but Statsig clearly surfaces the run time and utilization of resources to help you manage your warehouse bill for experimentation * Support: Statsig has best-in-class support, with access to our data science team for open discussion of methodology, approaches, and collaborative development of new features ## Trust Experiments drive important business decisions, and it's critical that you can trust the analysis and statistics being run. Statsig has a rigorous evaluation process for its methodologies, including peer review, simulations, and publishing the thought process behind statistical designs. This means that you can trust Statsig's results being accurate and reliable as they help to guide your decisions. How this manifests is: * Detailed blog posts on the rationale behind decisions in new features * Documentation of methodologies in blogs, and references to prior art referenced * A full suite of diagnostic health checks on experiment results to warn you when statistical assumptions, or data quality, have been compromised ## Flexibility Experimentation is not a one-size-fits-all tool. Depending on your industry, philosophy, or even the setup of a specific experiment, Statsig lets you configure your analysis to suite your needs, offering: * Standard T-Tests * Sequential Testing * Bayesian Tests * Switchback Tests * Multi-armed bandits Additionally, Statsig offers many options around tools to control for multiple comparisons, outliers, and regression adjustment. ## Power Effective experimentation relies on having trustworthy results in a short amount of time. Tools to reduce experiment run time and increase accuracy are of paramount importance, and Statsig has invested heavily into bringing the highest degree of accuracy and power available - meaning your results will be faster and more reliable. Some examples of features focused on the power of Statsig's stats engine are: * [CUPED](https://www.statsig.com/blog/cuped), which can significantly reduce experiment run-times and account for pre-experiment bias * [Stratified Sampling](/experiments-plus/stratified-sampling), which makes experiment results much more accurate and consistent, meaning you can trust the effects you see # Confidence Intervals Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/confidence-intervals How Statsig Warehouse Native computes confidence intervals for experiment metrics, including formulas, assumptions, and how to interpret them. Confidence intervals are an intuitive way to quantify the uncertainty in the observed metric deltas. A 95% confidence interval should contain the true effect 95% of the time. This means that if we ran an experiment 100 times, the true value of the metric delta should be inside the confidence intervals 95 times. Confidence interval visualization showing statistical significance In practical terms, a 95% confidence interval that doesn't contain zero (the green bar above) represents a statistically significant result (with *α = 0.05*). This is not always the case, though: There are cases when the p-value of the difference between test and control is statistically significant, but due to uncertainty in the control, a relative delta confidence interval may cross zero (using [The Delta Method](/experiments/statistical-methods/methodologies/delta-method)) or be represented as a point estimate (using [Fieller Intervals](/experiments/statistical-methods/methodologies/fieller-intervals) ) while the absolute difference's p-value is statistically significant. Only 5% of the time would we expect to see the confidence interval exclude zero if the true effect was zero (a.k.a. a false positive). Larger confidence intervals imply less certainty in the exact size of the effect with a larger range of likely values. ## Computing Confidence Intervals Confidence intervals in Statsig are calculated using a two-sample z-test. This test requires knowledge of the variance in the metric delta we're measuring, which is derived differently depending on the type of metric (details [here](/stats-engine/variance)). Once we've established the variance of the delta, it's straightforward to compute the confidence intervals. ### Two-Sided Tests For the **absolute metric delta**, the confidence interval is given by: $$ CI(\Delta \overline{X}) = \Delta \overline{X} \pm Z_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}} $$ where: * $Z_{\alpha/2}$ is the z-critical value for the desired significance level (1.96 for the standard $\alpha=0.05$ and 95% confidence interval) and we run a two-sided test * $var(\Delta \overline{X})$ is the variance of the absolute delta (details [here](/stats-engine/variance)) The confidence interval for the **relative metric delta** can use one of two methods, [Fieller Intervals](/stats-engine/methodologies/fieller-intervals) and the [Delta Method](/stats-engine/methodologies/delta-method). While customers can opt for either method to be used in their Statsig console, Statsig recommends and automatically opts-in all new customers to use Fieller Intervals by default. When using Fieller Intervals, the relative metric delta CI can be computed using: $$ CI(\% \Delta \overline{X} ) = \frac{1}{1-g} ( \frac{\overline{X_T}}{\overline{X_C}} - 1 \pm \frac{Z_{\alpha/2}}{\sqrt{n_C} \cdot \overline{X_C}} \sqrt{(1-g) \cdot \frac{var(X_T)}{n_T(n_T-1)} + \frac{\overline{X_T} var(X_C)}{\overline{X_C} n_C (n_C-1)}}) $$ When using the Delta Method, the confidence interval is: $$ \begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}}\\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot\sqrt{(\frac{\overline X_t}{\overline X_c})^{2} \cdot (\frac{var(X_c)}{n_c \cdot \overline X_c^2} + \frac{var(X_t)}{n_t \cdot \overline X_t^2})} \cdot 100\% \end{split} $$ If using the Delta Method and the control mean is not significantly away from zero, then it's simplified to: $$ \begin{split} CI(\Delta \overline X\%) &= \Delta \overline X\% \pm Z_{\alpha/2} \cdot\sqrt{{var(\Delta \overline X\%)}} \\ &= \frac{\Delta \overline X}{\overline X_c} \pm Z_{\alpha/2} \cdot \frac{\sqrt{{var\left(\Delta \overline X\right)}}}{\overline X_c} \cdot 100\% \end{split} $$ ### One-Sided Tests When running one-sided tests, the form of the confidence interval calculation changes slightly to account for a redistribution of desired false positive rate when looking for increases or decreases in the metric: $$ CI(\Delta \overline{X}) = \begin{cases} \left[\Delta \overline{X} - Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}}, \quad +\infty \right) & \text{if right-hand test}\\ \\ \left(-\infty, \quad \Delta \overline{X} + Z_{\alpha} \cdot \sqrt{{var(\Delta \overline{X})}} \: \right] & \text{if left-hand test} \end{cases} $$ where: * $Z_{\alpha}$ is the z-critical value for the desired significance level (1.645 for the standard $\alpha=0.05$ and 95% confidence interval) and we run a one-sided test * $var(\Delta \overline{X})$ is the same as for two-sided tests * the choice of confidence interval depends on if the one-sided test is looking for increases or decreases in the metric ## Welch's T-test for Small Sample Sizes For small sample sizes, we use Welch's t-test instead of a standard z-test. This statistical test is a better choice for handling samples of unequal size or variance without increasing the false positive rate. The structure of the confidence interval calculation remains the same as above (depending on 1- or 2-sided test), replacing the z-critical value with the t-critical value with degrees of freedom $\nu$. For a two-sided test, the confidence interval is therefore: $$ CI(\Delta \overline{X}) = \Delta \overline{X} \pm t_{\alpha/2} \cdot \sqrt{{var(\Delta \overline{X})}} $$ $$ \nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}\ = \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}} $$ Where $N_t$ and $N_c$ are the number of users in the test and control groups, respectively. Note that for large number of degrees of freedom, the t-statistic converges with the z-statistic. Therefore, Welch's t-test is used only when $\nu < 100$. ## Comparing Experiment Data to a Fixed Baseline: One-sample T-test Sometimes we want to answer questions like "Does my test variant lead to a click through rate higher than 0.5?". You can define a fixed-baseline comparison when adding metrics to the experiment. The confidence interval is calculated by $$ CI(\Delta \overline X) = (\overline X_{group} - fixed \ value) \pm Z \cdot\sqrt{{var( \overline X_{group})}} $$ # Benjamini–Hochberg Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/benjamini-hochberg-procedure How Statsig Warehouse Native applies the Benjamini-Hochberg procedure to control false discovery rate across many metrics in experiment scorecards. ## What it is The Benjamini-Hochberg Procedure ("BH" procedure) is a statistical method that reduces the probability of false positives by adjusting the significance level for multiple comparisons. It is not as extreme as a [Bonferroni Correction](/stats-engine/methodologies/bonferroni-correction), because instead of controlling the chance of at least one false positive (Family Wise Error Rate), BH controls the expected value of false positives when the null hypothesis has been rejected (False Discovery Rate). Like with many other analysis settings, you can enable BH procedure for individual experiments (or configure global Experiment Settings to default it). Benjamini-Hochberg procedure configuration interface ## Methodology The [Benjamini-Hochberg Procedure](https://www.statisticshowto.com/benjamini-hochberg-procedure/) updates the significance level to be used (modifying your pre-set $\alpha$). The new significance level to be applied is calculated by sorting the p-values of metrics in ascending order and comparing with a paired threshold. Each p-value’s paired threshold is the desired False Discovery Rate divided by the number of comparisons being evaluated multiplied by what rank a p-value is in the ordered list. The largest threshold value which is higher than its corresponding p-value is our new significance level ($\alpha$). The Benjamini-Hochberg Procedure can be applied based on: * The number of test groups (multiple treatment hypotheses). For each metric aggregate the list of p-values from each variant and complete the Benjamini-Hochberg procedure. * The number of metrics in the scorecard. For each variant aggregate the list of p-values from each metric and complete the Benjamini-Hochberg procedure. * Both the number of test groups and number of metrics in the scorecard. All p-values are aggregated to complete the Benjamini-Hochberg procedure. Statsig does not apply BH procedure when evaluating the p-values of any event-dimension or user-property experiment metric results. Only the top-line metric results are compared to the new significance level. ## How do my experiment metrics look now? In the experiment scorecard section, confidence intervals will be derived from (1 - adjusted α) for applicable metrics. If you hover over a confidence interval, the adjusted α will be displayed alongside other relevant metric details. In the experiment explore section, if applicable to the selections you make, a new adjusted α will be calculated and these confidence intervals will use (1 - adjusted α). # Bonferroni Correction Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/bonferroni-correction How Statsig Warehouse Native applies the Bonferroni correction to adjust p-values when testing multiple metrics or comparisons in an experiment. ## What is Bonferroni Correction? A Bonferroni Correction is a statistical method that reduces the probability of false positives by adjusting the significance level for multiple comparisons. If you run a tests with α = 0.05, the probability of a false positive will be 5%. If you run more comparisons at the same significance level, the chance of at least one false positive goes up because each comparison is an additional opportunity for false positive. Bonferroni corrections are an optional feature on Statsig experiments that reduces the probability of Type I errors (false positives) by adjusting the significance level (α). The significance level is divided by the number of comparisons being evaluated. You can choose to apply these based on one or both of the following: * The number of test groups (multiple treatment hypotheses). The significance level is divided by the number of variants being compared against control. * The number of metrics in the scorecard. Here you may select what percentage of your total α is divided evenly among the Primary Metrics, and the remaining α is split equally among Secondary Metrics. For example: * Significance level of 0.05 * 2 Primary Metrics and 4 Secondary Metrics * 60% of α applied to Primary Metrics * Each Primary Metric is calculated with α = 0.6 \* 0.05 / 2 = 0.015 * Each Secondary Metric is calculated with α = 0.4 \* 0.05 / 4 = 0.005 * If both corrections are selected, they're applied on top of each other. In the example above, if we also wanted to correct for having 2 tests groups, we would further divide each α by 2. When analyzing dimensions, if correction for metrics is enabled, it's applied separately for the dimensional breakdown. We use the number of dimensions as the total metric count to correct for *in the dimensional analysis*, but it does not impact topline metrics. Bonferroni correction configuration interface ## How do my experiment metrics look now? In the experiment scorecard section, confidence intervals will be derived from (1 - adjusted α) for applicable metrics. If you hover over a confidence interval, the adjusted α will be displayed alongside other relevant metric details. In the experiment explore section, if applicable to the selections you make, a new adjusted α will be calculated and these confidence intervals will use (1 - adjusted α). # CUPED Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/cuped How Statsig Warehouse Native uses CUPED variance reduction to improve experiment sensitivity by adjusting for pre-experiment user behavior on metrics. ## CUPED - Controlled-experiment Using Pre-Existing Data CUPED (short for Controlled-experiment Using Pre-Existing Data) is a technique which leverages user information from before an experiment to reduce the variance, and increase confidence in experimental metrics. At Statsig, this pre-experiment data is defined as the 7 days before each user's exposure rather than a fixed window before the experiment starts for all users. This can help to debias experiments which have meaningful pre-exposure bias (e.g. the groups were randomly different before any treatment was applied). Our Cloud product uses a 7-day window for CUPED calculation. For Warehouse Native customers, a 7-day window is recommended, but you have the flexibility to customize it to any length. See more at the [Variance Reduction](/experiments/statistical-methods/variance-reduction) page. To dive deep into the methodology, see CURE by Statsig. ## CUPED for Simple Aggregations The methodology for simple aggregations is described in the original [Microsoft paper](https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf), as well as our in-depth [article](https://www.statsig.com/blog/cuped) on the technique. The Cloud product uses stratification alongside CUPED to account for users who may not have pre-experiment data. Users are grouped into strata based on available pre-experimentation information. Treatment and control effects are first estimated within each stratum, then aggregated to produce an overall result. We then apply the standard difference-in-means and variance estimation. This approach allows us to retain users with missing pre-data while still benefiting from variance reduction where applicable. ## CUPED for Ratio Metrics The Microsoft paper also gives details on how to implement CUPED for those with a different analysis unit (Appendix B). On Statsig, we extend it to work for our ratio metrics, where each experiment unit is represented by a numerator and a denominator. The variance reduction process is performed by finding the variance of experiment data, pre-experiment data, and the covariance between the two. Denote the numerator, denominator, pre-experiment numerator, and pre-experiment denominator of a unit as $Y$, $N$, $X$, and $M$, respectively. Using the CUPED-reduced variance formula, $$ Var(\frac{Y_{cv}}{N_{cv}})=Var(\frac{Y}{N})+\theta^2 Var(\frac{X}{M})-2\theta Cov(\frac{Y}{N}, \frac{X}{M}) $$ where optimal $\theta$ is found as $$ \frac{Cov(\frac{Y}{N}, \frac{X}{M})}{Var(\frac{X}{M})} $$ expanded to \\ $$ \frac{Cov(\frac{Y}{\mu_N}-\frac{\mu_Y N}{\mu^2_N}, \frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})}{Var(\frac{X}{\mu_M}-\frac{\mu_X M}{\mu^2_M})} $$ At this point, we have $$ \frac{\hat{Y_{c}}}{\hat{N_{c}}}=\frac{Y_{c}}{N_{c}}-\theta( \frac{X_{c}}{M_{c}} - \mathbb{E}[R]) $$ $$ \frac{\hat{Y_{t}}}{\hat{N_{t}}}=\frac{Y_{t}}{N_{t}}-\theta( \frac{X_{t}}{M_{t}} - \mathbb{E}[R]) $$ While $\mathbb{E}[R]$ is hard to deduct, we recognized that the expectation term is the same for both group. We decided to substitute $\mathbb{E}[R]$ with $\frac{X_{c}}{M_{c}}$ so the formulas above are transformed to these following two: $$ \frac{Y_{cv}(control)}{N_{cv}(control)}=\frac{Y(control)}{N(control)} $$ $$ \frac{Y_{cv}(test)}{N_{cv}(test)} \\ :=\frac{Y(control)}{N(control)} - (\frac{Y(control)}{N(control)} - \theta \frac{X(control)}{M(control)}) + (\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)}) \\ :=\frac{Y(test)}{N(test)} - \theta\frac{X(test)}{M(test)} + \theta \frac{X(control)}{M(control)} $$ Using the optimal $\theta$, we are hoping to reduce group-level variance by plugging the parameter back in to calculate the adjustment. Please note that across-group $\theta$ does not necessarily reduce variance for one group, or the sum of variances of all groups, but in most cases it does. Our simulation shows that 98.3% of metrics saw a decrease by CUPED. Statsig will use CUPED variance when all of the following are met: * Core assumptions of the CUPED model are satisfied; this can be violated due to rounding error or other data artifacts * E(X\_hat) = E(X) * The pooled variance of the adjusted population across groups is \< the variance of the unadjusted population * Enough units have pre-experiment values (> 100) * Enough percentage of units have pre-experiment values (> 5%) # Delta Method Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/delta-method How Statsig Warehouse Native uses the delta method to compute variances for ratio metrics and non-linear functions of user-level data. ## Delta Method for Ratio Metics Statsig uses the delta method when calculating the variance for variables that have a numerator and denominator. The variance of ratio and mean metrics depends upon the numerator and denominator variables, which are typically correlated. For example, consider a *clicks per session* metric. The number of clicks and the number of sessions are two sets of observations coming from the same group of users, so they are not independent from each other. To properly account for this correlation, the variance of a ratio metric *R* is obtained using the delta method: Delta method variance formula where the variance of the numerator and denominator means are computed in the same way as detailed above for count metrics, and the covariance is Covariance calculation formula ## Delta Method for Relative Lifts Statsig may also use the delta method when calculating the confidence interval for relative lifts. The other methodology for calculating confidence intervals for relative lifts is [Fieller Intervals](/experiments/statistical-methods/methodologies/fieller-intervals) - the delta method is a heuristic for Fieller Intervals which converges with a large population. # Fieller Intervals Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/fieller-intervals How Fieller intervals build accurate confidence intervals for ratio metrics in Statsig Warehouse Native experiment scorecards and analysis views. ## Fieller Intervals Organizations can choose to use Fieller Intervals as the methodology of calculation for the confidence intervals for the relative change between test and control group. The Delta Method is an approximation for the variance of a ratio between two variables that is then used to establish a confidence interval, while Fieller Intervals are an exact solution for the confidence interval. In most cases though, Fieller Interval results are very similar to results from the Delta Method. Since Fieller Intervals are more accurate, we recommend that you opt into using this methodology! ## Calculation ### 1: Determine if a Fieller Interval is Well-Defined Before proceeding to applying Fieller’s Theorem, we need to check that the denominator of the relative lift metric $ \overline{X_C}$ is significantly distinct from 0. We do this by calculating the parameter $ g$: $$ g = \frac{Z_{\alpha/2}^2 \cdot \mathrm{var}(X_C)}{(n_C-1) \cdot \overline{X_C}^2} $$ Where: $ Z_{\alpha/2}$ is the critical value associated with the desired confidence level $ \mathrm{var}(X_C)$ is the variance of the control group metric values $ n_C$ is the number of units in the control group $ \overline{X_C}$ is the mean of the control group metric values When $ g$ \< 1, the control mean is significantly different from 0, and we can use Fieller intervals. ### 2A: Apply Fieller Interval Formula Since the control and test group results are independent of each other, covariance terms in Fieller's Theorem can be dropped. $$ CI(\% \Delta \overline{X} ) = \frac{1}{1-g} \left( \frac{\overline{X_T}}{\overline{X_C}} \pm \frac{Z_{\alpha/2}}{\overline{X_C}} \sqrt{ \frac{\overline{X_T}^2}{\overline{X_C}^2} \cdot \frac{\mathrm{var}(X_C)}{n_C-1} + (1-g)\frac{\mathrm{var}(X_T)}{n_T-1} } \right) - 1 $$ ### 2B: Edge Case: Control Mean not Statistically Distinct from Zero In rare cases (less than 5% of observed metric comparisons on Statsig), g $\geq$ 1, which means that the control group’s mean is not statistically distinguishable from 0. When $\overline{X_C}$ is not statistically different from zero, the denominator of our relative lift calculation is unstable. This means that the confidence interval for the percent difference between test and control is unbounded. When this happens, we surface the relative lift observed during the experiment. $$ \% \Delta \overline{X} = \frac{\overline{X_T}-\overline{X_C}}{\overline{X_C}} $$ ## Enabling on Statsig Controlling which relative confidence interval methodology you use is available in your Experimentation Settings at the Organization level, and changing this setting only impacts experiments created after the setting change. Experimentation settings configuration interface In many cases, the results will be effectively the same as using the [Delta Method](/stats-engine/methodologies/delta-method), but especially if you’re running experiments with small sample sizes or noisy denominators, Fieller Intervals are more reliable. Thus, we'd strongly recommend using Fieller Intervals. In the experiment scorecard, Fieller Intervals will look like this Experiment scorecard with Fieller intervals # One-Sided Tests Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/one-sided-test How Statsig Warehouse Native uses one-sided hypothesis tests in experiments to detect changes in a pre-specified direction with higher power. ## One-Sided Tests (aka One-Tailed Test, Non-Inferiority Test) A one-sided test lets you test for a metric moving in only one direction which you specify in advance. This trade off gives you additional sensitivity (or power). It differs from the standard Pulse results which show two-sided results by default. Use cases for one-sided testing include detecting regressions in guardrail metrics and testing for a change in which only one direction has any meaningful business impact. For example, you may be less interested in detecting if a new feature reduces crash rates, but are very interested in learning if the new feature increases crash rates. In this example, you are willing to forgo detecting the former in favor of better detecting the latter. One-sided tests completely disregard the possibility of detecting the metric moving in the direction that isn't specified, but they give you higher sensitivity in the direction you are looking (which allots all your alpha to testing statistical significance in the one direction of interest). This results in one-sided confidence intervals (CIs) that are narrower in the direction of interest than their two-sided counterparts. ## How to enable this When setting up an experiment and identifying metrics to measure, the default setting is to run a two-sided test. If you want to modify this, simply click on the metric name on the experiment setup screen. This will open a popup where you can modify the test type and indicate a desired direction you seek to measure. Our V1 doesn't support Bayesian testing yet. One-sided test configuration interface ## How to read this Metrics using one-sided tests will show up in Pulse very similarly to two-sided tests. The only difference will be that we show a one-sided CI rather than a two-sided CI. Reading one-sided CIs can be a bit confusing at first. They either extend to infinity or negative infinity, which is a bit unusual, but this is entirely expected since we only detect changes in the other direction. As usual with CIs, they indicate that the real mean value of the metric likely falls into this range. Since the CI for the one-sided metric analyzed is so wide, it can be equally useful to read the results as having high confidence the mean value does not fall in the range outside the CI. One-sided confidence interval visualization ## FAQ #### Why can't I just run two one-sided tests? It could be tempting to do this, but in reality it would result in less powerful test. One-sided tests work by allocating the entirety of our Type I error (alpha/significance) to one direction. If we were to add an additional one-sided test in the other direction, we would be re-introducing chance of making a Type I error in that direction. Doing so would mean we claim tighter confidence intervals around metric results that would actually result in higher rates of decision error than the specified confidence level (default: 95%). #### Why use a one-sided tests rather than a two-sided test? This comes down to your use case, metric of interest, and business impact of any decision. Selecting one-sided vs. two-sided tests depends on how you plan to interpret any change in the metric and whether detecting changes in either direction are equally valuable. # SRM Checks Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/srm-checks Sample ratio mismatch (SRM) detection in Statsig Warehouse Native flags skewed traffic splits in experiment scorecards so you can investigate and fix logging. ## SRM - Sample Ratio Mismatch Sample ratio mismatch (SRM for short) is when the observed allocation of **unique** users between test groups differs from the expected allocation or "split" of the test. We have a brief [rundown on this topic here](https://www.statsig.com/blog/sample-ratio-mismatch) on our blog. This is a signal that there could be some unknown bias in the test. This is a major problem because unless you can clearly diagnose the reason for the imbalance, there's not an easy way to know how much this bias impacts your results. ## SRM Checks Statsig runs SRM checks on all experiments and feature gates as part of our Health Checks (described [here](/experiments-plus/monitor)). We use a Chi-squared test to identify if the split of users between groups is indicative of a Sample Ratio Mismatch. SRM health check results interface We automatically analyze data by common dimensions logged by the Statsig SDK to identify potential drivers of SRM. These include sdk\_type, sdk\_version, reason, is\_bot, browser\_name, browser\_version, os, os\_version, and region to identify potential causes. SRM dimension analysis breakdown # Winsorization Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/methodologies/winsorization Apply winsorization in Statsig Warehouse Native to cap extreme metric values at chosen percentiles, stabilizing experiment scorecards against outliers. Winsorization is a common technique for removing noise in experiment results, specifically from outliers. Winsorization refers to the practice of measuring the percentile Px of a metric and setting all values over Px to Px. Statsig computes the Px value using all non-zero and non-null unit-level values of the metric; metrics are aggregated from rows or events, and then the Px'th unit's value is used as the threshold to adjust other units' values. At Statsig, the default percentile for winsorization is 99.9%. This reduces the influence of extreme outliers caused by factors such as logging errors or bad actors. Winsorization is applied to sum, event count, mean, ratio, and funnel metrics, including imported metrics. Winsorization will not be applied to Participation or User Accounting metrics. Statsig Warehouse Native lets you configure this per metric - and choose explicitly the upper and/or lower bounds to apply. Winsorization configuration interface Winsorization is applied to sum, event count, mean and ratio metrics. ## Metric Capping This is a very simple, but effective technique to handle outliers. With this capability, you can define max values for a metric for whatever unit type(s) are configured for this metric. Any value surpassing the set cap will automatically be adjusted downward to match it. For instance, if you determine that purchases greater than \$10,000 per day on your E-commerce platform should not skew analysis, any transaction exceeding this threshold will be adjusted downward to this limit, ensuring the integrity of your experiment analysis. Capped metrics are available for Event Count and Aggregation (sum) metric types. # Metric Deltas Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/metric-deltas How Statsig Warehouse Native computes metric deltas to compare absolute and relative differences between experiment groups in scorecards. ## Computing Metric Deltas A metric delta refers to the difference in metric values between two groups, by default the test and control groups. This is usually the impact we care about when looking at experiment results. To account for the different number of users (or units) that constitute each group, we compare the mean metric value per user, not the total. **Selecting Groups** We define all deltas here to be the difference between a "treatment" group as compared to a presumably unchanged "control" group. However, Statsig enables comparison between any two groups as desired. Two different metric deltas are available in Pulse. The **absolute delta** is simply the difference between the two means: $$ \Delta \overline{X}=\overline{X}_t-\overline{X}_c $$ It's often helpful to understand the impact relative to the baseline value of the metric. For example, an absolute delta of +1 clicks/user has different meanings with a baseline value of 1 (+100% increase) vs. a baseline value of 100 (+1% increase). The **relative delta** is computed using the mean of the control group as the baseline: $$ \Delta \overline{X} \%=\frac{\overline{X}_t-\overline{X}_c}{\overline{X}_c} \times 100 \% $$ If you reverse the order of group comparison in Pulse to be "control" vs "treatment", then all deltas will be reversed and the direction of change will be inverted. ## Computing Means Properly computing the groups means is critical for obtained meaningful metric deltas. The exact methodology for calculating the metric means depends on the type of metric. ### Event Count and Sum Metrics These metrics represent totals: Number of times an event occurs, sum of time spent, total purchase amount, etc. The mean is the average user-level total during the time period of the analysis. The mean value of the metric $X$ for a group is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} X_{i, d} $$ where: * $N$ is the number of users in the group * $n_i$ is the number of days during the analysis period that user $i$ was the experiment * $X_{i,d}$ is the metric value for user $i$ on day $d$ Only user metrics recorded after a user has been exposed to the experiment are included in the group mean. ### User Accounting and Event User Metrics (and legacy Event DAU) Event User metrics set to "Daily Participation Rate" capture the number of distinct users that have the event each day. In Pulse results, they are normalized by the number of days the user is in the experiment. This represents the probability that a user is daily active for that event, i.e. the daily participation rate. In the terms defined above, the group mean is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N \frac{1}{n_i} \sum_{d=0}^{n_i} X_{i, d} $$ where: * $X_{i,d}$ takes value 0 or 1 depending on if user $i$ has the event on a given day $d$. The following user accounting metrics are computed in the same may: *DAU, WAU, MAU\_28day, L7, L14, L28* For new user accounting (*new\_DAU, new\_WAU, new\_MAU\_28day*) we count users that are new xAU at some point during the analysis window. So the group mean is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N \max \left(X_i\right) $$ Where $\max(X_i)$ is the maximum value of the new xAU metric for user $i$. **event\_dau** metrics are now in legacy support only and are no longer created for new events. Existing event\_dau metrics will continue to be available for any of your new experiments and will continue to be computed daily. For all new events, you should create an event\_user metric to measure daily active users. ### Custom Ratios, Means, Retention and Stickiness Metrics These are metrics such as click through rate, average purchase value, sessions per user, etc. They're obtained by diving a numerator value, $X$, by a denominator value, $Y$. The mean value of a ratio metric $R$ for an experiment group is given by: $$ \overline{R}=\frac{\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} X_{i, d}}{\frac{1}{N} \sum_{i=0}^N \sum_{d=0}^{n_i} Y_{i, d}}=\frac{\overline{X}}{\overline{Y}} $$ Where $N$ is the number of users in the experiment group that participate in the metric, i.e. have a non-zero denominator value. $X_{i,d}$ and $Y_{i,d}$ are the $X$ and $Y$ values for user $i$ on day $d$. Different approaches exist for dealing with ratio metrics in experiments. This implementation was selected because it's statistically sound as well as interpretable, given that: * $R$ is the ratio of two means of independent observations: A set of user-level $X$ values and a set of user-level $Y$ values. This means the central limit theorem can be used to separately obtain the summary statistics of $X$ and $Y$. * The group means are computed in the same way as the topline metric value, making it easier to interpret the means and relate them to the topline metric. ### Event User One-Time Event For custom **event\_user** metrics with "One-Time Event" selected, statsig computes how many users have the event at any time after the user has been in the experiment. This result is not normalized by the number of days a user is in the experiment. The group mean is given by: $$ \overline{X}=\frac{1}{N} \sum_{i=0}^N X_{i} $$ where: * $N$ is the number of users in the group * $X_{i}$ takes value 0 or 1 depending on if user $i$ has the event at any point after entering the experiment # p-Value Calculation Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/p-value What p-values mean in Statsig Warehouse Native experiments, how they are computed, and how to interpret them alongside intervals and lift estimates. In Null Hypothesis Significance Tests, the p-value is the probability of observing an effect larger than or equal to the measured metric delta, under the assumption that the null hypothesis is true. In practice, a p-value that's lower than your pre-defined Type I Error threshold ($\alpha$) is treated as evidence for there being a true effect. The methodology used for p-value calculation depends on the number of degrees of freedom ($\nu$). A two-sample z-test is appropriate for most experiments. Welch's t-test is used for smaller experiments with $\nu < 100$. In both cases, the p-value depends on the metric [mean](/stats-engine/metric-deltas) and [variance](/stats-engine/variance) computed for the test and control groups. Typically, a p-value that indicates statistical significance (below the pre-determined threshold $\alpha$) could only occur with a confidence interval that does not cross 0. However, this phenomenon can occur in the Statsig UI, due to cases when the p-value of the difference between test and control is statsig, but due to uncertainty in the control, a relative delta confidence interval may cross zero (using [The Delta Method](/experiments/statistical-methods/methodologies/delta-method)) or be represented as a point estimate (using [Fieller Intervals](/experiments/statistical-methods/methodologies/fieller-intervals) ) while the absolute difference's p-value is statistically significant. ## Two-Sample Tests ### Two-Sided z-Test The z-statistic (a.k.a. z-score) of a two-sample z-test can be computed in multiple equivalent formats: $$ \begin{split} Z &= \frac{\overline X_t - \overline X_c}{\sqrt{var(\overline X_t)+ var(\overline X_c)}} \\ &= \frac{\overline X_t - \overline X_c}{\sqrt{var(\Delta \overline{X})}} \\ &= \frac{\overline X_t - \overline X_c}{\sqrt{\sigma_{\overline{X}_t}^2 + \sigma_{\overline{X}_c}^2}} \end{split} $$ where: * $Z$ is the observed z-statistic (not the z-critical value $Z_{\alpha/s}$) * $var(\Delta \overline{X})$ is the variance of the absolute delta of means * $var(\overline{X}_i)$ is the variance of sample means either control or treatment group (details [here](/stats-engine/variance)) * $\sigma_{\overline{X}_t}$ is the standard error of the mean of either control or treatment group (these are the terms you can find in Pulse under the Statistics tab of a metric) The two-sided p-value is obtained from the standard normal cumulative distribution function: $$ p-value = 2 \cdot \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{-|Z|}{e^{-t^2/2}dt} $$ ### Welch's t-test For smaller sample sizes, Welch's t-test is the preferred statistical test for lower false positive rates in cases of unequal sizes and variances. In Pulse, Welch's t-test is automatically applied when the degrees of freedom $\nu < 100$. We compute the t-statistic (a.k.a. t-score) identically as the two-sample z-statistic above. Additionally, we compute the degrees of freedom $\nu$ using: $$ \nu = \frac{\left(var(\overline X_t) + var(\overline X_c)\right)^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}}\ := \frac{var(\Delta\overline{X})^2}{\frac{var(\overline X_t)^2}{N_t - 1}+\frac{var(\overline X_c)^2}{N_c - 1}} $$ The p-value is then obtained from the t-distribution with $\nu$ degrees of freedom. ### One-Sided Z-Test The procedure for a one-sided z-test computes the z-statistic $Z$ in the same way as a two-sided test above. The one-sided p-value is obtained from the standard normal cumulative distribution function as well, but with slight differences: $$ p-value = \begin{cases} 1 - \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if right-hand test}\\ \frac{1}{\sqrt{2\pi}} \int \limits _{-\infty}^{Z}{e^{-t^2/2}dt} &\text{if left-hand test} \end{cases} $$ where: * $Z$ is computed above in the two-sided test. Note that this uses the signed z-statistic, not the absolute value of the z-statistic as in the two-sided p-value. # Pre-Experiment Bias Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/pre-experiment-bias How Statsig Warehouse Native detects pre-experiment bias caused by uneven user distributions between treatment and control groups before exposure. In some cases, users in two experiment groups can have meaningfully different average behaviors before your experiment applies any intervention to them. If this difference is maintained after your experiment starts, it's possible that experiment analysis will attribute that pre-existing difference to your intervention. This can make a result seem more or less "good" or "bad" than it really is. [CUPED](/experiments/statistical-methods/methodologies/cuped) is helpful in addressing this bias, but can't totally account for it. Additionally, some metrics like retention are not viable candidates for CUPED and can't be easily adjusted. Statsig proactively measures the pre-experiment values of all scorecard metrics for all experiment groups, and determines if the values are significantly different and could cause misinterpretations. If bias is detected, users are notified and a warning is placed on relevant Pulse results. ### How it works Statsig provides a "Days Since Exposure" view to help identify novelty effects and existing pre-experiment effects. For example, the test group of the experiment below had a consistently higher mean than the control group in the week before exposure for this metric Pre-experiment bias visualization showing test group with consistently higher mean than control group Statsig detects this bias by running the standard [pulse](/pulse/read-pulse) calculation on the pre-experiment term (looking back one week in cloud, and your configured CUPED lookback window in Warehouse Native), and calculating the p-value for the null hypothesis that the groups are identical. Relevant results will be flagged according to logic which balances awareness and false positives stemming from high numbers of scorecard metrics or groups. ### What to Do Pre-experiment bias can occur by chance and is not always a major issue. * If the total delta is small, it may not meaningfully influence your interpretation of results * If CUPED can account for the bias, then the bias should not impact your results In many cases, you can just use this warning as that - a warning - and proceed while treating impacted metrics with a grain of salt. This is often the correct path forward if the metric is not critical to the experiment, or if you care more about the directional movement than the exact number. Additionally, more time may alleviate the bias if there's no systemic source (which is generally the case), as the bias will be diluted by additional new users. However, if the metric is critical to your analysis and you care about the exact numerical value, you may want to consider resalting and restarting this experiment. # Topline and Projected Impact Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/topline-impact How Statsig Warehouse Native estimates the topline impact of an experiment on company metrics by scaling experiment lift to your total user base. The **topline impact** is the average daily effect that an experiment has on the overall metric value as compared between two groups. This is the real daily impact to a metric resulting from running the experiment, measured amongst the two groups being evaluated. The **projected launch impact** is an estimate of the daily impact we expect to see in the metric measured globally if a decision is made and the test group is launched to all users (beyond just those in the experiment). This impact is computed relative to the expected baseline value of the metric if the experiment wasn't running at all. Topline Impact and Projected Launch Impact are shown in both absolute and relative units. Topline Impact and Projected Launch Impact currently do not use CUPED when measuring any impact induced by your experiment. The reason is that CUPED already adjusts for pre-exposure data, which is what the topline metrics change from - mixing the two would double-count that adjustment. **Example**: Take a simple example experiment with a Control group of 1000 users and a Test group of another 1000 users, which ran for 30 days. For an **event\_count** metric, we observed an Experiment Delta of +1.0 events per user (abs). The Topline Impact for this metric would be +33.33 events per day (abs). ## Computing Topline Impact The topline impact is computed over the total duration of the experiment. This gives the most accurate estimate and tight confidence interval. The exact calculation depends on whether the metric represents an absolute quantity or a ratio: ### Count and Sum Metrics (event\_count, sum) The absolute topline impact is derived directly from the experiment results. It depends on the difference in means between test and control, and the average number of users in the test group per day. $$ Impact_{abs}=(X_t-X_c) \cdot N_t / n_{days} $$ Knowing the absolute impact and the overall metric value (as seen in the [metrics dashboard](/metrics/console)), we can compute the relative impact. This is the percentage change in the overall metric value over the rollup window that is attributed to the active experiment. $$ Impact_{rel}=\frac{Impact_{abs}}{Topline\_Value-Impact_{abs}} \times 100\% $$ ### Ratio and Mean Metrics To properly derive the topline impact on a ratio metric we must understand the impact on the numerator (*X*) and denominator (*Y*) separately. The topline impact is the current value of the ratio metric minus the baseline value we obtain by subtracting the numerator and denominator impacts: $$ Impact_{abs}=\frac{Topline\_X}{Topline\_Y}-Baseline\_Value $$ Where the baseline value is the expected value of the topline metric if the experiment wasn't running: $$ Baseline\_Value=\frac{Topline\_X-(\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline\_Y-(\bar{Y_t}-\bar{Y_c}) \cdot N_t} $$ The relative impact for ratio metrics is obtained by dividing the absolute impact by the baseline value: $$ Impact_{rel}=\frac{Impact_{abs}}{Baseline\_Value} \times 100\% $$ ## Computing Projected Launch Impact The layer allocation of the experiment and the size of the test group are used to estimate a scaling factor *m*, which represents the increase in absolute impact expected when a decision is made to launch the test group. The launch factor over a rollup window is calculated as $$ m_{rollup}=\frac{1}{\sum_{1}^{rollup}{layer\_alloc \times group\_pct}} \times rollup $$ to accommodate changes in allocation during the experiment. The targeting gate isn't factored in. The projected impact calculation assumes that the target gate remains the same after the experiment is launched. ### Count and Sum Metrics (event\_count, event\_dau, sum) For count and sum metrics, the projected absolute impact is simply the current topline impact scaled by a factor of *m*. For example: Consider an experiment running with 50% layers allocation and 50/50 test/control split, so that 25% of all users are in the test group. If the allocation has been changing during this experiment, we will use a weighted average based on historical allocations. If the topline impact is currently +10 events per day, then launching the experiment would lead to +40 events per day. $$ Projected_{abs}=Impact_{abs} \times m $$ The relative projected impact is expected percentage change in the topline metric, relative to the baseline value of the metric without the experiment running. $$ Projected_{rel}=\frac{Projected_{abs}}{Topline\_Value-Impact_{abs}} \times 100\% = Impact_{rel} \times m $$ ### Ratio and Mean Metrics Similar to the topline impact calculation above, the projected impact of ratio metrics depends on the numerator and denominator impacts. We use the same scaling factor *m* to obtain the projected impact for each term: $$ Projected_{abs}=\frac{Topline\_X+(m-1) \cdot (\bar{X_t}-\bar{X_c}) \cdot N_t}{Topline\_Y+(m-1) \cdot (\bar{Y_t}-\bar{Y_c}) \cdot N_t} - Baseline\_Value $$ Where the first term represents the projected metric value after launch. Finally, the projected relative impact of a ratio metric is the projected absolute impact divided by the baseline value of the ratio: $$ Projected_{rel}=(\frac{Projected_{abs}}{Baseline\_Value}) \times 100\% $$ ## Confidence intervals The confidence intervals for topline and projected impact are computed in the same way as the [confidence intervals](/stats-engine/confidence-intervals) for experiment deltas. $$ CI(Impact) = Impact \pm Z \cdot \sqrt{var(Impact)} $$ In the case of absolute impact of count and sum metrics, the variance calculation is simply a linear combination of the test and control variances: $$ var(Impact_{abs})=[var(\bar{X_t})+var({\bar{X_c}})] \cdot N_t^2 $$ And for projected launch impact we get: $$ var(Projected_{abs})=var(Impact_{abs}) \cdot m^2 $$ For ratio metrics and relative impacts, the variance is calculated using the Delta method. This properly accounts for the correlation between the various numerator and denominator terms, leveraging Taylor expansion to linearize expressions containing non-linear combinations of experiment variables. For example, the variance in the relative impact of a count metric is given by: $$ var(Impact_{rel})=var(Impact_{abs}) \cdot \frac{(Topline\_Value - 2 \cdot Impact_{abs})^2}{(Topline\_Value - Impact_{abs})^4} $$ # Standard Error & Mean Variance Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/variance How Statsig Warehouse Native computes variance for experiment metrics, including handling of ratio metrics, clustered data, and user-level aggregation. The standard error (sometimes denoted "SE" or "std err") of the mean of each group is required for computing the confidence interval and p-value of a metric delta between those groups. The standard error of the mean can be obtained by dividing the sample standard deviation of $X$ by the square root of the number of users in the group. $$ \sigma_{\overline X} = \frac{\sigma_{X}}{\sqrt{N}} = \sqrt{\frac{var(X)}{N}} = \sqrt{var(\overline{X})} $$ Note that standard deviation is the square root of the variance. Since variances are easier to manipulate algebraically, here we derive the variance for each metric type and then take the square root to obtain the confidence intervals. Pulse displays the standard error of the mean of each group alongside the units and mean of each group. ## Computing Variance The variance of the absolute metric delta is simply the sum of the variances of the test and control means: $$ var(\Delta \overline X) =var(\overline X_t - \overline X_c) = var(\overline X_t) + var(\overline X_c) $$ In other words, it comes down to correctly calculating the variance of the means for each group. ### Count and Sum Metrics For count and sum metrics, the variance of the sample mean for a given group is obtained directly from the sample variance: $$ var(\overline{X}) = \frac{var(X)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline{X})^2}{N} $$ Where: * $N$ is the number of users in the group * $X_i$ is the metric value for user $i$ * $\overline{X}$ is the user-level average of $X$ for users in that group ### Ratio and Mean Metrics Some metrics like ratio and mean metrics combines multiple variables $X$ and $Y$ instead of a single variable $X$. The variance of these metrics depends upon both the numerator and denominator variables, which are typically correlated. We'll call the metric of interest $R$ and we can compute the group mean $\overline{R}$ and group variance of the mean $var(\overline{R}$). For example, consider a *clicks per session* metric. The number of clicks and the number of sessions are two sets of observations coming from the same group of users, so they are not independent from each other. To properly account for any correlation, the variance of the mean of a ratio metric $R$ is obtained using the delta method: $$ var(\overline R) = var\left(\frac{\overline X}{\overline Y}\right) := \left(\frac{\overline X}{\overline Y}\right)^2 \cdot \left(\frac{var(\overline X)}{\overline X^2} + \frac{var(\overline Y)}{\overline Y^2} - 2 \cdot \frac{covar(\overline X, \overline Y)}{\overline X\cdot \overline Y} \right) $$ where the variance of the numerator and denominator means are computed in the same way as detailed above for count metrics, and the covariance is $$ covar(\overline X, \overline Y) = \frac{covar(X, Y)}{N} = \frac{\frac{1}{N-1}\sum_{i=0}^{N}(X_i-\overline X)\cdot (Y_i-\overline Y)}{N} $$ # Variance Reduction Source: https://docs.statsig.com/statsig-warehouse-native/features/statistics/variance-reduction Overview of variance reduction techniques in Statsig Warehouse Native, including CUPED, stratified sampling, and regression adjustment for sensitivity. ## Variance Reduction [Variance](/stats-engine/variance) is a measurement of dispersion which measures the amount of "noise" in a metric or experiment results. Higher variance is associated with larger confidence intervals, and leads to experiments requiring more sample size to consistently observe a statistically significant result on the same effect size. Reducing variance can lead to shorter experiment run times due to the lower sample required. Because of this, techniques have been developed to reduce the variance in experiment results in order to reduce run times and increase confidence. At Statsig, we use a form of CUPED based on a [2013 Microsoft paper](https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf) (Deng, Xu, Kohavi, & Walker). This is automatically applied to experiments at Statsig, and is run for the topline results on key metrics in Pulse. This observably leads to significant variance reduction in the large majority of metrics where CUPED can be applied. Refer to our [launch post for CUPED](https://blog.statsig.com/cuped-on-statsig-d57f23122d0e) for more details. ## CUPED - Controlled-experiment Using Pre-Existing Data CUPED (short for Controlled-experiment Using Pre-Existing Data) is a technique which leverages user information from before an experiment to reduce the variance, and increase confidence in experimental metrics. At Statsig, this pre-experiment data is defined as the 7 days before each user's exposure rather than a fixed window before the experiment starts for all users. This can help to debias experiments which have meaningful pre-exposure bias (e.g. the groups were randomly different before any treatment was applied). The Cloud product uses stratification alongside CUPED to account for users who may not have pre-experiment data. Users are grouped into strata based on available pre-experimentation information. Treatment and control effects are first estimated within each stratum, then aggregated to produce an overall result. We then apply the standard difference-in-means and variance estimation. This approach allows us to retain users with missing pre-data while still benefiting from variance reduction where applicable. ## Winsorization Another common technique for reducing noise is Winsorization, which is a way to manage the influence of outliers. Winsorization refers to the practice of measuring the percentile *Px* of a metric and setting all values over *Px* to *Px*.This reduces the influence of extreme outliers caused by factors such as logging errors or bad actors. ## Metric Selection The metrics you use can dramatically influence the sensitivity of your analysis. The transformations above, in addition to techniques like creating threshold-based flags, can let you trade-off exact numbers for significantly more power. Please refer to our [blog post](https://www.statsig.com/blog/understanding-and-reducing-variance-and-standard-deviation) on the topic for more information. ## Literature Here's a short list of useful content for understanding more about these techniques and its applications * [Deng, Xu, Kohavi, & Walker](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf) is the seminal paper on using this technique for online controlled experiments * [Booking.com](https://booking.ai/how-booking-com-increases-the-power-of-online-experiments-with-cuped-995d186fff1d) has an excellent blog post on the theory and practice of CUPED * [Improving the Sensitivity of Online Controlled Experiments: Case Studies at Netflix](https://www.kdd.org/kdd2016/papers/files/adp0945-xieA.pdf) # Stratified Sampling Source: https://docs.statsig.com/statsig-warehouse-native/features/stratified-sampling How Statsig Warehouse Native uses stratified sampling to reduce variance and improve experiment reliability in low volume or high variance scenarios. ## What is Stratified Sampling Stratified sampling involves dividing the entire population into homogeneous groups called strata (plural for stratum). Random samples are then selected from each stratum. e.g. If you had XS and XL customers and randomized them into two groups - Control and Test, you'd want both Control and Test to be balanced across XS and XL customers. You can also stratify based on a metric like Revenue/User. With large numbers, randomization typically solves this. However in B2B scenarios and other relatively low volume or high variance scenarios, stratified sampling is useful to ensure this balance. Statsig supports both automated and manual stratified sampling. On tests where a tail-end of power users drive a large portion of an overall metric value, stratified sampling meaningfully reduces false positive rates and makes your results more consistent and trustworthy. In our simulations, we saw around a 50% decrease in the variance of reported results. ## Automated Stratified Sampling ### How it works The Statsig SDKs use a *salt* to randomize or bucket experiment subjects ([learn more](/faq#how-does-bucketing-within-the-statsig-sdks-work)). When you enable stratified sampling, we'll try n different salts (100 for now) and evaluate how "balanced" your groups. We evaluate this balance based on either a metric you pick - or an attribute you give us describing your experiment subjects. We pick the best salt from this set and save this as the salt to use. [Learn more](https://statsig.com/blog/introducing-stratified-sampling). Stratified sampling algorithm diagram The selection space for the salts is sufficiently large - stratifying multiple experiments on the same metric will not result in overlap. In the simulations we ran, the groups were as independent as expected which matched up with the literature here. ### Enabling Stratified Sampling You can enable this on experiment under Advanced Settings on the experiment setup page. There are two ways you can "stratify" on Statsig. If you choose a metric to stratify using, we'll use that to balance the group. Stratified sampling metric selection interface If you instead choose an attribute or a classification (e.g. S, M, L, XL) we'll use that to balance the group. * On Statsig Cloud, you'll upload a CSV (in Beta) * On Statsig Warehouse Native, you'll use Entity Properties Entity properties configuration for stratified sampling Once you press the Stratify button, we'll analyze a set of salts and pick the best one. Stratification analysis results interface ## FAQ and Best Practices * **What population is used when balancing?** * When evaluating salts, Statsig computes balance using pre-experiment data for the entire targeted population of the experiment’s unit type (e.g., all `userID`s or all `customerID`s) over the selected lookback window. There is no filtering on exposure because the experiment has not started yet. * **How are new units handled after stratification?** * Units that were not present in the pre-experiment data are still assigned deterministically by the chosen salt, i.e., effectively at random with respect to the balancing metric. They do not influence the salt selection and may introduce some drift from the initial balance. * **Should I use stratified sampling for every experiment?** * Not necessarily. It’s most useful when you expect imbalance due to heterogeneous units (e.g., “whales”) or skewed metrics. The tradeoff is time/compute cost that scales with the number of units and adds steps before starting an experiment. If you don’t expect meaningful imbalance, a standard random split is generally recommended. * **Does salt evaluation assume 100% allocation? What about running at less than 100%?** * Yes. All candidate salts are evaluated assuming 100% of the targeted population is allocated. If you then run the experiment at an allocation below 100%, random sampling of that subset can reintroduce imbalance (e.g., by chance, some high-impact units may fall disproportionately into one arm). For the period you care most about inference, prefer 100% allocation to preserve the intended balance. Lower allocations are best used briefly for safe rollouts rather than for the full experiment duration. * **Across candidate salts, is it the same set of users being evaluated?** * Yes. Candidate salts are assessed over the same targeted population; only the randomization induced by the salt changes. * **How long does stratification take?** * Duration depends on the number of units and the metric/source being queried. There is no fixed SLA; larger populations take longer. ## Manual assignment for Stratified Sampling When setting up an experiment, you can configure overrides (e.g. force user X or Segment A into Control, force user Y or Segment B into Test). This is meant for testing; overridden users are excluded from experimental analysis in Pulse results. If you do want manual assignment for stratified sampling, you should check the *Include Overrides in Pulse* checkbox. This will include the users you've manually overridden into each variant in all metric lift analyses. You can configure 100% of experiment participants into your test variants manually, or configure some subset of participants into variants manually and randomly assign the rest of your participants. While you can add overrides for an ID type that is different than the ID type of the experiment, those ID evaluations will not be resolved to the id type of the experiment and will not contribute to pulse results. When you use the Statsig SDK for assignment, it takes care of randomization. When you control assignment of users, you're responsible for making sure users are balanced across experiment groups. Manual assignment override configuration ## Additional reading [Morgan and Rubin 2012](https://projecteuclid.org/journals/annals-of-statistics/volume-40/issue-2/Rerandomization-to-improve-covariate-balance-in-experiments/10.1214/12-AOS1008.full) walks through the history, the philosophy, and the proofs of re-randomization, especially how re-randomization reduces the randomization variance of the difference in means. It's worth noting that "Standard asymptotic-based analysis procedures that do not take the re-randomization into account will be statistically conservative" was called out in the paper. However, to maintain consistent and comparable results across different methods, we stay conservative with the t-test. [Lin & Ding 2019](https://arxiv.org/abs/1906.11291) is another interesting read for your reference. # Targeting Source: https://docs.statsig.com/statsig-warehouse-native/features/targeting Configure targeting in Statsig Warehouse Native experiments to limit exposure to specific users, segments, or qualifying events from your warehouse. ## How to Target Experiments Experiments integrate natively with Statsig's Feature Gates product in order to target interventions. Feature gates provide a rich language for targeting users by properties or segments, and experiments can be given a targeting gate in order to only test an intervention on units which pass that gate. Read more about [Feature Gates](/feature-flags/overview). ## When to Target an Experiment Targeting an experiment makes sense when * You want to gradually release the experiment to tiers of users * Part of your hypothesis is that the experiment intervention will only work for a target subset of users, e.g. mobile users # Turbo Mode Source: https://docs.statsig.com/statsig-warehouse-native/features/turbo Use turbo mode in Statsig Warehouse Native to reduce warehouse compute costs by trading off pipeline frequency for lower query volume on your warehouse. By default, Statsig and other warehouse native platforms calculate cumulative results for every day in an experiment. This lets users take a historical analysis and get a rich historical view, or make changes and see how those changes "would have" impacted an analysis on previous days. This is a heavily optimized flow, but at the end of the day does take additional compute to achieve. Many customers want to run large experiments with minimal compute cost, so Turbo Mode was built to give control of the tradeoff between compute and historical tracking. ## How to use Turbo When loading pulse, there is a checkbox to enable Turbo mode. Subsequent scheduled loads use the last setting. Note that using Turbo Mode changes the shape of the underlying data, so switching back to standard analysis requires a full reload with Turbo Mode disabled. There is also a project-level setting for Turbo Mode being on or off by default in a project's experimentation settings. ## What Turbo Mode Removes Turbo removes two functionalities from Statsig. The first is the ability to click into the date picker and see pulse results "as of" various historical dates. The second - and associated - is the cumulative timeseries. These will both exist if an experiment is loaded daily using incremental reloads, but they may miss dates if there are gaps in the loading schedule, and they will not retroactively update when a full reload is run. Cumulative Timeseries ## What Turbo Mode Keeps Turbo mode and standard loads both calculate identical pulse results (including CUPED and other advanced techniques) for the latest day of the experiment analysis, and Turbo mode still calculates the "daily" and "days since exposure" timeseries for diagnosis, as well as pre-experiment bias checks. ## What to expect This will vary depending on an experiment's settings, but generally there is a 40-80% reduction in load time and compute cost for turbo jobs on large, long-running experiments. This is likely biased and is best used as a ballpark estimate, since this observation comes from cases where standard loads were slower than desired and Turbo Mode was applied in response; there is not a good counterfactual for companies who default to turbo being on. ## When to Use Turbo Turbo Mode is more effective for long-running experiments using full reloads, since the job would have to recreate the entire history of "user state" on each day. Since Turbo Mode only calculates statistics for the latest snapshot, it can skip many calculations. Turbo Mode is also more effective for experiments with an unusually large number of users or metrics compared to other experiments in a project, since it prevents the warehouse from spilling to disk by reducing memory requirements; spill dramatically slows down jobs and adds expense because of that. Of note, holdouts benefit from Turbo Mode since they typically have a large battery of metrics, expose a large portion of a project's end users, and are commonly run for 3-6 months. Turbo Mode will be less effective for experiments that have scheduled/incremental reloads, or for smaller experiments. # Types of Experiments Source: https://docs.statsig.com/statsig-warehouse-native/features/types-of-experiments Statsig Warehouse Native offers many forms of experiment analysis, including standard A/B tests, switchbacks, geotests, holdouts, and pre-post analyses. ## Analysis Only ### A/B/n Analysis-only A/B/n tests run analysis on top of Assignment and Metric data from your warehouse. In these experiments, Statsig operates as a statistics engine to help make analyses more reproducible. ## Assign and Analyze ### A/B/n The most common Assign-and-Analyze experiments are A/B/n tests that integrate warehouse data with Statsig's live SDK. In these experiments, Statsig operates as an assignment tool, a real-time diagnostics tool, and as a statistics engine. Assignment related features supported include: > #### [Stratified Sampling](/experiments-plus/stratified-sampling) > > Dividing your population into homogeneous groups (based on a metric or classification). > #### Configurable Allocation Duration > > You can enroll users for a subset of an experiment's duration. If you are experimenting on a one-time experience (e.g. signup flows), this just works. If it is not a one-time experience (enrolled users need to keep being assigned), you will need to configure your SDKs to use Persistent Assignment (you provide a store to save user's enrollment states; the SDK manages this state). Learn more about Persistent Assignment on [Client](/client/concepts/persistent_assignment) and [Server](/server/concepts/persistent_assignment) SDKs. ### Switchback Experiments Statsig offers [Switchback](/experiments-plus/switchback-tests) tests, which are a powerful way to experiment in the presence of meaningful network effects or for ecosystems where changing an experience for one group will leak outside of that group (e.g. a ride-service app changing prices for some users will change driver demand for all users). Switchback test are configurable in Statsig's console, and use time periods (as well as optional buckets, like, city or country) to change experiments, and run a bootstrapping analysis to estimate test statistics. This includes advanced options like burn-in/burn-out periods, allocation windows, and configurable window lengths. ### Geo Testing Statsig has launched [Geotesting](/experiments-plus/geotests) to support marketing and product causal inference techniques where you can't run a traditional A/B test. Geotesting treats geographic units (e.g. postal codes and DMAs) as your unit of analysis, and unlocks new experimental methodologies like rigorous testing of paid marketing and/or search on platforms like Google and Facebook Ads. Geotesting relies on Synthetic Control methodologies built using the best-in-class open source package GeoLift from Meta, combined with the ease and simplicity of the Statsig platform. All your existing metrics and metric sources are available to use with the addition of some geographical labels. ### MABs * [Autotune](/statsig-warehouse-native/features/autotune) is Statsig's multi-armed bandit solution, which balances explore and exploit to deliver the optimal global treatment to your users. This is a useful way to explore a high number of options and dynamically adjust traffic to avoid over-delivering underperforming variants. * Statsig also offers a [Contextual Multi Armed Bandit](/autotune/contextual/introduction), which extends the multi-armed bandit by personalizing which experience is served to users depending on "context", or user/event attributes provided to the Statsig SDK. This balances explore and exploit by optimizing for potential upside in its predictions. Using both MABs is as simple as calling Statsig's getExperiment, and providing relevant attributes to the user object for the CMAB approach. Please reach out to our support team, your sales contact, or in our [Slack community](https://statsig.com/slack), if you're interested in exploring Contextual Multi-Armed Bandits. # Understanding Experiments Source: https://docs.statsig.com/statsig-warehouse-native/features/understanding-experiments Understand how experiments are structured and run in Statsig Warehouse Native, including assignment, exposures, metric sources, and result analysis. Running statistical analysis for an experiment can be complex. Much of this complexity is abstracted away from the end user, but this commonly leads to questions - things like: * What do different date ranges mean? * What metric data is included in the calculation? This page aims to give an overview of the basic settings in Statsig, and what they mean for the analysis that will be run. For more advanced settings, refer to [experiment configuration](./experiment-options) ## Date Ranges The date range of an experiment controls the filters used when querying both exposure and metric data. For example, an experiment with a start date of `2025-02-01` and an end date of `2025-02-14` will query assignment data between those two dates, and also query metric data between those two dates. There's a few exceptions: * Metric Bake Windows or Cohort Windows with `wait for...` enabled; in this case, cohorts which have not been in the experiment for the duration of their cohort/bake window will be ignored and not included in either the exposure count or metric data for the metric in question * If `allow cohort metrics to bake after experiment end` is enabled. This will push out the end date of analysis artificially - only for cohort metrics - to maximize data collection for experiments with a one-time intervention where it's expected that the experiment effect persists even once the treatment is turned off. ### Exposure Timing A given experimental unit (e.g. user)'s data is only included *after* they trigger the experiment. For example, if you're testing a new notification, metric data from before a given user ever sees the notification should NOT be included in the experimental analysis. However, there’s an exception: if a metric source uses a date column as its timestamp, then metric data from the day of exposure will not be included in the calculation. This happens because a timestamp cast from a date defaults to the first second of that date (00:00:00), which falls before the exposure time. To address this, the metric source has a setting called `Treat timestamp as date`: * If checked: Statsig will include day-0 data by converting the join condition to use the casted date (ignoring time). * If unchecked: The join between exposure and metric will exclude day-0 data for the reason described above. Generally Statsig recommends including day-0 data (checking `measure timestamp as day`) if data has been preprocessed at a daily grain. ### Explore Query Dates Explore queries can filter metric data or assignment data independently. The default filtering option is metric data; this allows you to exclude certain dates with buggy or non-representative data, or to scope an analysis to recent periods - e.g. if your hypothesis is that the treatment effect will take a long time to develop. It is also possible, under the advanced tab, to: * filter to exposures in a date range (or outside of) * filter to exposures within a certain cohort (e.g. X-Y days since exposure) ### Selecting Dates Regardless of [turbo mode](/statsig-warehouse-native/features/turbo) when using incremental reloads, the date picker next to pulse can be used to view pulse "as of" a certain date. This is useful for understanding historical discussion of an experiment, or for understanding how results have evolved over time. This data can also be viewed through the lens of the time series views in Statsig. As a reminder, this ability should not be used as a way to cherry-pick dates with desirable results! ## What is being calculated? For every experiment analysis, the basic flow is: * Identifying when units first saw or were "enrolled" into the experiment * Identifying what those units did (metric data - events, or other rollups) after seeing the experimental variant they were assigned to, up until the end of the experiment. This results in a tagged dataset, where metric data is associated with a group * Aggregating that metric data over the experiment duration. The [metrics documentation](/statsig-warehouse-native/configuration/metrics) has descriptions and SQL snippets describing this step * Calculating group-level statistics to be used in the final scorecard analysis - at a high level, the analysis requires the observed totals/means per-group, and the variance which is used to understand how meaningful (usually statistical significance in frequentist analysis, or probability of best in bayesian analysis) that difference is * Means are normalized per-unit. For basic aggregations this will impute 0s - that is, the calculation is the total value over the count of units exposed. For ratios and means, the means will be computed as the sum of the numerator over the sum of the denominator. # Practical Use Cases Source: https://docs.statsig.com/statsig-warehouse-native/features/use-case Common Statsig Warehouse Native use cases, including A/B testing on top of an existing warehouse, observational analysis, and metric exploration. ## How can I define and filter results to 'new users' only? ### Scenario I want to filter the experiment results to new users only: A user is 'new' if they had never visited the website before they were exposed to this experiment. In other words, if the first time that they visited the website was after the exposure timestamp in this experiment, they were 'new users'; otherwise they were existing users. ### Solution Assuming you have a logging table which contains every time when a user visits the website and relative timestamp `logging`, and you have another table with all user's user\_id `user`. First, you need to create an Entity Property with the following logic: ```sql expandable theme={null} select distinct user_id, 'new' as new_user, timestamp('1900-01-01') as timestamp from user -- you can add filters here to include only targeted population only union all select distinct user_id, 'existing' as new_user, timestamp as timestamp from logging ``` Click Run Query and will return a table like this. Click Save Results, then you can directly use this Entity Property across all experiments/gates to filter or group experiment results by whether they are a new users or not. | user\_id | new\_user | timestamp | | -------- | --------- | ------------------------- | | A | new | 1900-01-01T00:00:00+00:00 | | A | existing | 2024-01-01T10:10:18+00:00 | | B | new | 1900-01-01T00:00:00+00:00 | *** ## How can I analyze logged-in metrics when my experiment exposures are at logged-out grain? ### Scenario I want to run an experiment to find out which version of my website design leads to a higher signup rate among new visitors. The experiment assignment will occur when a logged-out user visits my website, and they will be exposed to one of the design variants. When a user decides to sign up, a new and unique logged-in user ID will be generated for them. To calculate the conversion rate (CVR) accurately and consistently, I need a reliable way to map the logged-out IDs to the corresponding logged-in IDs. This will allow me to attribute each signup to the correct experiment variant and evaluate which design performs better in driving conversions. ### Solution Each suited to different business scenarios, two commonly used approaches for achieving the mapping supported by Statsig are: **- Strict 1:1 Mapping:** This approach only keeps records where there is a *unique, unambiguous* mapping between the logged-out ID and the logged-in ID. Any records with duplication (e.g., multiple logged-out IDs mapping to the same logged-in ID or vice versa) are discarded. It's recommended when accuracy and clarity are the top priorities for your experiments, and when data duplication is rare. **- First Touch Mapping:** For cases where a logged-in ID maps to multiple logged-out IDs, retain only the first association and discard the rest. It is better suited for scenarios where you want to preserve as much data as possible, and duplications are common in your business settings (e.g., users frequently access the website from multiple devices or sessions). For whichever mapping approach, you will need to create a mapping between the logged-out id and the logged-in id by either setting up an Entity Property with both IDs present or creating an exposure assignment source with columns for both ID types. #### Step 1 - Advanced settings for your experiments When setting up your experiment, under the *Setup* tab within your experiments, go to *Advanced Settings* and pick your Secondary ID type (the log-in userid in this scenario). Experiment setup advanced settings selecting secondary ID #### Step 2 - Identify the mapping mode that suits your need ID resolution mapping mode options #### Step 3 - Choose your Entity Property Source Entity property source selection for ID mapping * **\[Recommended]** Create a new Entity Property Source for your ID Resolution mapping if you haven't already. Go to Data -> Entity Properties. Follow the steps, you can either enter an existing table with the ID mappings or write a new query to create the mappings with the following logic: ```sql theme={null} SELECT stable_id, user_id, timestamp FROM id_mapping ``` * You can also choose **"None"** by using your assignment source for the mapping. Create your new assignment source by going to Data -> Assignment Sources. Note that this is not recommended because it can get more complex to manage if you start to increase the scale of your experiments. # Get Started with Geotests Source: https://docs.statsig.com/statsig-warehouse-native/geotests/geotests-setup Set up a geotest in Statsig Warehouse Native to measure the impact of a treatment across geographic regions when user-level randomization isn't possible. This guide walks you through creating a Geotest experiment using Statsig. The steps include: 1. Define your Geo Types 2. Configure your Metric Source 3. Create and Configure 4. Evaluate Design Options 5. Run Analysis ## Define your Geo Types First decide on one more geo types relevant to your business. Some of the most common are Postal Codes and DMAs (Designated Market Areas), but Statsig allows you to define any arbitrary geo type you'd like. Geo types configuration interface ## Configure your Metric Source Assuming you've already added a [metric source](/statsig-warehouse-native/configuration/metric-sources/) to Statsig, you can next indicate which column(s) represent the geo type(s) you've created. Geo column mapping configuration screen ## Create and Configure * Navigate to your Experiments page. * Click the + Create button. * In the modal, select Analyze to analyze existing experiment data in your warehouse. Experiment creation modal with analyze option * Enter a name and description for your experiment. * Choose Geotest as the experiment type from the dropdown. Fill in the remaining info as you usually would Geotest experiment type selection dropdown In the Setup tab: * Define your hypothesis. * Set the Geo Type (e.g., country\_id) from one of the types you've defined in your project and metric sources * Choose your Primary Metrics. **Important Metric Types** GeoTest metrics are currently limited to SUM, COUNT, and COUNT\_DISTINCT types. This limitation is driven by the inference techniques involved in synthetic control methodologies. Geotest experiment setup configuration screen Configure: * Expected Treatment Start/End Dates * Pre-treatment duration (days) * Desired Effect Size % (this is the MDE you're hoping to measure; smaller MDEs will make finding a valid design harder) * Type of Test: 1-sided or 2-sided * Alpha level (desired type I error rate, inverse of Significance) * Optionally enable Budgeting * Additional advanced modeling parameters for users familiar with [Geolift's Advanced API](https://github.com/facebookincubator/GeoLift/blob/main/R/pre_test_power.R) Experiment configuration settings interface ## Generate Design Options When you're happy with your initial settings, click "Use Experiment Designer" to begin to design process. Experiment designer launch button The Experiment Designer automates the process of evaluating potential splits of geos. Creating a new design set involves 3 steps: 1. **Define your targeting** 2. **Set any Inclusion or Exclusion overrides** 3. **Define your date range** Geographic limitations configuration interface ### Define Targeting Targeting allows you to define the set of geos potentially eligible to be included in your experiment. By default it pulls the unique set of geo types from your metric source from the last 90 days. You can add additional custom filters that help you filter your geos down, based on columns in your metric source. For example, you can filter geos to those in a certain region or by transactions that occurred in a certain language. Geotest targeting filters for eligible regions You can also manually specify a set of geos yourself. The geo id types must match those in your metric source exactly. ### Set Inclusion or Exclusion Overrides After defining your targeting set, you can manually overwrite any of these geos into treatment or control groups as desired. Override interface assigning specific geos to treatment or control For example, when testing if a new marketing campaign is effective, you might know that you want include the city of Austin because a contract has already been signed. And you might know that you do not want to launch in New York, Los Angeles, or Washington D.C. because of local regulations. You can specify these constraints on the designer and it will ensure that all design options considered will conform to these rules. ### Define Date Range Lastly, you must define the data range of your design dataset. This date range defines what data gets pulled as historical baseline to train a synthetic control model. In general, a good rule of thump is that you design data duration should be at least 4x the expected duration of your treatment period in order to achieve good fits and maximize sensitivity and power of your experiments. ### Advanced Options * **Rolling Lookback Windows:** Control how many iterative simulations should be done to evaluate power. For instance, a value equal to 5 would simulate power for the last five possible tests in the date range. Corresponds to `lookback_window` in the GeoLift API. * **Dynamic Time Warping:** Control how much the synthetic control model depends on auto-correlations in the metric vs. correlations between different geos on the same days. A value of 1 focuses exclusively on the metric while a value of 0 (default) relies on correlations only. Corresponds to `dtw` in the GeoLift API. * **Control Cell Size Requirements:** Sets limits on the share of primary metric value from geos in any considered control group. If not set, all design options will be analyzed regardless of their size. For example, if designing on a Revenue metric, a control size range of \[50%, 90%] would mean that only design options where 50% to 90% of revenue was in the designated control geos. Corresponds to `holdout` in the GeoLift API. ## Evaluate Design Options Once a Design Option Set has been created (usually takes a few minutes), it will be shown in your Experiment Designs tab. All design option sets generated are shown here, enabling you to look through prior design iterations as needed. Experiment designer options display Click into a completed design set (e.g. “Design Set 3”). * Review the ranked recommendations for your best design options * Compare MDE, Power, Cost and Control/Test Allocation * Click View Cell Details to expand and see specific geo assignments * See additional model performance details * Select your desired design using the radio button, and clicking **Save Design to Experiment**. Design option selection interface ## Run Analysis Geotest analysis results dashboard # Geotests Source: https://docs.statsig.com/statsig-warehouse-native/geotests/introduction Introduction to geotests in Statsig Warehouse Native for measuring marketing and operational treatments across geographic regions without user-level testing. Sometimes you just can't run an A/B test at a per-user level. Some common causes are: * You don't have per-user control over who sees what, e.g. a third-party marketing platform like Meta or Google Ads. * You can't attribute metrics to individuals, e.g. total store foot traffic can't be tied to which users received your email campaign. Geotests help solve these problems. One of the most common use cases is marketing; you can't A/B test on 3rd party ad platforms, but geo-testing lets you evaluate the incrementality of your program. Geotests are specialized experiments that allow you to target users based on geographic location. These tests enable you to test features, content, or experiences across any type of geographic area you specify. By splitting your campaign up amongst DMAs or postal codes, for example, you can measure its incrementality even when an ads platform doesn't support it. Geotests are particularly valuable for businesses with global audiences who need to optimize their product for regional differences in user needs, preferences, or regulatory requirements. # Why use Geotests Statsig's Geotesting is designed to combine the flexibility and statistical rigor of statsig with your own data sources in your warehouse. Geotesting: 1. Uses the data already in your warehouse, avoiding expensive and time-consuming exports. 2. Includes an automated Experiment Designer to help you select where and when to run your campaign. 3. Help export your campaign definition to your campaign platform, making setup faster and less error-prone. 4. Automates analysis of the campaign, providing statistical rigor to measure incrementality. # GeoLift Statsig's Geotesting is built on top of Geolift, a best-in-class industry tool for running geospatial experimentation for marketing. [GeoLift](https://facebookincubator.github.io/GeoLift/) is an open-source package from Meta that’s used by many to infer causal relationships in timeseries data. It builds on advancements made in prior packages like Google’s [Causal Impact](https://google.github.io/CausalImpact/CausalImpact.html) package. Statsig is using GeoLift in our own implementation of Geotesting, and allowing you to easily connect your own internal metrics and feed them into the analysis package. # References * [GeoLift](https://facebookincubator.github.io/GeoLift/) * [CausalImpact](https://google.github.io/CausalImpact/CausalImpact.html) * [Synthetic Control Methods (wiki)](https://en.wikipedia.org/wiki/Synthetic_control_method) * [Causal inference using synthetic controls (medium)](https://medium.com/data-science-at-microsoft/causal-inference-using-synthetic-controls-d96a890c83a7) # Geotesting Methodology Source: https://docs.statsig.com/statsig-warehouse-native/geotests/methodology The methodology behind geotests in Statsig Warehouse Native, including synthetic control modeling, region selection, and treatment effect estimation. ## How it's different Standard A/B testing relies on a few core assumptions: 1. You can reliably and randomly split your users into similar groups 2. While each group has heterogeneity inside it, random user-level variations average out at larger sample sizes, resulting in homologous (similar) groups 3. You control how each group gets treated, and there is no interference between groups When these core assumptions are true, we assume the treatment effect will be the only observable difference between the groups. We can simply measure each group’s mean value and find the difference between them. There are times however when these assumptions don’t hold: * Your intended treatment can’t be controlled and scoped to the user level * You don’t know who the users are * You can’t track outcomes at the user level Running geographically-based campaigns is a prime example that often features at least some of these violations. Examples include: * Some classic marketing, like billboards, can’t be randomly split between test and control users at a given location. We can’t track who sees them and how they act afterwards. * Some digital campaigns might be able to split users 50/50 (if the platform allows it), but if you don't have user-level data that you can tie to outcomes of interest, you'll never know how the two groups performed. ## Using Synthetic Controls “Geotesting” as we refer to it here is an experimental framework that relies on a different basic setup than AB tests: 1. Split your users into geographical boundaries at some useful level (zip codes, states, countries, etc) 2. Apply a known treatment in some “treatment” geos, and no treatment in some “control” geos 3. Use the control geos to figure out what would have happened in the treatment geos had no treatment been applied (i.e. “synthetic” results) 4. Measure the delta in the treatment geos between the actual and synthetic values The “figuring out what would have happened” tends to be the hardest and nastiest part of this setup. Deducing things that don’t occur is usually a fool’s errand, but in some circumstances there’s enough predictability in a system to figure it out. Geotesting relies on the idea that among some sets of geos there are enough correlations between control and test geos that make predicting this “counterfactual” possible. We assume, for instance, that general trends and seasonality tend to be shared across some geos. Example: pretty much all zip codes in the United States exhibit similar quantities of mail sent on any given day. If you were to split zip codes 50/50, you could reasonably infer that the two groups would see similar trends. # Approach There are many ways to estimate unknown data from other known data, depending on a multitude of factors. In the case of Geotesting, we have: * timeseries data that is autocorrelated (i.e. data is correlated from one day to the next within a group) * timeseries data that is correlated between groups * regular seasonal patterns that repeat at regular intervals * granular data from among many distinct geographies Plotting some metric of interest for a bunch of relevant geographies can show these relationships clearly. Example data from the GeoLift package walkthrough. Example data from the GeoLift package walkthrough. But what happens when we run a campaign or make some change in just some of these geos? How can we tell if a fluctuation in this graph is random noise or real stat-sig result? A marketing campaign that starts on day 91 (black dotted line) could have affected the treatment cities of Chicago and Portland. But how can you tell any real effect from all the noise? A marketing campaign that starts on day 91 (black dotted line) could have affected the treatment cities of Chicago and Portland. But how do you distinguish any real effect from all the noise? We turn to causal inference modeling to answer this question. Thankfully, a variety of powerful models and methods have been developed to answer this question. The basic steps are the following: * Split your geos into test and control groups * Using the pre-treatment period data, train a model on the control data to predict how the treatment geos behave * Use the post-treatment period data for the control geos, create a “synthetic control” dataset to predict how the treatment geos would have behaved after the treatment event. * Subtract the synthetic data from the actual observed data to determine the induced effects of your treatment on the treatment geos When your training data is able to produce strong models that can predict metric outcome well, you end up with strong estimates of an induced effect: Geotest period split diagram showing pre-treatment and post-treatment phases Subtracting the modeled Synthetic Control values from the observed Treatment values reveals any incremental effect for the Treatment geos. Subtracting the modeled Synthetic Control values from the observed Treatment values reveals any incremental effect for the Treatment geos. # Synthetic A/A Test Source: https://docs.statsig.com/statsig-warehouse-native/guides/aatest Run an A/A test in Statsig Warehouse Native to validate your experimentation setup and confirm metrics behave correctly before running real A/B tests. To create a quick A/A test, you can randomly split existing users you already have events for. Create an assignment source using the script below - and you;ll be ready to analyze your A/A test in minutes and see Pulse scorecards light up. Example script to use - ```sql theme={null} SELECT user_id, timestamp, 'AA_Test_1' AS experiment_name, --CAST('AA_Test_1' as varchar) AS experiment_name for Redshift warehouse CASE WHEN THEN 'Control' ELSE 'Test' END AS GroupAssignment FROM ``` Replace `` with the following based your warehouse: * Bigquery: `mod(abs(farm_fingerprint(cast(user_id as string))), 100) < 50` * Redshift: `mod(abs(farmFingerprint64(cast(user_id as varchar))), 100) < 50` * Snowflake: `mod(abs(hash(cast(user_id as string))), 100) < 50` * Databricks: `mod(abs(hash(cast(user_id as string))), 100) < 50` * Athena: `mod(abs(cast(conv(substr(md5(cast(user_id as varchar)), 1, 16), 16, 10) as bigint)), 100) < 50` # Data Best Practices Source: https://docs.statsig.com/statsig-warehouse-native/guides/best-practices Best practices for using Statsig Warehouse Native in your data warehouse, covering metric design, pipeline scheduling, cost control, and experiment hygiene. ## Cost Management Statsig's pipelines leverage many SQL best practices in order to reduce cost, and smaller customers can run month-long analyses for a few pennies. Following these best practices will help keep costs under control and consistent. See [Compute Cost Transparency](/statsig-warehouse-native/guides/best-practices#compute-cost-transparency) to understand how much compute time your experiments use. ### Follow SQL Best Practices Statsig uses your SQL to connect to your source data. Here's some common issues: * Avoid using `SELECT *`, and only select the columns you'll need * `SELECT *` leads to a lack of clarity in what you are pulling/require for other users * For warehouse like Bigquery and Snowflake, it can increase the scan/size of materialized assets like CTEs in snowflake * This leads to higher query runtime and higher warehouse bills * Filter to the data you will need in your base query * This reduces the scan and amount of data required in future operations * You can use Statsig Macros (below) to dynamically prune date partitions in subqueries * Group common metric groups into a single Metric Source * Statsig will filter your source to the minimal set of data and then create materializations of experiment-tagged data. To maximize the effectiveness of this strategy, have metric sources that cover common suites of metrics that are usually pulled together. * Cluster/Partition tables to reduce query scope * Most warehouses offer some form of partitioning or indexing along frequently filtered keys. In warehouses with clustering, we recommend the following strategy: * Partition/Cluster Assignment Source tables by date, and then your experiment\_id column so experiment-level queries can be scoped to just that experiment * Partition/Cluster Metric Source tables based on date, and then the fields you expect to use for filters These best practices are generally true across all warehouses, especially as the datasets you use for experimentation scale. ### Materialize Tables/Views Since Statsig offers a flexible and robust metric creation flow, it's common for people to write joins or other complex queries in a metric source. As tables and experimental velocity scale, these joins can become expensive since they will be run across every experiment analysis. To reduce the impact of this, best practice is to: * Materialize the results of the join and reference that in Statsig, so you only have to compute the join once * Use Statsig macros to make sure partitions are pruned before the join, and you only join the data you need ### Use Incremental Reloads Statsig offers both Full and Incremental reloads. Incremental reloads will only process new data, and can be significantly cheaper on long-running experiments. A number of advanced settings in the pulse advanced settings section (and available at an org-level default) will help you make tradeoffs to reduce the total compute cost. For example "Only calculate the latest day of experiment results" skips timeseries, but can run large full reloads (e.g. 1-year on 100M users) in 5 minutes on a Large snowflake cluster. ### Reload Data Ad-Hoc Depending on your warehouse and data size, Statsig Pulse results can be available in as little as 45 seconds. Since there's flat costs to pipelines, reloading 5 days is not 5 times as expensive as loading one day. If cost is a high concern, being judicious about which results you schedule vs. load on-demand can significantly reduce the amount of processing your warehouse has to do. ### Leverage Statsig's Advanced Options By default, Statsig runs a thorough analysis including historical timeseries and context on your results. ### Utilize Metric Level Reloads Statsig offers Metric-level reloads; this allows you to add a new metric to an experiment and get its entire history, or restate a single metric after its definition has changed. This is cheaper than a full reload for experiments with many metrics, and is an easy way to check guardrails or analyze follow-up questions post-hoc. ### Use Statsig's Macros In Metric and Assignment sources, you can use Statsig Macros to directly inject a DATE() type which will be relative to the experiment period being loaded. * `{statsig_start_date}` * `{statsig_end_date}` For example, in an incremental reload from `2023-09-01` to `2023-09-03`, this query: ```sql theme={null} SELECT user_id, event, ts, dt FROM log_table WHERE dt BETWEEN {statsig_start_date} AND {statsig_end_date} ``` resolves to ```sql theme={null} SELECT user_id, event, ts, dt FROM log_table WHERE dt BETWEEN DATE('2023-09-01') AND DATE('2023-09-03') ``` This is a powerful tool since you can inject filters into queries with joins or CTEs and be confident that the initial scan will be pruned. ### Avoid Contention Resource contention is a common problem for Data Engineering teams. Usually, there will be large runs in the morning to calculate the previous day's data or reload tables. On warehouses that have flat resources or scaling limits, Pulse queries can be significantly slower during these windows, and additionally will slow down core business logic pipelines. The best practice is to assign a scoped resource to Statsig's service user. This has a few advantages: * Costs are easy to understand, since anything billed to that resource is attributable to Statsig * You can control the max spend by controlling the size of the resource, and independently scale the resource as your experimentation velocity increases * Statsig jobs will not effect your production jobs, and vice versa If this is not possible, it's a good idea to: * Schedule your Statsig runs after your main runs - this also ensures the data in your experiment analysis is fresh * Use API triggers to launch Statsig analyses after the main run is finished ## Analytics Optimization When using Statsig’s Metric Explorer to visualize the data within your warehouse, optimizing table layout and clustering configurations can greatly improve latency. This section serves to describe a set of best practices you can employ to improve the performance of analytics queries. Here are recommendations for some of the most commonly used warehouses. ### BigQuery #### Table Layout - Partitioning & Clustering We advise partitioning on event date and clustering on event when defining your events table. This will improve performance as the majority of queries will filter for the name of the event and the time it was logged. When defining the partition on event date, you should truncate the timestamp to day-level granularity instead of using the raw timestamp (which would otherwise have millisecond precision resulting in very high cardinality). ```sql theme={null} -- Create an events table partitioned by date and clustered by event. CREATE TABLE dataset.events ( ts TIMESTAMP NOT NULL, event STRING NOT NULL, ... ) PARTITION BY DATE(ts) CLUSTER BY event; ``` BigQuery's support for applying a cluster to an existing table is limited. Additionally, adding a cluster to an existing table will not automatically recluster the data right away. Given this, if you need to repartition on event date or add a cluster by event, you can create a new table with the correct partitions and clusters using your current table. ```sql theme={null} -- Using an existing events table, create a new table that is partitioned by date and clustered by event. CREATE OR REPLACE TABLE dataset.events_new PARTITION BY DATE(ts) CLUSTER BY event AS SELECT * FROM dataset.events; -- (Optional) Swap the name of your new table with the old one for consistency. DROP TABLE dataset.events; ALTER TABLE dataset.events_new RENAME TO events; ``` ### Databricks #### (Preferred) Use Liquid Clustering When making clustering decisions in your events table layout, [liquid clustering](https://docs.databricks.com/aws/en/delta/clustering) provides a flexible approach that allows you to modify your clustering keys without needing to manually rewrite existing data. We recommend employing liquid clustering for your events table with the following: ```sql theme={null} -- Enable liquid clustering for your events table. ALTER TABLE events SET TBLPROPERTIES ('delta.liquidClustering.enabled' = 'true'); -- Cluster on the event column. ALTER TABLE events ALTER CLUSTER BY (event); -- Trigger clustering using the OPTIMIZE command. OPTIMIZE events; ``` If your events table is frequently being updated, Databricks recommends scheduling `OPTIMIZE` jobs every 1-2 hours. This will incrementally apply liquid clustering to your table. #### (Alternative) Partitioning and ZORDER If you choose not to use liquid partitioning, our recommendation is to partition on a single low cardinality column such as the date of the event. Try to avoid adding more than one column on the partition unless absolutely necessary and don't partition on a column that has a cardinality that would exceed one thousand. We suggest using a generated column to simplify pruning: ```sql theme={null} -- Define a partition on the event date generated column. CREATE TABLE events (ts TIMESTAMP, event STRING, event_date DATE GENERATED ALWAYS AS (CAST(ts AS DATE))) USING DELTA PARTITIONED BY (event_date); ``` You can use `ZORDER` to colocate similar values within a file for a high cardinality column, which benefits query performance by improving data skipping. We recommend doing this on the event column: ```sql theme={null} -- ZORDER by the event column to improve data skipping. OPTIMIZE events ZORDER BY (event); ``` Using a scheduled job that runs `OPTIMIZE` on the last week's event data, you can improve query performance by compacting the number of small data files into fewer, larger files: ```sql theme={null} -- Periodically optimize the events table based on the last week of data. OPTIMIZE events WHERE event_date >= current_date() - INTERVAL 7 DAYS; ``` In general, avoid partitioning on a combination of event date and event. For even small queries, this can create a lot of overhead. Given 500 events, this would cause a 30-day query to hit 15,000 partitions. Instead, partition only by date as described above and `ZORDER` on the event. If extra parallelism is desired without introducing too many partitions, you can `ZORDER` by an additional bucket column. This bucket column can be defined as follows: ```sql theme={null} -- Add bucket column for extra parallelism. ALTER TABLE events ADD COLUMN event_bucket INT GENERATED ALWAYS AS (pmod(hash(event), 16)); -- Rewrite data into new files based on this ZORDER key. OPTIMIZE events ZORDER BY (event_bucket); ``` #### Partition Pruning Dynamic File Pruning will enable Spark to prune partitions based on filter values at runtime. It should be enabled if possible using: ```sql theme={null} -- Turn on Dynamic File Pruning. SET spark.databricks.optimizer.dynamicFilePruning = true; ``` #### Handling Skew and Joins To rebalance skew and adjust join strategy, we advise turning on Adaptive Query Execution: ```sql theme={null} -- Turn on Adapative Query Execution SET spark.sql.adaptive.enabled = true; ``` #### Compute Choices The Photon query engine allows for faster execution of queries with more efficient use of CPU and memory. If possible, it should be enabled for your compute cluster or SQL warehouse. ### Redshift #### Table Design - Distribution Style and Sort Key As Redshift does not support partitioning by column, we can instead employ use of a sort key. Given most event tables are generally append-only and time-based, we advise use of a compound sort key on (timestamp, event) so that the data is ordered by time. Given all analytics queries will filter on timestamp, this should allow for queries to read only the relevant blocks. Ordering secondarily by event will mean rows are grouped by event within each time block, which can reduce scan size when filtering by event. We want pruning to be primarily upon timestamp, which is why it is kept as the first key. Given there can be significant skew among event types, we generally advise using an automatic distribution style rather than distributing upon the event column. This will allow Redshift to make distribution decisions based on the size of the table and query patterns. You can create an events table with the above recommendations in mind as follows: ```sql theme={null} -- Create an events table with an automatic distrbution style and sort key on timestamp, event. CREATE TABLE events ( event VARCHAR NOT NULL, ts TIMESTAMP NOT NULL, ... ) DISTSTYLE AUTO COMPOUND SORTKEY (ts, event); ``` If you already have an events table and want to leverage the recommended distribution style and sort key, you won't be able to apply those changes by modifying the existing table. Instead, you can create a new table and copy over data: ```sql theme={null} -- Create a new events table with the preferred distribution style and sort key. CREATE TABLE events_new ( event VARCHAR NOT NULL, ts TIMESTAMP NOT NULL, ... ) DISTSTYLE AUTO COMPOUND SORTKEY (ts, event); -- Copy over data from the old events table. INSERT INTO events_new (event, ts, ...) SELECT event_id, ts, ... FROM events; -- Rename the two tables in a single transaction. BEGIN; ALTER TABLE events RENAME TO events_old; ALTER TABLE events_new RENAME TO events; COMMIT; -- Drop the old table. DROP TABLE events_old; ``` ### Snowflake #### Clustering Keys Given Snowflake does not allow for explicit partitioning, we recommend using clustering keys on the event date and event to improve performance as the majority of queries will filter for the name of the event and the time it was logged. Note that we advise clustering on the timestamp truncated to day-level granularity instead of the raw timestamp (which would otherwise have millisecond precision causing very high cardinality). ```sql theme={null} -- Cluster events on the timestamp truncated to day and event. ALTER TABLE events CLUSTER BY (DATE_TRUNC('day', ts), event); ``` If you have a high cardinality of unique event types, it might be advisable to add a search optimization on event instead of clustering by that column. See the [next section](#managing-high-cardinality-columns) for details. #### Managing High Cardinality Columns If there are high cardinality columns that you frequently reference in Metrics Explorer filters, consider using search optimization rather than clustering by it. This avoids an overly large range of values for each micro-partition generated by Snowflake, which would make pruning inefficient. For high cardinality columns, we can instead rely on the auxiliary search index generated through search optimization. ```sql theme={null} -- Add search optimization for fast lookups. ALTER TABLE events ADD SEARCH OPTIMIZATION ON EQUALITY(some_column); ``` #### Monitoring Clustering You can use `SYSTEM$CLUSTERING_INFORMATION` to check if your current clustering scheme is effective. Large values of average\_depth and average\_overlaps may indicate the existing table should be reclustered on different keys with lower cardinality. ```sql theme={null} -- Given an events table clustered on event date and event, check the clustering information. SELECT SYSTEM$CLUSTERING_INFORMATION('YOUR_DATABASE.YOUR_SCHEMA.EVENTS', '(DATE_TRUNC(''day'', ts), event)'); ``` #### Timestamp Column If possible, use the `TIMESTAMP_NTZ` (no timezone) type for your timestamp column. This saves on space as the offset metadata will not need to be stored. It additionally can help speed up queries as it allows Snowflake to avoid the extra work for timezone normalization internally, which would otherwise take place during filtering and pruning of micro-partitions. ### Questions? If you need additional support in optimizing your warehouse configuration for analytics, please reach out in the Slack support channel for your organization within Statsig Connect. ## Debugging Statsig shows you all the SQL being run, and any errors that occur. Generally these are caused by changing underlying tables or Metric Sources, causing a metric query to fail. Here's some best practices for debugging Statsig Queries. ### Use the Queries from your Statsig console If a pulse load fails, you can find all of the SQL queries and associated error messages in the Diagnostics tab. You can easily click the copy button to run/debug the query within your warehouse. Look out for common errors: * A query attempting to access a field that no longer exists on a table * A table not existing - usually due to the Statsig user not having permission on a new table ## Turbo Mode [Turbo Mode](../features/turbo) skips some enrichment calculations (in particular some time series rollups) in order to very cheaply compute the latest snapshot of your data. With this, customers have run experiments on 150+ million users in less than 5 minutes on a snowflake S cluster. ### Ask! Statsig's support is very responsive, and will be happy to help you fix your issue and build tools to prevent it in the future - whether it's due to system or user error. ## Compute Cost Transparency Statsig Warehouse Native now lets you get a birds eye view across the compute time experiment analysis incurs in your warehouse. Break this down by experiment, metric source or type of query to find what to optimize. Common customers we've designed the dashboard to be able to address include What Metric Sources take the most compute time (useful to focus optimization effort here; see tips here) What is the split of compute time between full loads vs incremental loads vs custom queries? How is compute time distributed across experiments? (useful to make sure value realized and compute costs incurred are roughly aligned) You can find this dashboard in the Left Nav under Analytics -> Dashboards -> Pipeline Overview Pipeline Overview dashboard interface This is built using Statsig Product Analytics - you can customize any of these charts, or build new ones yourself. A favorite is to add in your average compute cost, so you can turn slot time per experiment into \$ cost per experiment. At the end of every Pulse load / DAG, we'll upload a single row to the `pipeline_overview` table for each job executed as part of that run. This table has the following schema: | Column | Type | Description | | ---------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------ | | ts | timestamp | Timestamp at which the DAG was created. | | job\_type | string | Job type (see [Pipeline Overview](https://docs.statsig.com/statsig-warehouse-native/pipeline-overview/)) | | metric\_source\_id | string | Only applicable for 'Unit-Day Calculations' jobs - the ID of the metric source | | assignment\_source\_id | string | The Assignment Source ID of the experiment for which Pulse was loaded. | | job\_status | string | The final state of the job (`fail` or `success`) | | metrics | string | Metrics processed by the job | | dag\_state | string | Final state of the DAG (`success`, `partial_failure`, or `failure`) | | dag\_type | string | Type of DAG (`full`, `incremental`, `metric`, `power`, `custom_query`, `autotune`, `assignment_source`, `stratified_sampling`) | | experiment\_id | string | ID of the experiment for which Pulse was loaded, if applicable | | dag\_start\_ds | string | Start of the date range being loaded | | dag\_end\_ds | string | End of the date range being loaded | | wall\_time | number | Total time elapsed between DAG start and finish, in milliseconds | | turbo\_mode | boolean | Whether the DAG was run in Turbo Mode | | dag\_id | string | Internal identifier for the DAG | | dag\_duration | number | Number of days in the date range being loaded | | is\_scheduled | boolean | Whether the DAG was triggered by a scheduled run | # Setup Checklist Source: https://docs.statsig.com/statsig-warehouse-native/guides/checklist Onboarding checklist for Statsig Warehouse Native, covering warehouse connection, metric setup, assignment sources, and your first experiment launch. After you've connected your warehouse and set up both metrics and assignment sources, you can ensure your setup is complete and correct by checking the following items: 1. Primary keys 2. Timestamps 3. Duplication 4. Data availability Once you've completed these checks, your offline results should align with those in the Statsig console when advanced features are disabled. ## 1. Primary keys When setting up an experiment, you can select the unit of assignment, acting as the primary key to join the assignment with metrics. The assignment source and the metrics source must use the same primary key. In an Analyze-Only experiment, this primary key can be selected from the unit IDs defined by your assignment source. * Ensure the unit ID in your assignment source matches the unit ID in your metrics source. In an Assign and Analyze experiment, the primary key (unit ID) is generated by the Statsig SDK. * You can verify this unit ID in the statsig\_forwarded\_exposures table within the assignment sources. * You must either forward the unit ID to the SDK ([docs](/client/introduction)) or utilize the SDK to manage your features and correspondingly generate the metrics table. ## 2. Timestamps It is important to analyze metric data only after a user has been exposed to the experiment. Pre-experiment data should have no average treatment effect, and therefore its inclusion dilutes results. Statsig employs a timestamp-based join for this purpose, with an option for a date-based join for daily data. This should look like the SQL snippet below: ```sql expandable theme={null} WITH metrics as (...), exposures as (...), joined_data as ( SELECT exposures.unit_id, exposures.experiment_id, exposures.group_id, metrics.timestamp, metrics.value FROM exposures JOIN metrics ON ( exposures.unit_id = metrics.unit_id AND metrics.timestamp >= exposures.first_timestamp ) ) SELECT group_id, SUM(value) as value FROM joined_data GROUP BY group_id; ``` ## 3. Exposure duplication Exposure data must be de-duplicated before joining to ensure a single record per user. Many vendors further manage crossover users (users present in more than one experiment group), removing them from analysis and/or alerting them if this occurs with high frequency. ```sql expandable theme={null} SELECT unit_id, experiment_id, MIN(timestamp) as first_timestamp, COUNT(distinct group_id) as groups FROM GROUP BY unit_id, experiment_id, group_id HAVING COUNT(distinct group_id) = 1; ``` ## 4. Data availability When comparing a platform analysis to an **existing** experiment analysis that may have been run in the past, it's possible that the underlying data has since fallen out of retention or has been otherwise deleted. To check this, you can compare the table's retention policy to the analysis dates used in your original experiment analysis to make sure the data still exists. # Make sure results match After completing the above four steps, your offline analysis should produce results that match those in the Statsig console. Note that the Statsig Console includes several advanced features, such as [winsorization](/stats-engine/methodologies/winsorization#winsorization-statsig-whn), [CUPED](/stats-engine/variance_reduction#cuped), and employs [the delta method](/stats-engine/variance#ratio-and-mean-metrics) to address ratio metrics. We recommend disabling these features initially when comparing results. [This article](https://www.statsig.com/blog/how-to-analyze-an-experiment-from-databricks-tables) provides an example of conducting offline calculations in Databricks. If you have additional questions, just send us a Slack message. We are always here to help. # Connect Your Warehouse Source: https://docs.statsig.com/statsig-warehouse-native/guides/connect Guide to connecting your data warehouse to Statsig Warehouse Native, including supported warehouses, required permissions, and validation steps. Warehouse Native is part of Statsig's Enterprise tier. Please [contact us](https://statsig.com/contact/demo) to get started To run analysis on your warehouse, Statsig needs to connect to your warehouse via a service user. You control the access this user gets to your warehouse. In general, Statsig will require: * Read access to metric data and exposures data * Write access to an isolated Statsig Staging database for writing results and exporting data * Access to run jobs and queries ## Choose Your Warehouse The following warehouses/tools are currently supported in Statsig Warehouse Native: * [Snowflake](/statsig-warehouse-native/connecting-your-warehouse/snowflake) * [Athena](/statsig-warehouse-native/connecting-your-warehouse/athena) * [Bigquery](/statsig-warehouse-native/connecting-your-warehouse/bigquery) * [Databricks](/statsig-warehouse-native/connecting-your-warehouse/databricks) * [Redshift](/statsig-warehouse-native/connecting-your-warehouse/redshift) # Warehouse Costs Source: https://docs.statsig.com/statsig-warehouse-native/guides/costs How to manage warehouse compute costs with Statsig Warehouse Native, including reload cadence, turbo mode, query optimization, and pipeline scheduling tips. This page is a high level summary of how to think about Warehouse Costs and Total Cost of Ownership on Statsig Warehouse Native. There's some duplication with the [best practices](/statsig-warehouse-native/guides/best-practices) page; treat that page an implementation guide and this page as a high-level overview. ## Why do Costs Matter? Warehouse Native platforms require you to provide compute and storage to run data jobs like experiment analyses and analytics queries. This will be a cost center, and can meaningfully drive up your warehouse bill if queries are unoptimized or poorly integrated with your databases. This is a major concern for many customers considering a Warehouse Native solution for Experimentation or Analytics. Warehouse Native platforms are very flexible, but that flexibility can lead to unexpected data costs. Understanding total cost of ownership can be tricky, but is an important part of evaluating a platform as many platforms don't treat this as a primary focus. Statsig started as a cloud-only platform, and it's in our DNA to care about the costs of running our platform. From continuously optimizing pipelines on the cloud side of our business, we've also developed expertise with strategies that can be employed to save costs and try to pass that on to our Warehouse Native customers. ## Benchmarks ### Total Cost of Ownership Generally, we see that warehouse costs hover between 5% and 20% of customers' platform spend with Statsig. There are exceptions depending on usage, but these tend to be on the low end. To avoid cherry picking, the examples below are an anonymized list of the top 5 Statsig Warehouse Native customers on Snowflake, by total warehouse spend: | Company Profile | Experiments Run, 12 Months | Estimated % Contract Value Spent on Compute | | ------------------------------------------------------------------------- | -------------------------- | ------------------------------------------- | | Multi-sided marketplace (e.g. job search / contracts), 100M-1B in Revenue | 120 | 12% | | Multinational B2C business, 5B-20B in revenue | 210 | 9% | | B2B SaaS, 100M-1B in Revenue | 250 | 11% | | Online Entertainment, 100M-1B in Revenue | 180 | 6% | | Online Services, 50-100M in Revenue | 130 | 20% | This is fairly representative of what you can expect using Statsig at scale and with reasonable adherence to best practices. Note that costs do scale with # experiments; companies running less experiments will generally see a smaller relative TCO. Relative to Experiment Size, you will see a great deal of variation in spend depending on how many metrics you use, what kind of metrics, and how "dense" your metrics are, but generally we see numbers like the below for cost when teams follow best practices: | Experiment Size | Cost per Experiment Load | Lifetime Cost per Experiment (4 Weeks with Daily Refreshes & Drilldowns) | Lifetime Cost Using [Turbo Mode](../features/turbo) | | ------------------------------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------ | --------------------------------------------------- | | Small (50k or less units) | \< \$0.05 | \$0.25 - \$2.00 | \$0.25 - \$1.00 | | Medium | \< \$0.25 | \$1.00 - \$10.00 | \$1.00 - \$5.00 | | Large | \$0.25 - \$10.00 | \$5.00 - \$280.00 | \$3.00 - \$50.00 | | Extremely Large (500 Million+ experimental units) | Low Sample with only a handful of companies - Can't Generalize | " | " | Note that [Turbo Mode](../features/turbo) is especially efficient for large-scale experiments; [Turbo Mode](../features/turbo) dramatically reduces memory usage, keeping large jobs from spilling to disk and reducing the need for XL clusters to provide enough memory. ### Industry Statsig prides itself on being best-in-class when it comes to Warehouse Native costs for experimentation. Customers who evaluate multiple vendors consistently report that Statsig is 50-66% of the cost of other platforms. In cases where Statsig has been more expensive than competitors, it's considered a large issue and is quickly root-caused; for example, a recent customer accidentally used un-partitioned data for their Statsig PoC, driving up cost relative to another vendor where the tables were partitioned, leading to higher costs. We followed up to build warnings when source data is not partitioned, and after partitioning we went from 120% of the other vendor's cost to \<60%. Multiple customers have migrated to Statsig from other Warehouse Native Experimentation platforms due to concerns about cost on their current platform; please reach out to our Sales team if you're interested in running an evaluation of relative costs. If you have concerns, or a separate party claims their solution is more efficient, please reach out to us and we'll be happy to discuss. ## Tools for Understanding Cost Statsig tries to be as transparent as possible about costs. Numerous tools throughout the product help in this regard: ### History In every loaded experiment/gate, you can quickly see the cost of each load. This is an easy inline way to understand the rough cost of a load. The metrics shown will be relevant to your warehouse - e.g. slot hours or bytes scanned for bigquery, uptime for snowflake, etc. Pipeline History ### Pipeline Overview Dashboard Statsig automatically creates a `Pipeline Overview` dashboard in its MEx platform. The underlying data is written to your warehouse, so you can do deep dives and try to understand what is driving costs from Statsig. Many customers have built their own monitoring and alerting on top of this dataset and will flag to Statsig if they see any changes in cost. Pipeline Overview ### Vendor Tooling It's recommended to use standalone compute/storage resources for Statsig in your warehouse. This makes understanding the total cost of ownership trivial; while Statsig does its best to estimate costs, some info is invisible to the platform without administrator rights on your Warehouse. If there's a mismatch between Statsig's dashboard and your costs, always reach out to support! We take this seriously and will work to resolve the gap. ## Tools for Managing Cost ### Follow UI Hints Statsig will proactively flag to you if: * A data source is particularly slow * A metric source is malformed (e.g. a view definition which is broken within your warehouse) * A data source is not following best practices Paying attention to these and flagging them quickly prevents wasted compute. In most cases where PoC customers had performance issues, they boiled down to issues like their temp tables for the PoC not being partitioned, or not using macros to filter tables with extremely long data retention. ### Access Controls For large organizations, most cost issues are caused by poorly-optimized source tables, or using the wrong source when pulling data. Statsig offers access controls so that: * Your data team can control data sources and ensure that they are well-behaved * Verified sources are clear in the console for end users to use. ### Following Best Practices The [best practices](/statsig-warehouse-native/guides/best-practices) cover most situations that lead to high warehouse costs on Statsig. In short, you should aim to: * Cluster or Partition source tables * Use statsig date macros to guarantee filter push-down * Use Statsig's [Turbo Mode](../features/turbo) to reduce redundant compute, especially for large or long experiments # Warehouse Native Debugging Guide Source: https://docs.statsig.com/statsig-warehouse-native/guides/debugging Debug Statsig Warehouse Native experiment configurations and queries using query tools, pipeline logs, and metric source previews in the console. ## Debugging When you're interacting with complex data sources, it's common to run into strange artifacts or results that don't make sense. Statsig will proactively monitor for these and let you know if something looks off. This guide is to help with the next steps of identifying what went wrong, and for reconciling differences in results between Statsig's analysis and your own. ## Common Issues ### Mismatched IDs In many cases, you may have multiple versions of the same ID across tables. For example, the user `abc123` in one source might be `USER_abc123` in another; or, in some cases, certain loggers will hash ID values for privacy reasons. This will usually result in Statsig triggering an alert that it was unable to join data between sources. What we recommend doing is: * Going to the relevant sources and running the sample queries to check for obvious ID mismatches * Failing that, find the job which isn't able to join sources and copy SQL. By working through the SQL query you should be able to pinpoint where the mismatch is occurring ### Conflicting Filters Statsig allows a high degree of customization in explore queries and on explore pages. This can lead to scenarios where you add two conflicting filters that, together, will never pass. For example: * You have a metric with a cohort window from 7 to 13 days, but set up your experiment analysis to run on users' first 6 days. These two filters return no users. * You create a count metric filtered to `event='purchase`', and then create a local metric that filters to `event='checkout'`. This set of filters will also return a null set. Like the above, we recommend going to the Unit-Level Aggregations jobs (where filters are rendered) and checking the filter by searching for the metric name of interest in the SQL code. ## Common Points of Confusion Many data scientists run their own analyses using the Statsig staging data in their warehouse. In some cases, they can see very different results. There's a few common methodology differences that tend to explain this: ### Join conditions Statsig joins events/metrics to exposures based on the event or metric input data occurring after the user was exposed to the experiment that's being analyzed. If you check the Treat Timestamp as Date setting, this is done by comparing dates. By default, it uses a timestamp. Not having a time filter, or using a Date when Statsig uses a timestamp (or vice versa) can yield very different results in some cases. ### Winsorization/Capping Metrics can be configured to be capped or winsorized in Statsig. If you have extreme outliers, you will see significant differences in results if this procedure is not carried out. You can perform this procedure yourself, or create a local metric (or clone the metric) without winsorization. ### CUPED CUPED can greatly reduce variance, and for aggregation (non-ratio) metrics can shift the group-level mean estimates if there's pre-experiment bias. You can turn this off in the scorecard any time to view the non-cuped results for comparison. ### Metric Types In some cases, users have assumed that their input data is binomial - that is, the metric will only be 1-or-0 at a unit/user level. If this isn't the case, and the metric is configured as a count or sum, it can lead to much higher variance than expected. # Running Email Experiments Source: https://docs.statsig.com/statsig-warehouse-native/guides/email-experiments Run email experiments in Statsig Warehouse Native by mapping send and open events from your warehouse to assignment sources and metric definitions. Email marketing experimentation is a common use-case that can be handled elegantly by using Statsig Warehouse Native to analyze the downstream outcomes that occurred as a result of an email experiment. Throughout this guide, we will detail the various considerations in achieving an optimal configuration for running this type of experiment. ### Data Access offered by your marketing platform > Does the platform offer the ability to ETL email engagement events (sends, deliveries, email opens) to your data warehouse either directly, or via a CDP or other pipeline? This aspect may be critical to the viability and quality of your experiment, depending on your specific use-case. In particular, `email open` events are very useful for controlling the population of users that are counted for Analysis in Statsig. If you send 20,000 emails, and only a portion of those get delivered and opened (*e.g.*, 12,000), you will avoid diluting your population and achieve more statistical power by counting *only* the 12,000 email open users in the test population rather than the entire recipient list. You can use the email open events to either handcraft an assignment source for an `Analyze Only` experiment or to [filter your analysis population](/statsig-warehouse-native/configuration/qualifying-events/#filter-by-qualifying-events) for an `Analyze Only` experiment. ### Configuring your scorecard metrics > What are you key down-funnel (*e.g.*, on-site or in-app) success metrics for the experiment, and are you able to unify the email recipients with their subsequent down-funnel interactions? Email Open events can be useful both for filtering your test population or simply for the monitoring purposes, but is not typically a primary metric unless your experiment is testing out different subject lines. You should consider which down-funnel metrics should be included as primary metrics for making on a decision on your experiment. Attributing down-funnel interactions that occur prior to user login may be tricky, but can be solved by setting up [ID resolution in Statsig](/statsig-warehouse-native/features/id-resolution/). Down-funnel metrics that occur post-login will be simple to attribute back to the experiment, given you will have their user ID and email address which will map directly to the identifiers on your assignment source. These will work without much configuration. ### Designing your test groups and running the test > What are the options for defining your Test and Control groups natively within your email platform? Does the platform require you to upload a list of each test and group users, or does it natively allow you to define A/B cohorts in some way? If your platform offers native A/B split capabilities, this is ideal and will reduce the amount of work you'll need to do. If this is the case, it's best to configure your test as an `Analyze Only` experiment in Statsig. Under this model, you will hit the "Analyze" button and keep the "Experiment Completed" checkbox *unchecked*. This will allow us to continue computing Results starting at the first exposure time indefinitely, until you've made the decision to conclude your experiment. If you need to handcraft the test groups for one reason or another, you should set up an `Assign and Analyze` experiment and use Statsig's SDKs to generate your test and control lists by looping over all of the desired recipients in code and calling `getExperiment`. This will both (a) allow you to capture each test group assignment and (b) automatically write the assignment events to your data warehouse (via the [Statsig managed exposure pipeline](/statsig-warehouse-native/guides/forwarded-data/)). To do so, you will first need to hit the "Start" button, prior to running your assignments script, and then keep the test in a running state until you've made the decision to conclude your experiment. # Bootstrapping Your Experimentation Program Source: https://docs.statsig.com/statsig-warehouse-native/guides/experimentation-program Best practices for running an experimentation program on Statsig Warehouse Native, including org design, metric hygiene, review processes, and tooling. In this guide, we'll outline a straightforward and successful approach to starting an experimentation program using Statsig at your company. Best of all, you can get started completely free with Statsig’s free tier, which includes up to 5M events per month and no limit on the number of team members. ## 1. Generate Product Ideas Generating ideas for experimentation is crucial to running a successful program. There are various methods for brainstorming ideas, such as dogfooding your product, studying competitors, analyzing existing metrics, and gathering UX research and user feedback. Here are three common frameworks we use at Statsig, which are also popular among our customers: ### Generate Ideas Based on Your Business Context 1. **Turn Every Upcoming Feature into an Experiment**: * This is one of the simplest and most natural methods. At Statsig, we often put every new feature behind a feature flag. Statsig’s [Feature Flags](/feature-flags/overview) (also called "gates") automatically convert a feature rollout into an A/B test, measuring the impact on your key metrics as the rollout progresses. If your engineering team already uses feature flags for releases, this is the easiest path to your first A/B test. 2. **Work Backwards from Company Goals**: * Many of our customers begin by aligning experiments with strategic company goals. For example, if the goal is to drive adoption growth, break down the goal into actionable metrics (e.g., total revenue, monthly active users) and run experiments that can move those metrics incrementally. Instead of targeting broad goals like revenue, focus on specific, actionable metrics like checkout completions or user onboarding time. 3. **Break Down Your Business Levers**: * Decompose key drivers of your business to reveal more granular areas where you can experiment. For example, an e-commerce business might focus on increasing the checkout rate, which could lead to experiments on improving the CTA, reducing friction, or building urgency. Business levers breakdown diagram ### Prioritize Your Ideas Use a simple return-on-investment (RoI) approach to prioritize your experimentation ideas: * Choose ideas that are **simple to execute**, **easy to measure**, and **high in potential**. * Start with ideas that are easy to implement. Once you’ve run a few experiments, focus on higher-impact or more controversial ones. Generating a prioritized list of ideas is more than half the work in running a successful experimentation program. **(Optional) Create a Backlog** You can create an experimentation backlog from your list of ideas. As running experiments often generates more ideas, don't overthink it—consider it a "nice to have" for future testing. ## 2. Run a Simple A/B Test ### Clarify Your Hypothesis Start by formulating a hypothesis for one of your high-priority ideas. A well-framed hypothesis defines the expected outcomes of your experiment, making it easier to align with stakeholders and evaluate results objectively. A hypothesis consists of three key elements: **Action**, **Predicted Outcome**, and **Rationale**. Here’s an example: | Element | Example | | --------------------- | --------------------------------------------------------------------------------- | | **Action** | If we reduce the number of fields on the sign-up page | | **Predicted Outcome** | Then the percentage of users that complete sign-up will increase | | **Rationale** | Because users will spend less time and effort, leading to higher completion rates | ### Create Your Experiment Statsig makes it simple to create an A/B test. In the [Statsig Console](https://console.statsig.com/login): 1. Name your experiment. 2. Define the variants you want to test. 3. Set the percentage of users eligible to participate. ### Set Up Your Application for the Experiment To start the experiment: 1. Use a Statsig SDK to assign users to experiment variants. 2. Validate assignments through the **Diagnostics** tab in the Statsig console. Here's a [quick start guide](/guides/abn-tests) to help you set this up. To compute results, you can: * Log events directly from your application. * Pipe events to Statsig via a service like Segment. * Configure Statsig to ingest data from your warehouse. ### Validate Your Experiment Configuration Once your application is set up: * You can validate it immediately through the Statsig console, which provides a live exposure stream as users encounter different variants. * Statsig will flag issues, such as missing identifiers or mismatches between exposure and application events, ensuring you won’t miss any results. Once validated, hit **Start** to begin the experiment. **(Optional) Set a Target End Date** Set a target end date using Statsig's power calculator. This ensures that decisions are made only after the experiment has gathered enough data for reliable results. **Running Experiments in Registration Funnels** If you don't have user IDs for users who haven't signed up yet, create an experiment using a **stableID**, which Statsig SDKs auto-generate. This will act as the unit of analysis. **Selecting Key Metrics** When setting up an experiment, configure the key metrics to evaluate your hypothesis. Statsig will compute results for all your metrics, but focusing on key metrics helps avoid cherry-picking. ### Organize Regular Experiment Reviews Experiment reviews provide a structured environment to discuss results, share insights, and align on decisions. Run weekly or bi-weekly reviews to reinforce a culture of data-driven decision-making and experimentation rigor. ## 3. Share Your Results Sharing your results widely is key to building momentum for experimentation. Here are a few ways to share insights across your organization: * **Monthly Experimentation Update**: Summarize the number of experiments run, their business impact, and key learnings. * **Slack #experimentation Channel**: Use a dedicated channel to post real-time experiment learnings and foster discussion. * **Weekly Experiment Review**: Open these reviews to a broader audience to increase visibility and foster engagement. ## 4. Accelerate Your Speed of Experimentation ### Cement the Value of Experimentation As you gain confidence, try balancing ideas with more uncertain outcomes. Counterintuitive results can solidify the value of experimentation and lead to new discoveries. ### Increase Experimentation with Self-Serve Experiments Statsig’s results are trustworthy and easy to interpret, making it possible for any team to run experiments. Lower the barrier to entry for experimentation by educating team members on how to create and run their own experiments. ### Build Incentives to Run More Experiments Encourage engineers and teams to prioritize impact over shipping new features. Reward them for experimentation outcomes rather than the volume of shipped features, ensuring a focus on end-user value. ## 5. Build a Culture of Learning ### Accelerate Learning > “If you double the number of experiments you do per year, you're going to double your inventiveness.” > — Jeff Bezos, Founder of Amazon Frequent experimentation leads to faster learning, helping teams discover user preferences and make data-driven product decisions. Even inconclusive results can prompt deeper analysis and more informed hypotheses. ### Measure the Impact of Your Experimentation Program Evaluate your program’s success using key criteria such as: | Success Criteria | Score | Weight | | ---------------------------------------------------------- | ----- | ------ | | Time spent to set up an experiment | | | | Time spent to prepare experiment results | | | | Percentage of decisions made using trustworthy data | | | | Number of experiments run per week | | | | Quality of experiments run | | | | Time spent indexing/searching past results and discussions | | | Use these criteria to score your team's progress, set goals, and demonstrate the productivity boost from a well-implemented experimentation program. # How to Run a Playground Evaluation Source: https://docs.statsig.com/statsig-warehouse-native/guides/playground_eval Use the Statsig Warehouse Native playground to evaluate metric and experiment configurations with sample data before committing to production pipelines. Playground evaluations are only recommended for Enterprise-tier customers. Many customers evaluating Statsig have sensitive data & long lead times to test on production data due to privacy and data security process. Statsig offers a read-only [Demo Project](https://console.statsig.com/whn_demo), but this doesn't allow you to really experience the warehouse native product. To solve this, you can reach out to support to ask for access to a Playground Evaluation project. This is a project hosted by Statsig with fake, near-real time data sources that can be used to set up experiments, analyze metrics, and get a feel for what it's like to use Statsig and to get internal feedback. You'll be an admin, and can add teammates as desired to explore the shared project together. ## Playground Projects Playground projects come with some boilerplate connections hooked up: * A real-time event streaming metric source * A daily user-level metric source * An assignment source with a handful of experiments * A handful of pre-loaded metrics and experiments Playground projects do not come with any assign-and-analyze experiments, so some features that require SDK usage (e.g. Stratified Sampling) won't be available. A normal demo project can be used with an SDK on a local script to generate data for those use cases. ## Evaluation We recommend following steps 2+ from the [quick start](/statsig-warehouse-native/guides/quick-start) guide, starting with the tables available to the playground project. This will give you a sense of what it takes to run experiments and manage metrics in Statsig. Once that's done, refer to the [POC guide](/statsig-warehouse-native/guides/running_a_poc) and simulate the steps that are relevant to your evaluation. Customers have also found it useful to explore the Pulse UI together - the Statsig team is always happy to chat about any questions or feedback. # Warehouse Native Quickstart Source: https://docs.statsig.com/statsig-warehouse-native/guides/quick-start Quick start guide for Statsig Warehouse Native: connect your warehouse, define a metric source, run an A/A test, and analyze your first results. You can get experiment results in record time with Statsig Warehouse Native. This page walks you through connecting your data, configuring a metric, and getting experiment results. All you'll need is a table in your warehouse that has metric or event logging data. ## Step 1: Connect Your Warehouse Statsig will use your warehouse to store and analyze your experiment data; you have total control and visibility over the data itself. To connect your warehouse, visit your warehouse's setup page. * [Snowflake](/statsig-warehouse-native/connecting-your-warehouse/snowflake) * [Athena](/statsig-warehouse-native/connecting-your-warehouse/athena) * [BigQuery](/statsig-warehouse-native/connecting-your-warehouse/bigquery) * [Databricks](/statsig-warehouse-native/connecting-your-warehouse/databricks) * [Redshift](/statsig-warehouse-native/connecting-your-warehouse/redshift) ## Step 2: Connect to Data To connect your event or metric, data, you'll create a [Metric Source](/statsig-warehouse-native/configuration/metric-sources). Navigate to the Metric Sources page and click Create to make a new Metric Source. Create metric source button If you have a table, use the `Table` metric source type and put in the path to your table. Otherwise, you can write a query to access or generate some test data. Press analyze to generate samples from your table, and then map required columns (Timestamp and UserID) so that Statsig can connect your metric data to your assignment data. Metric source configuration mapping timestamp and user ID Save your changes, and you've connected to your data! ## Step 3: Make a Metric Now that you've connected to data, you can build metrics on top of this. Later, this can be configured programmatically, but for now navigate to your Metrics Catalog and click Create to make a new Metric. Metrics catalog create button Point to the metric source you just configured, name your metric, and press Create. This will take you to the new Metric's page, where you can configure your metric. To get started, we recommend just making a [count metric](/statsig-warehouse-native/metrics/count). This is the simplest kind of metric, which just counts the number of rows in your metric source. This is useful for event logging - for example, you might filter to a "purchase" event to count the number of times users purchased an item. Select "Count" and save - or, feel free to pause here and explore the options here. ## Step 4: Connect an Experiment Next, you'll connect to experiment data. If you have a table with exposures you've already logged, feel free to use that. You'll just need to make sure that you've logged the same identifier there as you used in your metric source. Otherwise, you can quickly follow our guide to setting up an [A/A test](/guides/aa-test), using the same data you used for your exposures. This should generate a neutral experiment result. Navigate to Assignment Sources and create a new one. Create assignment source page If using an AA test, follow the instructions to randomly assign users to groups. If you're using existing assignments, write a query to pull the data from your logs and map the unit, group, experiment, and timestamp columns. Assignment source query configuration Pressing save and scan will save your new source and detect experiments that exist on the source. This should only take seconds to a few minutes, and once it's done you can scroll down to look through the experiments Statsig found. Detected experiments table ## Step 5: Analyze Your Experiment Press Create on your experiment of interest to start creating your experiment. Experiment creation wizard Add a display name and hypothesis, and press Create. This will take you to the final setup step, where you specify the metrics for your experiment. Choose the metric you created in step 3). Statsig will automatically detect the group split, but if the detected split is incorrect you can manually adjust it to the intended value. Experiment setup selecting metrics and group split Press Save and Analyze, and Statsig will start calculating Pulse Results. You can track the progress in the loading bar at the bottom of the experiment's results page. ## Step 6: Read Results If everything worked, you should see: * Your hypothesis. This lives at the top of the results page to give context and guide interpretation of the results. * Cumulative exposures. This shows you the number of unique units exposed to each group, and the balance between groups * Your scorecard. This shows a quick summary of the observed differences in metrics between your experiment groups, with access to additional views and raw statistics Experiment results showing hypothesis and exposures There's a lot of features here - please explore the product and the docs to learn more! * Click into a result's error bar to view raw statistics, timeseries, and projected [timeline impact](/stats-engine/topline-impact) * Hover over a metric to get detailed context on its inputs and how the pulse result was calculated * Navigate to the diagnostics tab to see the [checks](/statsig-warehouse-native/features/monitor-an-experiment) Statsig automatically ran to make sure your experiment results were valid * Click into the reloaded timestamp to see the run time and query cost of your pulse analysis, as well as the SQL queries used to calculate the pulse results * Visit the explore tab to start filtering data, exploring results by dimensions, or running other follow-up analyses * Visit the summary tab to start putting together a [report](/statsig-warehouse-native/features/reports) to share out the results of your analysis * Start a discussion or add context, either in the discussion tab or with in-context comments on top of the results themselves Note that our experiment was not an AA test, and there was an experimental impact. In this case, we have a statistically significant result, with an estimated lift of +12.89% ±0.93% from our control group to our test group. We can also click into see additional details, like the expected change to our overall topline metric value if we were to ship this experiment. Detailed metric lift view with significance Congratulations - you've completed the Quick Start guide! # Running a Warehouse Native POC Source: https://docs.statsig.com/statsig-warehouse-native/guides/running_a_poc Plan a Statsig Warehouse Native proof of concept by understanding the solution components, key validation steps, and how to move from POC to production. ## Introduction **Statsig Warehouse Native** enables customers with existing metric logs to quickly run analysis on their existing metric data, and optionally bring previous assignment data/offline experiments into the platform quickly. Statsig WHN has two types of experiments:
    1. **Assign and Analyze**: You can run an experiment on **web/mobile/app** and use Statsig’s SDKs to assign (bucket or randomize) users, and then analyze results.
    2. **Analyze**: You can run an experiment elsewhere (***your own SDK, email, direct mail, sms, ivr, etc.***) and use Statsig to analyze that data and calculate experiment results. This assignment data can be read from your warehouse in [this format](/statsig-warehouse-native/configuration/assignment-sources#example-data) - we call these [Assignment Sources](/statsig-warehouse-native/configuration/assignment-sources).
    If you have a pre-existing experiment in your warehouse, we recommend getting started first with an **Analyze** experiment. This is an effective and quick way (less than 1 day) to get comfortable with establishing a connection between Statsig and your warehouse, and the experience of consuming [experiment results](/pulse/read-pulse) in the Statsig console. Then, we recommend running a **Assign and Analyze** experiment using Statsig's SDKs; typically an A/A test. With the A/A **Assign and Analyze** experiment, you can test Statsig's SDKs and implementation process with your engineering and product team. # Steps to running an effective Proof of Concept with Warehouse Native Warehouse Native POC workflow diagram Keep these high level steps in mind as you begin your planning your Warehouse Native implementation: 1. **Define your experiment(s) and metrics for validation** - Ultimately a proof of concept will determine if Statsig fits your experimentation needs thus running an experiment with Statsig is the quickest path for evaluation. — **Responsible Party**: Typically a product or engineering lead * Plan to run 1-2 production level experiments to validate. Past experiments, A/A tests or upcoming projects or product changes are great opportunities to implement a Statsig experiment! * Identify your hypothesis and [metrics](/statsig-warehouse-native/configuration/metrics) which will validate this hypothesis. These metrics will be joined with unit assignment/exposure data and run through the stats engine. * If your team plans on running **analysis only**, identify the user **assignment** data which will be joined with the metric data. * This approach can yield results for analysis in as little as **30 minutes,** assuming data is readily available for ingestion * If your team plans on utilizing the **Assign and Analyze** experimentation option, you’ll want to identify **where** the experiment will run. Typically **web based** experiments are easier to evaluate, however Statsig has SDK support for server and mobile SDKs as well. * **Note**: It’s important the implementing team understands how the SDKs operate prior to executing a proof of concept. Our [client](/client/introduction) and [server](/server/introduction) docs can help orient your team! * A typical evaluation takes **2-4 weeks** to account for experiment design, implementation, time to bake, and analysis. To ensure a successful POC, [have a well scoped plan](/guides/running-a-poc#2-phase-0-scope--prepare-your-poc) and ensure the right teams are included to assist along the way. * Read [experimentation best practices](https://statsig.com/blog/product-experimentation-best-practices) to get an idea of how to best succeed. 2. **Connect the Warehouse** - In order to query data and operate within your warehouse, you’ll need to allocate resources and connect to Statsig. You may choose to utilize an existing prod database or create a separate cluster specifically for experimentation (if you don’t already have one). * Statsig requires a role and the following access: * Read access to metric and exposure data * Write access so results and exposures can be written back to the warehouse * Access to run jobs and query data * Find more guidance on [connecting with your specific warehouse vendor here](/statsig-warehouse-native/guides/connect). * Review the [data pipeline overview here](/statsig-warehouse-native/pipeline-overview) to see how data flows for warehouse native jobs. 3. **Connect Metric Sources & Define Metrics** - Once the data warehouse has been connected, you can begin defining metric and assignment sources (if applicable) to Statsig. Our systems expect specific schemas in order to correctly map the data to our pipelines: Schema requirements diagram for metric sources Beyond these columns, the schema is flexible and can accept additional columns. **Metadata** can be used to filter metrics and also be utilized for more granular analysis. * Review guides for creating a [metric](/statsig-warehouse-native/configuration/metric-sources) and [assignment](/statsig-warehouse-native/configuration/assignment-sources) source * Follow our [data best practices](/statsig-warehouse-native/guides/best-practices) to ensure your queries are running efficiently. This section is important to review and can prevent unnecessary infrastructure costs! After metric sources have been connected, [metrics](/statsig-warehouse-native/configuration/metrics) are configured to perform various aggregations (E.g. Sum, Mean, Count, Unique Users) that represent what you’re trying to measure in your experiments. * [Supported metric types](/statsig-warehouse-native/configuration/metrics#metric-types) and ways to [configure them](/statsig-warehouse-native/configuration/metrics) * [Cohort metrics](/statsig-warehouse-native/features/cohort-metrics) can be used to measure impact during a certain time frame per user 4. **Create and Rollout an Experiment** - In step 1, you defined the planned experiment(s) and the metrics used to validate them. With metrics and assignment sources configured, the experiment(s) can now be created. A more detailed guide for experiment setup can be found [here](/statsig-warehouse-native/guides/experiments) but consider these things as you complete this step: * Create your hypothesis and select the **experiment (assignment) source** * If using the SDK for assignment, the SDK itself will be the assignment source * [Custom IDs](/guides/experiment-on-custom-id-types) can be used but they must first be configured - ex: device\_id, vehicleId, etc * Check out [advanced settings](/statsig-warehouse-native/features/configure-an-experiment#advanced-settings) to see the many ways you can configure your experiment * As you rollout your experiment, you can [monitor the status with health checks](/statsig-warehouse-native/features/monitor-an-experiment) and get a readout of live exposures as they come through the SDK. * If you’re hoping to quickly validate the platform, you can create and run a [quick A/A test](statsig-warehouse-native/guides/aatest.mdx). 5. **Read Results** - Once the experiment has been successfully run, it’s important to read the results and ensure everything looks reasonable. Was your hypothesis validated or are the results surprising? Are the results easy to interpret and navigate for the teams involved? Check out our [section on pulse](/statsig-warehouse-native/guides/pulse) to get an idea of the high level analytics capabilities. A few things to note here: * Results can be sliced further via the [explore tab](/statsig-warehouse-native/guides/pulse#explore) and enables you to break down results by specific user and event properties * Exposure and metric data can be configured to be forwarded to your warehouse * The [Health Checks (diagnostics) tab](/statsig-warehouse-native/guides/pulse#health-checks) surfaces the SQL used to generate results so you can validate any analysis performed on your systems. 6. **Finalize Evaluation and Next Steps -** Ultimately a POC is meant to validate a set of evaluation criteria that will determine whether or not Statsig is a good fit for your team’s workflows. The following graphic provides high-level guidance on what to look for during your evaluation phase. POC evaluation checklist graphic Should you decide to move forward, the next step becomes converting your POC environment to a production-level implementation. We have created [this guide](/guides/production) to give you a general sense of what that entails. Need assistance? We’re here to help! Statsig support is available via our [community slack channel](https://statsig.com/slack). # Working With the SDK Source: https://docs.statsig.com/statsig-warehouse-native/guides/sdks Use Statsig SDKs alongside Statsig Warehouse Native to assign users to experiments client-side while analyzing results from your data warehouse. Warehouse Native Works with any of Statsig's SDKs for logging events and getting feature flags or experiment assignments. This page is a brief overview of how Warehouse Native works with Statsig's SDKs. Refer to the [client](../../client/introduction) or [server](../../server/introduction) SDK docs for help setting up SDKs. ## Data Forwarding When you first set up your data connection, Statsig will create tables to forward datasets to, and generate an assignment/metric sources which includes any user-level metadata fields you log as part of your evaluation. You can find the configuration for these datasets and the table we create/output data to in the advanced section of the warehouse connection page. Choose Groups ## Exposures Statsig calculates deduplicated first-exposure rollups for you on a daily basis and exports that miniaturized dataset to your warehouse. Additionally, on every Pulse load on the first day of a experiment, deduplicated exposures for the day will be exported to your warehouse in near real-time, up to 1 million exposures. ## Log Events You can also use Statsig's powerful event logging to send events to the Statsig SDK and have them later exported to your warehouse for analysis. # View SQL Source: https://docs.statsig.com/statsig-warehouse-native/guides/sql Reference for the SQL dialects and conventions used by Statsig Warehouse Native, including supported functions and warehouse-specific syntax. Statsig Warehouse Native runs SQL in your warehouse to generate the experiment results, and the queries are fully transparent and made visible in console. This means that you can trace any results you see on the Statsig console back to its calculation, artifacts, and raw events. The [pipeline-overview](/statsig-warehouse-native/pipeline-overview) provides an overview of how the queries are orchestrated to produce the experiment results. If you are curious about the details or want to debug, you can find the experiment-specific SQL by going to the experiment’s Pulse view -> view history -> view details on a specific pulse load. You’ll be able to see all the jobs that went into that pulse load. Pulse view SQL query details interface There are a few queries with \[Health Check] as a prefix that automatically check the health of the experiment. You can view [this page](/statsig-warehouse-native/features/monitor-an-experiment) to understand what they do. For experiment result calculation queries, below are one-sentence summaries of what each query does. * First exposure: Generate the first exposure time of this experiment at the unit level, given the assignment source. * Exposure summary: Summarize the first exposures at the group level and generate cumulative exposures. * User-level calculations: Calculate unit-level metrics at unit \_ metric \_ day grain. * Windowed rollups: Rollup unit-level metrics at different time windows. * Plus calculation: Generate the sample parameters that are necessary for calculating treatment effects, such as units, total, mean, standard deviation, population variance, covariance, etc. # About Warehouse Native Source: https://docs.statsig.com/statsig-warehouse-native/introduction Introduction to Statsig Warehouse Native, a deployment model that runs experiment analysis directly on your data warehouse for privacy and control. Warehouse Native is part of Statsig's Enterprise tier. Please [contact us](https://statsig.com/contact/demo) to get started Statsig Warehouse Native is an enterprise-grade experimentation platform that runs analysis in your data warehouse. It integrates easily with your existing datasets and any source of experiment assignment data – including a powerful integration with Statsig's SDKs and real-time logging infrastructure. Warehouse Native shares core features with Statsig Cloud but focuses on specific scenarios around running experiments on top of your warehouse. ### Use Cases for Statsig Warehouse Native * End-to-end experimentation platform covering targeting, experiment setup and assignment, and analysis. * Features include Feature Flagging, Automated/Protected Rollouts, Native Holdouts, Mutual Exclusion, and comprehensive analysis tools. * Modern statistical engine for running analyses on existing experiments from third-party or internal systems. * Statsig offers a full suite of experiment measurement tools, including CUPED, Stratified Sampling, Switchback Tests, and more. If you’re unsure whether Statsig Warehouse Native or Statsig Cloud is a better fit for your experimentation needs, reach out to our support team, your sales contact, or in our [Slack community](https://statsig.com/slack), and we'll help you decide. ## How Warehouse Native Works Statsig Warehouse Native runs experimentation compute jobs directly in your data warehouse, using your existing datasets to calculate metrics and enrich experiment analysis based on your data. Warehouse Native data flow architecture diagram ### SDK Integration Using [Statsig SDKs](/sdks/getting-started) with Statsig Warehouse Native is similar to integrating with Statsig Cloud. Here’s a quick overview of the process: 1. **Set Up Targeting and Experiments**: Create and manage experiments using the Statsig console. 2. **Initialize the SDK**: Integrate the Statsig SDK on your client or server-side applications. 3. **Targeting and Assignment**: Call the Statsig SDK to assign users to variants. * Optionally provide a logging callback to store logs in your warehouse or use Statsig's real-time infrastructure for instant diagnostics and safe rollouts. The resulting assignment and (optional) event logging data ends up in your warehouse, where you can connect it to other datasets for analysis. If you log to Statsig, data is exported on-demand for real-time analysis or in scheduled batch jobs. ### Experiment Setup Running experiments on Warehouse Native involves several key steps: 1. **[Connect Statsig to Your Warehouse](/statsig-warehouse-native/guides/quick-start#step-1-connect-your-warehouse)**: Integrate Statsig with your data warehouse to access relevant datasets. 2. **Create Metrics**: Define the input data and configure experiment metrics. 3. **Log Exposures**: Log experiment exposures with Statsig or point to existing assignments in your warehouse. 4. **State Your Hypothesis**: Formulate a hypothesis and choose scorecard metrics to evaluate the experiment. 5. **Run Pulse Analysis**: Execute a Pulse analysis to test your hypotheses and measure the impact on scorecard metrics. ### Advanced Analysis Tools Statsig’s [data analysis tools](/pulse/read-pulse) run directly in your data warehouse. All queries, intermediate datasets, and final results generated by Statsig will be available in your warehouse for auditing and custom analysis. * **Pulse Analysis**: Pulse helps you test your hypotheses and measure the impact of changes on your scorecard metrics. * **Metrics Explorer**: Visualize experiment metrics for your entire population and drill down into user behavior. Metrics Explorer is part of a broader suite of product observability tools, including Session Replay, advanced filtering, and custom dashboards. * **Exposure Analysis**: Analyze your experiment's first exposures to debug and gain insights on your experiments in Metrics Explorer. Drill down into certain properties or understand how your experiments interact with one another. ## Next Steps We love helping people use Statsig. Don't hesitate to reach out to our support team, your sales contact, or in our [Slack community](https://statsig.com/slack). Check our getting started guides to start running or analyzing an experiment in minutes: Connect Statsig to your warehouse and get test results in minutes Evaluate how Statsig Warehouse Native can work for you Learn about Statsig's modern Stats Engine *** # Composite Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/composite Composite metrics in Statsig Warehouse Native combine two or more aggregated metric sources by addition or subtraction to produce a single result. ### Use Cases Composite metrics are a flexible metric type designed to sum aggregated metric sources at the user level. Use them when you need to add or subtract two or more aggregrated values into a single signal, such as: * Net value (e.g., revenue minus refunds) * Unit-level experiment change (latestvalue minus first value) ## Calculation At the unit level, composite metrics first compute the aggregations of each component. The aggregated results are then added or subtracted according to the formula specified (for example: `A + B` or `A - B + C`). This calculation is done at a daily, 7-day, and cumulative level during experiment analysis. Composite Metrics do not currently support daily rollups in turbo mode. At a group level, the mean is calculated as the average of the unit-level composite aggregation. ### Example If you define a composite gap metric as `Max Price - Min Price`, we compute `Max Price` and `Min Price` for each user, then subtract to get each user's net value. The experiment group mean is the average of those per-user results. The SQL for the individual components would look like the following: MAX Component: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, MAX(source_data.value_column) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` MIN Component: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, MIN(source_data.value_column) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` Composite Aggregation: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, MAX(source_data.value_column) - MIN(source_data.value_column) AS value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ## Options * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed * Winsorization * Specify a lower and/or upper percentile bound to winsorize at. These bounds will be applied separately to the top level metric *and* its individual components. * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) allow you to specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig will delay loading the data until the window has elapsed, and only calculate pulse results for that metric if a unit's metric has matured. # Count Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/count Count metrics in Statsig Warehouse Native count the number of records in a metric source, per unit and aggregated across experimental units for analysis. ### Use Cases Counts are an extremely common metric type, especially used for measuring how often events occurred. Common examples are: * Counting click frequency by counting event logs filtered to the `click` event * Making a threshold metric of if a user read more than 2 articles ## Calculation At the unit level, count metrics run a COUNT(1) across their metric source. At the group level, the mean is calculated as the SUM of the unit-level count, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, COUNT(1) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ### Methodology Notes Count metrics are simple, and will use the sql COUNT aggregation. However, there are many advanced options you can apply. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Multi Source * Counts can be built with inputs from multiple metric sources; think of this as a UNION in SQL. This can be useful if you have the same measure in sharded or disparate tables * Winsorization * Specify a lower and/or upper percentile bound to winsorize at. All values below the lower threshold, or above the upper threshold, will be clamped to that threshold to reduce the outsized impact of outliers on your analysis * Capping * Specify an exact cap value (per unit type) to limit a unit's daily value for the count. For example, you might want daily purchases to be capped at a high value like 100 on an e-commerce site to ignore reseller behavior * CUPED * Specify if you want to calculate CUPED, and the lookback window for CUPED's pre-experiment data inputs * Thresholding * Turn this metric into a 1/0 unit count metric counting if the unit's total count equals to or surpasses (>=) a given threshold * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) allow you to specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig will delay loading the data until the window has elapsed, and only calculate pulse results for that metric if a unit's metric has matured. # Count Distinct Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/count-distinct Count Distinct metrics in Warehouse Native measure unique values in a metric source column per unit, totaling the unique unit-value pairs observed. Count-distinct metrics are more expensive to compute than [count](./count) or [unique unit count](./unit-count-once) metrics, especially for very long experiments If you want to count distinct occurrences of the experiment's unit of assignment (e.g. the user\_id in a user\_id experiment), you should use a unit\_count metrics instead. This achieves the same result, but more efficiently calculates and stores the metric data. In many cases a count metric serves as a close proxy to count-distinct; you can also set up a data source to track unique instances of a key to avoid re-running the distinct operation across experiment analyses. ### Use Cases Count distinct metrics have two primary use cases: * Measuring interactions and surface area. For example, you might to count the number of entities a user has engaged with on a video streaming platform, or measure if your new recommendation engine increases the diversity of products clicked * As a denominator in ratio metrics, especially common when you want to normalize by a unit other than your experiment's unit of analysis. For example, a B2B experiment might run an experiment at the company level, but measure "Clicks per USER" by making a ratio metric of COUNT(clicks)/COUNT\_DISTINCT(user\_id). ## Calculation At the unit level, count distinct metrics use COUNT\_DISTINCT on their input column. At the group level, the mean is calculated as the SUM of the unit-level COUNT\_DISTINCTs, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, COUNT(distinct source_data.value_column) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ### Methodology Notes In the metrics page view, we use APPROX\_COUNT\_DISTINCT (or equivalent) to avoid massive compute jobs on analytical count distinct, and because the approximate error becomes acceptably small for the topline estimate. For experiment result loads, the calculation is analytical and exact to avoid jitter or bias from approximation error. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Winsorization * Specify a lower and/or upper percentile bound to winsorize at. All values below the lower threshold, or above the upper threshold, will be clamped to that threshold to reduce the outsized impact of outliers on your analysis * CUPED * Specify if you want to calculate CUPED, and the lookback window for CUPED's pre-experiment data inputs * Thresholding * Turn this metric into a 1/0 unit count metric counting if the unit's total count equals to or surpasses (>=) a given threshold * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed ## Limits Count distinct metrics are available in most experiments, except for Switchbacks. # Funnel++ Source: https://docs.statsig.com/statsig-warehouse-native/metrics/funnel Funnel metrics in Statsig Warehouse Native measure conversion rates through a defined sequence of steps for multi-step user journey analysis. Funnel++ is a Statsig Warehouse Native feature. ## Summary Funnel metrics measure user journeys through a series of steps. There's additional guidance [here](/statsig-warehouse-native/features/funnel-metrics) on how to configure and use funnel metrics. Statsig allows you to use advanced functionalities - normally reserved for product analytics tools - within the rigorous statistical framework used in pulse analysis. This includes: * Configurable completion windows per-step * Session controls - going beyond user-based conversion * Built-in allowance for timestamp noise ### Use Cases Funnel metrics are a powerful tool to understand how users move through your product. For example, you might want to measure: * User conversion through a subscription flow, e.g. Start -> Description Page -> Payment Info -> Confirm, so you can measure if an experiment causes users to drop of less, and at which steps * User conversion from "logged out visitor" to "first logged-in-article-read" leveraging Statsig's [ID Resolution](/statsig-warehouse-native/features/funnel-metrics) * Session-level conversion of checking out a vacation property, so you can understand what the conversion looks like for every unique checkout flow -- not just at the user level ## Calculation At the unit level, funnel metrics will calculate, for each step of the funnel, if the unit completed that step some time after all previous steps were completed in order. This creates a series of step flags. If using session funnels, those step flags are instead counts of unique sessions. At the group level, the stepwise mean is calculated as the units for the next step divided by the units for the current step. The overall mean is calculated as the units/sessions that completed the funnel divided by the unit/sessions that started the funnel. For each step, the *first* occurrence after the previous step is treated as the canonical trigger and timestamp for that event going forward for subsequent timestamp comparisons. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level, per step SELECT distinct source_data.unit_id, source_data.funnel_session_id, -- optional source_data.funnel_step_id, IF(``, 1, 0) as numerator, IF(``, 1, 0) as denominator FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp ; --Group Level SELECT group_id, funnel_step_id, SUM(numerator)/SUM(denominator) as mean FROM unit_data GROUP BY group_id; ``` ### Methodology Notes Conversion metrics require adjustment due to potential unit-level covariance between the numerator and the denominator. Statsig uses the delta method to estimate this adjustment. By default, Statsig only includes numerators from metrics with non-null, non-zero denominators. This is configurable in the advanced settings. Funnels in experiment-analysis order strictly. For example, in the funnel A->B->C, all subsequent timestamp comparisons are based on the FIRST occurrence of A. If a user has an A event on day 0, with no other events, and then A/B/C all occur in order on day 5, this funnel will not count as completed if there is a 1-day conversion window from A->B since the time from the first A to the first B is 5 days. ## Options * Conversion window * A step-level setting specifying how long this step has to occur after the previous step. For example, in the funnel A->B->C, if B has a conversion window of one hour, it will only be counted if it occurs within 1 hour of A. * Use Strict Event Ordering * Whether to use >= or > when comparing step timestamps. Strict mode allows you to have two subsequent steps of the same event without it "automatically" passing * Timestamp Allowance * Allow some amount of buffer, in milliseconds, between funnel steps. Event logs can have noise on timestamps, so sometimes events may be logged slightly out of order. This setting helps to correct for this issue. * Count Distinct Mode * Whether to count sessions or units. For sessions, you must provide a session identifier field on each step * Calculation window (optional) * How long a unit has to complete the funnel, once started, and if the funnel starts when the unit is exposed to the experiment or when they trigger the first event in the funnel * Treat Exposure as Initial Funnel Event * With this setting enabled, the first step of the funnel is the exposure event of the experiment. This makes it easy to measure the conversion rate to the first event, and additionally normalizes the final outcome per experiment-user. Note that this is incompatible with session-based funnels. * Measure time to complete * Switches the funnel mode into measuring the average time for users to complete a funnel. This will create a ratio metric, where the numerator is the sum of funnel seconds-to-complete, and the denominator is the number of completed funnels. This can be useful in isolation, or when paired with a conversion measurement to understand the change to both completion rate and time to complete. ## Visualization In addition to breaking down lift by funnel steps, it is also possible to create funnel visualizations by step. Funnel visualization by step breakdown Funnel step analysis chart # First or Latest Value Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/latest-value First (or Latest) Value metrics calculate the first/latest value of a metric source for each unit, and then average it over the experiment population. ## Use Cases ### First Value First value metrics are used to track the initial experience a user has with your product. This could be first purchase value, initial time-to-load, or any other "initial" result you ar expecting to influence with your change. ### Latest Value Latest value metrics are used to track how an experiment is impacting the state of your population. For example, you might want to measure if the test group in an experiment has a higher net account balance, or if the current loyalty rewards balance is higher for users in one arm of an experiment. You can use latest value metrics to approximate user 'statuses' as well, such as "is\_subscriber", if you have a Metric Source with a 1/0 flag for all users. We recommend using [the unit count](./unit-count-latest) equivalent for this use case, since you can filter on a sparse dataset and Statsig handles imputing/tracking 0s. ## Calculation At the unit level, first value metrics will calculate each day's first non-null value within cohort bounds. Similarly, latest value metrics calculate each day's latest non-null value within any cohort bounds. For pulse, the first value will be carried forward through from the first date with data, and the latest value will be determined by taking the latest value from the latest day available. At the group level, the mean is calculated as the SUM of the unit-level values, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, LATEST_VALUE(source_data.value_field) as value -- or FIRST_VALUE FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp WHERE value_field IS NOT NULL GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ### Methodology Notes Users without a value will be treated as 0s; note that if there is an existing value, the user will "stay at" that value unless a 0 or another new value is provided later. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed * [CUPED](/experiments/statistical-methods/methodologies/cuped) * Specify if you want to calculate CUPED, and the lookback window for CUPED's pre-experiment data inputs ### Special Case: Surrogate Metrics You can use latest value metrics to implement surrogate metrics. Surrogate metrics (aka proxy metrics or predictive metrics) are a prediction of some long term metric that's impractical to measure over the duration of an experiment, and have some inherent prediction error associated with the model used to derive the metric values. Under advanced settings in latest value metrics, you can indicate a metric as a surrogate metric with a mean squared error (MSE). This means that the prediction accuracy can be accounted for in calculating variance and thus adjusts p-values and confidence intervals accordingly. Consider the variable X to be the true north metric which is being predicted by the surrogate metric S. The surrogate metric S is assumed to be an unbiased estimator with an error term $\epsilon$. $$ \mu_{X} = \mu_{S} = \overline{S} $$ $$ Var(X) = Var(S + \epsilon) = Var(S) + MSE $$ $$ Var(\overline{X}) = \frac{Var(X)}{n} = \frac{Var(S) + MSE}{n} $$ # Log Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/log Log metrics in Warehouse Native apply a logarithm to unit-level values from Sum or Count metrics before calculating Pulse results, reducing variance. ### Use Cases This can be useful for understanding if the distribution of a log-normal or tail-driven metric has shifted. This is calculated as a conditional mean - implicitly, it is a ratio metric where the numerator is the sum of unit-level log values, and the denominator is a 1 for units with a valid log. Records with a 0 denominator are accordingly filtered out. This is done because imputing 0s for logs does not work without a treatment such as an inverse hyperbolic sine function. This might be revenue, time spent, or some other metric where a small portion of users drives most of the metric value, but it's important to drive "bulk" improvements. Log metrics can be thought of as measuring relative change per unit, since an increase of 1 corresponds to a multiplication by the log's base. ## Calculation At the unit level, count metrics run a COUNT(1) or SUM(`value`) across their metric source. At the group level, the mean is calculated as the SUM of the log of the unit-level value, divided by the count of units with a unit-level value that log is valid for (exists, and is greater than 0). This would look like the SQL below, for a count metric: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, COUNT(1) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY unit_id, group_id; -- Group Level SELECT group_id, -- divide the sum of the logged values by the count of participating units SUM(LOG(value, ))/COUNT(1) as mean FROM unit_data WHERE value > 0 -- the filter is implicit from the CTE, but let's make it explicit -- a sum metric might have negative values GROUP BY group_id; ``` ### Methodology Notes Log metrics can be tricky to interpret and to extrapolate out to topline values. We highly encourage you to use this in conjunction with the raw or winsorized SUM and COUNT metric. There's a few ways to handle 0s in a log metric; a transformation like [IHS](https://en.wikipedia.org/wiki/Inverse_hyperbolic_functions) can approximate the behavior of log for large values while accepting 0s as inputs, or you can scope the analysis to non-zero units. We've chosen the second for ease of interpretation (log properties are broadly understood). This does mean there's potentially a confounding factor of participation rate; to mitigate this, results are presented the same as for [ratio](/statsig-warehouse-native/metrics/ratio) metrics and present statistics for the overall result as well as the implicit numerators and denominators. ## Options Non-log options will be based on whether or not the metric is a Sum or Count. * Custom log Base * You can configure a custom base for the log operation. Defaults to LN # Max/Min Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/max-min Max and Min metrics in Warehouse Native compute the largest or smallest value of a metric source column at the unit level for experiment analysis. ### Use Cases Max/min metrics allow you to easily track users' extremes during an experiment. Common examples are: * Measuring the impact of performance changes on users' worst experiences by analyzing the maximum latency per user. * Measuring the effect of game changes on user performance by calculating the peak high score per user. * Counting the number of users who ever left a 2-star review or lower by applying MIN(review\_score) with a threshold condition. ## Calculation for MAX/MIN At the unit level, max/min metrics take the max/min of their input column. At the group level, the max/min is calculated as the MEAN of the unit-level sums, divided by the count of UNIQUE UNITS exposed to the experiment. The MAX would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, MAX(source_data.value_column) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` The MIN would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, MIN(source_data.value_column) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ### Methodology Notes Max/min metrics are simple and there are many advanced options you can apply. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Winsorization * Specify a lower and/or upper percentile bound to winsorize at. All values below the lower threshold, or above the upper threshold, will be clamped to that threshold to reduce the outsized impact of outliers on your analysis * CUPED * Specify if you want to calculate CUPED, and the lookback window for CUPED's pre-experiment data inputs * Thresholding * Turn this metric into a 1/0 unit count metric counting if the unit's max/min equals to or surpasses (>=) a given threshold * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) allow you to specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig will delay loading the data until the window has elapsed, and only calculate pulse results for that metric if a unit's metric has matured. # Mean Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/mean Mean metrics in Statsig Warehouse Native calculate the average value of a numeric column from a metric source per unit for experiment analysis. ### Use Cases This is most often used on event-level data, e.g. measure the "Average Time to Load", or "Average Purchase Value". ## Calculation At the unit level, mean metrics SUM their value column, and COUNT records where the value column is non-null. At the group level, the mean is calculated as the SUM of the unit-level sums, and the SUM of the unit-level counts. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT source_data.unit_id, exposure_data.group_id, SUM(source_data.value_column) as value, COUNT(source_data.value_column) as records FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp WHERE value_column IS NOT NULL GROUP BY source_data.unit_id, exposure_data.group_id; -- Group Level SELECT group_id, SUM(value)/SUM(records) as mean FROM unit_data GROUP BY group_id; ``` ### Methodology Notes Under the hood, mean metrics function like a SUM/COUNT [Ratio](./ratio) metric. Mean metrics have the delta method applied to account for covariance between unit-level numerators and denominators. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Winsorization * Specify a lower and/or upper percentile bound to winsorize at. All values below the lower threshold, or above the upper threshold, will be clamped to that threshold to reduce the outsized impact of outliers on your analysis * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) allow you to specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig will delay loading the data until the window has elapsed, and only calculate pulse results for that metric if a unit's metric has matured. # Normalized Metrics (aka Clustered Experiments) Source: https://docs.statsig.com/statsig-warehouse-native/metrics/normalized-metrics Normalized metrics in Statsig Warehouse Native scale unit-level metric values by a normalization factor so per-unit comparisons account for differences. # Normalized Metrics ## When to use it With normal A/B tests the unit of randomization (e.g. UserID) matches the unit of analysis. In some cases it is useful for these to be different. The most common example is with B2B experiments - where you want to randomize by BusinessID - but measure by average metric per userID, not average metric per businessID. For example - you've added image support to a collaborative commenting feature in your product and want to A/B test it before rollout. You randomize it using businessID. You cannot randomize by userID, since you need everyone within a single business to either have this new feature or not. If you simply compared # of comments per businessID, this data would be skewed by large companies. A business with 1000 employees, but 10 comments would "contribute more" than a business with 5 employees who made 5 comments. Normalizing a metric in this case - is normalizing by users exposed to the experiment. In this instance if 1000 and 5 users were exposed from each business, the first business would have a comments/user rate of 0.01, while the second company would have a comments/user rate of 1. This is reasonable now to compare across companies of many different sizes. ## What it does Under the covers, normalizing a metric simply creates a ratio metric. The numerator is the metric you're normalizing. The denominator is a COUNT DISTINCT of the UnitID you're normalizing to. If you wish to create this ratio metric yourself and use it in experiments, you can follow this [Cluster Experiments](/metrics/different-id) guide. ## How to do it You can create Normalized Metrics when adding metrics to an Experiment. Experiment metrics configuration interface You can select the unitID to normalize using next. ## Where to use it You can create and use Normalized Metrics anywhere you can run a ratio metric and a count-distinct metric. # Percentile Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/percentile Percentile metrics in Statsig Warehouse Native compute p50, p90, p95, and custom percentiles of a numeric column for latency and tail-value analysis. ### Use Cases In many cases, the mean of a metric can be acceptable, but the tail end (e.g. 99th percentile or more) can be unacceptable; imagine a website with an average TTL of 300ms, but a p99 TTL of 1 minute. A small portion of the population is getting an unusable experience that does not show up in the mean, so it's hard to measure if this p99 value moved with traditional A/B/n metrics. Percentiles are popular for guardrail metrics around performance regression, as well as measuring improvements gained from investments in performance and infrastructure. ## Calculation For percentile metrics, there is no unit-level calculation; the analysis is run at the group level. This would look like the SQL below: ```sql expandable theme={null} -- Group Level SELECT exposures_data.group_id, PERCENTILE(user_data.value, percentile_level) as value, COUNT(distinct user_data.user_id) as population FROM user_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp WHERE value IS NOT NULL GROUP BY group_id; ``` ### Methodology Notes Percentile metrics use the outer CI method to estimate a confidence interval and significance. Deng Et. Al. have a good description of the methodology in section 4 of [this paper](https://arxiv.org/pdf/1803.06336). Note that some metrics are not well formed for this approach; there's an assumption that the underlying distribution is continuous. For example, if your data has 1/3 of its rows with a value of 0, 1/3 with a value of 5, and 1/3 with a value of 10, we will not calculate results for significance for a median or p99 metric since there's no local variability. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results # Ratio Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/ratio Ratio metrics in Statsig Warehouse Native measure the ratio of two underlying metrics (Count, Sum, Count Distinct, or Unit Count) for experiment analysis. ### Use Cases Ratio metrics are used to add a more nuanced understanding - revenue went up, as did purchase volume, but did revenue/purchase go up in a way that was meaningful? Ratio metrics are also used to normalize metrics. For example, if you're a company that rents out devices like scooters, you might run a scooter-level experiment but want to measure average revenue per distinct rider. You could make a metric of SUM(revenue)/COUNT\_DISTINCT(rider\_id) to calculate this normalized metric and have the metric be less influenced by scooters in popular areas which get lots of riders. ## Calculation At the unit level, ratio metrics will calculate both component metric's unit level aggregation. At the group level, the mean is calculated as the total group calculation of the first metric, divided by the total group value of the second metric. The denominator is **not** the number of units in the experiment; the normalization is by the denominator metric. This would look like the SQL below: ```sql expandable theme={null} -- Denominator (Checkouts) SELECT source_data.unit_id, exposure_data.group_id, COUNT(1) as denominator FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Numerator (Revenue) SELECT unit_id, group_id, SUM(revenue) as numerator FROM source_data GROUP BY unit_id, group_id; -- Group Level SELECT group_id, SUM(numerator)/SUM(denominator) as mean FROM denominator -- full outer join depending on settings LEFT JOIN numerator USING (group_id) GROUP BY group_id; ``` ### Methodology Notes Ratio metrics require adjustment due to potential unit-level covariance between the numerator and the denominator. Statsig uses the delta method to estimate this adjustment. By default, Statsig treats ratio metrics as a conversion rate (unordered). That is, we only count numerator events for units that also performed the denominator event. On Statsig Warehouse Native this is a configurable setting- available via Advanced settings at the metric level- where you can switch to a simple ratio. ## Options * Treat as conversion rate (unordered). * Control whether to include numerator events only if the unit also performed the denominator event, regardless of order. Uncheck for a simple ratio, which counts numerator events for all units, even if they never performed the denominator event. * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) (Numerator and Denominator) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed * Winsorization * Specify a lower and/or upper percentile bound to winsorize at. Winsorization and its thresholds can be specified for both the numerator and denominator of the ratio metric independently. All values below the lower threshold, or above the upper threshold, will be clamped to that threshold to reduce the outsized impact of outliers on your analysis * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) allow you to specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig will delay loading the data until the window has elapsed, and only calculate pulse results for that metric if a unit's metric has matured. # Retention Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/retention Retention metrics measure the rolling retention rate across a configured time window for a given event - or between two different events. ### Use Cases Retention metrics are an easy and powerful way to measure user stickiness, conversion, and growth over the duration of experiments and holdouts. For example, this retention metric can evaluate the change of "Current User Retention", "Notification Retention", "Video Viewer Retention" or more over the course of the experiment, and be broken down in timeseries and days-since-exposure views to understand how this shifted over time. It's fairly typical for platforms to limit retention metrics to checking if a unit was active between days X and X+Y since exposure. This is useful for new-user or marketing experiments, but is incomplete and is notably less useful for experiments targeted at an existing userbase. [This article in Lenny's newsletter](https://www.lennysnewsletter.com/p/how-duolingo-reignited-user-growth) provides a view into how people are using these metrics to drive user growth. We highly recommend using this metric type for any change aimed at increasing user stickiness - e.g. anything that touches notifications, reactivation campaigns, or quality work. ### Setup and Definition Retention metrics are defined with a duration and a lookback window. The period is measured backwards from the end - so "Lookback = 7, Duration = 14" or L7D14 would measure the week ending 14 days after the start event Retention Setup This is a rolling calculation. Each day a user triggers the start event, they get a 1 in their metric denominator. If they are active in the corresponding completion window, their numerator will be 1 for that day. Only days with completed windows will be included in pulse. For example if the duration is 7, the last week of data is excluded from pulse to avoid diluting the metric since an L3D7 metric would always have a numerator of 0 for those days. Using the `allow cohort metrics to mature after experiment end` setting in advanced experiment settings allows for post-experiment data to complete the analysis, meaning units exposed later in the analysis can be included. This is appropriate in cases where the treatment is one-time and doesn't need to be re-applied in order to impact users. Retention Explanation ## Calculation ```sql expandable theme={null} -- Denominator - 1/0 flag for activity on a day WITH denominator AS ( SELECT source_data.unit_id, source_data.date, exposure_data.group_id, MAX(1) as denominator FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp WHERE GROUP BY ALL; ), -- Numerator Candidates - 1/0 flag for success activity on a day -- Note by default this is equivalent to the denominator CTE numerator_candidates AS ( SELECT source_data.unit_id, source_data.date, exposure_data.group_id, MAX(1) as denominator FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp WHERE GROUP BY ALL ), -- Numerators, deduplicated - 1/0 flag for success per denominator -- Now we have a 1-0 numerator flag per denominator-day joined_data AS ( SELECT denominator.unit_id, denominator.date, den.group_id, 1 as denominator MAX(CASE WHEN numerator_candidates.unit_id IS NOT NULL THEN 1 ELSE 0 END) as numerator FROM denominator LEFT JOIN numerator_candidates ON denominator.unit_id = numerator_candidates.unit_id AND numerator_candidates.date BETWEEN denominator.date + INTERVAL '' DAY AND denominator.date + INTERVAL '' DAY GROUP BY ALL ) -- Group Level SELECT group_id, SUM(denominator) as unit_days_started, SUM(numerator) as unit_days_completed, SUM(numerator)/SUM(denominator) as mean FROM joined_data GROUP BY ALL ``` ### Methodology Notes Retention metrics are [ratio metrics](/statsig-warehouse-native/metrics/ratio) for the purposes of pulse calculations; the only distinction is that the metric date is attributed to the denominator date. The ratio components for retention metrics reflect the rolling metric definition: * the denominator is the average number of days per user where the "retention start" event was triggered * the numerator is the average number of days per user where a "retention start" event had a corresponding "retention end" event in its retention period. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Retention Lookback Window (Days) * The length of the "Completion Event" collection window * Retention Period End (Days) * When to stop measuring retention completion events * Use a different start and completion event for retention calculations * Choose a secondary event for completion windows. By default, retention measures a behavior's retention to itself. Toggling this allows you to measure a secondary event instead - for example if you have user accounting flags you could measure "IS CHURNED" -> "IS REACTIVATED" as well as "IS REACTIVATED" -> "IS CHURNED" to measure both reactivation and falloff of a long-term marketing test. * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results # Sum Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/sum Sum metrics in Statsig Warehouse Native calculate the sum of a numeric column from a metric source per unit, aggregated across experimental units. ### Use Cases Sums are an extremely common metric type, being the most fitting way to compare behavior across groups in many cases. Common examples are: * Totalling revenue by summing a `revenue` column from purchase logs * Totalling time spent by summing an aggregated `time_spent` metric on a user-day fact table ## Calculation At the unit level, sum metrics take the SUM of their input column. At the group level, the mean is calculated as the SUM of the unit-level sums, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT unit_id, group_id, SUM(value_column) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ### Methodology Notes Sum metrics are simple, and will use the sql SUM aggregation. However, there are many advanced options you can apply. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Multi Source * Sums can be built with inputs from multiple metric sources; think of this as a UNION in SQL. This can be useful if you have the same measure in sharded or disparate tables * Winsorization * Specify a lower and/or upper percentile bound to winsorize at. All values below the lower threshold, or above the upper threshold, will be clamped to that threshold to reduce the outsized impact of outliers on your analysis * Capping * Specify an exact cap value (per unit type) to limit a unit's daily value for the sum. For example, you might want daily purchases to be capped at a high value like 100 on an e-commerce site to ignore reseller behavior * CUPED * Specify if you want to calculate CUPED, and the lookback window for CUPED's pre-experiment data inputs * Thresholding * Turn this metric into a 1/0 unit count metric counting if the unit's total sum equals to or surpasses (>=) a given threshold * [Cohort Windows](/statsig-warehouse-native/features/cohort-metrics) * You can specify a window for data collection after a unit's exposure. For example, a 0-1 day cohort window would only count actions from days 0 and 1 after a unit was exposed to an experiment * **Only include units with a completed window** can be selected to remove units out of pulse analysis for this metric until the cohort window has completed * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) * [Baked Metrics](/statsig-warehouse-native/features/cohort-metrics#metric-bake-windows) allow you to specify how long a metric needs to mature. This is common in situations like chargebacks or cancellations. Statsig will delay loading the data until the window has elapsed, and only calculate pulse results for that metric if a unit's metric has matured. # Unit Count (Latest Value) Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/unit-count-latest Unit count metrics with the latest rollup type in Statsig Warehouse Native measure whether a unit performed an action on the most recent observed day. ### Use Cases This is a powerful metric type for measuring state. For example, you might want to understand if your test variant made it so that you currently have more active subscribers in test than in control, or if more users are currently active. ## Calculation At the unit level, unit count metrics create a 1/0 flag for if they participated on a given day. This is carried forward to the current date if data is behind. At the group level, the mean is calculated as the SUM of the unit-level flags, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT unit_id, group_id, if(passes_filter, 1, 0) as value FROM ( SELECT source_data.*, exposures_data.group_id, source_data.date = MAX(source_data.date) over (partition by source_data.unit_id) as is_latest_date FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp ) WHERE is_latest_date = 1 GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Multi Source * Sums can be built with inputs from multiple metric sources; think of this as a UNION in SQL. This can be useful if you have the same measure in sharded or disparate tables * Rollup Mode * Rollup Mode controls the specific way that Unit Count metrics are aggregated # Unit Count (One-Time Event) Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/unit-count-once Unit count metrics with the one-time event rollup type measure if a unit performed an action any time after being exposed to the experiment. ### Use Cases This is an extremely common metric type, used to measure participation rates among users in the experiment. ## Calculation At the unit level, unit count metrics create a 1/0 flag for if they participated. At the group level, the mean is calculated as the SUM of the unit-level flags, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT distinct source_data.unit_id, exposure_data.group_id, 1 as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp ; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ### Methodology Notes Note that daily/days-since-exposure views for user-day metrics are calculated per-day. That is, if a user is active on all 14 days of an experiment, they would contribute 1 to the overall cumulative numerator, but 1 to each day of the daily view; that is, they are not deduped in that view. This generally yields a more intuitive interpretation, and leads to less sparse timeseries data on long experiments. However, this can lead to mix-shift effects where the daily trend goes in the opposite direction of the cumulative timeseries since the daily returning users have already been seen and do not contribute additional metric value to the cumulative view. Unit count metrics are simple, and will use the sql SUM aggregation. However, there are many advanced options you can apply. ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Multi Source * Sums can be built with inputs from multiple metric sources; think of this as a UNION in SQL. This can be useful if you have the same measure in sharded or disparate tables * Rollup Mode * Rollup Mode controls the specific way that Unit Count metrics are aggregated # Unit Count (Daily Participation) Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/unit-count-rate Unit count metrics with the daily participation rate rollup type measure the fraction of days that the user participated in an action during the experiment. ### Use Cases This metric type is an analogue for DAU, and is useful for growth accounting and for measuring upkeep of usage and retention. ## Calculation At the unit level, unit count metrics create a 1/0 flag for if they participated on a given day. At the group level, the mean is calculated as the SUM of the unit-level flags divided by each unit's days in th experiment, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT distinct source_data.unit_id, exposure_data.group_id COUNT(distinct source_data.date) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; -- Experiment SELECT group_id, unit_id, date_diff(first_exposure_ds, today()) + 1 AS days_exposed FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value/days_exposed)/COUNT(distinct unit_id) as mean FROM unit_data JOIN group_data USING (group_id, unit_id) GROUP BY group_id; ``` ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Multi Source * Sums can be built with inputs from multiple metric sources; think of this as a UNION in SQL. This can be useful if you have the same measure in sharded or disparate tables * Rollup Mode * Rollup Mode controls the specific way that Unit Count metrics are aggregated # Unit Count (Window) Metrics Source: https://docs.statsig.com/statsig-warehouse-native/metrics/unit-count-window Windowed unit count metrics in Warehouse Native measure whether a unit performed an action within a defined time window after experiment exposure. ### Use Cases This is an extremely common metric type, used to measure participation rates early in experiments, or to measure retention later into an experiment (e.g. did a user come back in their second week). ## Calculation At the unit level, unit count metrics create a 1/0 flag for if they participated during the time window. The time window is defined relative to the unit's first exposure. Subsequent exposures are not considered. At the group level, the mean is calculated as the SUM of the unit-level flags, divided by the count of UNIQUE UNITS exposed to the experiment. This would look like the SQL below: ```sql expandable theme={null} -- Unit Level SELECT distinct source_data.unit_id, exposure_data.group_id, MAX(if(source_data.in_time_window, 1, 0)) as value FROM source_data JOIN exposure_data ON -- Only include users who saw the experiment source_data.unit_id = exposure_data.unit_id -- Only include data from after the user saw the experiment -- In this case exposure_data is already deduped to the "first exposure" AND source_data.timestamp >= exposure_data.timestamp GROUP BY source_data.unit_id, exposure_data.group_id; ; -- Experiment SELECT group_id, COUNT(distinct unit_id) total_units FROM exposure_data GROUP BY group_id; -- Group Level SELECT group_id, SUM(value)/SUM(total_units) as mean FROM unit_data JOIN group_data USING (group_id) GROUP BY group_id; ``` ## Options * Metric Breakdowns * You can configure Metadata Columns to group results by, getting easy access to dimensional views in pulse results * Multi Source * Sums can be built with inputs from multiple metric sources; think of this as a UNION in SQL. This can be useful if you have the same measure in sharded or disparate tables * Rollup Mode * Rollup Mode controls the specific way that Unit Count metrics are aggregated # Comparing Warehouse Native and Cloud Source: https://docs.statsig.com/statsig-warehouse-native/native-vs-cloud Compare Statsig Cloud and Statsig Warehouse Native to choose the right deployment model based on your data warehouse, infrastructure, and privacy requirements. Statsig Warehouse Native and Statsig Cloud share many capabilities, but there are some differences between the platforms. This page is intended to make those differences clear, and to help you make the right choice for your experimentation needs. If there's any confusion, or if a feature is critical for your evaluation, we're always open to prioritizing requests. Don't hesitate to reach out to our support team, your sales contact, or in our [Slack community](https://statsig.com/slack). ## Feature Gates | Feature | Cloud vs. WHN Status | | ------------------------- | -------------------- | | Targeting | Identical | | Overrides | Identical | | SDKs | Identical | | Diagnostics | Identical | | Pulse | Identical | | Gate lifecycle management | Identical | ## E2E Experiments (with SDK) | Feature | Cloud vs. WHN Status | | ------- | ---------------------------- | | Pulse | WHN offers on-demand reloads | ## Pulse (Scorecards) | Feature | Cloud vs. WHN Status | | ----------------------- | ------------------------------------------------ | | CUPED | Identical | | CURE | WHN only for now | | Topline Impact | Identical | | Daily Time Series | Identical | | Frequentist vs Bayesian | Identical | | Sequential Testing | Identical | | BF Correction | Identical | | Power Analysis | Identical | | Custom Queries | Identical | | Guardrail Metric Alerts | Available in WHN, based on your pipeline cadence | | Export to CSV | Identical | | Share Links | Identical | ## Analysis Only Experiments (without SDK) | Feature | Cloud vs. WHN Status | | --------------------------- | -------------------- | | Overrides | Not relevant w/o SDK | | Diagnostics (SDK Related) | Not relevant w/o SDK | | Diagnostics (Data specific) | WHN has additional | | Capture Hypothesis | Identical | | Capture Images | Identical | | Experiment Groups | Identical | | Experiment Parameters | Not relevant w/o SDK | | SRM Checks | Identical | | View SQL | WHN-only | ## Core Features | Feature | Cloud vs. WHN Status | | -------------- | -------------------- | | Layers | Identical | | Dynamic Config | Identical | | Dashboards | Identical | | Segments | Identical | | Holdouts | Identical | ## Insights | Feature | Cloud vs. WHN Status | | ------------------------- | -------------------- | | Experiment Timeline | Identical | | Knowledge Bank | Identical | | Metric Insights | Identical | | Metric Correlations | Identical | | Metric Impacts | Identical | | Active feature analysis | Identical | | Historic feature analysis | Identical | ## Metrics | Feature | Cloud vs. WHN Status | | ----------------------------------------- | ----------------------- | | Metrics Explorer | Substantially Identical | | SDK Events Logstream | Identical | | Autocreate metric from SDK Events | Cloud only for now | | User Accounting Metrics (DAU, Stickiness) | Cloud only for now | | Automated A/A Tests | Cloud only for now | | Entity Property Source ID mapping | WHN only | ## Organization & Settings | Feature | Cloud vs. WHN Status | | ----------------- | -------------------- | | SSO | Identical | | Experiment Policy | Identical | | Change Reviews | Identical | | Custom UnitIDs | Identical | | Custom RBAC | Identical | ## Integrations & Admin | Feature | Cloud vs. WHN Status | | ---------------------------------------- | -------------------- | | Change mgmt (Slack, Datadog, Github etc) | Identical | | CDBs (Segment/Rudderstack etc) | Not relevant for WHN | | Custom Environments | Identical | | Target Applications | Identical | | Console API | Identical | | Audit Log | Identical | | Audit Log API | Identical | | Autotune (MAB) | Identical | | Users Tab | Identical | # Warehouse Storage Source: https://docs.statsig.com/statsig-warehouse-native/warehouse-management/storage Manage storage in Statsig Warehouse Native, including where intermediate datasets live in your warehouse and how to control object lifetimes and costs. ## Introduction Statsig uses its sandbox in your warehouse to cache intermediate tables and result tables. This allows incremental reloads - not recalculating metrics for every day of the experiment on each load - and allows the use of these tables for additional ad-hoc analysis. These tables will be stored in the sandbox schema or dataset configured for statsig. You can use this to track storage footprint and manage permissions easily. ## Conventions / Usage Tables are generally sharded by entity ID. For example, the experiment `early_user_journey_acceleration` would have that identifier in its associated table names for scorecard loads. This is an easy way to look up tables for a given experiment. Statsig has special tables that it writes to. These can be found in metric sources or assignment sources: * pipeline overview, where the performance statistics for the jobs Statsig runs are written * statsig\_forwarded\_events, where events logged via statsig.log\_event are forwarded * statsig\_forwarded\_exposures, where exposures from experiments, gates, autotunes, and holdouts are sent * statsig\_forwarded\_switchback\_exposures, where switchback-formatted exposures are sent * statsig\_daily\_results, where rendered results with statistics like p-value are sent Some of the above have pre-set table names, and some table names are configured in data connection settings. Many users ingest these tables as part of internal pipelines. We recommend configuring lookback windows such that mutable data does not cause issues, as data is regularly updated in the tables above, and in some cases will be backfilled up to several days in cases of data delays or repairs. Additionally, exposures do not necessarily dedupe, since fast-forwarded exposures will duplicate records from daily exports, and statsig only holds on to 30d of history for warehouse native projects - meaning after 30 days a given unit's exposure will be "new" and re-exported. ## Volume Scorecard loads will generate a varied number of tables depending on factors like the number of metric sources accessed and the types of metrics loaded. Additionally, depending on data volumes Statsig may choose to materialize intermediate tables before or after large operations; this dramatically reduces compute cost incurred. This can lead to a large number of artifacts; customers running 300+ experiments have run into default quota limits on vendors like databricks. This can be managed by requesting a free quota increase or utilizing the TTLs described below in the management settings. ## Management Transient tables have a short ttl - usually 1-2 days - and will be automatically cleaned up. Other tables are permanent by default, and can be cleaned up from the experiment in statsig's console or as part of launching an experiment. Additionally, TTLs can be configured for tables by type in the data connection section of a project's settings. You should also plan to manage storage programmatically via your own warehouse tools in addition to Statsig's systems, e.g. cleaning up entities which have not been accessed or modified in the last month. Ideally this is not necessary given TTLs, but there are many known cases where Statsig's internal tracking can consider a table dropped when it still has a storage footprint, and Statsig cannot guarantee that all tables will be removed. **How TTLs work** When Statsig creates or modified a table it manages, it will use the current TTL setting to schedule a cleanup of that table at the current time + the TTL. For example, if a Result table is being written on 6/1, and Result tables are configured with a 14-day TTL, a deletion will be scheduled for 6/15. If that table were modified on 6/7 (e.g. through a scorecard reload), the deletion request would be set to 6/21, overwriting the existing one. In this way, incremental updates on long-running experiments keep their staging data until the experiment stops. This does mean that changing TTLs will not retroactively impact existing tables' deletion requests until a scorecard load is run for the relevant experiment. **Types of Tables for TTL** * `Result Datasets` are the final tables Statsig creates at the end of an experiment or gate reload that contain the aggregated group-metric level data. These are generally pretty small (1 row per metric/day/group/dimension) and useful for post-hoc analysis * `Intermediate Tables` are all the other tables Statsig writes to throughout the course of an experiment reload; these can be large since they can contain user-level data. These are re-used for incremental and metric reloads. * `Transient Datasets` refer to tables created for one-off queries (most commonly Explore queries and Power Analyses), or temporary datasets used while creating `Intermediate Tables` as a performance optimization. By default, these are dropped after 2-3 days (unless overridden with the setting above) **Explore Query Dependencies**: Explore queries rely specifically on permanent staging tables for functionality. These tables are used to reduce the need to re-compute data for analysis that was already performed by the scorecard run. Unlike results tables which are cached locally on Statsig servers, permanent staging tables must be maintained in your warehouse for explorer queries to function properly; this is to avoid regressing large volumes of data that may contain PII or other sensitive information. ## Troubleshooting Storage Issues ### Missing Data Errors Warehouse Native users may encounter `TABLE_OR_VIEW_NOT_FOUND` errors, or similar, when required data tables are missing from your warehouse. This typically occurs when: * **Permanent staging tables have been dropped**: Explore queries and advanced analysis rely specifically on permanent staging tables, not results or transient staging tables * **TTL settings have expired tables**: Tables with configured time-to-live (TTL) settings may be automatically cleaned up * **Incomplete data loads**: Initial experiment setup or data pipeline issues may prevent table creation #### Resolution Steps **For Missing Staging Tables:** Missing permanent staging tables require a full reload to recreate the staging dataset. **For General Missing Tables:** 1. Check your warehouse's TTL settings in the data connection configuration 2. Verify that permanent staging tables exist in your configured sandbox schema 3. If tables were manually dropped, trigger a full data reload 4. Contact support if tables continue to be missing after reload, or if you didn't drop them #### Understanding Storage Dependencies Warehouse Native uses several types of tables with different storage patterns: * **Permanent staging tables**: Required for explore queries and advanced analysis functionality * **Transient staging tables**: Short-lived intermediate tables with a mix of automatic cleanup (1-2 days TTL) and permanent storage (small tables that are useful for ad-hoc analysis like regression coefficients). * **Results tables**: Output statistics from the pipeline, which are copied and cached locally on Statsig servers Vacuum jobs do not affect staging tables used by Statsig. # Commands in Statsig CLI Source: https://docs.statsig.com/statsigcli/commands Reference for Statsig CLI commands, including authentication, gate management, experiment management, and project configuration operations. ## Getting Help Running `siggy` with no arguments will invoke help which will print out the usage and currently supported commands and options ```bash theme={null} $ siggy # Response Usage: siggy [options] [command] Statsig CLI For information on schema, see /console-api/introduction Options: -V, --version output the version number -h, --help display help for command Commands: config [options] view/edit configuration settings gates create/list/edit gates dyncon create/list/edit dynamic configs segments create/list/edit segments experiments create/list/edit experiments help [command] display help for command ``` Calling each command with `--help` option will print out the corresponding help content for that specific command. ```bash theme={null} $ siggy gates --help # Response Usage: siggy gates [options] [command] create/list/edit gates Options: -h, --help display help for command Commands: create create a new feature gate get retrieve gate details list [options] list all gates update update a gate delete [options] delete a gate check [options] check if the current state of the gate for a user help [command] display help for command ``` ## Other commands You can list, create, update, delete entities like Gates, Experiments, Dynamic Configs, Segments and more. The help content is kept up to date in code and you will get the most relevant help by running the CLI tool with the help option. For those commands that require a JSON body, the schema is defined here: [Console API](/console-api/introduction) # Walkthrough guide for Gate Management with CLI Source: https://docs.statsig.com/statsigcli/gate-management Use the Statsig CLI to manage feature gates programmatically, including listing, creating, editing, and archiving gates from your terminal or CI. The walkthrough guide assumes you have Statsig CLI installed and configured with the right API keys. Please refer to the [Statsig CLI Overview](/statsigcli) to get started ## Create a new gate You can create a new empty gate with no rules by calling `create` command on gates ```bash theme={null} $ siggy gates create my-first-gate # Response { id: 'my-first-gate', name: 'my-first-gate', description: '', idType: 'userID', lastModifierID: '..', ... } ``` ## Update new rules New rules could be updated by passing a rule object to the `update` command. This will replace the existing rules as a whole. ```bash theme={null} $ siggy gates update my-first-gate '{ "rules": [ { "name": "all employees", "passPercentage": 100, "conditions": [ { "type": "email", "operator": "str_contains_any", "targetValue": [ "@statsig.com" ] } ] } ] }' # Response { id: 'my-first-gate', name: 'my-first-gate', ... rules: [ { id: '729Qb4MVDs0YrIjNR5aOSm', name: 'all employees', passPercentage: 100, conditions: [ { type: 'email', targetValue: [ '@statsig.com' ], operator: 'str_contains_any' } ], } ], ... } ``` ## Check if the gate works When the Client API key is configured correctly, you can invoke the gate for different users and validate the gate works Passing in no user object will create an empty user object and evaluate the gate against that ```bash theme={null} $ siggy gates check my-first-gate # Response { name: 'my-first-gate', value: false, rule_id: 'default', group_name: null } ``` You can also pass a user object crafted as JSON using the `--user` option ```bash theme={null} $ siggy gates check my-first-gate --user '{ "email": "siggy@statsig.com" }' # Response { name: 'my-first-gate', value: true, rule_id: '729Qb4MVDs0YrIjNR5aOSm', group_name: null } ``` ## List all gates ```bash theme={null} $ siggy gates list # Response [ { id: 'my-first-gate', name: 'my-first-gate', lastModifiedTime: 1718222637700, lastModifierName: 'CONSOLE API' }, { id: 'from_siggy', name: 'from_siggy', lastModifiedTime: 1717807438090, lastModifierName: 'CONSOLE API' } ... ] ``` ## Delete gate You could also delete this gate via the CLI. By default there is a confirmation prompt that requires interaction. ```bash theme={null} $ siggy gates delete my-first-gate # Response Are you sure you want to delete gate (id: my-first-gate)? (y/n): ``` You could override it by using the `--force` option. ```bash theme={null} $ siggy gates delete my-first-gate --force # Response Gate deleted successfully. ``` # Statsig CLI ("Siggy") Source: https://docs.statsig.com/statsigcli/introduction Introduction to the Statsig CLI for managing feature gates, experiments, and project configuration from your terminal and CI environments. ## Overview The Statsig CLI is a command-line interface that helps with management of Feature Gates, Experiments, and Dynamic Configs within the Statsig platform. This tool allows you create, manage, and delete configs, all from the command line. This tool can also be used within scripts. ## Why Use the Statsig CLI? ### Efficiency The Statsig CLI provides a fast way to interact with Statsig, reducing the need for manual actions through our Console interface. It allows for quick execution of tasks with simple commands. ### Automation Our CLI can be integrated into CI/CD pipelines to automate the management of Feature Gates and Experiments. ### Consistency Using the CLI allows for scriptable and repeatable actions, promoting best practices and reducing the risk of human error. ## Installation To get started with the Statsig CLI, follow these steps: ### Prerequisites * Node.js (version 14 or higher) * npm (Node Package Manager) ### Steps 1. Install the Statsig CLI via npm: ```bash theme={null} $ npm install -g @statsig/siggy ``` 2. Check install ```bash theme={null} $ siggy --version ``` If you get a `command not found` error, you might need to add your node global bin folder to your path. One way to do that is by running `export PATH=$PATH:$(npm get prefix -g)/bin` If that doesn't work, as a workaround, you can run the CLI by prefixing `npx` in the command line ```bash theme={null} $ npx siggy --version ``` 2. Configure the API keys by running: ```bash theme={null} $ siggy config -c $ siggy config -k ``` You can retrieve these keys from your Statsig project. In order to get these keys, login to Statsig Console here: [https://console.statsig.com](https://console.statsig.com) and navigate to the Settings page ([https://console.statsig.com/settings](https://console.statsig.com/settings)) Once you're there, select the **Keys & Environments** panel within **Project Settings**, then copy both the Console API Key and Client API Key, and paste them in the Settings dialog. Statsig Console API keys configuration screen ### Next up: [Commands in CLI](/statsigcli/commands) # Support Options Source: https://docs.statsig.com/support-options Learn about Statsig support channels including Slack, email, and the console, and compare support plans like Premium Support and Enterprise SLAs. Statsig uses automated tooling in the Slack Connect channel to triage and prioritize inbound support messages. All support at Statsig is done by actual, full-time engineers. While Statsig maintains a monitored support email, we strongly encourage Slack as the primary channel for support and communication to ensure the fastest response times. Our preferred methods of raising a support request: 1. Slack message via your dedicated slack channel or Slack Community 2. Email via [support@statsig.com](mailto:support@statsig.com) is discouraged but available for sensitive topics ## Community vs Standard vs Premium support | Feature | Community | Standard | Premium | | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------- | --------- | --------------- | | Access to Statsig University certifications and webinar training sessions | Yes | Yes | Yes | | Access to Statsig's global Slack community for best practices and peer insights | Yes | Yes | Yes | | Basic assistance during Statsig business hours | Yes | Yes | Yes | | Customer member access to Private Slack channel with Statsig engineers[1](#note-1) | No | No | Up to 4 members | | Critical/High Priority support with 4 hour response time during Statsig business hours[2](#note-2) | No | No | Yes | | 24 hour response time for all other inquiries during Statsig business hours | No | No | Yes | | Dedicated Account Manager to support ongoing needs | No | Yes | Yes | | Check-ins with the Statsig team experts to align on goals and progress | No | As needed | Quarterly | | Data science consultations for advisory and experiment reviews | No | No | Up to 4 | | Early access invitation to events hosted by Statsig and partners | No | No | Yes | \[1] Standard or Premium support is only available for [Enterprise plans](https://www.statsig.com/pricing)
    \[2] Statsig standard business hours of 8am-5pm PT ## Learning resources * **Statsig University**: A comprehensive collection of onboarding and training content, including a video Resource Library and a schedule of user training Webinars. * **Statsig Status Page**: Monitor Statsig's operational status and configure [Slack alerts](/integrations/slack) to be notified of updates. * **Community**: Join our Slack Community to connect with other users and get help from the Statsig team. * **Blog**: Stay updated with the latest developments in product experimentation and feature management on our [Blog](https://statsig.com/blog). * **Use Cases**: Discover [Customer Stories](https://statsig.com/customers) to learn why customers love Statsig and how they use it to power their product development. # Platform Overview Source: https://docs.statsig.com/understanding-platform Learn what Statsig is used for and how to set it up using the Cloud or Warehouse Native deployment models, including key concepts and onboarding steps. ## What do I use Statsig for? Statsig's goal is to be the single platform to ship, measure, and learn from the products you build. The most popular features of Statsig are: * [Feature Flags](/feature-flags/overview): Expose new features to select user groups, roll them out (and roll them back, when things go wrong), and measure the impact they have. * [Experiments](/experiments/overview): Run randomized, controlled experiments on different variations of your product, and measure the exact impact on your users. Customers use Statsig to run thousands of experiments each year, and iterate in the right direction. * [Product Analytics](/product-analytics/overview): Understand the trends of your core business metrics, user behavior, and more. * [Infra Analytics](/infra-analytics/overview): Monitor and debug service health alongside product outcomes. Ingest metrics and traces with OpenTelemetry, search and group logs, and set alerts to catch regressions fast. While these three are some of our most popular, Statsig offers other tools like [Session Replay](/session-replay/overview), [Web Analytics](/webanalytics/overview/) and all of these features together are greater than the sum of the parts. *** ## How do I setup Statsig? While Statsig is flexible to many setups models, the most common setup approach is to integrate [Statsig's SDKs](/sdks/quickstart) which let you integrate flags/experiments, and track your core business metrics. While SDK installation is most common, you might want to bring existing data to Statsig. If so, you have a few options: * **I'd like to bring existing data from my Data Warehouse:** You'll likely want to use [Statsig Warehouse Native](/statsig-warehouse-native/introduction), a zero-ETL model for running experiments and product analytics on top of your existing data. Alternatively, you can import warehouse data into Statsig Cloud with our [warehouse ingestions](/data-warehouse-ingestion/introduction). * **I'd like to bring existing data from my Segment, Rudderstack, Amplitude, or another platform:** Consider one of Statsig's [integrations](/integrations/introduction), which can port your events straight into Statsig Cloud. * **I'd like to import my existing experiment assignment data, and use Statsig for Analysis:** You'll need to use [Statsig Warehouse Native](/statsig-warehouse-native/introduction). Statsig has two models to leverage its core products based on your needs: Statsig Cloud (where we host your data) and Statsig Warehouse Native (where you host your data in your own warehouse) a little more on each of these: *** ## Statsig Cloud With Statsig Cloud, setting up is simple. Install the Statsig SDK and configure event logging—we handle everything else. * You get feature flags and 2 million metered events for free. * Enjoy powerful analytics tools such as Dashboards, Metrics Explorer, and Insights. * For more details on the pricing, check [our pricing page](https://www.statsig.com/pricing). Statsig Cloud is a great choice for those who want to get started quickly without needing to manage infrastructure or data warehousing. *** ## Statsig Warehouse Native (WHN) If your events and metrics already reside in your own data warehouse and you have a dedicated data team, Statsig Warehouse Native (WHN) may be a better option. * WHN allows you to host Statsig's Stats Engine within your warehouse, enabling you to calculate metric lifts on your pre-existing datasets. * You can choose between two methods: 1. **Using 3rd party or your own SDKs**: You handle feature assignment and provide us exposure data (you randomize the users). 2. **Using Statsig SDKs**: We handle randomization and write data into your warehouse for you. The first method helps you scale analysis, while the second can 10x your experimentation velocity. WHN is available only with Enterprise contracts. If you're interested in this option, check [this link](/statsig-warehouse-native/introduction) or [schedule a demo](https://www.statsig.com/contact/demo) with our Sales team. *** ## Which Model is Right for You? Below is a summary of key criteria to consider when making your decision between the two modes of deployment: | Criteria | Cloud-hosted | Warehouse native (WHN) | | --------------------- | ------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | | Data Source | Primary source of metrics come from Statsig SDKs or CDPs like Segment. Some metrics can still come from a warehouse. | Warehouse is the primary source of metrics, making WHN ideal when wanting to reuse existing data pipelines and computation. | | Analysis needs | Automated experimentation for every experiment and product launch, especially with metrics derived from event logging. | Flexible analysis on top of your existing source of truth metric data. | | Data team involvement | Involvement is optional but recommended for experiment design and readouts. | Necessary for setting up the warehouse connection and configuring core metrics, but not mandatory for every experiment. | | Costs | TCO is slightly lower. No warehouse costs involved. | TCO includes Statsig license + costs incurred for computation and storage in your warehouse. | | Modularity | An integrated end-to-end platform that spans SDKs for feature rollout, experiment execution, analysis, and experiment readouts. | Modular: You can opt for the integrated end-to-end platform or choose to use only a subset of capabilities, such as assignment or experiment analysis. | Still unsure! Read this blog post for further information: [Statsig Cloud vs Warehouse Native](https://www.statsig.com/blog/deciding-cloud-hosted-versus-warehouse-native-experimentation-platforms). ## Next steps Once you've decided whether Statsig Cloud or Statsig Warehouse Native fits your organization's needs, choose the appropriate *getting started* guide for your first use case: * [Getting Started with Statsig Cloud](/sdks/getting-started) * [Getting Started with Statsig Warehouse Native](/statsig-warehouse-native/guides/quick-start) Have a question or need help getting set up? Our Engineering, Data, and Product teams are ready to answer questions in our [Slack community](https://www.statsig.com/slack). # Autocapture Metrics Source: https://docs.statsig.com/webanalytics/autocapture Learn how Statsig autocapture collects web analytics events, the metadata attributes it records, and how to configure tracking in your application. The JavaScript, React, and Angular SDKs support autocapture, which automatically captures events and their attributes without requiring you to manually instrument your code. ## Attributes Captured ### Common Metadata (on all events) Statsig automatically captures comprehensive metadata with each autocapture event to provide rich context for analysis: * To filter events, you can use the `eventFilterFunc` option when initializing the Statsig client. * To sanitize events and its metadata, you can use listen for the `pre_logs_flushed` client event and modify the event object before it is sent to Statsig. (See [Client Event Emitter](/client/javascript-sdk#client-event-emitter) for more details.) #### Page and Environment Metadata: | Property | Description | | -------------------------- | -------------------------------------------------- | | `metadata.title` | Page title from `` tag | | `metadata.current_url` | Full current URL | | `metadata.hostname` | Domain name of the current page | | `metadata.pathname` | Path portion of the URL | | `metadata.user_agent` | Browser user agent string (truncated to 200 chars) | | `metadata.locale` | Browser language setting | | `metadata.timezone` | User's timezone | | `metadata.timezone_offset` | Timezone offset in minutes | | `metadata.timestamp` | Event timestamp in milliseconds | #### Screen and Viewport Data: | Property | Description | | -------------------------- | --------------------------------- | | `metadata.screen_width` | Screen width in pixels | | `metadata.screen_height` | Screen height in pixels | | `metadata.viewport_width` | Browser viewport width in pixels | | `metadata.viewport_height` | Browser viewport height in pixels | #### Network Information: | Property | Description | | ------------------------------------ | ----------------------------------- | | `metadata.effective_connection_type` | Connection speed estimate | | `metadata.rtt_ms` | Round-trip time in milliseconds | | `metadata.downlink_mbps` | Download speed in Mbps | | `metadata.save_data` | Whether user has data saver enabled | #### Referrer and Traffic Source Data: | Property | Description | | -------------------------- | -------------------------- | | `metadata.referrer` | Full referrer URL | | `metadata.referrer_domain` | Domain of the referrer | | `metadata.referrer_path` | Path of the referrer URL | | `metadata.searchEngine` | Detected search engine | | `metadata.searchQuery` | Search query from referrer | #### Campaign and Marketing Attribution: Statsig automatically captures 25+ marketing attribution parameters including: | Property | Description | | ----------------------- | ----------------------- | | `metadata.utm_source` | UTM source parameter | | `metadata.utm_medium` | UTM medium parameter | | `metadata.utm_campaign` | UTM campaign parameter | | `metadata.utm_term` | UTM term parameter | | `metadata.utm_content` | UTM content parameter | | `metadata.utm_id` | UTM ID parameter | | `metadata.gclid` | Google Click ID | | `metadata.fbclid` | Facebook Click ID | | `metadata.msclkid` | Microsoft Bing Click ID | | `metadata.ttclid` | TikTok Click ID | | `metadata.li_fat_id` | LinkedIn Click ID | | `metadata.epik` | Pinterest Click ID | **Additional Platform IDs Captured:** * `gclsrc`, `wbraid`, `gad_source` (Google) * `dclid` (DoubleClick) * `irclid` (Impact) * `igshid` (Instagram) * `_kx` (Klaviyo) * `mc_cid`, `mc_eid` (Mailchimp) * `qclid` (Quora) * `rdt_cid` (Reddit) * `sccid` (Snapchat) * `ttc`, `ttc_id` (TikTok) * `twclid` (Twitter) ### Action Events (Click, Form Submit, etc.) | Property | Description | | ----------- | ----------------------------------------------------------- | | `tagName` | The HTML tag of the clicked element | | `classList` | The list of CSS classes on the clicked element | | `class` | The class attribute of the clicked element | | `id` | The ID attribute of the clicked element | | `ariaLabel` | The aria-label attribute of the clicked element | | `selector` | A CSS selector that uniquely identifies the clicked element | depending on the event element, the following properties may be captured: * `action` * `method` * `formName` * `formId` * `href` * `content` * `inputName` * `textContent` * `selectedText` ...and more ### Page View Events PageViewStart: | Property | Description | | -------------------- | ----------------------------- | | `last_page_view_url` | The URL of the last page view | PageViewEnd: | Property | Description | | ---------------------- | --------------------------------------------------------------------------------- | | `page_view_length` | The length of the page view in milliseconds | | `lastScrollY` | The last scroll position in pixels | | `maxScrollY` | The maximum scroll position reached during the page view in pixels | | `lastScrollPercentage` | The last scroll percentage (0-100) of the page | | `maxScrollPercentage` | The maximum scroll percentage reached during the page view (0-100) | | `scrollDepth` | The scroll depth percentage (0-100) - deprecated, use maxScrollPercentage instead | | `dueToInactivity` | Whether the page view ended due to inactivity | ### Dead Click Events | Property | Description | | ------------------------ | ----------------------------------------------------------------------------------------------- | | `scrollTimeout` | Boolean indicating if a scroll timeout occurred | | `selectionChangeTimeout` | Boolean indicating if a selection change timeout occurred | | `mutationTimeout` | Boolean indicating if a mutation timeout occurred | | `absoluteTimeout` | Boolean indicating if an absolute timeout occurred | | `scrollDelayMs` | The number of milliseconds to wait before considering a scroll event as a dead click | | `selectionChangeDelayMs` | The number of milliseconds to wait before considering a selection change event as a dead click | | `mutationDelayMs` | The number of milliseconds to wait before considering a mutation event as a dead click | | `absoluteDelayMs` | The number of milliseconds to wait before considering an absolute timeout event as a dead click | ### Rage Click Events | Property | Description | | ----------- | ----------------------------------------------- | | `x` | The x-coordinate of the rage click | | `y` | The y-coordinate of the rage click | | `timestamp` | The timestamp of the rage click in milliseconds | ### Web Vitals Events | Property | Description | | -------- | ---------------------------------------------------------------------------------------------- | | `name` | The name of the web vital ('CLS', 'FCP', 'LCP', 'TTFB') | | `value` | The current measurement value (e.g., milliseconds for LCP, FCP, etc.; a numeric score for CLS) | | `delta` | The change from the previously reported value—used to calculate cumulative values over time | | `id` | The ID of the web vital event | ### Console Log Events | Property | Description | | ----------- | ------------------------------------------------------------------------------------------------------- | | `status` | The status of the console log ('debug', 'info', 'warn', 'error') | | `log_level` | The level of the console log ('debug', 'info', 'log', 'warn', 'error') | | `payload` | The payload of the console log (array of strings) | | `timestamp` | The timestamp of the console log in milliseconds (epoch time) | | `trace` | A stack trace (if available) showing where in the code the console log was triggered (array of strings) | | `source` | The source of the console log ('js-auto-capture') | #### Query Parameters: All URL query parameters are automatically captured as individual metadata fields with their parameter names as keys. # Statsig Web Analytics Overview Source: https://docs.statsig.com/webanalytics/overview Overview of Statsig Web Analytics for tracking page views, autocapture events, conversions, and running A/B tests on your website. # Web Analytics Overview [Web analytics](https://www.statsig.com/web-analytics) lets you track and watch key measures for your website easily. It is different from product analytics because it's simpler and more direct, making it great for marketers, web site maintainers, or anyone familiar with tools like Google Analytics. With Web Analytics and Statsig Dashboards you can easily gather insights such as number of visitors, views, sessions, how long sessions last, error rates, usage journey, and more. You can see the events flowing in your metrics page once you add web analytics to your application: [https://console.statsig.com/metrics/events](https://console.statsig.com/metrics/events) <Frame> <img alt="Metrics events dashboard showing web analytics data" /> </Frame> ## Getting Started Follow these simple steps to get started with Web Analytics. ### Option 1: HTML Script Installation The simplest way to get started is by adding a single script tag to your website: ```html theme={null} <script src="https://cdn.jsdelivr.net/npm/@statsig/js-client@3/build/statsig-js-client+session-replay+web-analytics.min.js?apikey=[YOUR_CLIENT_KEY]"></script> ``` Get YOUR\_CLIENT\_KEY from Project Settings -> Keys & Environments. Reveal the Client API Key, copy, and paste it over the \[YOUR\_CLIENT\_KEY] in the snippet above. <img alt="Project settings showing client API key configuration" /> ### Option 2: HTML Script with Advanced Control For more control over initialization and configuration, you can manually initialize the Statsig client: ```html theme={null} <script src="https://cdn.jsdelivr.net/npm/@statsig/js-client@3/build/statsig-js-client+session-replay+web-analytics.min.js"></script> <script> const { StatsigClient, StatsigAutoCapturePlugin, StatsigSessionReplayPlugin } = window.Statsig; const client = new StatsigClient( 'YOUR_CLIENT_KEY', // put your client sdk key here - "client-XXXX" { userID: 'optional' }, // set a userID here if you have one { plugins: [ new StatsigAutoCapturePlugin(), new StatsigSessionReplayPlugin() ]} ); client.initializeAsync().catch((err) => console.error(err)); </script> ``` ### Option 3: JavaScript Client SDK (React Bindings) If using javascript client SDK or other frameworks, you can use the corresponding SDK with the autocapture plugin (JS, React, Angular, etc.): ```javascript theme={null} import * as React from 'react'; import { StatsigProvider } from '@statsig/react-bindings'; import { StatsigAutoCapturePlugin } from '@statsig/web-analytics'; return ( <StatsigProvider sdkKey={YOUR_CLIENT_KEY} user={{ userID: 'a-user' }} loadingComponent={ <div>Loading...</div> } options={{ plugins: [ new StatsigAutoCapturePlugin() ] }}> <div>Hello</div> </StatsigProvider> ); ``` ## Autocaptured Events Statsig Web Analytics automatically captures the following events without any additional code: ### Page Navigation Events * **`auto_capture::page_view`** - Triggered when a user navigates to a new page. Includes page URL, query parameters, referrer information, and metadata. * **`auto_capture::page_view_end`** - Triggered when a user leaves a page. Includes engagement metrics like scroll depth, time on page, and whether the exit was due to inactivity. ### User Interaction Events * **`auto_capture::click`** - Captures all click events with target element information, coordinates, and page context. * **`auto_capture::rage_click`** - Detects rapid repeated clicks in the same area, often indicating user frustration. * **`auto_capture::dead_click`** - Identifies clicks that don't result in any page changes or navigation, potentially indicating broken functionality. * **`auto_capture::form_submit`** - Captures form submission events with form metadata. * **`auto_capture::copy`** - Tracks when users copy text from your page, including the selected text content. ### Technical Performance Events * **`auto_capture::error`** - Automatically captures JavaScript errors with stack traces, error messages, and context. * **`auto_capture::web_vitals`** - Captures Core Web Vitals metrics (see the [Web Vitals Events](#web-vitals-events) section below). * **`auto_capture::performance`** - Collects page load performance metrics including load times, DOM interactive time, and network transfer data. (deprecated, use `auto_capture::web_vitals` instead) ### Session Events * **`auto_capture::session_start`** - Marks the beginning of a new user session. ### Console Logging (Disabled by Default) * **`statsig::log_line`** - Captures console log output when explicitly enabled. This event is disabled by default and must be configured to activate. For details on how to enable console log autocapture, see [Console Log Capture Configuration](#console-log-capture-configuration). ## Event Filtering and Console Configuration ### Disabling Events via Console You can disable specific autocapture events from the Statsig console under Project Settings > Analytics & Session Replay: <img alt="Analytics event filtering configuration interface" /> ### Programmatic Event Filtering For more granular control, you can filter events programmatically using a custom filter function: ```javascript theme={null} const client = new StatsigClient( 'YOUR_CLIENT_KEY', { userID: 'optional' }, { plugins: [ new StatsigAutoCapturePlugin({ eventFilterFunc: (event) => { // Filter out events from admin pages if (event.metadata.pageUrl && event.metadata.pageUrl.includes('/admin/')) { return false; } return true; } }) ]} ); ``` ### Console Log Capture Configuration Console log capture is disabled by default but can be enabled with the following configuration: ```javascript theme={null} const client = new StatsigClient( 'YOUR_CLIENT_KEY', { userID: 'optional' }, { plugins: [ new StatsigAutoCapturePlugin({ consoleLogAutoCaptureSettings: { enabled: true, } }) ]} ); ``` There are a few additional optional configuration settings available when capturing console logs: * **`logLevel`**: minimum level to capture (`debug` | `info` | `log` | `warn` | `error`) * **`sampleRate`**: fraction between 0 and 1 to sample captured logs * **`maxKeys`**: maximum number of keys to serialize from logged objects * **`maxDepth`**: maximum nesting depth when serializing objects * **`maxStringLength`**: maximum number of characters for stringified values ```javascript theme={null} const client = new StatsigClient( 'YOUR_CLIENT_KEY', { userID: 'optional' }, { plugins: [ new StatsigAutoCapturePlugin({ consoleLogAutoCaptureSettings: { enabled: true, logLevel: 'warn', // capture warn and error sampleRate: 0.5, // capture 50% of matched logs maxKeys: 50, // limit number of keys per object maxDepth: 3, // limit nested object depth maxStringLength: 2000, // truncate long strings } }) ]} ); ``` ## Web Vitals Events Statsig automatically captures Core Web Vitals metrics that are essential for measuring user experience and SEO performance: ### Core Web Vitals Metrics * **CLS (Cumulative Layout Shift)** - Measures visual stability by tracking unexpected layout shifts during page load. Lower scores indicate better user experience. * **FCP (First Contentful Paint)** - Measures loading performance by tracking when the first text or image is painted on the screen. Faster FCP times indicate better perceived performance. * **LCP (Largest Contentful Paint)** - Measures loading performance by tracking when the largest content element becomes visible. This metric correlates strongly with user-perceived load times. * **TTFB (Time to First Byte)** - Measures server response time by tracking how long it takes to receive the first byte of response from the server. This indicates server and network performance. * **INP (Interaction to Next Paint)** - Measures responsiveness by tracking the longest latency across user interactions. Lower values mean more consistent, responsive experiences. These metrics are automatically collected and sent as `auto_capture::web_vitals` events, providing insights into your website's performance and user experience quality. ## Single Page Application (SPA) Support Statsig Web Analytics automatically supports single page applications without any additional configuration. The system intelligently detects route changes through: ### Automatic Route Change Detection * **Browser Navigation**: Listens for `popstate` events to detect when users use browser back/forward buttons * **Programmatic Navigation**: Automatically proxies `history.pushState` calls to detect when your application programmatically navigates to new routes * **Automatic Page Views**: Triggers new `auto_capture::page_view` events whenever route changes are detected This means that modern SPA frameworks like React Router, Vue Router, Angular Router, and others work seamlessly with Statsig Web Analytics without requiring manual page view tracking. ## Exploring using Metrics Explorer In [Metrics Explorer](https://console.statsig.com/metrics/explore), you can dig deeper and explore your events by using complex filters and dimensions. You can then explore your analysis to your dashboard. More information on using Metrics Explorer like [Funnels and Retention is here](/product-analytics/overview). <img alt="Metrics Explorer interface for analyzing web analytics data" /> You can see some example autocapture event details [here](/webanalytics/autocapture). ## Autocapture Settings The autocapture plugin or manual setup can take an optional `options` parameter to customize the autocapture settings. * **`eventFilterFunc`**: `Optional[Callable[[AutoCaptureEvent], bool]]` - A function to filter events based on their metadata. * **`consoleLogAutoCaptureSettings`**: `Optional[ConsoleLogAutoCaptureSettings]` - An object to configure console log capture. **consoleLogAutoCaptureSettings**: * **`enabled`**: `Optional[bool]` - A boolean to enable or disable console log capture. * **`logLevel`**: `Optional[str]` - A string to set the minimum log level for console log capture. * **`sampleRate`**: `Optional[float]` - A number to set the sample rate for console log capture. (0-1) * **`maxKeys`**: `Optional[number]` - Maximum number of keys to serialize from logged objects. * **`maxDepth`**: `Optional[number]` - Maximum nesting depth when serializing objects. * **`maxStringLength`**: `Optional[number]` - Maximum number of characters for stringified values. **eventFilterFunc**: * **`event`**: `AutoCaptureEvent` - The event object that is being captured. * **`return`**: `bool` - A boolean to return true to capture the event or false to ignore it. # Statsig Overview Source: https://docs.statsig.com/welcome Statsig is a unified platform for feature flags, A/B testing, and product analytics. Ship, measure, and learn with tools used by leading tech companies. ## Quick Links <Columns> <Card title="SDK Quickstart" icon="bolt" href="/sdks/quickstart"> Install the Statsig SDK and get started in minutes. </Card> <Card title="Intro to Statsig" icon="flag-checkered" href="/understanding-platform"> Learn more about core concepts and features of Statsig. </Card> <Card title="Warehouse Native" icon="database" href="/statsig-warehouse-native/introduction"> Read about how to use Statsig with your data warehouse. </Card> </Columns> ## Key features <Columns> <Card title="Feature Flags" href="/feature-flags/overview"> Control feature rollouts and manage deployments safely with feature flags. </Card> <Card title="Experimentation" href="/experiments/overview"> Run A/B tests and experiments to optimize your product decisions. </Card> <Card title="Product Analytics" href="/product-analytics/overview"> Analyze user behavior and product metrics with powerful analytics tools. </Card> </Columns> You can use each of these features on top of your own data warehouse through [Warehouse Native](/statsig-warehouse-native/introduction) with no ETL, or host your data in our infrastructure. *** ## Explore SDKs Statsig offers SDKs for a wide variety of platforms to suit any codebase or deployment preference: ### Client SDKs <Columns> <Card title="JavaScript" icon="js" href="/client/javascript-sdk"> Browser JavaScript </Card> <Card title="React" icon="react" href="/client/React"> Client-Side React </Card> <Card title="React Native" icon="react" href="/client/ReactNative"> Bare React Native SDK </Card> <Card title="Next.js" icon="n" href="/client/Next"> Next.js SSR, SSG & Client-Side </Card> <Card title="Angular" icon="angular" href="/client/Angular"> Angular bindings for Javascript SDK </Card> <Card title="Swift" icon="swift" href="/client/iosClientSDK"> iOS, MacOS, tvOS SDK </Card> <Card title="Android" icon="android" href="/client/Android"> Android Kotlin/Java SDK </Card> <Card title=".NET Client" icon="https://mintcdn.com/statsig-4b2ff144/ZtEzUK-pMS9sfX1d/images/dotnet.svg?fit=max&auto=format&n=ZtEzUK-pMS9sfX1d&q=85&s=28b20fecaf010359f2454a14bd6e2158" href="/client/DotNet"> Client SDK for .NET framework </Card> <Card title="Roku" icon="r" href="/client/Roku"> Roku Brightscript SDK </Card> <Card title="Unity" icon="unity" href="/client/Unity"> Unity game engine SDK </Card> <Card title="Dart/Flutter" icon="mobile-screen" href="/client/Dart"> Flutter/Dart Mobile App SDK </Card> <Card title="C++ Client" icon="code" href="/client/CPP"> C++ client-side SDK </Card> </Columns> ### Server Side SDKs <Columns> <Card title="Node.js" icon="node-js" href="/server-core/node-core"> Node.js server SDK </Card> <Card title="Java" icon="java" href="/server-core/java-core"> Java server SDK </Card> <Card title="Python" icon="python" href="/server-core/python-core"> Python server SDK </Card> <Card title="Go" icon="golang" href="/server/golangSDK"> Go server SDK </Card> <Card title="Ruby" icon="gem" href="/server/rubySDK"> Ruby server SDK </Card> <Card title=".NET Server" icon="microsoft" href="/server-core/dotnetCoreSDK"> .NET server SDK </Card> <Card title="PHP" icon="php" href="/server-core/php-core"> PHP server SDK </Card> <Card title="Rust" icon="rust" href="/server-core/rust-core"> Rust server SDK </Card> <Card title="C++ Server" icon="code" href="/server-core/cpp-core"> C++ server SDK </Card> </Columns> ### Integrations <Columns> <Card title="Webflow" icon="w" href="/guides/webflow-sidecar-ab-test"> Webflow integration </Card> <Card title="Shopify" icon="shopify" href="/guides/shopify-ab-test"> Shopify integration </Card> <Card title="Segment" icon="chart-pie" href="/integrations/data-connectors/segment"> Segment data connector </Card> <Card title="Rudderstack" icon="layer-group" href="/integrations/data-connectors/rudderstack"> Rudderstack connector </Card> <Card title="Hightouch" icon="cloud-arrow-up" href="/integrations/data-connectors/hightouch"> Hightouch integration </Card> <Card title="mParticle" icon="share-nodes" href="/integrations/data-connectors/mparticle"> mParticle connector </Card> <Card title="Framer" icon="f" href="/guides/framer-analytics"> Framer integration </Card> <Card title="Slack" icon="slack" href="/integrations/slack"> Slack notifications </Card> <Card title="Integrations" icon="puzzle" href="/integrations/introduction"> View more integrations </Card> </Columns> If you don't see the SDK or framework you need, feel free to reach out directly in our [Slack Community](https://statsig.com/slack). *** ## Learning resources * **Statsig University**: A comprehensive collection of onboarding and training content, including a video <a href="https://learn.statsig.com/pages/resource-library">Resource Library</a> and a schedule of user training <a href="https://learn.statsig.com/pages/upcoming-webinars">Webinars</a>. * **Community**: Join our <a href="https://statsig.com/slack">Slack Community</a> to connect with other users and get help from the Statsig team. * **Blog**: Stay updated with the latest developments in product experimentation and feature management on our [Blog](https://statsig.com/blog). * **Use Cases**: Discover [Customer Stories](https://statsig.com/customers) to learn why customers love Statsig and how they use it to power their product development.