Configs

A config stores the model, prompt messages, and parameters for one LLM task in your app, such as classifying a support ticket. Configs on this page are Agent Experimentation configs, not Dynamic Configs. Your app fetches the config at runtime with a Statsig AI SDK, so you can change the model or the prompt without deploying code. Each config has versions. Your app serves the Live version, and you evaluate other versions before you promote one to Live.

Use a config to serve, evaluate, and roll out a prompt in production. To compare prompts side by side before you commit to one, use the Playground instead.

The Statsig SDKs and the Console API still call configs "prompts." For example, the Node AI SDK fetches a config with getPrompt.

Find a config

Go to Agent Experimentation > Configs to open the list of configs in your project. The list has these columns:

Search the list by name, or sort it by Name or Last Modified. Filter it by Creator, Tag, or Target Application. The Target Application filter appears only when your project uses target applications.

Click a config to open it. The config page has three tabs: Setup, Runs, and Traces.

Create a config

1. Go to Agent Experimentation > Configs and click Create. You can also click Create on the Statsig home page, then click AI Config.
2. In the Create New Config dialog, enter a name. Statsig shows the config ID under the name, after ai_config_name:. The ID is the name in lowercase, with underscores in place of spaces. Your app uses the ID to fetch the config.
3. (Optional) Enter a description.
4. (Optional) Choose a Team, Tags, and Target Apps. By default, a new config uses your first team and that team's default target applications.
5. Click Create.

Statsig creates the config and opens its Setup tab. The config starts with one Live version named "Version 1," with an empty System message and an empty User message. Saved changes to Version 1 reach your app right away, because Version 1 is Live.

To turn prompts that you tested in a playground into a config, go to Promote prompts to a config.

Configure a version

Configure a config version on the config's Setup tab. The tab is one page with two groups of settings:

• Config Setup: The selected version's settings: Version, Custom workflow, Model, Prompt, and Parameters. Choose the version to configure from the Version list.
• Run Setup: The evaluators and the dataset that offline runs use. These settings apply to every version of the config. The group's Schedule section can also start offline runs of the versions you choose every day. For details, go to Schedule daily runs.

To add evaluators, go to Evaluators. Evaluators also decide a version's overall score. The config's primary evaluator sets the score, and a critical evaluator fails a run when it scores 0. To add a dataset, go to Add a dataset to a config.

Changes on the Setup tab stay unsaved until you click Save. Click Discard Changes to undo them.

Choose a model

Under Model, choose a Provider and a Model. Statsig maintains the list of models for each provider: OpenAI, Anthropic, and Google.

Some models limit the other settings:

Statsig charges previews and runs to your Included Credits unless your company has an API key for the provider. Without a key, a notice under Model shows an Add API key link. For details, go to Credits and API keys.

Write the prompt messages

Under Prompt, each message has a role: System, User, or Assistant. Click the role on a message to change it. A config version holds up to 10 messages.

• To add a message, click Add New Message.
• To copy or remove a message, open the message's menu and click Clone Prompt or Delete Prompt.

Messages can't be empty. When the config has a dataset, at least one message must reference a dataset column with a variable.

A variable is a dataset column name in double braces, such as {{input}}. In previews and offline runs, Statsig replaces each variable with the row's value from that column. The Variables list under the messages shows every column in the config's dataset. Click a variable to insert it at the cursor. A variable uses underscores where its column name has spaces.

When a column holds JSON, {{column}} inserts the whole value. To insert one field, add a dot path of keys and array indexes, such as {{order.items.0.name}}. A negative index counts from the end of the array. A path that doesn't exist inserts an empty string. A path that points to an object or an array inserts it as JSON.

Preview output

Results Preview runs the version on the first 5 rows of the config's dataset and shows each output. Use the preview to check a prompt before you start a full run. The preview uses the current settings, including unsaved changes.

To run the preview, click Preview Output under the messages. To run it again after a change, click Reload Preview. The config needs a dataset first.

Improve the prompt with Prompt Advisor

Prompt Advisor uses Statsig AI to rate the prompt messages and suggest a rewrite. When Prompt Advisor is on, a badge under the messages rates them as Weak, Acceptable, or Great.

1. Click the badge to show the Prompt Suggestion comments.
2. Click Rewrite with suggestions to get rewritten messages.
3. Click Apply to replace all of the version's messages with the suggestion.

An admin turns on Prompt Advisor in Settings > Product Configuration > AI Features. In the AI Configs section, turn on Prompt Advisor.

Set parameters

Under Parameters, set the values that Statsig sends to the model. Top P, Frequency Penalty, and Presence Penalty each have a switch. Statsig sends one of these values only when its switch is on.

Under Output Format, choose the format of the model's response:

• Text: Plain text.
• JSON Object: A JSON object.
• Structured Output (JSON Schema): JSON that matches a schema you enter. The schema is a JSON object with a name field and a schema field. The schema field needs type and properties fields.

Call your own endpoint with a custom workflow

A custom workflow sends each row of an offline run to your own endpoint instead of to the model. Use a custom workflow when your app produces the output with more than one model call, such as an agent that calls tools or retrieves documents. Statsig grades your endpoint's response. Your app still receives the version's model and prompt through the SDK.

To set up a custom workflow, open the config's Setup tab and turn on Call your own endpoint for more complex workflows under Custom workflow. Then fill in these fields. The version's Prompt messages become optional.

To check the settings, click Test Connection. Statsig sends one request and shows the response, or the HTTP error code.

During a run, Statsig adds two headers to each request. x-statsig-eval-run holds the run's ID, so your endpoint can skip side effects for requests from a run. traceparent holds the W3C trace context of the row's trace in Traces. A header that you set with the same name replaces Statsig's. Test Connection doesn't send these headers. To show your endpoint's spans in the row's trace, go to Join a workflow run's trace.

The workflow body accepts dataset columns, such as {{input}}, and these variables from the version:

For example, this body sends the messages, the model, and the row's input column:

{
  "messages": {{prompt.prompts}},
  "model": "{{prompt.model}}",
  "question": "{{input}}"
}

The SDK receives the workflow URL, body, and output field path with the version. The headers, auth headers, and batch size stay on Statsig's servers.

Manage versions

Each version of a config holds its own model, messages, and parameters. Your app serves the Live version by default, so you can test a change in another version first. Manage versions in the Version section of the config's Setup tab.

Version types

Every version has one of four types. The type decides whether your app receives the version and whether you can promote it.

Create and change versions

To create a version, open the Version list and click New version. A new version starts as a Draft copy of the Live version.

The menu next to the Version list has these actions:

Promote a version to Live

Promote to Live compares the version's latest completed offline run with the Live version's latest completed offline run. If the Live version has no completed run, Statsig promotes the version without asking. Otherwise, Statsig asks you to confirm in these cases:

Click Promote Anyway to promote the version.

Edit the Live version

Saving a change to the Live version changes what your app serves right away. If your app fetched the config recently, Statsig asks you to confirm in the Save Changes to Live Version dialog. To test a change before your app serves it, click New version, make the change in the new version, and evaluate the new version.

Start an offline run

In an offline run, Statsig runs versions on every row of the config's dataset and grades each output. The config needs a dataset and at least one evaluator first.

1. In the config's header, click Run Evaluators.
2. In the Start Evaluation dialog, select the versions to evaluate. Save any changes to a version before you evaluate it.
3. Click Start Evaluation.

While a run is in progress, Run Evaluators becomes Stop Evaluation. Results appear on the Runs tab. For how to read them, go to Offline Evaluation.

Create an experiment from a config

An experiment serves different versions of a config to different users and compares their metrics. You need at least two versions that are Live or Candidate.

1. In the config's header, click Create Experiment.
2. In the Create Experiment dialog, select at least two Live or Candidate versions, then click Create Experiment.
3. In the experiment form, choose the Control version and one or more Treatment versions.
4. Name the experiment, add a hypothesis, and create the experiment.

Statsig creates the experiment in the config's layer. The control group serves the Control version. Each Treatment version gets a test group, and Statsig splits traffic evenly across the groups. For each evaluator on the config, Statsig adds a primary metric named after the experiment and the evaluator.

The experiment appears in the config's header and in the Linked Experiments list in the Details panel. When you create an experiment from the Experiments page instead, the form shows a "Testing prompts or agents?" notice that links to Agent Experimentation.

Other config actions

Besides Run Evaluators and Create Experiment, the config's header has these actions. History, Edit Targeting Rules, and Code Snippets are icon buttons. The other actions are in the header's menu.

The Included Credits indicator next to the tabs shows the percentage of your Included Credits that your company has used. Hover over the indicator to find the Add API key link. The indicator disappears when your company has an API key for every provider in AI Provider API Keys.

Review results on the Runs and Traces tabs

• Runs: The results of offline runs for one version, or for two versions side by side. For details, go to Offline Evaluation.
• Traces: The traces your app sends for the config, with online evaluator scores. For details, go to Traces and Online Evaluation.

Serve a config in your app

Your app fetches a config by its ID with a Statsig AI SDK, then reads the Live version or the Candidate versions. In the Node AI SDK, call getPrompt. In the Python AI SDK, call get_prompt. To copy code for a specific config, click Code Snippets in the config's header.

Manage configs programmatically

To create, edit, and evaluate configs from code or from an AI agent, use the Console API or the Statsig MCP server. The Console API's config endpoints are in the Prompts section of the reference, starting with List Prompts.