Agent Experimentation Overview

Statsig Agent Experimentation serves the prompts and models behind your AI app or agent, and grades their output before and after release. Each LLM task's model, prompt messages, and parameters live in a config. A config here is an Agent Experimentation config, not a Dynamic Config. Your app fetches the config at runtime, so you can change a prompt or model without deploying code. Evaluators score output from 0 to 1, on test datasets before a release and on a sample of live traffic after it.

Use Agent Experimentation to find out whether a new prompt or model gives better answers. To measure whether the change also moves product metrics, such as retention, create an experiment from the config. For details, go to Use configs with gates, experiments, and analytics.

The console renamed AI Configs to Configs, Graders to Evaluators, Evals to Runs, and AI Traces to Traces. The Statsig SDKs and the Console API still use the older names for configs ("prompts"), evaluators ("graders"), and run groups ("evals"), so the Node AI SDK, for example, fetches a config with getPrompt.

Parts of Agent Experimentation

Agent Experimentation has these parts:

The Node AI SDK and the Python AI SDK connect your app to Agent Experimentation. They fetch configs, send model calls to Traces as spans, and log grades that your own code computes.

How the parts fit together

A typical workflow uses the parts in this order:

1. Try prompts and models in the Playground, then promote the best prompts to a config.
2. Add a dataset and evaluators to the config. Start an offline run of each version, then promote the best version to Live.
3. Fetch the config in your app with a Statsig AI SDK, and trace its model calls.
4. Add online rules to your evaluators to grade production traffic.
5. Review spans on the Traces page. Add spans with poor answers to a dataset, so the next offline run tests them.

For a step-by-step walkthrough, go to Set up Agent Experimentation.

Offline and online grading

Offline runs of a config and runs on the Runs page grade every row of a dataset, so an evaluator can compare the output with an expected answer from the dataset. Production traffic has no expected answer. Online evaluation therefore uses evaluators that score each answer on its own, such as LLM judges and Python evaluators. Each evaluator's targeting rules decide where the evaluator runs. For details, go to Choose where an evaluator runs.

Use configs with gates, experiments, and analytics

Configs work with the rest of Statsig:

• Feature gates: To release an AI feature to some users first, check a feature gate in your code before you fetch the config. To serve a different config to users who match a rule, click Edit Targeting Rules in the config's header.
• Experiments: To measure how config versions affect product metrics, click Create Experiment in the config's header. You need at least two versions that are Live or Candidate. Statsig creates the experiment in the config's layer, gives each version a group, and splits traffic evenly across the groups. Statsig also adds a primary metric for each evaluator on the config. These evaluator metrics read only grades that your code logs with logEvalGrade. For the steps, go to Create an experiment from a config.
• Analytics: Statsig logs online grades and logEvalGrade grades as statsig::eval_result events in your project. For the event's fields, go to Grade events in your project.

LLM as a judge

Some checks follow a fixed rule. For example, when an answer must be High, Medium, or Low, a String Comparison evaluator checks for an exact match. Other checks need judgment. "Your ticket has been escalated" and "This ticket has been escalated" mean the same thing, but an exact match scores them as different.

An LLM as a Judge evaluator sends the output and your judge prompt to a model. The model returns a score from 0 to 1 with its reasoning. For example, the prompt can ask the model to score how close the answer is to an ideal answer. A judge can be wrong, but it applies the same prompt to every output and needs no human reviewers. Judges make it practical to compare versions across many outputs.

A judge can use any provider and model in the evaluator's Evaluation Model list, such as OpenAI, Anthropic, or Google models. New evaluators use OpenAI gpt-5-mini. To choose a model and write a judge prompt, go to Choose the evaluation model.

Credits and API keys

Agent Experimentation calls model providers to generate and grade output. Each Statsig organization gets $100 of Included Credits for these calls, shared by all of its projects. A project that isn't in an organization gets its own $100. Statsig charges each model call that runs on Statsig's provider keys against the credits, based on the model's token prices.

Included Credits are for getting started. To keep using Agent Experimentation after they run out, add your own API keys for the model providers you use. Statsig doesn't meter calls that use your own keys.

What Included Credits cover

These activities use Included Credits when your project has no API key for the provider they call:

Two kinds of evaluator call a provider. LLM as a Judge evaluators call the provider in Evaluation Model. Text Similarity evaluators that use Cosine call OpenAI for text-embedding-3-large embeddings. String Comparison, Python, and the other Text Similarity functions run inside Statsig at no cost. Config versions with a custom workflow call your own endpoint, so they don't use credits. For a summary by evaluator type, go to Evaluator costs.

Track Included Credits

The Included Credits indicator next to the tabs on each config page shows what percentage of the Included Credits your organization has used. Hover over the indicator to find the Add API key link. The indicator disappears when your project has an API key for every provider in AI Provider API Keys: OpenAI, Anthropic, Google, and Meta.

When a model uses a provider that has no API key, a notice under the model also shows an Add API key link. The notice appears in the Model section of a config's Setup tab and under Evaluation Model on an evaluator's Setup tab.

Add your own API keys

API keys belong to a project, and only admins can add them.

1. Go to Settings > Product Configuration > General. Console messages call this page Settings > Products.
2. In the AI Provider API Keys row, click Edit.
3. In the Edit AI Provider API Keys dialog, click Add API Key.
4. Choose a Provider, enter the key in API Key, and click Add API Key. Repeat for each provider that your configs, evaluators, and playgrounds use.
5. Click Save.

After you save, Statsig uses your key for every call to that provider. The provider bills your account for those calls.

When Included Credits run out

When your organization's Included Credits run out, Statsig stops making calls on its own provider keys. Work that doesn't need those keys keeps running: calls on your own API keys, config versions with a custom workflow, and evaluators that call no provider.

• Offline work: Statsig blocks previews, offline runs, runs on the Runs page, and Playground runs that call a provider without an API key. The console shows "Included Credits have run out" and names the provider that needs a key. Statsig also skips scheduled runs that need credits.
• Online evaluation: Statsig keeps running evaluators that use your own API keys and evaluators that call no provider, such as Python evaluators. Statsig skips the other evaluators, and the trace drawer explains that Statsig didn't grade the span.

Judges that use other providers

Statsig calls LLM as a Judge evaluators that use a provider other than OpenAI through Statsig's model gateway. If your project has an API key for that provider, the gateway call uses your key, and Statsig doesn't meter it. Without a key, the call uses Statsig's key and spends Included Credits.

Data residency

Statsig processes traces for online evaluation in the US only. Statsig plans to support EU data residency.