---
title: Agent Experimentation Overview
description: Overview of Statsig Agent Experimentation for evaluating the prompts and models behind AI applications with offline and online evaluators.
product: general
lang: en
token_estimate: 2798
---
# Agent Experimentation Overview

Statsig Agent Experimentation serves the prompts and models behind your AI app or agent, and grades their output before and after release. Each LLM task's model, prompt messages, and parameters live in a config. A config here is an Agent Experimentation config, not a [Dynamic Config](https://docs.statsig.com/dynamic-config/overview). Your app fetches the config at runtime, so you can change a prompt or model without deploying code. Evaluators score output from 0 to 1, on test datasets before a release and on a sample of live traffic after it.

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

Use Agent Experimentation to find out whether a new prompt or model gives better answers. To measure whether the change also moves product metrics, such as retention, create an experiment from the config. For details, go to [Use configs with gates, experiments, and analytics](#use-ai-configs-with-gates-experiments-and-analytics).

The console renamed AI Configs to Configs, Graders to Evaluators, Evals to Runs, and AI Traces to Traces. The Statsig SDKs and the Console API still use the older names for configs ("prompts"), evaluators ("graders"), and run groups ("evals"), so the Node AI SDK, for example, fetches a config with `getPrompt`.

## Parts of Agent Experimentation

Agent Experimentation has these parts:

| Part | What it does |
| --- | --- |
| [Configs](https://docs.statsig.com/agent-experimentation/configs) | Store the model, prompt messages, and parameters for one LLM task, in versions. Your app serves the Live version. |
| [Playground](https://docs.statsig.com/agent-experimentation/playground) | Runs up to 10 prompts side by side on the same inputs. You can promote the prompts to a new config. |
| [Datasets](https://docs.statsig.com/agent-experimentation/datasets) | Hold test cases for runs to grade, such as user questions and expected answers. |
| [Evaluators](https://docs.statsig.com/agent-experimentation/evaluators) | Score output from 0 to 1 with an LLM judge, Python code, a string comparison, or a text similarity function. Evaluators live in one catalog, so every config and run can reuse them. |
| [Offline evaluation](https://docs.statsig.com/agent-experimentation/offline-evaluation) | Run config versions on every dataset row and grade the outputs. Compare versions before you promote one to Live. |
| [Runs](https://docs.statsig.com/agent-experimentation/runs) | Grade outputs that already exist in a dataset, such as answers from your own agent, and track scores across runs. |
| [Online evaluation](https://docs.statsig.com/agent-experimentation/online-evaluation) | Grade a sample of production traffic about one minute after your app sends each span. |
| [Traces](https://docs.statsig.com/agent-experimentation/traces) | Lists the LLM spans your app sends, with their messages, model, tokens, cost, and evaluator scores. You can save spans to a dataset. |

The [Node AI SDK](https://docs.statsig.com/agent-experimentation/node) and the [Python AI SDK](https://docs.statsig.com/agent-experimentation/python) connect your app to Agent Experimentation. They fetch configs, send model calls to Traces as spans, and log grades that your own code computes.

### How the parts fit together

A typical workflow uses the parts in this order:

1. Try prompts and models in the Playground, then promote the best prompts to a config.
2. Add a dataset and evaluators to the config. Start an offline run of each version, then promote the best version to Live.
3. Fetch the config in your app with a Statsig AI SDK, and trace its model calls.
4. Add online rules to your evaluators to grade production traffic.
5. Review spans on the _Traces_ page. Add spans with poor answers to a dataset, so the next offline run tests them.

For a step-by-step walkthrough, go to [Set up Agent Experimentation](https://docs.statsig.com/agent-experimentation/setup).

### Offline and online grading

Offline runs of a config and runs on the _Runs_ page grade every row of a dataset, so an evaluator can compare the output with an expected answer from the dataset. Production traffic has no expected answer. Online evaluation therefore uses evaluators that score each answer on its own, such as LLM judges and Python evaluators. Each evaluator's targeting rules decide where the evaluator runs. For details, go to [Choose where an evaluator runs](https://docs.statsig.com/agent-experimentation/evaluators#choose-where-a-grader-runs).

## Use configs with gates, experiments, and analytics

Configs work with the rest of Statsig:

- **Feature gates**: To release an AI feature to some users first, check a [feature gate](https://docs.statsig.com/feature-flags/overview) in your code before you fetch the config. To serve a different config to users who match a rule, click **Edit Targeting Rules** in the config's header.
- **Experiments**: To measure how config versions affect product metrics, click **Create Experiment** in the config's header. You need at least two versions that are Live or Candidate. Statsig creates the experiment in the config's layer, gives each version a group, and splits traffic evenly across the groups. Statsig also adds a primary metric for each evaluator on the config. These evaluator metrics read only grades that your code logs with `logEvalGrade`. For the steps, go to [Create an experiment from a config](https://docs.statsig.com/agent-experimentation/configs#create-an-experiment-from-an-ai-config).
- **Analytics**: Statsig logs online grades and `logEvalGrade` grades as `statsig::eval_result` events in your project. For the event's fields, go to [Grade events in your project](https://docs.statsig.com/agent-experimentation/online-evaluation#grade-events-in-your-project).

## LLM as a judge

Some checks follow a fixed rule. For example, when an answer must be High, Medium, or Low, a String Comparison evaluator checks for an exact match. Other checks need judgment. "Your ticket has been escalated" and "This ticket has been escalated" mean the same thing, but an exact match scores them as different.

An LLM as a Judge evaluator sends the output and your judge prompt to a model. The model returns a score from 0 to 1 with its reasoning. For example, the prompt can ask the model to score how close the answer is to an ideal answer. A judge can be wrong, but it applies the same prompt to every output and needs no human reviewers. Judges make it practical to compare versions across many outputs.

A judge can use any provider and model in the evaluator's **Evaluation Model** list, such as OpenAI, Anthropic, or Google models. New evaluators use OpenAI `gpt-5-mini`. To choose a model and write a judge prompt, go to [Choose the evaluation model](https://docs.statsig.com/agent-experimentation/evaluators#choose-the-evaluation-model).

## Credits and API keys

Agent Experimentation calls model providers to generate and grade output. Each Statsig organization gets $100 of Included Credits for these calls, shared by all of its projects. A project that isn't in an organization gets its own $100. Statsig charges each model call that runs on Statsig's provider keys against the credits, based on the model's token prices.

Included Credits are for getting started. To keep using Agent Experimentation after they run out, add your own API keys for the model providers you use. Statsig doesn't meter calls that use your own keys.

### What Included Credits cover

These activities use Included Credits when your project has no API key for the provider they call:

| Activity | Model calls that use credits |
| --- | --- |
| Previews of config output and evaluator scores | The model of the config version or of the judge |
| Offline runs from a config, including scheduled runs | The version's model for each dataset row, and evaluators that call a provider |
| Runs on the _Runs_ page | Evaluators that call a provider |
| Online evaluation | Evaluators that call a provider |
| Playground runs | The model of each prompt |

Two kinds of evaluator call a provider. LLM as a Judge evaluators call the provider in **Evaluation Model**. Text Similarity evaluators that use Cosine call OpenAI for `text-embedding-3-large` embeddings. String Comparison, Python, and the other Text Similarity functions run inside Statsig at no cost. Config versions with a custom workflow call your own endpoint, so they don't use credits. For a summary by evaluator type, go to [Evaluator costs](https://docs.statsig.com/agent-experimentation/evaluators#grader-costs).

### Track Included Credits

The _Included Credits_ indicator next to the tabs on each config page shows what percentage of the Included Credits your organization has used. Hover over the indicator to find the **Add API key** link. The indicator disappears when your project has an API key for every provider in _AI Provider API Keys_: OpenAI, Anthropic, Google, and Meta.

When a model uses a provider that has no API key, a notice under the model also shows an **Add API key** link. The notice appears in the _Model_ section of a config's _Setup_ tab and under **Evaluation Model** on an evaluator's _Setup_ tab.

### Add your own API keys

API keys belong to a project, and only admins can add them.

1. Go to _Settings > Product Configuration > General_. Console messages call this page _Settings > Products_.
2. In the **AI Provider API Keys** row, click **Edit**.
3. In the _Edit AI Provider API Keys_ dialog, click **Add API Key**.
4. Choose a **Provider**, enter the key in **API Key**, and click **Add API Key**. Repeat for each provider that your configs, evaluators, and playgrounds use.
5. Click **Save**.

After you save, Statsig uses your key for every call to that provider. The provider bills your account for those calls.

### When Included Credits run out

When your organization's Included Credits run out, Statsig stops making calls on its own provider keys. Work that doesn't need those keys keeps running: calls on your own API keys, config versions with a custom workflow, and evaluators that call no provider.

- **Offline work**: Statsig blocks previews, offline runs, runs on the _Runs_ page, and Playground runs that call a provider without an API key. The console shows "Included Credits have run out" and names the provider that needs a key. Statsig also skips scheduled runs that need credits.
- **Online evaluation**: Statsig keeps running evaluators that use your own API keys and evaluators that call no provider, such as Python evaluators. Statsig skips the other evaluators, and the trace drawer explains that Statsig didn't grade the span.

### Judges that use other providers

Statsig calls LLM as a Judge evaluators that use a provider other than OpenAI through Statsig's model gateway. If your project has an API key for that provider, the gateway call uses your key, and Statsig doesn't meter it. Without a key, the call uses Statsig's key and spends Included Credits.

## Data residency

Statsig processes traces for online evaluation in the US only. Statsig plans to support EU data residency.