---
title: Set up Agent Experimentation
description: "Set up Statsig Agent Experimentation end to end: create a config, grade it with an offline run, serve it from a Node.js app, trace model calls, and grade production traffic with online evaluation."
product: general
lang: en
token_estimate: 3729
---
# Set up Agent Experimentation

This walkthrough sets up Statsig Agent Experimentation from start to finish. You create a config and grade its first version against a dataset. A config here is an Agent Experimentation config, not a [Dynamic Config](https://docs.statsig.com/dynamic-config/overview). Then you serve the config from a Node.js app and grade the app's production traffic with online evaluation. Each step covers the minimum you need and links to the page with the full details.

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

Use this walkthrough to release one LLM task with evaluation before and after it reaches users. To compare prompts side by side before you create a config, use the [Playground](https://docs.statsig.com/agent-experimentation/playground) instead.

The Statsig SDKs and the Console API still call configs "prompts." For example, the Node AI SDK fetches a config with `getPrompt()`.

A [Server Secret Key](https://docs.statsig.com/access-management/api-keys#server-secret-keys) for the Statsig project, which your app uses to fetch configs and send traces.

Included Credits or your own model provider API keys. Statsig charges model calls from previews, runs, and LLM as a Judge evaluators to your organization's Included Credits. Calls on your project's own provider API keys don't use credits. An admin adds keys in _Settings > Product Configuration > General_, under **AI Provider API Keys**. For details, go to [Credits and API keys](https://docs.statsig.com/agent-experimentation/overview#credits-and-api-keys).

A project or org admin for the steps in [Grade production traffic](#grade-production-traffic).

A Node.js server app that calls OpenAI, Anthropic, or Google Gemini. The code on this page uses the [Node AI SDK](https://docs.statsig.com/agent-experimentation/node). From another language, send traces with OpenTelemetry instead. For details, go to [Send traces to the Traces page](https://docs.statsig.com/agent-experimentation/traces#send-traces-to-ai-traces).

## Create a config

A config stores the model, prompt messages, and parameters for one LLM task, such as answering support questions. Your app fetches the config at runtime, so you can change the model or the prompt without deploying code.

1. Go to _Agent Experimentation > Configs_ and click **Create**.
2. In the _Create New Config_ dialog, enter a name, such as `Support Bot`. Statsig shows the config ID under the name, after `ai_config_name:`. For `Support Bot`, the ID is `support_bot`. Your code fetches the config by this ID.
3. Click **Create**.

Statsig opens the config's _Setup_ tab. The config starts with one Live version, named "Version 1." Your app serves the Live version, so saved changes to Version 1 reach your app right away. For the optional fields, such as team and tags, go to [Create a config](https://docs.statsig.com/agent-experimentation/configs#create-an-ai-config).

## Configure the version

Configure Version 1 in the _Config Setup_ group of the config's _Setup_ tab:

1. Under _Model_, choose a **Provider** and a **Model**. Statsig maintains the list of models for OpenAI, Anthropic, and Google.
2. Under _Prompt_, fill in the System message and the User message. For example:

   - System: `Answer customer support questions in two sentences or fewer.`
   - User: `{{input}}`
3. (Optional) Under _Parameters_, change **Temperature** or **Max tokens**.
4. Click **Save**.

`{{input}}` is a variable. In offline runs, Statsig replaces the variable with each dataset row's value from the `input` column. Evaluators read the same column as their own `{{input}}` variable. In your app, you pass the variable's value to `getPromptMessages()`.

For model limits, output formats, Prompt Advisor, and custom workflows, go to [Configure a version](https://docs.statsig.com/agent-experimentation/configs#configure-a-version).

## Add a dataset and an evaluator

In an offline run, Statsig runs a version on every row of the config's dataset, then scores each output with the config's evaluators. Set the dataset and evaluators in the _Run Setup_ group of the _Setup_ tab. They apply to every version of the config.

To add a dataset:

1. Create a CSV file with an `input` column and one test question in each row. For example:

   ```csv
   input
   How do I reset my password?
   Can I change the email address on my account?
   My order arrived damaged. What should I do?
   ```
2. Under _Dataset_, click **Select a Dataset**, then click **Upload Dataset**.
3. In the _Upload Dataset_ drawer, click **Upload CSV** and choose the file. Check the preview, then click **Upload**.
4. Click **Save**.

An upload accepts one CSV file of up to 2 MB and 1,000 rows. To use a global dataset or generate rows instead, go to [Add a dataset to a config](https://docs.statsig.com/agent-experimentation/datasets#add-a-dataset-to-an-ai-config).

To add an evaluator:

1. Under _Evaluators_, click **Add evaluator**, then click **Create new evaluator**.
2. Choose **LLM as a Judge**.
3. In the _Create evaluator_ drawer, enter a **Name**, such as `Helpfulness`, and write the judge prompt. For example:

   ```text
   Score how well the answer resolves the request, from 0 to 1.

   <request>
   {{{input}}}
   </request>
   <answer>
   {{{output}}}
   </answer>
   ```
4. Click **Add evaluator**, then click **Save**.

`{{output}}` holds the model's answer. Three braces pass a variable's text without HTML escaping. This evaluator reads only `{{input}}` and `{{output}}`, so it can also grade production traffic in [Grade production traffic](#grade-production-traffic). The first evaluator you add becomes the config's primary evaluator, whose score is each run's overall score.

Evaluators you create from a config join your company's _Evaluators_ catalog. For evaluator types, templates, and evaluation models, go to [Evaluators](https://docs.statsig.com/agent-experimentation/evaluators).

## Start an offline run

1. In the config's header, click **Run Evaluators**.
2. In the _Start Evaluation_ dialog, keep Version 1 selected and click **Start Evaluation**.

Statsig opens the _Runs_ tab on the new run. A run can take a few minutes. When it finishes, the tab shows these results:

- **Model, Score, and Prompt cards**: The model and prompt messages the run used, and the run's overall score from 0 to 1.
- **Offline evaluators chart**: Each evaluator's score for the run, which is the mean of its row scores.
- **Results table**: One row for each dataset row, with the output and each evaluator's score. Click a row to open _Row Details_, which shows the judge's reasoning.

If you can't click **Run Evaluators**, hover over the button to read the reason. For each reason and its fix, go to [Why Run Evaluators is disabled](https://docs.statsig.com/agent-experimentation/offline-evaluation#why-run-eval-is-disabled).

To improve on Version 1, create a new version, start an offline run of both versions, and click **Compare** on the _Runs_ tab. Promote the better version with **Promote to Live**. For details, go to [Manage versions](https://docs.statsig.com/agent-experimentation/configs#manage-versions) and [Compare two versions](https://docs.statsig.com/agent-experimentation/offline-evaluation#compare-two-versions).

## Serve the config from your app

Your app fetches the config with the [Node AI SDK](https://docs.statsig.com/agent-experimentation/node), then calls the model with the Live version's settings. Install the SDK and the OpenAI client:

```shell
npm install @statsig/statsig-ai @statsig/statsig-node-core openai
```

For pnpm and Yarn commands, go to [Set up the SDK](https://docs.statsig.com/agent-experimentation/node#set-up-the-sdk).

Then fetch the config and call the model inside `withLive()`. This code reads your Server Secret Key from the `STATSIG_SDK_KEY` environment variable.

```ts
import OpenAI from "openai";
import { StatsigAI, StatsigUser } from "@statsig/statsig-ai";

const statsigAI = StatsigAI.newShared({ sdkKey: process.env.STATSIG_SDK_KEY! });
await statsigAI.initialize();

const openai = new OpenAI();

export async function answerQuestion(userID: string, question: string) {
  const user = new StatsigUser({ userID });
  const prompt = statsigAI.getPrompt(user, "support_bot");

  // withLive passes the Live version to your function
  return prompt.withLive(async (version) => {
    const response = await openai.chat.completions.create({
      model: version.getModel({ fallback: "gpt-5-mini" }),
      messages: version.getPromptMessages({
        input: question,
      }) as OpenAI.ChatCompletionMessageParam[],
    });
    return response.choices[0].message.content;
  });
}
```

`getPromptMessages()` returns the version's messages with each variable replaced, so `{{input}}` becomes the user's question. In TypeScript, cast the messages to OpenAI's message type. `withLive()` also tags the traced model calls inside it with the config and version, which online evaluation needs. For the other version getters, go to [Get a config](https://docs.statsig.com/agent-experimentation/node#getting-a-prompt). The Python AI SDK fetches configs with `get_prompt()`. For details, go to [Python AI SDK](https://docs.statsig.com/agent-experimentation/python#getting-a-prompt).

## Trace model calls

Online evaluation grades the spans that your app sends to Statsig. Each span needs its input and output messages, plus the config and version that produced it. To send these spans, add tracing to the code in [Serve the config from your app](#serve-the-ai-config-from-your-app):

1. Call `initializeTracing()` when your app starts. Call it before you wrap your model client.
2. Wrap the OpenAI client with `wrapOpenAI`, and turn on `captureOptions` for input and output messages.

```ts
import OpenAI from "openai";
import {
  StatsigAI,
  StatsigUser,
  initializeTracing,
  wrapOpenAI,
} from "@statsig/statsig-ai";

// Start tracing before you wrap the client. Reads the key from STATSIG_SDK_KEY.
initializeTracing({ serviceName: "support-bot" });

const statsigAI = StatsigAI.newShared({ sdkKey: process.env.STATSIG_SDK_KEY! });
await statsigAI.initialize();

// Record the input and output messages that online graders read
const openai = wrapOpenAI(new OpenAI(), {
  captureOptions: {
    capture_input_messages: true,
    capture_output_messages: true,
  },
});

// answerQuestion() stays the same
```

`answerQuestion()` already calls the model inside `withLive()`, so each span records the config in `statsig.llm.prompt_name` and the version in `statsig.llm.prompt_version_id`. Note these points about tracing:

- A client that you wrap before you call `initializeTracing()` doesn't send spans. The SDK logs `NoopOtelSingleton instance is being used` when you wrap it.
- The SDK doesn't record message content unless you turn on `captureOptions`. Online evaluation skips spans without input and output messages.
- `wrapAnthropic` and `wrapGemini` take the same options for Anthropic and Gemini clients.
- In a short-lived process, such as a script or a serverless function, call `await statsigAI.flushEvents()` before the process exits.

Run your app, then go to _Agent Experimentation > Traces_ to check that the spans arrive. For every tracing option, go to [Trace model calls](https://docs.statsig.com/agent-experimentation/node#trace-model-calls). To grade a multi-step agent turn as one span, go to [Grade a multi-step agent turn](https://docs.statsig.com/agent-experimentation/online-evaluation#grade-a-multi-step-agent-turn).

## Grade production traffic

An online evaluator scores a sample of your app's spans about one minute after Statsig receives them. Each evaluator's online targeting rules decide which spans the evaluator scores. Add an online rule that names your config to the evaluator from [Add a dataset and an evaluator](#add-a-dataset-and-a-grader). A project or org admin must save the rule.

1. Go to _Agent Experimentation > Evaluators_ and open your evaluator.
2. On the evaluator's _Setup_ tab, under _Targeting rules_, click **Add rule** and check **Online**.
3. Add a condition: choose **Config**, choose **is any of**, and select your config.
4. In **Sample**, enter the percentage of matching traces to grade. New rules start at 10%. While you test with a few requests, enter 100 so that Statsig grades every trace.
5. Click **Save Evaluator**.

The evaluator's _Targeting rules_ also list an offline rule for your config, which **Add evaluator** created. You can check **Online** on that rule instead of adding a rule.

Statsig grades at most 1,000 of the config's spans per hour when every rule that grades them names the config. Statsig processes traces for online evaluation in the US only, and plans to support EU data residency. For every rule field, go to [Online rules](https://docs.statsig.com/agent-experimentation/evaluators#online-rules). For every limit, including the cap that other spans share, go to [Online grading limits](https://docs.statsig.com/agent-experimentation/online-evaluation#online-grading-limits).

## Review online evaluation results

Statsig shows online grades on the config's _Traces_ tab:

1. Open the config and click the **Traces** tab.
2. Read the Online evaluators chart above the span list. Choose **Single version** to show each evaluator's average score over time for one version. Choose **Compare** to show each evaluator's score change relative to the Live version.
3. Click a span in the list to open the trace drawer. The _Trace_ tab shows the span's messages, and the _Details_ tab shows the model, tokens, and cost. The _Evaluators_ tab shows each evaluator's score, with the judge's reasoning.

The span list's Evaluators column shows a pass or fail chip for each evaluator, or `Not graded`. If a span still shows `Not graded` after a few minutes, check these causes:

- The span has no messages. Turn on `captureOptions` for input and output messages.
- The rule's **Sample** percentage skipped the span's trace.
- Your organization's Included Credits ran out, and your project has no API key for the evaluator's provider.

If your spans don't appear on the _Traces_ tab at all, check that the model call runs inside `withLive()`. For every requirement a span must meet, go to [What Statsig grades](https://docs.statsig.com/agent-experimentation/online-evaluation#what-statsig-grades). For the trace drawer, go to [Inspect a trace](https://docs.statsig.com/agent-experimentation/traces#inspect-a-trace).

To grade a Candidate version on production traffic before you promote it, run the version in the background with `prompt.withVersion()`. Users never see the Candidate version's answers. For an example, go to [Grade a single model call](https://docs.statsig.com/agent-experimentation/online-evaluation#grade-a-single-model-call).