# Set up AI Evals

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

Statsig AI Evals score the output of your prompts, both before and after you ship them. This walkthrough covers the full path in the Statsig console and in your code: create a prompt, score it against a dataset, serve it from your app, and grade real traffic.

AI Evals give you two feedback loops:

- **Offline evals**: Grade a prompt against a fixed dataset before you ship, so you catch regressions early.
- **Online evals**: Grade real production output after you ship, including shadow-running candidate versions your users never see.

## Create a prompt

**1. Create the prompt**

Go to _AI Evals > Prompts_ and select **Create New Prompt**. Give it a **Name** (this is the name your code references), an optional **Description**, and a **Target Application**. You can also set a **Team** and **Tags**.

**2. Configure the version**

Open the prompt. On the **Versions** tab, **Prompt Setup** subtab, configure:

- **Provider** and **Model**: The available models come from your project's configuration, so the list reflects what your project supports.
- **Parameters**: Set **Temperature** and **Max tokens**, plus optional **Top P**, **Frequency Penalty**, and **Presence Penalty**. Some reasoning models use a fixed temperature and disable the control.
- **Output Format**: Choose `Text`, `JSON Object`, or `Structured Output (JSON Schema)`.
- **Messages**: Add **System**, **User**, and **Assistant** messages. Insert variables with double-brace macros like `{{input}}`. A macro name matches a column in your eval dataset, or a value you pass at runtime. Statsig reserves `{{output}}` for the model's response.

**3. Understand versions**

A prompt has four version types. New versions start as **Draft**:

| Type | Served to your SDK? | Shown to users? | Use it for |
| --- | --- | --- | --- |
| **Draft** | No | No | Iterating in the console before you serve a version. |
| **Candidate** | Yes | No | Shadow-running and grading a version without exposing it. |
| **Live** | Yes | Yes | The version Statsig actively serves to users. |
| **Archive** | No | No | Inactive versions you keep for reference. |

Your SDK only receives **Live** and **Candidate** versions. Promote versions from the version menu with **Promote to Live**, **Set to Candidate**, **Set to Draft**, or **Archive**.

## Run an offline eval

Offline evals score a prompt version against a fixed dataset. Configure a dataset and graders on the prompt's **Eval Setup** tab, then run the eval from the **Versions** tab.

**1. Add a dataset**

On the **Eval Setup** tab, under **Dataset**, select **Select a Dataset**, then either **Upload Dataset** (a CSV) or browse an existing one. The first CSV row holds the column names, and you define the columns yourself, for example an `input` column and a `reference_output` column. A `categories` column (comma-separated) breaks results down by segment. You can also build a dataset by hand with **Add Row** and **Add Column** on the _Datasets_ page.

**2. Add graders**

Under **Graders**, select **Add New Grader** and pick a type:

- **LLM as a Judge**: Another model scores the output against a rubric you write. Reference the model output with `{{output}}`.
- **String Comparison**: Compare `{{output}}` to a column with **Equals**, **Does not equal**, **Contains**, or **Contains (ignore case)**.
- **Text Similarity**: Score how close `{{output}}` is to a column using a function like fuzzy match or cosine similarity.
- **Python**: Score the output with your own Python.

Every grader returns a score between 0 and 1. Mark one grader **Primary** to drive the version's overall score. Mark a grader **Critical** when it's a must-pass check: if a critical grader scores 0, Statsig fails the whole run.

**3. Run the eval**

Go to the **Versions** tab, **Prompt Setup** subtab, and select **Start Evaluation**. Select the versions you want to score and confirm with **Run Eval**. The button stays disabled until the prompt has both a dataset and at least one grader.

**4. Read the results**

Results appear on the **Eval Results** subtab: an overall **Score** gauge, a card per grader, a breakdown by category, and a per-row table. Select any row to inspect its input, the model's output, and the score. Use **Compare** to diff versions, and **Promote to Live** when a version is ready.

> **Tip:**
>
> Iterate by creating new versions of the prompt, running the eval on each, and comparing scores. Promote the best version to **Live** to serve it.

## Serve the prompt in your app

Retrieve the prompt at runtime with the [Statsig AI SDK](https://docs.statsig.com/ai-evals/node) and run a completion with your provider. First, install and initialize the SDK.

#### Node

```js
npm install @statsig/statsig-ai
```

#### Python

```python
pip install statsig-ai
```

#### Node

```js
import { StatsigAI } from '@statsig/statsig-ai';

const statsigAI = new StatsigAI({ sdkKey: 'YOUR_SERVER_SECRET_KEY' });
await statsigAI.initialize();
```

#### Python

```python
from statsig_ai import StatsigAI, StatsigCreateConfig

statsig_ai = StatsigAI(statsig_source=StatsigCreateConfig(server_secret_key='YOUR_SERVER_SECRET_KEY'))
statsig_ai.initialize()
```

Fetch the prompt and use its **Live** version. Statsig serves the prompt config; your app makes the completion call with your own provider client (OpenAI here, but any provider works). `getPromptMessages` fills your `{{macros}}` and returns messages ready to pass to the provider.

#### Node

```js
import { StatsigUser } from '@statsig/statsig-ai';

const user = new StatsigUser({ userID: 'a-user' });

const prompt = statsigAI.getPrompt(user, 'support_summary');
const live = prompt.getLive();

const response = await openai.chat.completions.create({
  model: live.getModel({ fallback: 'gpt-4.1' }),
  temperature: live.getTemperature(),
  max_tokens: live.getMaxTokens(),
  messages: live.getPromptMessages({ input: userInput }),
});
```

#### Python

```python
from statsig_ai import StatsigUser

user = StatsigUser(user_id='a-user')

prompt = statsig_ai.get_prompt(user, 'support_summary')
live = prompt.get_live()

response = openai.chat.completions.create(
    model=live.get_model(fallback='gpt-4.1'),
    temperature=live.get_temperature(),
    max_tokens=live.get_max_tokens(),
    messages=live.get_prompt_messages({'input': user_input}),
)
```

## Grade production traffic with online evals

Online evals grade real production output, without a ground-truth answer to compare against. You can also shadow-run **Candidate** versions in the background, so you gather grades on a new version before any user sees it.

### Serve the live and candidate versions

Show the **Live** output to your user, and run **Candidate** versions in the background without showing them. Statsig serves each version's config; your app makes the completion calls, so your code runs both the live and the candidate completions.

```js
const prompt = statsigAI.getPrompt(user, 'support_summary');
const live = prompt.getLive();
const candidates = prompt.getCandidates();

// Show the live output to the user
const liveResponse = await openai.chat.completions.create({
  model: live.getModel({ fallback: 'gpt-4.1' }),
  temperature: live.getTemperature(),
  max_tokens: live.getMaxTokens(),
  messages: live.getPromptMessages({ input: userInput }),
});

// Shadow-run each candidate in the background. Don't show these to the user.
for (const candidate of candidates) {
  const candidateResponse = await openai.chat.completions.create({
    model: candidate.getModel({ fallback: 'gpt-4.1' }),
    temperature: candidate.getTemperature(),
    max_tokens: candidate.getMaxTokens(),
    messages: candidate.getPromptMessages({ input: userInput }),
  });
  // Grade candidateResponse and log the grade (covered below)
}
```

### Choose how to produce grades

There are two ways to turn production output into grades. You can use either or both.

Automatic grading lets Statsig grade a sample of your live traffic for you:

1. On the prompt's **Eval Setup** tab, turn on **Enable for online eval** for a grader.
2. Under **Online Configuration**, set a **Sampling Rate**, the percentage of traffic Statsig grades.

Statsig grades the sampled traffic from the OpenTelemetry traces your application sends, so you need telemetry flowing. The [Node AI SDK](https://docs.statsig.com/ai-evals/node) provides an `initializeTracing` function and a `wrapOpenAI` helper to emit these traces (Python tracing support is coming soon). Online graders have no ground-truth column to compare against, so they grade from the model output alone. Use an LLM-as-a-judge grader whose only variable is `{{output}}`, or a Python grader that doesn't read dataset columns.

Manual grading lets you grade the output in your own code and log the score. Scores must be between 0 and 1.

#### Node

```js
statsigAI.logEvalGrade(user, live, 0.92, 'helpfulness', {
  sessionId: sessionId,
});

// Flush eval grade events to Statsig
await statsigAI.flushEvents();
```

#### Python

```python
statsig_ai.log_eval_grade(user, live, 0.92, 'helpfulness', {
    'session_id': session_id,
})

# Flush eval grade events to Statsig
statsig_ai.flush().wait()
```

### View online results

Open the prompt, go to the **Versions** tab, **Eval Results** subtab, and switch to **Online Results**. Each version's exposures and grader scores appear over time. Compare **Candidate** versions against **Live** for grader deltas relative to the **Live** baseline, before you promote a candidate.

## Next steps

- [Prompts & Graders](https://docs.statsig.com/ai-evals/prompts): Go deeper on prompt versions, graders, and critical graders.
- [Offline Evals](https://docs.statsig.com/ai-evals/offline-evals): Walk through the full offline eval workflow with screenshots.
- [Online Evals](https://docs.statsig.com/ai-evals/online-evals): Learn more about grading production traffic.
- [Node AI SDK](https://docs.statsig.com/ai-evals/node) and [Python AI SDK](https://docs.statsig.com/ai-evals/python): Reference for the AI SDKs.