---
title: Offline Evaluation
description: "Start offline runs in Statsig Agent Experimentation to grade every version of a config against a dataset, compare versions with a baseline, and catch regressions before you ship."
product: general
lang: en
token_estimate: 3348
---
# Offline Evaluation

An offline run grades AI output for every row of a fixed dataset of test cases, before any user sees that output. Statsig scores each row with the evaluators you choose, then reports each evaluator's score and an overall score for the run. Use offline runs to compare [config](https://docs.statsig.com/agent-experimentation/configs) versions and catch regressions before you promote a version to Live. To grade a sample of live production traffic instead, use [online evaluation](https://docs.statsig.com/agent-experimentation/online-evaluation). To try a few prompts side by side without grading, use the [Playground](https://docs.statsig.com/agent-experimentation/playground).

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

## Choose where to run offline evaluation

Agent Experimentation runs offline evaluation in two places. Choose the one that matches the outputs you want to grade:

|  | Offline run from a config | Run on the Runs page |
| --- | --- | --- |
| What it grades | New outputs that each config version generates for every dataset row | Outputs already stored in a dataset's output column |
| Use it to | Compare prompt, model, and parameter changes between versions | Grade outputs from your own agent or from production traces, without generating new ones |
| Where results appear | The config's _Runs_ tab | The run group's _Runs_ and _Compare_ tabs |

This page covers offline runs from a config. For grade-only runs on stored outputs, go to [Runs](https://docs.statsig.com/agent-experimentation/runs). To start a run from your own code with `Eval()`, go to the [Node AI SDK](https://docs.statsig.com/agent-experimentation/node).

## Set up offline runs on a config

Every version of a config shares one set of evaluators and one dataset, so each version's run grades the same test cases in the same way. Set both on the config's _Setup_ tab, in the _Run Setup_ group. You need a config with at least one version. To create one, go to [Configs](https://docs.statsig.com/agent-experimentation/configs).

### Add evaluators

1. Go to _Agent Experimentation > Configs_ and open the config.
2. On the _Setup_ tab, go to the _Evaluators_ section of the _Run Setup_ group.
3. Click **Add evaluator**, then choose one option:

   - **An evaluator from the list**: Click its name. The list shows the evaluators in your company's catalog.
   - **Create new evaluator**: Choose an evaluator type, define the evaluator in the _Create evaluator_ drawer, and click **Add evaluator**.
4. Click **Save**.

A config's _Run Setup_ holds at most 10 evaluators. Evaluators that you create from a config join your company's catalog. For evaluator types and how to define them, go to [Evaluators](https://docs.statsig.com/agent-experimentation/evaluators).

Each evaluator appears as a card. To change a saved evaluator's definition, click the card's edit icon, which opens the evaluator's page. The card's menu sets the evaluator's role in this config:

- **Set as Primary**: Use this evaluator's score as each run's overall score. For details, go to [What is a primary evaluator](https://docs.statsig.com/agent-experimentation/evaluators#what-is-a-primary-grader).
- **Set as Critical Evaluator**: Fail the run when this evaluator's score for the run is 0. For details, go to [What is a critical evaluator](https://docs.statsig.com/agent-experimentation/evaluators#what-is-a-critical-grader).
- **Clone Evaluator**: Add a copy of the evaluator to this config.
- **Remove Evaluator**: Remove the evaluator from this config.

### Add a dataset

1. On the config's _Setup_ tab, go to the _Dataset_ section of the _Run Setup_ group.
2. Click **Select a Dataset**, then choose one option:

   - **Upload Dataset**: Upload a CSV file as a local dataset for this config.
   - **Browse from Existing Datasets**: Pick a global dataset.
   - **AI Dataset Generation**: Generate test cases from a description.
3. Click **Save**.

The button then shows the dataset name with its row count. A local dataset shows as "Custom dataset." For dataset limits, row editing, and the differences between global and local datasets, go to [Add a dataset to a config](https://docs.statsig.com/agent-experimentation/datasets#add-a-dataset-to-an-ai-config).

Each dataset column is a variable that the version's prompt can reference by name, such as `{{question}}`. For each row, Statsig fills the variables with the row's values before it calls the model. When a column holds JSON, `{{column}}` inserts the whole value, and a dotted path such as `{{column.items.0.name}}` inserts one field. Evaluators can reference columns too, such as `{{expected}}`.

An offline run from a config generates its own output for each row, so its dataset doesn't need an output column.

## Start an offline run

1. In the config's header, click **Run Evaluators**.
2. In the _Start Evaluation_ dialog, select the versions to evaluate. Statsig selects every version by default and lists the Live version first.
3. Click **Start Evaluation**.

Statsig starts one run for each selected version and opens the config's _Runs_ tab on the first run. While a run is in progress, the tab shows how many rows are complete. A run can take a few minutes. To stop a version's run, select the version on the _Setup_ tab and click **Stop Evaluation** in the header.

To evaluate one version, select it in the _Version_ section of the _Setup_ tab, open the version's menu, and click **Start Evaluation**.

If the config has unsaved changes, you can't select the version you're editing. When you start runs on other versions, Statsig asks you to discard the unsaved changes first. To include the edited version, save your changes before you click **Run Evaluators**.

### Schedule daily runs

1. On the config's _Setup_ tab, go to the _Schedule_ section of the _Run Setup_ group.
2. Next to **Start offline runs**, select **Daily**.
3. Choose the hour, in your time zone, and the versions to run. Statsig selects the Live version by default.
4. Click **Save**.

Every day at that hour, Statsig starts one run for each selected version. Statsig skips a scheduled run when your Included Credits have run out and your company has no API key for a provider that the run calls. To stop daily runs, select **Manually** and save.

### Model calls and Included Credits

For each row, Statsig calls the version's model provider to generate the output. A version with a custom workflow calls your workflow endpoint instead. LLM as a Judge evaluators call the provider set in each evaluator's **Evaluation Model**. Statsig charges these model calls to your Included Credits unless your company has its own API key for the provider. For details, go to [Credits and API keys](https://docs.statsig.com/agent-experimentation/overview#credits-and-api-keys).

### Why Run Evaluators is disabled

Hover over the disabled button to see the reason:

| Message | Cause | Fix |
| --- | --- | --- |
| No evaluation dataset available | The config has no saved dataset, or its dataset has no rows. | Add a dataset that has at least one row, and save. |
| No evaluators configured | The config has no evaluators. | Add an evaluator, and save. |
| Changes must be saved before starting an evaluation | The version you're editing has unsaved changes, and the config has no other version. | Save or discard your changes. |
| Changes must be saved before this version can be used | In the _Start Evaluation_ dialog, the version you're editing has unsaved changes. | Close the dialog, save your changes, and start the run again. |
| Select at least one version | You haven't selected a version in the _Start Evaluation_ dialog. | Select a version. |
| Included Credits have run out | Your Included Credits have run out, and your company has no API key for a provider that the run calls. | Ask an admin to add an API key for the provider in _Settings > Product Configuration > General_, under **AI Provider API Keys**. |

## Review results on the Runs tab

The config's _Runs_ tab shows offline run results for one version at a time. At the top of the tab, choose **Single version** or **Compare**, then choose a version and a run. Statsig numbers each version's runs in order, such as Run #1 and Run #2.

### Review a single run

With **Single version** selected, the _Runs_ tab shows these results for the chosen run:

- **Model, Score, and Prompt cards**: The model that the run used, the run's overall score, and the prompt messages that the run used. The overall score is the primary evaluator's score for the run, from 0 to 1. Click the **Prompt** card to read the full messages.
- **Offline evaluators chart**: Each evaluator's score for the run, which is the mean of its row scores. A score above 0.66 counts as a pass.
- **Results table**: One row for each dataset row, with the output, the latency, and each evaluator's score. Filter the table with **Search rows**, and use **Columns** to show or hide dataset columns. To export the table, click the **Download results as CSV** icon.

Click a row in the results table to open _Row Details_. The drawer shows the row's dataset values, the output, the latency, and a card for each evaluator with its score. For LLM as a Judge evaluators, the card lists the judge's reasoning steps.

### Compare two versions

To compare two versions' runs, click **Compare**. Choose the version and run to compare. Then, after "relative to," choose the baseline version and run. The baseline is the Live version's latest finished run by default.

The comparison shows:

- Model, Score, and Prompts cards for both runs.
- An Offline evaluators chart that rates the compared run against the baseline run for each evaluator: Better, Worse, or No significant change.
- A results table with both runs' outputs and scores for each row. _Row Details_ shows the two runs side by side.

Statsig compares two runs only when both runs have finished, both have a primary evaluator score, and both used the same dataset version. Otherwise, the tab explains what's missing, for example "Both runs must use the same dataset version." Each upload, generated dataset, or edit to the config's dataset starts a new dataset version. After you change the dataset, start new runs on both versions before you compare them.

### Stale runs and critical evaluator failures

Two indicators on the _Runs_ tab mean that a run's score needs attention:

- **Stale**: This chip appears when the version changed after the run. The run's results reflect the earlier version. To grade the current version, start a new run.
- **Critical evaluator failure**: The banner "This run failed due to the critical evaluator" appears, with the evaluator's name, when a critical evaluator's score for the run is 0. The run's overall score is then 0. For details, go to [What is a critical evaluator](https://docs.statsig.com/agent-experimentation/evaluators#what-is-a-critical-grader).

### Rows that fail to generate

If Statsig can't generate a row's output, for example because the model provider or workflow endpoint returns an error, the results table shows the error in place of the output. The row's evaluators show "Not graded," and its latency shows N/A. Statsig leaves the row out of evaluator scores, the run's overall score, version comparisons, and the scores in the CSV download.

An evaluator error is different from a failed row. If an evaluator fails on a row, such as a Python evaluator that raises an exception, the row scores 0 for that evaluator. For details, go to [Write a Python evaluator](https://docs.statsig.com/agent-experimentation/evaluators#write-a-python-grader).

## Find offline run traces on the Traces page

Each row of an offline run produces one trace in [Traces](https://docs.statsig.com/agent-experimentation/traces):

- The root span has the config's name. Its child span, named `chat` and the model name, such as `chat gpt-5-mini`, records the model call that generated the output. A version with a custom workflow has no child span, because your workflow endpoint makes its own model calls. To show your endpoint's spans under the root span, go to [Join a workflow run's trace](https://docs.statsig.com/agent-experimentation/node#join-a-workflow-eval-trace).
- Both spans set `statsig.environment` to `eval`, and the Service column shows `statsig-offline-evals`.
- The root span records the run, dataset, and row in `statsig.eval.run_id`, `statsig.eval.dataset_id`, and `statsig.eval.dataset_row_id`.
- When a row fails to generate, its spans have an error status.

The _Traces_ page hides `eval` spans by default. To show them, remove `eval` from the default environment filter in the search bar. For details, go to [Environments](https://docs.statsig.com/agent-experimentation/traces#environments).

Click a span to open the trace drawer. The _Evaluators_ tab shows each evaluator's score for the row, with the judge's reasoning for LLM as a Judge evaluators.

Statsig keeps offline run scores out of the online evaluator results on the config's _Traces_ tab, so dataset runs don't change the scores for your production traffic. Online evaluation also skips the spans that your workflow endpoint sends to the same project when they join a row's trace, so an offline run doesn't pay for judge calls twice.