Offline Evaluation

An offline run grades AI output for every row of a fixed dataset of test cases, before any user sees that output. Statsig scores each row with the evaluators you choose, then reports each evaluator's score and an overall score for the run. Use offline runs to compare config versions and catch regressions before you promote a version to Live. To grade a sample of live production traffic instead, use online evaluation. To try a few prompts side by side without grading, use the Playground.

Choose where to run offline evaluation

Agent Experimentation runs offline evaluation in two places. Choose the one that matches the outputs you want to grade:

This page covers offline runs from a config. For grade-only runs on stored outputs, go to Runs. To start a run from your own code with Eval(), go to the Node AI SDK.

Set up offline runs on a config

Every version of a config shares one set of evaluators and one dataset, so each version's run grades the same test cases in the same way. Set both on the config's Setup tab, in the Run Setup group. You need a config with at least one version. To create one, go to Configs.

Add evaluators

1. Go to Agent Experimentation > Configs and open the config.
2. On the Setup tab, go to the Evaluators section of the Run Setup group.
3. Click Add evaluator, then choose one option:
   • An evaluator from the list: Click its name. The list shows the evaluators in your company's catalog.
   • Create new evaluator: Choose an evaluator type, define the evaluator in the Create evaluator drawer, and click Add evaluator.
4. Click Save.

A config's Run Setup holds at most 10 evaluators. Evaluators that you create from a config join your company's catalog. For evaluator types and how to define them, go to Evaluators.

Each evaluator appears as a card. To change a saved evaluator's definition, click the card's edit icon, which opens the evaluator's page. The card's menu sets the evaluator's role in this config:

• Set as Primary: Use this evaluator's score as each run's overall score. For details, go to What is a primary evaluator.
• Set as Critical Evaluator: Fail the run when this evaluator's score for the run is 0. For details, go to What is a critical evaluator.
• Clone Evaluator: Add a copy of the evaluator to this config.
• Remove Evaluator: Remove the evaluator from this config.

Add a dataset

1. On the config's Setup tab, go to the Dataset section of the Run Setup group.
2. Click Select a Dataset, then choose one option:
   • Upload Dataset: Upload a CSV file as a local dataset for this config.
   • Browse from Existing Datasets: Pick a global dataset.
   • AI Dataset Generation: Generate test cases from a description.
3. Click Save.

The button then shows the dataset name with its row count. A local dataset shows as "Custom dataset." For dataset limits, row editing, and the differences between global and local datasets, go to Add a dataset to a config.

Each dataset column is a variable that the version's prompt can reference by name, such as {{question}}. For each row, Statsig fills the variables with the row's values before it calls the model. When a column holds JSON, {{column}} inserts the whole value, and a dotted path such as {{column.items.0.name}} inserts one field. Evaluators can reference columns too, such as {{expected}}.

An offline run from a config generates its own output for each row, so its dataset doesn't need an output column.

Start an offline run

1. In the config's header, click Run Evaluators.
2. In the Start Evaluation dialog, select the versions to evaluate. Statsig selects every version by default and lists the Live version first.
3. Click Start Evaluation.

Statsig starts one run for each selected version and opens the config's Runs tab on the first run. While a run is in progress, the tab shows how many rows are complete. A run can take a few minutes. To stop a version's run, select the version on the Setup tab and click Stop Evaluation in the header.

To evaluate one version, select it in the Version section of the Setup tab, open the version's menu, and click Start Evaluation.

If the config has unsaved changes, you can't select the version you're editing. When you start runs on other versions, Statsig asks you to discard the unsaved changes first. To include the edited version, save your changes before you click Run Evaluators.

Schedule daily runs

1. On the config's Setup tab, go to the Schedule section of the Run Setup group.
2. Next to Start offline runs, select Daily.
3. Choose the hour, in your time zone, and the versions to run. Statsig selects the Live version by default.
4. Click Save.

Every day at that hour, Statsig starts one run for each selected version. Statsig skips a scheduled run when your Included Credits have run out and your company has no API key for a provider that the run calls. To stop daily runs, select Manually and save.

Model calls and Included Credits

For each row, Statsig calls the version's model provider to generate the output. A version with a custom workflow calls your workflow endpoint instead. LLM as a Judge evaluators call the provider set in each evaluator's Evaluation Model. Statsig charges these model calls to your Included Credits unless your company has its own API key for the provider. For details, go to Credits and API keys.

Why Run Evaluators is disabled

Hover over the disabled button to see the reason:

Review results on the Runs tab

The config's Runs tab shows offline run results for one version at a time. At the top of the tab, choose Single version or Compare, then choose a version and a run. Statsig numbers each version's runs in order, such as Run #1 and Run #2.

Review a single run

With Single version selected, the Runs tab shows these results for the chosen run:

• Model, Score, and Prompt cards: The model that the run used, the run's overall score, and the prompt messages that the run used. The overall score is the primary evaluator's score for the run, from 0 to 1. Click the Prompt card to read the full messages.
• Offline evaluators chart: Each evaluator's score for the run, which is the mean of its row scores. A score above 0.66 counts as a pass.
• Results table: One row for each dataset row, with the output, the latency, and each evaluator's score. Filter the table with Search rows, and use Columns to show or hide dataset columns. To export the table, click the Download results as CSV icon.

Click a row in the results table to open Row Details. The drawer shows the row's dataset values, the output, the latency, and a card for each evaluator with its score. For LLM as a Judge evaluators, the card lists the judge's reasoning steps.

Compare two versions

To compare two versions' runs, click Compare. Choose the version and run to compare. Then, after "relative to," choose the baseline version and run. The baseline is the Live version's latest finished run by default.

The comparison shows:

• Model, Score, and Prompts cards for both runs.
• An Offline evaluators chart that rates the compared run against the baseline run for each evaluator: Better, Worse, or No significant change.
• A results table with both runs' outputs and scores for each row. Row Details shows the two runs side by side.

Statsig compares two runs only when both runs have finished, both have a primary evaluator score, and both used the same dataset version. Otherwise, the tab explains what's missing, for example "Both runs must use the same dataset version." Each upload, generated dataset, or edit to the config's dataset starts a new dataset version. After you change the dataset, start new runs on both versions before you compare them.

Stale runs and critical evaluator failures

Two indicators on the Runs tab mean that a run's score needs attention:

• Stale: This chip appears when the version changed after the run. The run's results reflect the earlier version. To grade the current version, start a new run.
• Critical evaluator failure: The banner "This run failed due to the critical evaluator" appears, with the evaluator's name, when a critical evaluator's score for the run is 0. The run's overall score is then 0. For details, go to What is a critical evaluator.

Rows that fail to generate

If Statsig can't generate a row's output, for example because the model provider or workflow endpoint returns an error, the results table shows the error in place of the output. The row's evaluators show "Not graded," and its latency shows N/A. Statsig leaves the row out of evaluator scores, the run's overall score, version comparisons, and the scores in the CSV download.

An evaluator error is different from a failed row. If an evaluator fails on a row, such as a Python evaluator that raises an exception, the row scores 0 for that evaluator. For details, go to Write a Python evaluator.

Find offline run traces on the Traces page

Each row of an offline run produces one trace in Traces:

• The root span has the config's name. Its child span, named chat and the model name, such as chat gpt-5-mini, records the model call that generated the output. A version with a custom workflow has no child span, because your workflow endpoint makes its own model calls. To show your endpoint's spans under the root span, go to Join a workflow run's trace.
• Both spans set statsig.environment to eval, and the Service column shows statsig-offline-evals.
• The root span records the run, dataset, and row in statsig.eval.run_id, statsig.eval.dataset_id, and statsig.eval.dataset_row_id.
• When a row fails to generate, its spans have an error status.

The Traces page hides eval spans by default. To show them, remove eval from the default environment filter in the search bar. For details, go to Environments.

Click a span to open the trace drawer. The Evaluators tab shows each evaluator's score for the row, with the judge's reasoning for LLM as a Judge evaluators.

Statsig keeps offline run scores out of the online evaluator results on the config's Traces tab, so dataset runs don't change the scores for your production traffic. Online evaluation also skips the spans that your workflow endpoint sends to the same project when they join a row's trace, so an offline run doesn't pay for judge calls twice.