---
title: Runs
description: "Grade outputs already stored in a dataset with Statsig Agent Experimentation runs, track scores across a run group, compare runs, and record results from production traces or your own code."
product: general
lang: en
token_estimate: 3731
---
# Runs

Runs in Statsig Agent Experimentation grade outputs that already exist, and a run group keeps every run, so you can track scores over time. Each run applies [evaluators](https://docs.statsig.com/agent-experimentation/evaluators) to the output column of a [dataset](https://docs.statsig.com/agent-experimentation/datasets). The outputs can be answers your agent produced, responses from a model you plan to replace, or production traces. Compare runs to find out whether a change to your agent, model, or evaluators raised or lowered scores.

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

Use a run on the _Runs_ page when you already have the outputs. To generate outputs from [config](https://docs.statsig.com/agent-experimentation/configs) versions and grade them, start an [offline run from the config](https://docs.statsig.com/agent-experimentation/offline-evaluation) instead. To grade live traffic as it arrives, use [online evaluation](https://docs.statsig.com/agent-experimentation/online-evaluation).

## Runs and offline runs from a config

Both kinds of run grade every row of a dataset. They differ in where the output comes from and in what they compare.

| Difference | Run on the _Runs_ page | Offline run from a config |
| --- | --- | --- |
| Output it grades | The output column of the dataset, or results you score in your own code | Output that Statsig generates by running each config version on the dataset's inputs |
| Dataset | Needs at least one output column | Needs inputs only |
| Evaluators | Chosen for each run | Attached to the config |
| Results | The run group's _Runs_ and _Compare_ tabs | The config's _Runs_ tab |
| What you compare | Runs of the run group | Versions of the config |

## Create a run group

A run group holds a series of runs. Create one run group for each thing you want to track, such as an agent or a regression suite.

1. In the Statsig console, go to _Agent Experimentation > Runs_.
2. Click **Create**. If your project has no runs yet, click **Create a Run**.
3. In the _Create New Eval_ dialog, enter a name and an optional description.
4. Click **Create**.

Statsig opens the new run group on its _Runs_ tab, with the form for its first run.

The _Runs_ page lists each run group with its name, last modified date, and creator. Search the list, filter it by creator, or sort it by name or last modified date.

To delete a run group, open it, open the more actions menu (**⋯**) in the header, and click **Delete**. Statsig deletes the results of every run in the run group.

## Create a run

A run grades every row of one dataset with the evaluators you choose. The dataset needs at least one [output column](https://docs.statsig.com/agent-experimentation/datasets#output-and-image-columns), which holds the output to grade.

1. Open the run group's _Runs_ tab. A new run group already shows the run form. Otherwise, click **Create** above the list of runs.
2. In **Run Name**, enter a name for the run, such as the change it tests.
3. Under _Dataset_, click **Select a dataset**. Pick a dataset from the list, or click **Create new local dataset** to upload a CSV file and mark its output columns.
4. Click **Add Evaluator**, and then pick an evaluator from the catalog or click **Create new evaluator**. Repeat for each evaluator. Evaluators you create here join the [Evaluators](https://docs.statsig.com/agent-experimentation/evaluators) catalog.
5. (Optional) To change an evaluator's role in the run, open its more actions menu (**⋯**) and click **Set as Primary** or **Set as Critical Evaluator**.
6. Click **Create Run**.

Statsig grades the rows in the background. The run's card shows a progress bar until grading finishes.

The dataset list shows global datasets and the datasets of earlier runs in this run group. You can't pick a dataset that has no output column. A local dataset that you create in the run form belongs to the run group and doesn't appear on the _Datasets_ page.

**Create Run** stays disabled until the run has a name, a dataset, and at least one evaluator. It also stays disabled when your company has used up its Included Credits and has no OpenAI API key. For details, go to [Evaluator costs](https://docs.statsig.com/agent-experimentation/evaluators#grader-costs).

### What evaluators read from each row

Evaluators read the row's output column as `{{output}}`. If the dataset has more than one output column, `{{output}}` holds a JSON object that maps each output column name to its value. A column named `input` fills `{{input}}`, and a column named `trajectory` fills `{{trajectory}}`. Evaluators can also reference any other column by name. For the full list of variables, go to [Write a judge prompt](https://docs.statsig.com/agent-experimentation/evaluators#write-a-judge-prompt).

### Primary and critical evaluators in a run

- **Primary evaluator**: The run's card shows the primary evaluator's score. The first evaluator you add is the primary evaluator.
- **Critical evaluator**: A must-pass check. When a critical evaluator's score for the run is 0, the run's score is 0, whatever the other evaluators score. For more about critical evaluators, go to [What is a critical evaluator](https://docs.statsig.com/agent-experimentation/evaluators#what-is-a-critical-grader).

## Review a run

The _Runs_ tab lists the run group's runs on the left. Each run card shows the run's name, its score, and how long ago it started. Search the list with **Search runs**, and click a run to open it. To delete a run, open the more actions menu (**⋯**) on its card and click **Delete**.

A run has two tabs: _Run Results_, which opens first, and _Run Setup_.

### Results for each evaluator

The _Run Results_ tab shows one card for each evaluator. The card header shows the evaluator's name, type, and score for the run. For evaluators that call a model, the header also shows the average number of tokens the evaluator used for each row. A Primary badge or a Critical chip marks those roles.

Click a card to expand it:

- **Score**: A gauge with the evaluator's mean score. It shows "No Score" when the evaluator scored no rows.
- **Categories**: When the dataset's rows have [categories](https://docs.statsig.com/agent-experimentation/datasets#categorize-rows), click **All** or a category tab to filter the score and the rows. A bar chart shows the score for each category. Click a bar to open that category's tab.
- **Rows**: A table with the dataset's columns, including the output column, and the evaluator's Score for each row. Search the rows, filter them by score, pick the columns to show with **Columns**, or click **Download results as CSV**.

A row that an evaluator couldn't score shows N/A. Statsig leaves unscored rows out of the evaluator's score and its category scores.

Click a row to open _Row Details_. The drawer shows the row's columns and the evaluator's score. For LLM as a Judge evaluators, _Judge Response_ shows the judge's reasoning. Expand _Evaluation Prompt_ to read the evaluator's prompt with the row's values filled in. Press the up and down arrow keys to move between rows.

### What a run used

The _Run Setup_ tab shows what the run used:

- **Evaluators**: A card for each evaluator, with the version that the run used. "Latest version" means the evaluator hasn't changed since the run. Otherwise, the card shows "Version N of M, used for this run" and a **View current evaluator** link. Expand a card to read the evaluator's definition and targeting rules. Editing an evaluator later doesn't change the run's results.
- **Dataset**: A preview of the dataset's rows, with search. Icons mark the output and image columns. Click the dataset's name to open its page.
- **Parameters**: Any parameters that your code sent with its results. Refer to [Send results from your code](#send-results-from-your-code).

## Compare runs

The _Compare_ tab shows how scores change across the run group's finished runs. It leaves out runs that are still grading and runs that failed. What the tab shows depends on whether runs share a dataset:

- **Runs that share a dataset**: Statsig compares them in a card named after the dataset. Choose a **Baseline** run and one or more **Runs to compare**. The baseline starts as the earliest run, and the comparison starts as the latest run. For each evaluator and category, Statsig compares the scores of the rows that both runs scored and shows whether the change is statistically significant.
- **Runs that don't share a dataset**: Each evaluator gets a chart of its mean score over time, after at least two of these runs finish. Each chart has an **All** tab and a tab for each category. Runs from production traces, and results from your code that don't name a dataset, usually appear here, because each of these runs has its own dataset or none.

Until a second run finishes, the tab shows "A second finished run shows a comparison here."

## Grade the past hour of production traces

A project or org admin can turn the past hour of a config's production traces into a run. Statsig grades the run with the config's [online evaluators](https://docs.statsig.com/agent-experimentation/evaluators#online-rules). Use a run from traces to check a config's live outputs on demand.

Start the run in either place:

- **From the config**: In the config's header actions menu, click **Enable Online Eval**. In the _Enable online eval_ dialog, click **Run online graders for the last hour of traces instead**.
- **From the config's run group**: Open the more actions menu (**⋯**) in the run group's header and click **Run eval from traces**. **Navigate to original prompt** in the same menu opens the config.

Statsig builds the run in four steps:

1. Statsig collects the config's spans from the past hour that have input and output messages. When a trace has a workflow span, Statsig uses only that span. It skips tool spans and the spans that Statsig generates itself.
2. Statsig builds a local dataset with one row per span. The columns are `Input`, `Trajectory`, and `Output`, plus `Prompt Version ID` and `Template Variables`. `Output` is the output column.
3. Statsig grades the dataset with the config's online evaluators.
4. Statsig adds the run to the config's run group. The first time, Statsig creates that run group with the name `auto_eval_from_traces_eval:` followed by the config's name.

The run's name is the hour it covers, as a timestamp such as `2026-11-04T15:00:00.000Z`. Its card shows an Auto chip, and its _Run Setup_ tab explains where the run came from. The scores also appear on the original spans in [Traces](https://docs.statsig.com/agent-experimentation/traces).

The config needs at least one online evaluator. If no span from the past hour qualifies, Statsig doesn't create a run.

## Send results from your code

You can score outputs in your own code and record the results as runs in a run group:

- **AI SDKs**: The `Eval()` function in the [Node](https://docs.statsig.com/agent-experimentation/node#programmatic-evaluation) and [Python](https://docs.statsig.com/agent-experimentation/python#programmatic-evaluation) AI SDKs runs your task and scorers on each record, and then sends the results to the run group you name.
- **Console API**: The [Send eval results](https://docs.statsig.com/api-reference/evals/send-eval-results) endpoint, `POST /console/v1/evals/send_results/{name}`, accepts results that you scored anywhere. `Eval()` sends its results through this endpoint.

Both methods need an existing run group with the name you send. The Console API calls run groups "evals." Create the run group first, on the _Runs_ page or with [Create Eval](https://docs.statsig.com/api-reference/evals/create-eval). If no run group has the name, Statsig rejects the results with a 404 error. Give each run group a unique name, because Statsig finds the run group by name. Requests authenticate with a Console API key.

Each call adds one run to the run group. Statsig names the run with the name you send, or `Eval Run N` when you don't send one. The results appear on the run as follows:

- **Scores**: Each named score becomes an evaluator card on _Run Results_. The first score is the primary evaluator unless the request sets `primaryGrader`. Scores listed in the request's `criticalGraders` are critical.
- **Categories**: Each record's categories become category tabs.
- **Summary scores**: These appear in a _Summary Scores_ card on _Run Results_.
- **Parameters**: These appear on _Run Setup_.
- **Trace IDs**: When a result includes a `traceID` and `spanID`, its scores also appear on that span in [Traces](https://docs.statsig.com/agent-experimentation/traces) as offline run scores.

To compare runs row by row, name a [dataset](https://docs.statsig.com/agent-experimentation/datasets) in the request's `dataset` field, by its name or ID, and give each result the `rowID` of a different row of that dataset. Statsig rejects the results when a `rowID` is missing, repeated, or not a row of the dataset. Runs on the same dataset appear in the dataset's card on the _Compare_ tab. In the Node AI SDK, pass `statsigDataset()` as `Eval()`'s `data`. For details, go to [Run an eval on a Statsig dataset](https://docs.statsig.com/agent-experimentation/node#run-an-eval-on-a-statsig-dataset). Runs without a dataset appear on the mean score charts on the _Compare_ tab.

To start a run on a dataset with catalog evaluators from code, call [Create Eval Run](https://docs.statsig.com/api-reference/evals/create-eval-run), and then read the results with [Get Eval Run Results](https://docs.statsig.com/api-reference/evals/get-eval-run-results). The Console API calls configs prompts, so to link a run group to a config, pass the config's name or ID as `promptID` when you create the run group.

## Traces for graded rows

Statsig links each graded row of a run that you create on the _Runs_ page to a trace in [Traces](https://docs.statsig.com/agent-experimentation/traces). The trace shows the row's input and output next to its scores.

- **First run**: Statsig creates a small trace for the row in the `eval` [environment](https://docs.statsig.com/agent-experimentation/traces#environments).
- **Later runs**: Statsig adds their scores to the same trace, so the row's scores from every run stay together.
- **Edited rows**: After you edit a row's values, its next run creates a new trace.
- **Rows copied from production traces**: A row that Statsig copied from a production trace, and that you haven't edited, adds its scores to the original trace.

These scores appear on the trace's _Evaluators_ tab as offline run scores, with a **View runs** link to the run. They don't count toward online evaluation results. The _Traces_ page hides the `eval` environment by default, so remove that filter to find these traces.