Evals

An eval in Statsig Agent Experimentation grades outputs that already exist and keeps every run, so you can track scores over time. Each run applies graders to the output column of a dataset. The outputs can be answers your agent produced, responses from a model you plan to replace, or production traces. Compare runs to find out whether a change to your agent, model, or graders raised or lowered scores.

Use an eval when you already have the outputs. To generate outputs from AI Config versions and grade them, run an offline eval from the AI Config instead. To grade live traffic as it arrives, use online evals.

Evals and AI Config evals

Both kinds of eval grade every row of a dataset. They differ in where the output comes from and in what they compare.

Create an eval

An eval holds a series of runs. Create one eval for each thing you want to track, such as an agent or a regression suite.

1. In the Statsig console, go to Agent Experimentation > Evals.
2. Click Create. If your project has no evals yet, click Create an Eval.
3. In the Create New Eval dialog, enter a name and an optional description.
4. Click Create.

Statsig opens the new eval on its Runs tab, with the form for its first run.

The Evals page lists each eval with its name, last modified date, and creator. Search the list, filter it by creator, or sort it by name or last modified date.

To delete an eval, open it, open the more actions menu (⋯) in the header, and click Delete. Statsig deletes the results of every run on the eval.

Create a run

A run grades every row of one dataset with the graders you choose. The dataset needs at least one output column, which holds the output to grade.

1. Open the eval's Runs tab. A new eval already shows the run form. Otherwise, click Create above the list of runs.
2. In Run Name, enter a name for the run, such as the change it tests.
3. Under Dataset, click Select a dataset. Pick a dataset from the list, or click Create new local dataset to upload a CSV file and mark its output columns.
4. Click Add Grader, and then pick a grader from the catalog or click Create new grader. Repeat for each grader. Graders you create here join the Graders catalog.
5. (Optional) To change a grader's role in the run, open its more actions menu (⋯) and click Set as Primary or Set as Critical Grader.
6. Click Create Run.

Statsig grades the rows in the background. The run's card shows a progress bar until grading finishes.

The dataset list shows global datasets and the datasets of earlier runs on this eval. You can't pick a dataset that has no output column. A local dataset that you create in the run form belongs to the eval and doesn't appear on the Datasets page.

Create Run stays disabled until the run has a name, a dataset, and at least one grader. It also stays disabled when your company has used up its Included Credits and has no OpenAI API key. For details, go to Grader costs.

What graders read from each row

Graders read the row's output column as {{output}}. If the dataset has more than one output column, {{output}} holds a JSON object that maps each output column name to its value. A column named input fills {{input}}, and a column named trajectory fills {{trajectory}}. Graders can also reference any other column by name. For the full list of variables, go to Write a judge prompt.

Primary and critical graders in a run

• Primary grader: The run's card shows the primary grader's score. The first grader you add is the primary grader.
• Critical grader: A must-pass check. When a critical grader's score for the run is 0, the run's score is 0, whatever the other graders score. For more about critical graders, go to What is a critical grader.

Review a run

The Runs tab lists the eval's runs on the left. Each run card shows the run's name, its score, and how long ago it started. Search the list with Search runs, and click a run to open it. To delete a run, open the more actions menu (⋯) on its card and click Delete.

A run has two tabs: Run Results, which opens first, and Run Setup.

Results for each grader

The Run Results tab shows one card for each grader. The card header shows the grader's name, type, and score for the run. For graders that call a model, the header also shows the average number of tokens the grader used for each row. A Primary badge or a Critical chip marks those roles.

Click a card to expand it:

• Score: A gauge with the grader's mean score. It shows "No Score" when the grader scored no rows.
• Categories: When the dataset's rows have categories, click All or a category tab to filter the score and the rows. A bar chart shows the score for each category. Click a bar to open that category's tab.
• Rows: A table with the dataset's columns, including the output column, and the grader's Score for each row. Search the rows, filter them by score, pick the columns to show with Columns, or click Download results as CSV.

A row that a grader couldn't score shows N/A. Statsig leaves unscored rows out of the grader's score and its category scores.

Click a row to open Row Details. The drawer shows the row's columns and the grader's score. For LLM as a Judge graders, Judge Response shows the judge's reasoning. Expand Evaluation Prompt to read the grader's prompt with the row's values filled in. Press the up and down arrow keys to move between rows.

What a run used

The Run Setup tab shows what the run used:

• Graders: A card for each grader, with the version that the run used. "Latest version" means the grader hasn't changed since the run. Otherwise, the card shows "Version N of M, used for this run" and a View current grader link. Expand a card to read the grader's definition and targeting rules. Editing a grader later doesn't change the run's results.
• Dataset: A preview of the dataset's rows, with search. Icons mark the output and image columns. Click the dataset's name to open its page.
• Parameters: Any parameters that your code sent with its results. Refer to Send results from your code.

Compare runs

The Compare tab shows how scores change across the eval's finished runs. It leaves out runs that are still grading and runs that failed. What the tab shows depends on whether runs share a dataset:

• Runs that share a dataset: Statsig compares them in a card named after the dataset. Choose a Baseline run and one or more Runs to compare. The baseline starts as the earliest run, and the comparison starts as the latest run. For each grader and category, Statsig compares the scores of the rows that both runs scored and shows whether the change is statistically significant.
• Runs that don't share a dataset: Each grader gets a chart of its mean score over time, after at least two of these runs finish. Each chart has an All tab and a tab for each category. Runs from production traces and results from your code usually appear here, because each of these runs has its own dataset or none.

Until a second run finishes, the tab shows "A second finished run shows a comparison here."

Grade the past hour of production traces

A project or org admin can turn the past hour of an AI Config's production traces into a run. Statsig grades the run with the AI Config's online graders. Use a run from traces to check an AI Config's live outputs on demand.

Start the run in either place:

• From the AI Config: In the AI Config's header actions menu, click Enable Online Eval. In the Enable online eval dialog, click Run online graders for the last hour of traces instead.
• From the AI Config's eval: Open the more actions menu (⋯) in the eval's header and click Run eval from traces. Navigate to original prompt in the same menu opens the AI Config.

Statsig builds the run in four steps:

1. Statsig collects the AI Config's spans from the past hour that have input and output messages. When a trace has a workflow span, Statsig uses only that span. It skips tool spans and the spans that Statsig generates itself.
2. Statsig builds a local dataset with one row per span. The columns are Input, Trajectory, and Output, plus Prompt Version ID and Template Variables. Output is the output column.
3. Statsig grades the dataset with the AI Config's online graders.
4. Statsig adds the run to the AI Config's eval. The first time, Statsig creates that eval with the name auto_eval_from_traces_eval: followed by the AI Config's name.

The run's name is the hour it covers, as a timestamp such as 2026-11-04T15:00:00.000Z. Its card shows an Auto chip, and its Run Setup tab explains where the run came from. The scores also appear on the original spans in AI Traces.

The AI Config needs at least one online grader. If no span from the past hour qualifies, Statsig doesn't create a run.

Send results from your code

You can score outputs in your own code and record the results as runs on an eval:

• AI SDKs: The Eval() function in the Node and Python AI SDKs runs your task and scorers on each record, and then sends the results to the eval you name.
• Console API: The Send eval results endpoint, POST /console/v1/evals/send_results/{name}, accepts results that you scored anywhere. Eval() sends its results through this endpoint.

Both methods need an existing eval with the name you send. Create the eval first, on the Evals page or with Create Eval. If no eval has the name, Statsig rejects the results with a 404 error. Give each eval a unique name, because Statsig finds the eval by name. Requests authenticate with a Console API key.

Each call adds one run to the eval. Statsig names the run with the name you send, or Eval Run N when you don't send one. The results appear on the run as follows:

• Scores: Each named score becomes a grader card on Run Results. The first score is the primary grader unless the request sets primaryGrader. Scores listed in the request's criticalGraders are critical.
• Categories: Each record's categories become category tabs.
• Summary scores: These appear in a Summary Scores card on Run Results.
• Parameters: These appear on Run Setup.
• Trace IDs: When a result includes a traceID and spanID, its scores also appear on that span in AI Traces as offline eval scores.

These runs have no dataset, so they appear on the mean score charts on the Compare tab.

To start a run on a dataset with catalog graders from code, call Create Eval Run, and then read the results with Get Eval Run Results. The Console API calls AI Configs prompts, so to link an eval to an AI Config, pass the AI Config's name or ID as promptID when you create the eval.

Traces for eval rows

Statsig links each graded row of a run that you create on the Evals page to a trace in AI Traces. The trace shows the row's input and output next to its scores.

• First run: Statsig creates a small trace for the row in the eval environment.
• Later runs: Statsig adds their scores to the same trace, so the row's scores from every run stay together.
• Edited rows: After you edit a row's values, its next run creates a new trace.
• Rows copied from production traces: A row that Statsig copied from a production trace, and that you haven't edited, adds its scores to the original trace.

These scores appear on the trace's Graders tab as offline eval scores, with a View eval link to the run. They don't count toward online eval results. AI Traces hides the eval environment by default, so remove that filter to find these traces.