Evaluators
Create and configure evaluators in Statsig Agent Experimentation to score AI output in offline runs and online evaluation, with LLM judges, Python, string, and similarity checks.
An evaluator scores AI output with a number from 0 to 1. Statsig runs evaluators in two places: offline runs, which grade every row of a dataset, and online evaluation, which grades a sample of live traffic. Evaluators live in one catalog for your company. Any config or run can reuse an evaluator, so its scores stay comparable across versions.
Use an evaluator when you want Statsig to compute the score. To record a score that your own code computes, such as user feedback, call logEvalGrade from the Node AI SDK instead.
The Console API still calls evaluators "graders." For example, you create an evaluator with Create Eval Grader.
Evaluator types
Every evaluator has one of four types. The type decides how the evaluator scores output and whether it can run in online evaluation.
| Type | How it scores | Runs online |
|---|---|---|
| LLM as a Judge | A model reads your prompt and the output, then returns a score from 0 to 1 with its reasoning. | Yes, when the prompt uses only {{input}}, {{output}}, and {{trajectory}} |
| String Comparison | Compares {{output}} with a dataset column using Equals, Does not equal, Contains, or Contains (ignore case). Scores 1 when the comparison holds and 0 when it doesn't. | No |
| Text Similarity | Scores how similar {{output}} is to a dataset column, using Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, or Cosine. Cosine embeds both texts with the OpenAI text-embedding-3-large model. | No |
| Python | Runs your grade(sample, item) function, which returns a number. | Yes, when the code reads only item["input"] and item["trajectory"] |
A row or span passes an evaluator when it scores 0.67 or higher. String Comparison and Text Similarity compare the output with a dataset column. Live traffic has no dataset, so these types run only in offline runs.
Find an evaluator
The Evaluators page lists every evaluator in your company. Go to Agent Experimentation > Evaluators to open it.
The list has the columns Name, Type, Last Modified, and Creator. Search the list, filter it by Creator, or sort it by Name or Last Modified. Click an evaluator to open its page, which has three tabs: Setup, Results, and Traces.
Create an evaluator
Start from a template when one matches your check. Start from scratch to write your own.
- Go to Agent Experimentation > Evaluators and click Create. If your company has no evaluators yet, click Create an Evaluator.
- In the Create Evaluator dialog, click a template under Create from Template, or click Create from Scratch.
- Enter an Evaluator Name. Evaluator names must be unique in your company.
- (Optional) Enter a description.
- Click Create.
Statsig creates the evaluator and opens its Setup tab. A template evaluator starts with the template's prompt or code saved. An evaluator created from scratch starts as an LLM as a Judge evaluator with an empty prompt.
You can also create an evaluator from a config. On the config's Setup tab, in the Run Setup section, click Add evaluator, then click Create new evaluator. Evaluators you create this way join the same catalog. For more about configs, go to Configs.
Evaluator templates
The Create Evaluator dialog offers six templates. Each template is a starting point that you can edit after you create the evaluator.
| Template | Type | What it checks | Runs online |
|---|---|---|---|
| Helpfulness | LLM as a Judge | Whether the response resolves the request | Yes |
| Tone Check | LLM as a Judge | Whether the response is polite and on-brand. Edit the voice list in the prompt to match your brand. | Yes |
| Groundedness | LLM as a Judge | Whether the request or tool results back every claim | Yes |
| Faithfulness | LLM as a Judge | Whether the output contradicts the request or tool results | Yes |
| Exact Match | Python | Whether the output equals the value in a dataset column, expected by default | No, offline runs only |
| JSON Validity | Python | Whether the output parses as JSON, with optional required fields | Yes |
Set up an evaluator
The Setup tab holds the evaluator's definition and its targeting rules. It has these fields:
- Name: The evaluator's name in the catalog.
- Type: One of the four evaluator types.
- Evaluation Model: The provider and model that score output. This field appears for LLM as a Judge evaluators only.
- Prompt or Definition: The judge prompt for LLM as a Judge evaluators, or the comparison or code for the other types.
- Targeting rules: Where the evaluator runs. For details, go to Choose where an evaluator runs.
Choose the evaluation model
An LLM as a Judge evaluator uses the model in Evaluation Model to score each row or span. Choose a Provider, such as OpenAI, Anthropic, or Google, and then a Model. New evaluators use OpenAI gpt-5-mini.
Judge calls use your company's Included Credits unless your company has an API key for that provider. Without a key for the provider, the Setup tab shows a notice with an Add API key link. For details, go to Evaluator costs.
Write a judge prompt
The judge prompt tells the evaluation model what to check and how to score it. Reference the output and its context with these variables. To insert a variable, click it in the Variables list under the prompt.
| Variable | Value |
|---|---|
{{input}} | The input messages, one role: text line for each message. In offline runs, the dataset's Input column. |
{{output}} | The final answer text. |
{{trajectory}} | The tool calls, tool results, and reasoning before the final answer. Empty when the output has none. |
Wrap a variable in three braces, such as {{{output}}}, to pass its text unchanged. With two braces, Statsig HTML-escapes the text, so the judge reads characters such as " and < as " and <. The templates use three braces. For example:
Score how well the answer resolves the request, from 0 to 1.
<request>
{{{input}}}
</request>
<answer>
{{{output}}}
</answer>
A prompt can also reference a dataset column by name, such as {{expected}}. An evaluator whose prompt references any variable other than {{input}}, {{output}}, and {{trajectory}} can be used in offline runs only.
Define a String Comparison or Text Similarity evaluator
String Comparison and Text Similarity evaluators compare {{output}} with a column from the dataset that a run uses.
- String Comparison: Choose an operation and the column to compare with. The operations are Equals, Does not equal, Contains, and Contains (ignore case).
- Text Similarity: Choose a function and the column to compare with. The functions are Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, and Cosine.
The Column list shows the columns of a config's dataset, so create these evaluators from the config that runs them. On the config's Setup tab, click Add evaluator, then click Create new evaluator.
Write a Python evaluator
A Python evaluator runs your code once for each row in an offline run, or once for each graded span in online evaluation. The code must define a grade(sample, item) function that returns a number. For example, this evaluator passes outputs of 500 characters or fewer:
def grade(sample, item):
output = sample["output_text"]
return 1.0 if len(output) <= 500 else 0.0
| Part | Contract |
|---|---|
| Function | grade(sample, item), defined at the top level of your code |
| Output | sample["output_text"] holds the output text. |
| Item | item holds the dataset row's columns by name. item["input"] and item["trajectory"] hold the input and trajectory in both offline runs and online evaluation. |
| Return value | An int or float that isn't NaN or infinite. A score of 0.67 or higher passes. |
| Errors | If the code has no grade function, raises an exception, or returns anything other than a finite number, the row scores 0. Statsig records the error message as the evaluator's response for that row. |
| Time limit | 10 seconds for each row. A row that runs longer scores 0. |
| Source limit | 100,000 characters. Code over the limit fails every row. |
| Environment | A sandbox with the Python standard library. Statsig clears environment variables. |
To run online, a Python evaluator can read only item["input"] and item["trajectory"] from item, written as literal keys. Code that reads any other key, or a key held in a variable, can be used in offline runs only.
Save an evaluator version
Edits on the Setup tab stay unsaved until you click Save Evaluator. Click Discard Changes to undo them.
Saving a change to the evaluator's definition creates a new version of the evaluator. When configs use the evaluator, Statsig first shows the Update this evaluator? dialog. Saving updates the evaluator for the latest version of each config that uses it. Runs that already finished keep the evaluator version they used.
Choose where an evaluator runs
Targeting rules decide which offline runs include an evaluator and which live traffic it grades online. Set them in the Targeting rules section of the evaluator's Setup tab. Each rule is an offline rule, an online rule, or both.
To add a rule:
- On the evaluator's Setup tab, under Targeting rules, click Add rule.
- Check Offline, Online, or both.
- Set the rule's conditions. For each condition, choose a field, an operator, and one or more values.
- For an online rule, enter the percentage of matching traffic to grade in Sample.
- Click Save Evaluator.
A rule can be both offline and online when its only condition is "Config is any of." The Sample percentage then applies to online grading only, because offline runs grade every row.
Offline rules
An offline rule adds the evaluator to the runs of configs. It takes exactly one condition, "Config is any of," with no sampling. Check Critical to make the evaluator a critical evaluator for these configs. On each config's Setup tab, the evaluator card shows an Offline badge.
Adding an evaluator from a config's Setup tab with Add evaluator has the same effect as a "Config is any of" rule. A run of a config uses at most 10 evaluators.
Runs on the Runs page don't use offline rules. Each run uses the evaluators you pick when you create the run.
Online rules
An online rule grades a sample of live traffic as it arrives. Only evaluators that can run online, as listed in Evaluator types, can have online rules.
Online evaluation grades one span for each agent turn: the workflow span when the trace has one, or else the outermost LLM call. The span must carry input and output messages, and tool spans are never graded. Spans don't need a config, so a rule can also grade traced LLM calls that your app makes without one. For how to send traces, go to Online Evaluation.
An online rule matches a span when all of its conditions match. A rule with no conditions matches every span. Each condition uses one of these fields:
| Field | Matches |
|---|---|
| Config | The span's config, from statsig.llm.prompt_name |
| Environment | statsig.environment |
| Span name | The span's name |
| Model | The requested model, from gen_ai.request.model |
| Span attribute | Any span attribute. Enter the attribute name, then the values. |
Each condition uses the operator is any of or is not. Values match exactly, with no wildcards, and can't contain %. A span that doesn't have the attribute counts as an empty value, so is not matches it.
Each evaluator grades a span at most once, under the first enabled online rule that matches the span. Use the arrows on a rule to move it up or down. Sample sets the percentage of matching traces that the rule grades, from 0 to 100. New rules start at 10%. Statsig samples by trace: it hashes the trace ID, so every span in a trace gets the same sampling decision.
Online grading has these limits:
- Statsig grades at most 1,000 spans per hour for each config. A span counts toward its config's limit when every rule that grades it has a "Config is any of" condition that includes the config. Other spans that online rules grade share one limit of 1,000 spans per hour for your company.
- Statsig processes traces for online evaluation in the US only. Statsig plans to support EU data residency.
Grade a config's production traffic
Online rules alone decide which of a config's spans Statsig grades. To grade a config's spans, save an enabled online rule with a "Config is any of" condition that includes the config. To stop, disable the rule or remove the config from the condition. Rules that don't name the config can also grade its spans when their conditions match.
Disable an online rule
To stop an online rule without deleting it, clear its Online checkbox and save. Statsig keeps the rule, and the evaluator cards on configs show it with a Disabled badge.
Who can edit targeting rules
Enabled online rules start grading live traffic, so a project or org admin must save any change to them. This includes adding, editing, disabling, or deleting an enabled online rule. Anyone who can edit the evaluator can change its offline rules and its disabled online rules.
Targeting rule errors
Save Evaluator stays disabled until every rule is valid. These messages explain what to fix:
| Message | Fix |
|---|---|
| Check Offline, Online or both | Check at least one of Offline and Online. |
| Offline rules need one "Config is any of" condition | Reduce the offline rule to one "Config is any of" condition. |
| Every condition needs at least one value | Add a value to each condition, or remove the empty condition. |
| Span attribute conditions need an attribute name | Enter the attribute name in the Span attribute condition. |
| Condition values cannot contain % | Remove % from the value. Values match exactly. |
| Sampling rate must be between 0% and 100% | Enter a Sample value from 0 to 100. |
Online rules need an evaluator that reads just {{input}}, {{output}} and {{trajectory}} | Remove the online rules, or change the definition so the evaluator can run online. |
What is a primary evaluator
A primary evaluator is the evaluator whose score becomes a config version's overall score, the number you compare across versions. Each config has one primary evaluator. To change it, open the menu on an evaluator card in the config's Run Setup section and click Set as Primary. If you remove the primary evaluator, the next evaluator becomes primary.
What is a critical evaluator
A critical evaluator is a must-pass check for a config's runs. When a critical evaluator's score for a run is 0, the whole run fails. The run shows "This run failed due to the critical evaluator" with the evaluator's name, and the version's overall score is 0. When a critical evaluator scores above 0, it behaves like any other evaluator.
For example, if your app parses a config's output as JSON, mark a JSON Validity evaluator critical for that config. A version whose output never parses then fails its run, whatever its other evaluators score.
You set Critical separately for each config. Make an evaluator critical for a config in either place:
- On the config's Setup tab, open the menu on the evaluator card and click Set as Critical Evaluator.
- On the evaluator's Setup tab, check Critical on an offline rule with a "Config is any of" condition.
Critical grading applies to offline runs only.
Review an evaluator's results
The evaluator's Results and Traces tabs show where it runs and how it scores.
- Results: The Used by table lists each config and Playground that uses the evaluator. Its Type column shows Config or Playground. The Runs table lists each run group that ran the evaluator, with columns for Score, Run Group, Type, Config, and Last run. Type is Online for online evaluation and On Demand for all other run groups. An evaluator that nothing uses shows "This evaluator is currently unused."
- Traces: The traces that the evaluator scored, from the last day by default. Change the time range or add filters, and click a trace to open its details. Statsig tracks scores by evaluator ID, so renaming an evaluator keeps its history.
Evaluator costs
LLM as a Judge evaluators and Text Similarity evaluators that use Cosine call a model provider. Statsig charges these calls to your Included Credits unless your company has its own API key for the provider. Other evaluators run inside Statsig at no cost.
| Evaluator | Provider it calls | Cost |
|---|---|---|
| LLM as a Judge | The provider in Evaluation Model | Included Credits, or your own key for that provider |
| Text Similarity with Cosine | OpenAI, for text-embedding-3-large embeddings | Included Credits, or your own OpenAI key |
| String Comparison, Python, and other Text Similarity functions | None | Free |
Statsig doesn't meter calls on your own API keys. An admin adds keys in Settings > Product Configuration > General, under AI Provider API Keys. When your Included Credits run out, online evaluation keeps running the evaluators that use your own keys and skips the rest. For details on Included Credits, go to Credits and API keys.
Was this helpful?
For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.