Graders
Create and configure graders in Statsig Agent Experimentation to score AI output in offline and online evals, with LLM judges, Python, string, and similarity checks.
A grader scores AI output with a number from 0 to 1. Statsig runs graders in two places: offline evals, which grade every row of a dataset, and online evals, which grade a sample of live traffic. Graders live in one catalog for your company. Any AI Config or eval can reuse a grader, so its scores stay comparable across versions.
Use a grader when you want Statsig to compute the score. To record a score that your own code computes, such as user feedback, call logEvalGrade from the Node AI SDK instead.
Grader types
Every grader has one of four types. The type decides how the grader scores output and whether it can run in online evals.
| Type | How it scores | Runs online |
|---|---|---|
| LLM as a Judge | A model reads your prompt and the output, then returns a score from 0 to 1 with its reasoning. | Yes, when the prompt uses only {{input}}, {{output}}, and {{trajectory}} |
| String Comparison | Compares {{output}} with a dataset column using Equals, Does not equal, Contains, or Contains (ignore case). Scores 1 when the comparison holds and 0 when it doesn't. | No |
| Text Similarity | Scores how similar {{output}} is to a dataset column, using Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, or Cosine. Cosine embeds both texts with the OpenAI text-embedding-3-large model. | No |
| Python | Runs your grade(sample, item) function, which returns a number. | Yes, when the code reads only item["input"] and item["trajectory"] |
A row or span passes a grader when it scores 0.67 or higher. String Comparison and Text Similarity compare the output with a dataset column. Live traffic has no dataset, so these types run only in offline evals.
Find a grader
The Graders page lists every grader in your company. Go to Agent Experimentation > Graders to open it.
The list has the columns Name, Type, Last Modified, and Creator. Search the list, filter it by Creator, or sort it by Name or Last Modified. Click a grader to open its page, which has three tabs: Setup, Results, and Traces.
Create a grader
Start from a template when one matches your check. Start from scratch to write your own.
- Go to Agent Experimentation > Graders and click Create. If your company has no graders yet, click Create a Grader.
- In the Create Grader dialog, click a template under Create from Template, or click Create from Scratch.
- Enter a Grader Name. Grader names must be unique in your company.
- (Optional) Enter a description.
- Click Create.
Statsig creates the grader and opens its Setup tab. A template grader starts with the template's prompt or code saved. A grader created from scratch starts as an LLM as a Judge grader with an empty prompt.
You can also create a grader from an AI Config. On the AI Config's Setup tab, in the Eval Setup section, click Add grader, then click Create new grader. Graders you create this way join the same catalog. For more about AI Configs, go to AI Configs.
Grader templates
The Create Grader dialog offers six templates. Each template is a starting point that you can edit after you create the grader.
| Template | Type | What it checks | Runs online |
|---|---|---|---|
| Helpfulness | LLM as a Judge | Whether the response resolves the request | Yes |
| Tone Check | LLM as a Judge | Whether the response is polite and on-brand. Edit the voice list in the prompt to match your brand. | Yes |
| Groundedness | LLM as a Judge | Whether the request or tool results back every claim | Yes |
| Faithfulness | LLM as a Judge | Whether the output contradicts the request or tool results | Yes |
| Exact Match | Python | Whether the output equals the value in a dataset column, expected by default | No, dataset evals only |
| JSON Validity | Python | Whether the output parses as JSON, with optional required fields | Yes |
Set up a grader
The Setup tab holds the grader's definition and its targeting rules. It has these fields:
- Name: The grader's name in the catalog.
- Type: One of the four grader types.
- Evaluation Model: The provider and model that score output. This field appears for LLM as a Judge graders only.
- Prompt or Definition: The judge prompt for LLM as a Judge graders, or the comparison or code for the other types.
- Targeting rules: Where the grader runs. For details, go to Choose where a grader runs.
Choose the evaluation model
An LLM as a Judge grader uses the model in Evaluation Model to score each row or span. Choose a Provider, such as OpenAI, Anthropic, or Google, and then a Model. New graders use OpenAI gpt-5-mini.
Judge calls use your company's Included Credits unless your company has an API key for that provider. Without a key for the provider, the Setup tab shows a notice with an Add API key link. For details, go to Grader costs.
Write a judge prompt
The judge prompt tells the evaluation model what to check and how to score it. Reference the output and its context with these variables. To insert a variable, click it in the Variables list under the prompt.
| Variable | Value |
|---|---|
{{input}} | The input messages, one role: text line for each message. In offline evals, the dataset's Input column. |
{{output}} | The final answer text. |
{{trajectory}} | The tool calls, tool results, and reasoning before the final answer. Empty when the output has none. |
Wrap a variable in three braces, such as {{{output}}}, to pass its text unchanged. With two braces, Statsig HTML-escapes the text, so the judge reads characters such as " and < as " and <. The templates use three braces. For example:
Score how well the answer resolves the request, from 0 to 1.
<request>
{{{input}}}
</request>
<answer>
{{{output}}}
</answer>
A prompt can also reference a dataset column by name, such as {{expected}}. A grader whose prompt references any variable other than {{input}}, {{output}}, and {{trajectory}} runs in offline evals only.
Define a String Comparison or Text Similarity grader
String Comparison and Text Similarity graders compare {{output}} with a column from the dataset that an eval runs on.
- String Comparison: Choose an operation and the column to compare with. The operations are Equals, Does not equal, Contains, and Contains (ignore case).
- Text Similarity: Choose a function and the column to compare with. The functions are Fuzzy match, BLEU, GLEU, METEOR, ROUGE-1 through ROUGE-5, ROUGE-L, and Cosine.
The Column list shows the columns of an AI Config's dataset, so create these graders from the AI Config that runs them. On the AI Config's Setup tab, click Add grader, then click Create new grader.
Write a Python grader
A Python grader runs your code once for each row in an offline eval, or once for each graded span in an online eval. The code must define a grade(sample, item) function that returns a number. For example, this grader passes outputs of 500 characters or fewer:
def grade(sample, item):
output = sample["output_text"]
return 1.0 if len(output) <= 500 else 0.0
| Part | Contract |
|---|---|
| Function | grade(sample, item), defined at the top level of your code |
| Output | sample["output_text"] holds the output text. |
| Item | item holds the dataset row's columns by name. item["input"] and item["trajectory"] hold the input and trajectory in both offline and online evals. |
| Return value | An int or float that isn't NaN or infinite. A score of 0.67 or higher passes. |
| Errors | If the code has no grade function, raises an exception, or returns anything other than a finite number, the row scores 0. Statsig records the error message as the grader's response for that row. |
| Time limit | 10 seconds for each row. A row that runs longer scores 0. |
| Source limit | 100,000 characters. Code over the limit fails every row. |
| Environment | A sandbox with the Python standard library. Statsig clears environment variables. |
To run online, a Python grader can read only item["input"] and item["trajectory"] from item, written as literal keys. Code that reads any other key, or a key held in a variable, runs in offline evals only.
Save a grader version
Edits on the Setup tab stay unsaved until you click Save Grader. Click Discard Changes to undo them.
Saving a change to the grader's definition creates a new version of the grader. When AI Configs use the grader, Statsig first shows the Update this grader? dialog, which calls AI Configs "prompts." Saving updates the grader for the latest version of each AI Config that uses it. Eval runs that already finished keep the grader version they used.
Choose where a grader runs
Targeting rules decide which offline evals include a grader and which live traffic it grades online. Set them in the Targeting rules section of the grader's Setup tab. Each rule is an offline rule, an online rule, or both.
To add a rule:
- On the grader's Setup tab, under Targeting rules, click Add rule.
- Check Offline, Online, or both.
- Set the rule's conditions. For each condition, choose a field, an operator, and one or more values.
- For an online rule, enter the percentage of matching traffic to grade in Sample.
- Click Save Grader.
A rule can be both offline and online when its only condition is "AI config is any of." The Sample percentage then applies to online grading only, because offline evals grade every row.
Offline rules
An offline rule adds the grader to the version evals of AI Configs. It takes exactly one condition, "AI config is any of," with no sampling. Check Critical to make the grader a critical grader for these AI Configs. On each AI Config's Setup tab, the grader card shows an Offline badge.
Adding a grader from an AI Config's Setup tab with Add grader has the same effect as an "AI config is any of" rule. A version eval runs at most 10 graders.
Runs on the Evals page don't use offline rules. Each run uses the graders you pick when you create the run.
Online rules
An online rule grades a sample of live traffic as it arrives. Only graders that can run online, as listed in Grader types, can have online rules.
Online evals grade one span for each agent turn: the workflow span when the trace has one, or else the outermost LLM call. The span must carry input and output messages, and tool spans are never graded. Spans don't need an AI Config, so a rule can also grade traced LLM calls that your app makes without one. For how to send traces, go to Online Evals.
An online rule matches a span when all of its conditions match. A rule with no conditions matches every span. Each condition uses one of these fields:
| Field | Matches |
|---|---|
| AI config | The span's AI Config, from statsig.llm.prompt_name |
| Environment | statsig.environment |
| Span name | The span's name |
| Model | The requested model, from gen_ai.request.model |
| Span attribute | Any span attribute. Enter the attribute name, then the values. |
Each condition uses the operator is any of or is not. Values match exactly, with no wildcards, and can't contain %. A span that doesn't have the attribute counts as an empty value, so is not matches it.
Each grader grades a span at most once, under the first enabled online rule that matches the span. Use the arrows on a rule to move it up or down. Sample sets the percentage of matching traces that the rule grades, from 0 to 100. New rules start at 10%. Statsig samples by trace: it hashes the trace ID, so every span in a trace gets the same sampling decision.
Online grading has these limits:
- Statsig grades at most 1,000 spans per hour for each AI Config that has online eval turned on. Other spans that online rules grade share one limit of 1,000 spans per hour for your company.
- Statsig processes online-eval traces in the US only. Statsig plans to support EU data residency.
Turn on online eval for an AI Config
A rule with an "AI config is any of" condition grades that AI Config's spans only while online eval is on for the AI Config. To turn it on, an admin opens the actions menu in the AI Config's header and clicks Enable Online Eval. The AI Config needs at least one saved online grader first. Pause Online Eval in the same menu turns online eval off.
While online eval is off for an AI Config, each rule that names the AI Config skips the AI Config's spans. Statsig checks those spans against the grader's later rules instead. The rule shows a warning that online evals are off for the AI Config, and the AI Config's grader card shows a Paused badge. Rules that don't name the AI Config keep grading its spans.
Disable an online rule
To stop an online rule without deleting it, clear its Online checkbox and save. Statsig keeps the rule, and AI Config grader cards show it with a Disabled badge.
Who can edit targeting rules
Enabled online rules start grading live traffic, so a project or org admin must save any change to them. This includes adding, editing, disabling, or deleting an enabled online rule. Anyone who can edit the grader can change its offline rules and its disabled online rules.
Targeting rule errors
Save Grader stays disabled until every rule is valid. These messages explain what to fix:
| Message | Fix |
|---|---|
| Check Offline, Online or both | Check at least one of Offline and Online. |
| Offline rules need one "AI config is any of" condition | Reduce the offline rule to one "AI config is any of" condition. |
| Every condition needs at least one value | Add a value to each condition, or remove the empty condition. |
| Span attribute conditions need an attribute name | Enter the attribute name in the Span attribute condition. |
| Condition values cannot contain % | Remove % from the value. Values match exactly. |
| Sampling rate must be between 0% and 100% | Enter a Sample value from 0 to 100. |
Online rules need a grader that reads just {{input}}, {{output}} and {{trajectory}} | Remove the online rules, or change the definition so the grader can run online. |
What is a primary grader
A primary grader is the grader whose score becomes an AI Config version's overall score, the number you compare across versions. Each AI Config has one primary grader. To change it, open the menu on a grader card in the AI Config's Eval Setup section and click Set as Primary. If you remove the primary grader, the next grader becomes primary.
What is a critical grader
A critical grader is a must-pass check for an AI Config's version evals. When a critical grader's score for a run is 0, the whole run fails. The run shows "This eval run failed due to the critical grader" with the grader's name, and the version's overall score is 0. When a critical grader scores above 0, it behaves like any other grader.
For example, if your app parses an AI Config's output as JSON, mark a JSON Validity grader critical for that AI Config. A version whose output never parses then fails its eval, whatever its other graders score.
You set Critical separately for each AI Config. Make a grader critical for an AI Config in either place:
- On the AI Config's Setup tab, open the menu on the grader card and click Set as Critical Grader.
- On the grader's Setup tab, check Critical on an offline rule with an "AI config is any of" condition.
Critical grading applies to offline evals only.
Review a grader's results
The grader's Results and Traces tabs show where it runs and how it scores.
- Results: The Used by table lists each AI Config and Playground that uses the grader. Its Type column shows Prompt for an AI Config. The Eval runs table lists each eval that ran the grader, with columns for Score, Eval, Type, Config, and Last run. Type is Online for online evals and On Demand for all other evals. A grader that nothing uses shows "This grader is currently unused."
- Traces: The AI traces that the grader scored, from the last day by default. Change the time range or add filters, and click a trace to open its details. Statsig tracks scores by grader ID, so renaming a grader keeps its history.
Grader costs
LLM as a Judge graders and Text Similarity graders that use Cosine call a model provider. Statsig charges these calls to your Included Credits unless your company has its own API key for the provider. Other graders run inside Statsig at no cost.
| Grader | Provider it calls | Cost |
|---|---|---|
| LLM as a Judge | The provider in Evaluation Model | Included Credits, or your own key for that provider |
| Text Similarity with Cosine | OpenAI, for text-embedding-3-large embeddings | Included Credits, or your own OpenAI key |
| String Comparison, Python, and other Text Similarity functions | None | Free |
Statsig doesn't meter calls on your own API keys. An admin adds keys in Settings > Products, under AI Provider API Keys. When your Included Credits run out, online evals keep running the graders that use your own keys and skip the rest. For details on Included Credits, go to Credits and API keys.
Was this helpful?
For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.