Online Evals
Grade production traffic in Statsig Agent Experimentation with online graders, which score a sample of agent turns about a minute after your app sends their spans.
Online evals grade your AI app's production traffic. Statsig samples the traces your app sends and picks one span for each agent turn. About one minute after that span arrives, Statsig runs your online graders on it. Online graders score each answer on its own, without an expected answer to compare against. Use online evals to compare an AI Config's Live version with its Candidate versions on real requests. To score versions against a dataset of expected answers before you release them, use offline evals instead.
How online evals work
An online eval has four stages:
- Your app answers each request with the AI Config's Live version. It can also run Candidate versions on the same input without showing their answers to users. Your code runs the candidates. Statsig doesn't run them for you.
- Your app sends a span for each model call to Statsig, tagged with the AI Config and version that produced it.
- Each grader's online rules decide which spans the grader scores and what percentage of traces it samples.
- Statsig grades each sampled span once. Scores appear on the span and on the AI Config's Traces tab.
Your code can load an AI Config's Live version and its Candidate versions. Draft and Archived versions never reach your code. To manage versions, go to AI Configs.
Instrument your app
Online grading reads the spans your app sends. Each span needs its input and output messages, its AI Config, and its version. With the Node AI SDK, do the following:
- Wrap your model client with
wrapOpenAI,wrapAnthropic, orwrapGemini, and turn oncaptureOptionsfor input and output messages. The SDK doesn't record message content by default, and Statsig skips spans without messages. - Call the model inside
prompt.withLive()orprompt.withVersion(). Both functions tag each span with the AI Config name and version. A span without an AI Config doesn't show on any AI Config's Traces tab. - Wrap each multi-step agent turn in one workflow span with
wrap({ type: "workflow" }).
The SDK still calls an AI Config a prompt, so you load one with getPrompt(). To send spans without the SDK, set the attributes in the span attribute reference on each span yourself.
The examples on this page share this setup:
import OpenAI from "openai";
import { StatsigAI, initializeTracing, wrapOpenAI } from "@statsig/statsig-ai";
// Sends spans to Statsig with the server secret key in STATSIG_SDK_KEY
initializeTracing({ serviceName: "support-bot" });
const statsigAI = new StatsigAI({ sdkKey: process.env.STATSIG_SDK_KEY });
await statsigAI.initialize();
// Record the input and output messages that online graders read
const openai = wrapOpenAI(new OpenAI(), {
captureOptions: {
capture_input_messages: true,
capture_output_messages: true,
},
});
Grade a single model call
When an agent turn makes one model call, Statsig grades that call. The SDK adds the statsig.span.llm_root marker to each model call that isn't inside another model call or a workflow span. An HTTP span or another app span around the call doesn't change the marker. Statsig grades the spans that carry this marker.
The following example answers with the Live version. It also runs each Candidate version on the same question in the background. Each call produces its own span, tagged with its version, so Statsig grades each version separately.
// Calls the model with one AI Config version
function callModel(version, question) {
return openai.chat.completions.create({
model: version.getModel({ fallback: "gpt-5-mini" }),
messages: version.getPromptMessages({ question }),
});
}
async function answerQuestion(user, question) {
const prompt = statsigAI.getPrompt(user, "support_bot");
// Candidates answer in the background. Users never see these answers.
for (const candidate of prompt.getCandidates()) {
prompt
.withVersion(candidate, (version) => callModel(version, question))
.catch(console.error);
}
const response = await prompt.withLive((version) =>
callModel(version, question),
);
return response.choices[0].message.content;
}
Grade a multi-step agent turn
When one turn makes several model calls or tool calls, wrap the turn in a workflow span. The workflow span ends after every step in the turn, so it can hold the turn's input and final answer. Statsig grades the workflow span as the turn's answer. Statsig never grades tool spans, such as spans from wrap({ type: "tool" }).
The SDK doesn't record messages on workflow spans. Before the workflow function returns, set gen_ai.input.messages and gen_ai.output.messages on the active span to the turn's input and final answer. Without these attributes, Statsig skips the workflow span. The SDK doesn't mark the model calls inside a workflow span, so Statsig then grades nothing in the turn. Start the turn inside withLive() or withVersion() so the workflow span records the AI Config and version.
import { trace } from "@opentelemetry/api";
import { wrap } from "@statsig/statsig-ai";
// One workflow span for each agent turn
const runTurn = wrap(
{ type: "workflow", name: "support_turn" },
async (version, question) => {
const model = version.getModel({ fallback: "gpt-5-mini" });
// Step 1: draft an answer from the AI Config's prompt
const draft = await openai.chat.completions.create({
model,
messages: version.getPromptMessages({ question }),
});
// Step 2: revise the draft
const revision = await openai.chat.completions.create({
model,
messages: [
{
role: "system",
content: "Correct any errors in this answer and shorten it.",
},
{ role: "user", content: draft.choices[0].message.content ?? "" },
],
});
const answer = revision.choices[0].message.content ?? "";
// The SDK doesn't record messages on workflow spans, so set the turn's
// question and final answer for online graders to read
trace.getActiveSpan()?.setAttributes({
"gen_ai.input.messages": JSON.stringify([
{ role: "user", content: question },
]),
"gen_ai.output.messages": JSON.stringify([
{ role: "assistant", content: answer },
]),
});
return answer;
},
);
async function answerWithAgent(user, question) {
const prompt = statsigAI.getPrompt(user, "support_bot");
// Start the turn inside withLive so the workflow span records the AI Config and version
return prompt.withLive((version) => runTurn(version, question));
}
In @statsig/statsig-ai versions 0.0.1-beta.30 through 0.0.1-beta.33, the SDK also marks each model call inside a workflow span with statsig.span.llm_root. Statsig checks each span on its own, so it grades the workflow span and each model call inside it. Version 0.0.1-beta.34 doesn't mark these calls. If you can't upgrade, add a Span attribute condition to the grader's online rule to grade only workflow spans. Set the attribute name to statsig.span.gen_ai.type, the operator to is any of, and the value to workflow. A rule with this condition doesn't grade turns that make a single model call.
Turn on online grading
Online grading starts when a grader has an online rule that matches your spans. When the rule names an AI Config, online eval must also be on for that AI Config.
- Open a grader that can run online, or create one. LLM as a Judge and Python graders can run online when they read only
{{input}},{{output}}, and{{trajectory}}. For details, go to Grader types. - On the grader's Setup tab, under Targeting rules, click Add rule and check Online.
- Add conditions that match your spans. For example, choose AI config, choose is any of, and select your AI Config.
- In Sample, enter the percentage of matching traces to grade. New rules start at 10%.
- Click Save Grader. A project or org admin must save any change to an enabled online rule.
- If the rule names an AI Config, turn on online eval for that AI Config. An admin must do this step: in the AI Config's header, open the actions menu, click Enable Online Eval, and then click Enable.
For every rule field and operator, go to Online rules.
Rules and the AI Config's online eval setting interact in these ways:
- Each grader grades a span under the first enabled online rule that matches the span.
- Statsig samples by trace. Statsig hashes the trace ID, so every span in a trace gets the same sampling decision.
- Enable Online Eval needs at least one grader with an enabled online rule that names the AI Config. Only admins can enable or pause online eval.
- Pause Online Eval, in the same menu, pauses only the rules that name the AI Config. Those rules skip the AI Config's spans, and the grader's later rules can still grade them. Rules that don't name the AI Config keep grading its spans.
- A rule without an AI config condition can also grade spans that don't belong to any AI Config, such as model calls outside
withLive()andwithVersion(). Scores for these spans appear in the trace drawer and on the grader's Traces tab, but not on an AI Config.
What Statsig grades
Statsig checks each span when it arrives. Statsig grades a span when every requirement in this table holds:
| Requirement | Span attributes |
|---|---|
| The span is an LLM span. | statsig.span.type is gen_ai. |
| The span is a workflow span or an outermost model call. Statsig never grades tool spans. | statsig.span.gen_ai.type is workflow, or statsig.span.llm_root is "1" and statsig.span.gen_ai.type isn't tool. |
| The span has input and output messages. | gen_ai.input.messages and gen_ai.output.messages aren't empty. |
| Statsig didn't generate the span. | statsig.environment isn't eval, shadow, or online_eval_judge. Statsig treats a span without statsig.environment as production. |
| An online rule matches the span and samples its trace. | The attributes in the rule's conditions. |
Statsig assigns each score to the AI Config version in statsig.llm.prompt_version_id. For spans from older SDKs that record only the version name, Statsig finds the version by its name.
A grader reads three values from the span. {{input}} holds the input messages, {{output}} holds the final assistant answer, and {{trajectory}} holds the tool calls and results before that answer. For message formats and every attribute, go to the span attribute reference. For the environment values, go to Environments.
Online grading limits
Online grading has these limits:
- Latency: A score appears on its span about one minute after Statsig receives the span. The Online graders chart can take longer to show the score.
- Hourly cap: Statsig grades at most 1,000 spans per hour for each AI Config that has online eval on. A span counts toward its AI Config's cap when every rule that grades it names the AI Config. All other graded spans, including spans without an AI Config, share one cap of 1,000 spans per hour for your company. Statsig doesn't grade spans over a cap, and the trace drawer shows
No grader scores this span. - One grade per span: Each grader grades a span at most once. Statsig retries failed judge calls. If a grader still fails, the span has no score from that grader.
- Included Credits: When your company's Included Credits run out, Statsig keeps running the graders that use your own provider API key. Statsig skips the other graders. The trace drawer explains that Statsig didn't grade the span. An admin adds API keys in Settings > Product Configuration > General, under AI Provider API Keys. For which graders use credits, go to Grader costs. For how Included Credits work, go to Credits and API keys.
- Data residency: Statsig processes online-eval traces in the US only. Statsig plans to support EU data residency.
Review online eval results
Statsig shows online grades on the AI Config's Traces tab, on each span, and on each grader's page.
Online graders chart
To open the chart, go to Agent Experimentation > AI Configs, open the AI Config, and click the Traces tab. The Online graders chart appears above the span list and covers the last day by default. Choose a view:
- Single version: Each grader's average score over time, for the version you select or for All versions. Grades that your code logs with
logEvalGradeappear as separate lines, labeled with the grade name and "(SDK)". - Compare: Select the versions to compare with the Live version. For each grader, the chart shows the score change relative to the Live version. The legend marks each change as Better, Worse, or No significant change.
How Statsig compares versions
The Compare view tests each grader's score change for statistical significance at p < 0.05. The test depends on where the scores come from:
| Score source | Comparison |
|---|---|
| Online graders | Welch's t-test, unpaired. Each graded span is one sample, because each trace comes from one version. Each version needs at least two scores. |
logEvalGrade (SDK) | Paired by sessionId. Statsig compares only sessions that have a grade for both versions. |
| Offline evals | Not included. Offline evals grade dataset rows, not production traffic. |
Scores on a span
To open a span's scores, click the span in the list, then click the Graders tab in the trace drawer. Each score shows its grader and its score. Online grades show the grader type, such as LLM as a Judge or Python. Grades from logEvalGrade show SDK. For the full tab, go to Graders tab.
A grader's Results and Traces tabs also list its online grades. Statsig tracks scores by grader ID, so a renamed grader keeps its history. For details, go to Review a grader's results.
Log grades from your code
Use logEvalGrade to record a score that your own code computes, such as a user's rating or a business outcome. Statsig shows these grades as SDK lines on the Online graders chart and as SDK scores in the trace drawer.
statsigAI.logEvalGrade(user, version, 1, "ticket_resolved", {
sessionId: conversationId,
traceId,
spanId,
reasoning: "The customer closed the ticket without escalating.",
metadata: { channel: "chat" },
});
// Send the grade events to Statsig
await statsigAI.flushEvents();
logEvalGrade takes these arguments, in order:
| Argument | Description |
|---|---|
user | The StatsigUser for the request. |
version | The AI Config version that produced the answer, such as the version that withLive() passes to your function. |
score | A number from 0 to 1. The SDK skips scores outside this range and logs a warning. |
graderName | The name of the grade, such as ticket_resolved. |
evalData | An object with optional fields: sessionId, activityId, traceId, spanId, reasoning, and metadata. Always pass this argument. Pass {} when you set no fields. |
The evalData object accepts these fields:
| Field | Description |
|---|---|
sessionId | Pairs grades across versions in the Compare view. Use the same ID for the live and candidate answers to one request. |
activityId | Adds a statsig.activity_id custom ID to the user. |
traceId, spanId | The span that the grade belongs to in the trace drawer. When you omit both, the SDK uses the active span. |
reasoning | Your explanation of the score, logged with the grade. |
metadata | Extra string fields to log with the grade. |
To grade a workflow span after it ends, such as when a user rates an answer later, save the span's IDs during the turn. Inside the workflow function, trace.getActiveSpan()?.spanContext() returns the workflow span's traceId and spanId.
Grade events in your project
Statsig logs each online grade as a statsig::eval_result event in your project. The Online graders chart reads these events. The event's value is the AI Config name, or empty for a span without an AI Config. The events have no user ID.
| Metadata field | Value |
|---|---|
score | The score, from 0 to 1. |
version_id, version_name | The AI Config version that produced the span. |
grader_id, grader_name, grader_version | The grader and the grader version that scored the span. |
ai_config_name | The AI Config name. |
source | online_eval. |
trace_id, span_id | The graded span. |
Statsig logs these events with a server secret key that has the description AI_EVALS. Statsig creates the key in your project the first time it logs an online grade.
logEvalGrade logs the same event name with source set to sdk. Its events also carry session_id, is_live, and the fields you pass in metadata. Grader metrics in experiments that you create from an AI Config read only grades from logEvalGrade. Online grades don't feed those metrics.
Was this helpful?
For AI agents: a documentation index is available at /llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.