---
title: AI Traces
description: "Inspect, filter, and review the LLM spans your app sends to Statsig in AI Traces, and look up which span attributes power each view, filter, and online grader."
product: general
lang: en
token_estimate: 3673
---
# AI Traces

AI Traces lists the LLM spans your app sends to Statsig: model calls, tool calls, and agent steps, each with its messages, model, token count, cost, and grader scores. Use AI Traces to debug a single agent turn, find slow or failing calls, review outputs by hand, and save useful examples to a dataset. To query every span your services send, including HTTP and database spans, use the [Traces Explorer](https://docs.statsig.com/infra-analytics/send-traces) instead.

> For AI agents: a documentation index is available at [/llms.txt](/llms.txt). Append `.md` to any page URL for markdown, or send `Accept: text/markdown`.

## Where traces appear

Statsig shows LLM spans in three places in the console:

- **AI Traces page**: Every LLM span in your project. Open it from _Agent Experimentation > AI Traces_.
- **Traces tab on an AI Config**: Spans tagged with that AI Config, with a filter for the AI Config version. The tab also shows online grader results, described in [Online Evals](https://docs.statsig.com/ai-evals/online-evals).
- **Traces tab on a grader**: Spans that the grader scored. The tab opens on the last day of spans.

In each of these views, click a span to open the trace drawer.

## Send traces to AI Traces

AI Traces shows only spans whose `statsig.span.type` attribute is `gen_ai`. Spans you send by following the generic [Traces Explorer quick start](https://docs.statsig.com/infra-analytics/send-traces) don't carry that attribute, so they appear in the Traces Explorer but not in AI Traces.

Choose one way to send LLM spans:

- **Node AI SDK**: Call `initializeTracing()` and wrap your model client with `wrapOpenAI`, `wrapAnthropic`, or `wrapGemini`. The SDK sets the required attributes for you. The SDK doesn't record message content by default. Turn on `captureOptions` to see messages in the trace drawer and to make spans eligible for online grading. Call the model inside `withLive()` or `withVersion()` so each span records its AI Config and version. Go to [Node AI SDK](https://docs.statsig.com/ai-evals/node) for setup.
- **Raw OpenTelemetry**: Export spans to Statsig over OTLP/HTTP, as the [Traces Explorer quick start](https://docs.statsig.com/infra-analytics/send-traces) shows, and set the attributes in the [span attribute reference](#span-attribute-reference) on each LLM span yourself.

The following raw OpenTelemetry span sets the attributes that AI Traces and online grading read. OpenTelemetry attributes can't hold nested objects, so serialize message arrays as JSON strings. `callModel` stands in for your own model call.

```js
import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("support-bot");

await tracer.startActiveSpan("chat gpt-5", async (span) => {
  const answer = await callModel(question);
  span.setAttributes({
    "statsig.span.type": "gen_ai",
    "statsig.span.gen_ai.type": "generate_content",
    "statsig.span.llm_root": "1", // a string, not a number
    "statsig.llm.prompt_name": "support_bot",
    "gen_ai.request.model": "gpt-5",
    "gen_ai.input.messages": JSON.stringify([
      { role: "user", parts: [{ type: "text", content: question }] },
    ]),
    "gen_ai.output.messages": JSON.stringify([
      { role: "assistant", parts: [{ type: "text", content: answer.text }] },
    ]),
    "gen_ai.usage.input_tokens": answer.inputTokens,
    "gen_ai.usage.output_tokens": answer.outputTokens,
  });
  span.end();
});
```

## Find spans in the explorer

The AI Traces page opens on the past hour of spans. The search bar starts with `#environment!:eval,shadow,online_eval_judge`, which hides the spans Statsig generates itself. Remove that filter to include those spans. For what each value means, go to [Environments](#environments).

The span scope menu next to the search bar sets which spans the list includes:

- **All Spans**: Every LLM span.
- **Trace Root Spans**: Only the first span in each trace. A trace whose first span isn't an LLM span, such as an HTTP request span that wraps your model calls, doesn't appear in this view.

### Columns

| Column | Shows |
| --- | --- |
| Timestamp | When the span started. |
| Duration | How long the span took. |
| Name | The span name. |
| Service | The service that sent the span, from the `service.name` resource attribute. |
| Config | The AI Config the span belongs to, from `statsig.llm.prompt_name`. |
| Graders | One chip per grader score, or `Not graded` when the span has no scores. |

Each grader chip shows the grader name and `pass` or `fail`. A score above 0.66 counts as `pass`. The trace drawer shows the exact score.

### Filters

Filters in the left rail narrow the span list.

| Filter | Filters on |
| --- | --- |
| Duration (ms) | Span duration. |
| Config | The AI Config, from `statsig.llm.prompt_name`. |
| Grader | The graders that scored the span. |
| Environment | The `statsig.environment` attribute. Only equality filters work. |
| Status | Span status, such as `error`. |
| Span name | The span name. |
| Service | The `service.name` resource attribute. |
| Cluster | The `cluster_name` resource attribute. |
| Has activity ID | Whether the span has a `statsig.activity_id` custom ID. |

## Inspect a trace

Click a span in the list to open the trace drawer. The drawer header shows the trace ID, duration, total tokens, cost in USD, and service, plus the review actions described in [Review and save spans](#review-and-save-spans).

### Span tree

The left side of the drawer lists every span in the trace. Use **Search spans** to filter the list by span name, and switch between **Tree** and **Timeline** to show the spans as a hierarchy or on a time axis. Each row shows an icon for the span type (agent or workflow, tool, model call, or step), the span name, the tool or AI Config name, duration, cost, and up to three grader chips.

When the selected span has a `statsig.activity_id` custom ID, the drawer loads every span with that activity ID, even across traces. Use an activity ID to group a multi-turn conversation or a long-running job into one view.

### Trace tab

The **Trace** tab shows the selected span's messages. Switch between two views:

- **Formatted**: A chat transcript that Statsig builds from `gen_ai.system_instructions`, `gen_ai.input.messages`, and `gen_ai.output.messages`. Each turn has a role label, such as System, User, Assistant, or Tool. A tool call appears as `Function call` with the tool name and its input, and a tool result appears as `Function output`. The tab shows `This span has no messages.` when the span has no message attributes.
- **JSON**: Input and Output cards with the raw messages, and a Metadata card with every span attribute. Attributes that hold JSON strings expand into objects.

### Details tab

The **Details** tab lists the span's Start, Duration, Service, Model, Agent, Trace ID, Tokens, and Cost (USD). Model, Agent, and Tokens appear only when the span sets the matching attribute. Go to the [span attribute reference](#span-attribute-reference) for the source of each field.

### Graders tab

The **Graders** tab shows one card per score on the selected span. The tab label shows a count of scores. Each card shows:

- The grader name and its score.
- A label for the grader type, such as LLM as a Judge, String Comparison, Text Similarity, Python, or Programmatic Grader. Scores with no grader type show their source instead: Online eval, Offline eval, SDK, User feedback, Reviewer, or Other.
- The judge's reasoning steps, for LLM-as-a-judge scores.
- **View eval**, for scores from an eval run. The link opens that run.

A span with no scores shows `No grader scores this span.` If your organization runs out of Included Credits, the tab explains that Statsig didn't grade the span and how to add your own API key.

## Review and save spans

Use review actions to mark spans for follow-up and to build datasets from production traffic. Actions in the drawer header apply to the open span.

- **Thumbs up or thumbs down**: Record a human review of the span.
- **Add tag**: Apply an existing tag, or choose **Create New Tag**. Choose **Manage Tags** to rename or remove tags.
- **Add Annotation**: Write a comment on the span. Edit or delete an annotation from its menu.
- **Add to dataset**: Choose a dataset, or choose **Create new Dataset** and enter a **Dataset Name**.

To act on many spans at once, hover over a row in the list and select its checkbox. A bar shows the number of selected spans with **Add to Dataset**, **Add Tag**, and thumbs up or down.

When you add spans to a dataset, each span becomes a row. The `input` column holds the input messages, the `output` column holds the final answer, and the `metadata` column holds the model, token counts, and trace and span IDs. Spans with annotations add an `annotations` column, and a span's tags become categories on its row.

## Environments

The `statsig.environment` span attribute records which deployment environment a span came from. AI Traces, the Environment filter, and online grading all read this attribute.

A span gets its `statsig.environment` value in one of two ways:

- The span sets `statsig.environment` itself.
- The span doesn't set it, and the `deployment.environment.name` resource attribute names a non-production environment, such as `staging`. Statsig copies that name into `statsig.environment`.

The Node AI SDK sets `deployment.environment.name` from the `environment` option of `initializeTracing()`, or from `NODE_ENV` when you don't pass one.

Production spans usually have no `statsig.environment` value, because Statsig copies only non-production names. Online grading treats a span with no value as production, and can grade spans from any of your environments.

Statsig reserves three values for spans it generates itself. The AI Traces page hides these spans by default, and online grading never grades them.

| Value | Spans |
| --- | --- |
| `eval` | Offline eval runs, one trace per dataset row. |
| `online_eval_judge` | The model calls an online grader makes while it scores your spans. |
| `shadow` | Other runs that Statsig generates. |

Don't use these values for your own spans.

## Span attribute reference

These span attributes control what AI Traces shows and which spans online grading can score. The Node AI SDK sets them for you. If you send raw OpenTelemetry spans, set them yourself.

### Span type and structure

| Attribute | Value | What it powers |
| --- | --- | --- |
| `statsig.span.type` | `gen_ai` | Required. Only spans with this value appear in AI Traces, in the Traces tabs, and in online grading. |
| `statsig.span.gen_ai.type` | `workflow`, `agent`, `tool`, `generate_content`, `step` | The icon in the span tree. Online grading grades `workflow` spans and never grades `tool` spans. Without this attribute, Statsig reads `gen_ai.operation.name`, then the first word of the span name, such as `execute_tool` or `invoke_agent`. |
| `statsig.span.llm_root` | `"1"` (string) | Marks the outermost LLM call in an agent turn. Online grading grades an LLM span only when it carries this marker or is a `workflow` span. |

### AI Config

| Attribute | Value | What it powers |
| --- | --- | --- |
| `statsig.llm.prompt_name` | The AI Config name you pass to `getPrompt()` | The Config column and filter, and the AI Config's Traces tab. The AI Config's online eval grades only spans with this attribute. Without it, only a grader's grading rules can grade the span. |
| `statsig.llm.prompt_version_id` | The AI Config version ID | The version filter on the AI Config's Traces tab, and the version online grading assigns each score to. Older SDKs send `statsig.llm.prompt_version_name` or `statsig.llm.prompt_version`. |

The SDK, the API, and these attribute names use `prompt`, the earlier name for an AI Config.

### Messages

Set message attributes to JSON strings. AI Traces accepts OpenTelemetry GenAI messages (`role` with `parts`) and OpenAI chat messages (`role` with `content`).

| Attribute | What it powers |
| --- | --- |
| `gen_ai.system_instructions` | System turns in the Formatted view. |
| `gen_ai.input.messages` | Input turns in the Formatted view, the Input card in the JSON view, the dataset `input` column, and the `{{input}}` grader variable. Online grading skips spans without input messages. |
| `gen_ai.output.messages` | Output turns in the Formatted view and the Output card. The final assistant answer becomes the dataset `output` column and the `{{output}}` grader variable. The tool calls and results before it become `{{trajectory}}`. Online grading skips spans without output messages. |

### Model, tokens, and cost

| Attribute | What it powers |
| --- | --- |
| `gen_ai.request.model` | Model in the Details tab. Statsig also uses it to estimate cost when `gen_ai.response.model` is missing. |
| `gen_ai.response.model` | The model Statsig uses first to estimate cost. |
| `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` | Cost estimates. |
| `gen_ai.usage.input_cached_tokens` or `gen_ai.usage.cache_read.input_tokens` | Cost estimates. Statsig prices cached input tokens at the model's cached rate. |
| `statsig.gen_ai.usage.total_tokens`, `gen_ai.usage.total_tokens` | Tokens in the Details tab. |
| `statsig.gen_ai.usage.cost_usd` | Cost (USD) in the drawer and the span tree. If the span doesn't set a cost, Statsig estimates one from the model and token counts for models it has prices for. A cost you send always takes precedence. |

### Agents and tools

| Attribute | What it powers |
| --- | --- |
| `gen_ai.agent.name` | Agent in the Details tab. |
| `gen_ai.tool.name` | The tool name on the span's row in the span tree. |
| `gen_ai.operation.name` | The span type icon, when the span doesn't set `statsig.span.gen_ai.type`. |

### Grouping and environment

| Attribute | What it powers |
| --- | --- |
| `statsig.environment` | The Environment filter, the default filter that hides Statsig-generated spans, and which spans online grading skips. Go to [Environments](#environments). |
| `statsig.custom_ids` | Custom IDs for the span, as a JSON string. Include a `statsig.activity_id` key, for example `{"statsig.activity_id":"conversation-123"}`, to power the Has activity ID filter and group spans across traces in the drawer. |

### Resource attributes

Set these on the OpenTelemetry resource, not on individual spans.

| Attribute | What it powers |
| --- | --- |
| `service.name` | The Service column and filter, and Service in the drawer. |
| `cluster_name` | The Cluster filter. |
| `deployment.environment.name` | The environment. Statsig copies non-production names into `statsig.environment`. Go to [Environments](#environments). |