---
title: OpenTelemetry
description: Configure OpenTelemetry destinations, content capture, and third-party exports.
url: "https://eve.dev/docs/observability/otel"
docs_index: /llms.txt
---

> For an index of all documentation, see [/llms.txt](/llms.txt).

eve emits OpenTelemetry traces for agent turns, and `eve dev` records them locally by default. Configure OpenTelemetry with files under `agent/instrumentation/`: `otel()` for process-wide settings, and `otelIntegration()` for each third-party destination.

## Configure OpenTelemetry

Both are optional. Declare `otel()` at most once, in `agent/instrumentation/otel.ts`, for settings such as the resource, sampler, propagators, or trace capture policy. Declare `otelIntegration()` in one file per third-party exporter or processor chain.

```ts title="agent/instrumentation/otel.ts"
import { otel } from "eve/instrumentation/otel";

export default otel({
  resource: { "deployment.environment": process.env.VERCEL_ENV ?? "development" },
  tracePolicy: ({ audience, environment }) => ({
    emit: true,
    recordInputs: audience === "public" || environment === "development",
    recordOutputs: audience === "public" || environment === "development",
  }),
});
```

`otel({ tracePolicy })` sets the most content any OTel destination can receive. No destination can add back content it excludes. It doesn't affect instrumentation files created with `defineInstrumentation()`. See [Audience](/docs/channels#audience) for how channels classify the conversation passed to this policy.

The same capture decision sets the session's default [Workflow run-data retention](/docs/agent-config#run-data-retention):

- Allowing both inputs and outputs keeps the World's default retention.
- Dropping the trace or excluding either direction deletes stored payloads and streams when the run ends.

So under the default policy, completed private and unknown-audience transcripts are unreadable outside development. Set `experimental.workflow.retention: "default"` on the agent to keep completed sessions regardless of tracing. Destination `exportPolicy` filters don't change Workflow retention, and Workflow deletion doesn't delete exported traces.

## Add an OpenTelemetry destination

To send traces to a third-party destination, install `@vercel/otel` for its exporters:

```bash
pnpm add @vercel/otel
```

```ts title="agent/instrumentation/honeycomb.ts"
import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

export default otelIntegration({
  exportPolicy: {
    span: () => ({ redact: true, inputs: true, outputs: true }),
  },
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: "https://api.honeycomb.io/v1/traces",
    headers: {
      "x-honeycomb-team": process.env.HONEYCOMB_API_KEY!,
    },
  }),
});
```

The example's `exportPolicy` keeps span metadata but removes inputs and outputs from this destination. Pass `spanProcessors` to `otelIntegration()` when a custom destination needs filtering or transformation. Processors and exporters from any `@opentelemetry/*` build satisfy the `SpanProcessor` and `SpanExporter` types from `eve/instrumentation/otel`. `SpanProcessorOrName` also accepts `"auto"`.

## Filter a destination

`otelIntegration()` and `localTraces()` accept one `exportPolicy` object or an array applied in order. Each policy filters spans and attributes before that destination's processors receive them:

```ts title="agent/instrumentation/local.ts"
import { localTraces } from "eve/instrumentation/otel";

export default localTraces({
  exportPolicy: {
    span: ({ name }) => ({ emit: name !== "internal.cache.refresh" }),
    attribute: ({ key }) => (key === "customer.id" ? { emit: false } : { emit: true }),
  },
});
```

| Callback    | Return                                                           | Effect                                 |
| ----------- | ---------------------------------------------------------------- | -------------------------------------- |
| `span`      | `{ emit: true }`                                                 | Keep the span unchanged.               |
| `span`      | `{ emit: false }`                                                | Omit the span from this destination.   |
| `span`      | `{ redact: true }` with `inputs: true`, `outputs: true`, or both | Keep the span and remove that content. |
| `span`      | Throws                                                           | Drop the span from this destination.   |
| `attribute` | `{ emit: true }`                                                 | Keep the attribute.                    |
| `attribute` | `{ emit: false }`                                                | Remove the attribute.                  |
| `attribute` | `{ replace: true, value }`                                       | Change the attribute's value.          |

A redaction decision implies emission and needs at least one direction.

## Local traces

Omit `agent/instrumentation/local.ts` to keep `eve dev`'s local traces. Export
`localTraces(...)` from that file to reconfigure them, or `disableInstrumentation()` to turn them off:

```ts title="agent/instrumentation/local.ts"
import { disableInstrumentation } from "eve/instrumentation";

export default disableInstrumentation();
```

## Enable tracing on Vercel

For Agent Runs sampling defaults and existing-deployment setup, see
[Enable tracing on Vercel](./agent-runs#enable-tracing-on-vercel).

## Instrument Node.js dependencies

Pass OpenTelemetry instrumentation instances to `otel({ instrumentations })`.
For example, install the PostgreSQL instrumentation alongside `pg`:

```bash
pnpm add pg @opentelemetry/instrumentation-pg
```

```ts title="agent/instrumentation/otel.ts"
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { otel } from "eve/instrumentation/otel";

export default otel({
  instrumentations: [new PgInstrumentation()],
});
```

Keep the instrumented package external so Node.js module hooks can intercept
it. Add `pg` to your agent's [build configuration](/docs/agent-config#other-defineagent-fields):

```ts title="agent/agent.ts"
import { defineAgent } from "eve";

export default defineAgent({
  model: "openai/gpt-5.4-mini",
  build: { externalDependencies: ["pg"] },
});
```

## Add runtime context

Use `runtimeContext` to add JSON values to AI SDK spans for each model attempt.
For typed channel metadata, import the channel definition and narrow with
`isChannel`:

```ts title="agent/instrumentation/support.ts"
import { isChannel } from "eve/channels";
import { otelIntegration } from "eve/instrumentation/otel";

import supportChannel from "../channels/support";

export default otelIntegration({
  runtimeContext(input) {
    if (!isChannel(input.channel, supportChannel)) return undefined;

    return {
      "support.channel_id": input.channel.metadata.channelId ?? "",
      "support.user_id": input.channel.metadata.triggeringUserId ?? "",
    };
  },
});
```

The resolver receives the channel, session, final model input, `run: { id, attempt }`, and optional turn. Its input type is `InstrumentationModelStartedInput` from `eve/instrumentation`. Model input content is `unknown`, so narrow provider-specific messages before you read them. Content that `otel({ tracePolicy })` excludes isn't available to the resolver.

Your values appear under `ai.settings.context.*`, next to `ai.settings.context.eve.environment`. Keys that begin with `eve.` are reserved. Channel, session, step, turn, and version IDs are regular span attributes, not runtime context.

## Trace topology

An ordinary agent turn produces this trace:

```text
invoke_agent <agent>
  └── agent.step
        ├── chat <model>
        └── execute_tool <tool>
```

Each root agent turn starts a new trace. A local subagent's first turn joins the caller's trace, as a child of the dispatch span, when eve receives the caller's trace context. Otherwise it starts a new trace. Later turns in the child session start new traces without the dispatch span as parent.

A remote child starts a separate trace with a root `invoke_agent` span. Its first turn links to the caller with `eve.link.type=agent.dispatch`, and the receiving deployment applies its own sampler. Local dispatches have no dispatch link, so query them by `parentSpanId`, or by `agent.parent_run.id` and `agent.parent_call.id`.

| Attribute                | Use                                                         |
| ------------------------ | ----------------------------------------------------------- |
| `gen_ai.conversation.id` | Find the root session and its child sessions across traces. |
| `agent.run.id`           | Identify one agent session, not the full conversation.      |
| `gen_ai.tool.call.id`    | Match tool spans with their approval spans.                 |
| `error.type`             | Classify tool failures.                                     |
| `gen_ai.usage.*`         | Token counts on model and activation spans.                 |
| `agent.usage.*`          | Token counts on step and tool spans.                        |

The trace schema is version 4.

eve adds `gen_ai.conversation.id` to spans created during model, tool, and memory work, including spans from third-party instrumentation. The conversation ID grants no access to message history, agent state, or credentials, and no authorization. Remote requests carry it in W3C Baggage. `trustedForwarders` controls whether the receiver accepts parent session data and the trace-content policy. See [Remote agents](../guides/remote-agents#preserving-trace-content).

Gateway-routed model calls pass the configured OpenTelemetry context on each provider request. eve removes its own baggage first. AI Gateway decides whether its spans are sampled and which destinations receive them.

When AI Gateway reports [session affinity](https://vercel.com/docs/ai-gateway/models-and-providers/session-affinity) for a model call, the `agent.step` span records it, even when the trace content policy omits outputs.

`vercel.ai_gateway.client_session_id` is the session ID AI Gateway used for affinity. It isn't the eve session ID, which `vercel.session_id` records. Unless you set `gateway.sessionId` yourself, it equals `gen_ai.conversation.id`. AI Gateway hashes a conversation ID longer than 256 characters. eve records an authored `gateway.sessionId` exactly as written, so don't put personal data in it.

| Attribute                                       | Value                                                           |
| ----------------------------------------------- | --------------------------------------------------------------- |
| `vercel.ai_gateway.client_session_id`           | Session ID AI Gateway used                                      |
| `vercel.ai_gateway.client_session_id_source`    | `explicit`, `agent`, or `fingerprint`                           |
| `vercel.ai_gateway.affinity.outcome`            | What affinity did, such as `promoted`, `confirmed`, or `no_pin` |
| `vercel.ai_gateway.affinity.pinned_provider`    | Provider the session's pin named                                |
| `vercel.ai_gateway.affinity.displaced_provider` | Provider that would have gone first, when the pin moved it      |

## Session handover events

When a session moves to a new workflow run, after a deployment change or [compaction](../concepts/execution-model-and-durability#compaction-handoff), Workflow step spans include these events:

- `eve.session.handover.forwarding_started`: the previous run starts sending its buffered messages, even when the buffer is empty.
- `eve.session.handover.part_submitted`: eve saved one buffered part for the new run.
- `eve.session.handover.final_submitted`: eve saved the final marker. The new run may not have processed it yet.
- `eve.session.handover.inbox_unblocked`: a turn step in the new run sees that every buffered part and the final marker were applied.

Join the two runs with `eve.session.id` and `eve.handoff.attempt_key`. `workflow.run.id` identifies the run that emitted the event, and submission events also include `eve.handoff.destination_run_id`. Counts and part indexes describe the transfer without message content. Step retries can repeat events, so tell them apart by step ID and `eve.handoff.step_attempt` instead of counting events.

The new run records `eve.handoff.applied_at_ms` on Workflow's replay-stable clock (`eve.handoff.applied_at_clock: workflow`). That value is logical event time, not host time. The event's own timestamp is when the next turn step exports it, so the new run emits no event until a later turn step runs. An abandoned handover emits no success event.

## What to read next

- [Instrumentation](/docs/observability/instrumentation): handle eve lifecycle events.
- [Agent Runs](./agent-runs): configure Vercel's built-in trace destination.
- [Local development](../guides/dev-tui): inspect local traces in the TUI.

---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)