---
title: OpenTelemetry
description: Configure OpenTelemetry destinations, content capture, and third-party exports.
url: "https://eve.dev/docs/observability/otel"
docs_index: /llms.txt
---

> For an index of all documentation, see [/llms.txt](/llms.txt).

`eve dev` records local traces by default. Use `otel()` for process-wide
settings and `otelIntegration()` for third-party destinations.

## Session handover events

Existing Workflow step spans include these events when tracing is active:

- `eve.session.handover.forwarding_started`: the previous owner starts submitting its buffered messages, including an empty buffer.
- `eve.session.handover.part_submitted`: a buffered part was durably submitted.
- `eve.session.handover.final_submitted`: the final marker was durably submitted. This does not confirm that the successor has processed it.
- `eve.session.handover.inbox_unblocked`: an existing turn step observes that the inbox applied every buffered part and the final marker.

Join both owners with `eve.session.id` and `eve.handoff.attempt_key`.
`workflow.run.id` identifies the emitting owner; submission events also include
`eve.handoff.destination_run_id`. Counts and part indexes describe the transfer
without recording message content. Step IDs and `eve.handoff.step_attempt`
distinguish retries; these events are not exactly-once counters.

The receiver records `eve.handoff.applied_at_ms` using Workflow's replay-stable
clock, identified by `eve.handoff.applied_at_clock: workflow`. This is logical
event time, not the host time when replay applied the marker. The event's own
timestamp is when the next turn step exports the observation. No receiver
observation is exported if no subsequent turn step runs. An abandoned handover
does not emit a successful receipt. No additional Workflow steps or writes are
created for tracing.

## Configure OpenTelemetry

`otel()` and `otelIntegration()` are optional:

- Use `otel()` only when you need process-wide settings such as the resource,
  sampler, propagators, or trace capture policy. Declare it at most once.
- Use `otelIntegration()` only when adding a third-party OpenTelemetry
  destination. Declare one file per exporter or processor chain.

Add `agent/instrumentation/otel.ts` when you need process-wide settings or
want to control the content eve writes to OpenTelemetry spans:

```ts title="agent/instrumentation/otel.ts"
import { otel } from "eve/instrumentation/otel";

export default otel({
  resource: { "deployment.environment": process.env.VERCEL_ENV ?? "development" },
  tracePolicy: ({ audience, environment }) => ({
    emit: true,
    recordInputs: audience === "public" || environment === "development",
    recordOutputs: audience === "public" || environment === "development",
  }),
});
```

The OpenTelemetry `tracePolicy` is a capture ceiling shared by every OTel
destination. A destination cannot restore content excluded by this policy. It
does not affect lifecycle instrumentation created with `defineInstrumentation()`.
See [Audience](../channels/overview#audience) for how channels classify the
conversation passed to this policy.

The session's resolved capture decision also selects its default
[Workflow run-data retention](/docs/agent-config#run-data-retention). Allowing
both inputs and outputs uses the World's default; dropping the trace or excluding
either direction requests deletion of durable payloads and streams when the run
ends. This makes completed private and unknown-audience transcripts unreadable
under the default policy outside development. Set
`experimental.workflow.retention: "default"` on the agent to retain completed
sessions independently of tracing. Destination `exportPolicy` filters do not
change Workflow retention, and Workflow deletion does not delete exported traces.

## Instrument Node.js dependencies

Pass OpenTelemetry instrumentation instances to `otel({ instrumentations })`.
For example, install the PostgreSQL instrumentation alongside `pg`:

```bash
pnpm add pg @opentelemetry/instrumentation-pg
```

```ts title="agent/instrumentation/otel.ts"
import { PgInstrumentation } from "@opentelemetry/instrumentation-pg";
import { otel } from "eve/instrumentation/otel";

export default otel({
  instrumentations: [new PgInstrumentation()],
});
```

Keep the instrumented package external so Node.js module hooks can intercept
it. Add `pg` to your agent's [build configuration](/docs/agent-config#other-defineagent-fields):

```ts title="agent/agent.ts"
import { defineAgent } from "eve";

export default defineAgent({
  model: "openai/gpt-5.4-mini",
  build: { externalDependencies: ["pg"] },
});
```

## Add an OpenTelemetry destination

Use this section when sending traces to a third-party destination. Install
`@vercel/otel` before using its exporters:

```bash
pnpm add @vercel/otel
```

```ts title="agent/instrumentation/honeycomb.ts"
import { OTLPHttpProtoTraceExporter } from "@vercel/otel";
import { otelIntegration } from "eve/instrumentation/otel";

export default otelIntegration({
  exportPolicy: {
    span: () => ({ redact: true, inputs: true, outputs: true }),
  },
  traceExporter: new OTLPHttpProtoTraceExporter({
    url: "https://api.honeycomb.io/v1/traces",
    headers: {
      "x-honeycomb-team": process.env.HONEYCOMB_API_KEY!,
    },
  }),
});
```

Pass `spanProcessors` to `otelIntegration()` when a custom destination needs
filtering or transformation. The example's `exportPolicy` preserves span
metadata while removing inputs and outputs from this destination.

## Add runtime context

Use `runtimeContext` to add JSON values to AI SDK spans for each model attempt.
For typed channel metadata, import the channel definition and narrow with
`isChannel`:

```ts title="agent/instrumentation/support.ts"
import { isChannel } from "eve/instrumentation";
import { otelIntegration } from "eve/instrumentation/otel";

import supportChannel from "../channels/support";

export default otelIntegration({
  runtimeContext(input) {
    if (!isChannel(input.channel, supportChannel)) return undefined;

    return {
      "support.channel_id": input.channel.metadata.channelId ?? "",
      "support.user_id": input.channel.metadata.triggeringUserId ?? "",
    };
  },
});
```

The resolver receives the channel, session, final model input, step, and turn.
Keys beginning with `eve.` are reserved. Content excluded by
`otel({ tracePolicy })` is not available to the resolver.

Authored values appear under `ai.settings.context.*`. Framework channel, session,
step, turn, and version identifiers use structural span attributes instead of
runtime-context copies. `ai.settings.context.eve.environment` remains available.

## Trace topology

OpenTelemetry destinations receive the following trace topology for an ordinary
agent turn:

```text
invoke_agent <agent>
  └── agent.step
        ├── chat <model>
        └── execute_tool <tool>
```

Each root agent turn starts a new trace. If eve receives a local caller's trace
context, the first local subagent turn uses that trace. Its span is a
child of the dispatch span and has a different span ID. Without caller trace
context, the first subagent turn starts a new trace. Later turns in the child
session start new traces. They do not use the original dispatch span as their
parent.

Use `gen_ai.tool.call.id` to correlate tool and approval spans. Tool failures use
`error.type`. These attributes replace the duplicate `agent.action.call_id` and
`agent.action.error.code` fields without changing trace schema version 4.

Model and activation spans record token counts under `gen_ai.usage.*`, without
duplicate `agent.usage.*` token fields. Step and tool spans retain `agent.usage.*`.
Cost values remain unchanged.

A remote child starts a separate trace with a root `invoke_agent` span. Its first
turn links to the caller with `eve.link.type=agent.dispatch`. The receiving
deployment applies its sampler. Queries for local dispatches must use
`parentSpanId` or `agent.parent_run.id` and `agent.parent_call.id` instead of a
dispatch link.

Use `gen_ai.conversation.id` to find the root session and its child sessions
across traces. eve adds this ID to spans created during model, tool, and memory
execution. This includes spans from third-party instrumentation. `agent.run.id`
identifies one agent session, not the full conversation.

The shared conversation ID does not give access to message history, agent state,
or credentials. It does not give authorization or require the same worker for
each turn. Remote requests carry the conversation ID in W3C Baggage.
`trustedForwarders` controls whether the receiver accepts parent session data
and the trace-content policy. See
[Remote agents](../guides/remote-agents#preserving-trace-content).
Gateway-routed model calls propagate the configured OpenTelemetry context at
each provider request. Eve-owned baggage is removed before forwarding. Gateway
controls whether its spans are sampled and which destinations receive them.

When AI Gateway reports
[session affinity](https://vercel.com/docs/ai-gateway/models-and-providers/session-affinity)
for a model call, the `agent.step` span records it. These attributes appear even
when the trace content policy omits outputs.

`vercel.ai_gateway.client_session_id` is the session ID AI Gateway used for
affinity. It is not the eve session ID that `vercel.session_id` records. It
equals `gen_ai.conversation.id` unless you set `gateway.sessionId` yourself or
the ID is longer than 256 characters, in which case AI Gateway hashes it. An
authored `gateway.sessionId` is recorded verbatim, so don't put personal data in
it.

| Attribute                                       | Value                                                           |
| ----------------------------------------------- | --------------------------------------------------------------- |
| `vercel.ai_gateway.client_session_id`           | Session ID AI Gateway used                                      |
| `vercel.ai_gateway.client_session_id_source`    | `explicit`, `agent`, or `fingerprint`                           |
| `vercel.ai_gateway.affinity.outcome`            | What affinity did, such as `promoted`, `confirmed`, or `no_pin` |
| `vercel.ai_gateway.affinity.pinned_provider`    | Provider the session's pin named                                |
| `vercel.ai_gateway.affinity.displaced_provider` | Provider that would have gone first, when the pin moved it      |

## Local traces

`eve dev` records local traces by default. Omitting
`agent/instrumentation/local.ts` preserves that default. Export
`localTraces(...)` from that file to reconfigure it, or disable it explicitly:

```ts title="agent/instrumentation/local.ts"
import { disableInstrumentation } from "eve/instrumentation";

export default disableInstrumentation();
```

## Enable tracing on Vercel

For Agent Runs sampling defaults and existing-deployment setup, see
[Enable tracing on Vercel](./agent-runs#enable-tracing-on-vercel).

## Filter a destination

`otelIntegration()` and `localTraces()` accept one
`exportPolicy` object or an array applied in order. Each policy filters spans
and attributes before that destination's processors receive them:

```ts title="agent/instrumentation/local.ts"
import { localTraces } from "eve/instrumentation/otel";

export default localTraces({
  exportPolicy: {
    span: ({ name }) => ({ emit: name !== "internal.cache.refresh" }),
    attribute: ({ key }) => (key === "customer.id" ? { emit: false } : { emit: true }),
  },
});
```

Return `{ emit: true }` from `span` to retain a span unchanged, or
`{ emit: false }` to omit it from this destination. To retain the span while
redacting content, return `{ redact: true }` with `inputs: true`,
`outputs: true`, or both. A redaction decision implies emission and requires
at least one direction. For example, return
`{ redact: true, inputs: true, outputs: true }` to redact both directions.
A throwing `span` callback drops the span from this destination. Return
`{ emit: true }`, `{ emit: false }`, or
`{ replace: true, value }` from `attribute` to retain, remove, or change
one attribute.

## What to read next

- [Instrumentation](/docs/observability/instrumentation): handle eve lifecycle events.
- [Agent Runs](./agent-runs): configure Vercel's built-in trace destination.
- [Migrate instrumentation](/docs/observability/instrumentation-migration): replace `agent/instrumentation.ts`.
- [Local development](../guides/dev-tui): inspect local traces in the TUI.

---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)