Observability

Instrumentation

Configure lifecycle instrumentation, handle runtime events, and control the content each destination receives.

Add lifecycle instrumentation as files under agent/instrumentation/. Each file handles eve runtime events independently. Configure trace exporters and shared span settings with the built-in OpenTelemetry APIs.

The instrumentation directory replaces agent/instrumentation.ts. See Migrate instrumentation when upgrading from eve < 0.62.0.

Add instrumentation

The filename identifies the instrumentation slot. A lifecycle-event file must default-export defineInstrumentation(...) or disableInstrumentation(). See OpenTelemetry for the supported OTel declarations.

agent/instrumentation/
  audit.ts  lifecycle instrumentation

This instrumentation records tool timing and identity without receiving tool arguments or results:

agent/instrumentation/audit.ts
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  events: {
    "tool.call.started": (event, ctx) => {
      ctx.state.set({ name: event.toolName, startedAt: Date.now() });
    },
    "tool.call.completed": (event, ctx) => {
      const started = ctx.state.get() as { name: string; startedAt: number } | undefined;
      if (started === undefined) return;

      console.log({
        tool: started.name,
        durationMs: Date.now() - started.startedAt,
        outcome: event.outcome,
      });
    },
  },
});

ctx.state is JSON storage scoped to this file and operation. It survives durable suspension and is released after the terminal event.

Use setup for asynchronous initialization. In eve dev and built servers, eve awaits every provider's setup before loading the server entry. A setup failure stops startup. Modules imported by an instrumentation file load before setup, so keep dependencies that need initialization out of those imports.

Control inputs and outputs

Each lifecycle file created with defineInstrumentation(...) has an independent tracePolicy. It decides whether that file receives a trace and whether its events include input or output content. OpenTelemetry destinations share the process-wide policy declared by otel(...).

agent/instrumentation/audit.ts
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  tracePolicy: ({ audience, environment }) => ({
    emit: true,
    recordInputs: audience === "public" || environment === "development",
    recordOutputs: audience === "public" || environment === "development",
  }),
  events: {
    "model.call.started": (event) => {
      console.log("input", event.input);
    },
    "model.call.completed": (event) => {
      console.log("output", event.content);
    },
  },
});

The function receives agentName, channel, audience, environment, and principalType. audience is "public", "private", or "unknown"; environment is "development", "preview", or "production".

Without a policy, or when it returns true, eve sends metadata for every trace. It includes input and output content in these cases:

EnvironmentAudienceContent
DevelopmentAnyInputs and outputs
Preview or productionpublicInputs and outputs
Preview or productionprivate or unknownMetadata only

Return { emit: false } to not sample the trace for this file. Return { emit: true, recordInputs, recordOutputs } to choose the two content directions explicitly. Inputs include model prompts, tool arguments, channel input, and user responses. Outputs include model responses, tool results, requests for user input, model provider metadata, and error details. Omitted content fields are unavailable to the handler.

The channel assigns the audience once when it creates the session. It controls content capture, not access. See Audience for how the channel derives public, private, and unknown.

Redact fields

Lifecycle events are immutable snapshots. Copy the fields you need into a destination-specific payload and redact that copy before sending it:

agent/instrumentation/audit.ts
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  tracePolicy: () => ({
    emit: true,
    recordInputs: true,
    recordOutputs: false,
  }),
  events: {
    "tool.call.started": async (event) => {
      await sendAuditRecord({
        id: event.idempotencyKey,
        input: redactApiKey(event.input),
        kind: event.kind,
        name: event.toolName,
      });
    },
  },
});

function redactApiKey(value: unknown): unknown {
  if (typeof value !== "object" || value === null || Array.isArray(value)) return value;

  const record = value as Record<string, unknown>;
  return "apiKey" in record ? { ...record, apiKey: "[redacted]" } : record;
}

async function sendAuditRecord(record: unknown): Promise<void> {
  // Send the sanitized record to your destination.
  void record;
}

Prefer recordInputs: false or recordOutputs: false when the destination does not need an entire content direction. Use field-level redaction only when the destination needs part of that content.

Lifecycle events

defineInstrumentation({ events }) supports the following event names. A start and its terminal event share an idempotencyKey; use it as a stable operation key when upserting records across replays.

LifecycleEventsMeaning
Sessionsession.started, session.waiting, session.completed, session.failedTrack a session from creation through suspension or termination. session.waiting is resumable; session.completed and session.failed are terminal.
Turnturn.started, turn.completed, turn.cancelled, turn.failedTrack one turn. Cancellation is not a failure and commonly leaves the session waiting for another turn.
Channel deliverychannel.delivery.started, channel.delivery.completed, channel.delivery.cancelled, channel.delivery.failedTrack an inbound channel delivery while eve processes its turn. These events are emitted when the channel provides delivery metadata.
Step attemptstep.attempt.started, step.attempt.completed, step.attempt.failed, step.attempt.metadataTrack each model-step attempt; retries have separate attempts. step.attempt.metadata is emitted after a step only when the AI SDK supplies provider metadata.
Model callmodel.call.started, model.call.completed, model.call.failedTrack an individual call to the model provider. Completion includes usage and finish reason; content fields follow the trace policy.
Tool calltool.call.started, tool.call.completed, tool.call.failedTrack a tool, skill load, local subagent, or remote-agent call through durable settlement. A failed event's outcome distinguishes failure, rejection, cancellation, and abandonment.
User inputinput.requested, input.resolvedTrack a question, tool approval, or session-limit request and its resolution. The resolution includes an outcome such as answered, approved, denied, or cancelled.
Memory operationmemory.operation.started, memory.operation.completed, memory.operation.failedTrack a memory-store search or write. Events include the operation name, phase, and memory slot; completion may include a record count and records.

Each call emits one tool.call.* lifecycle to authored providers. SDK execution callbacks do not emit a second lifecycle. Tool calls can remain open across approvals, parked turns, and worker replacement.

For local and remote subagents, tool.call.completed and tool.call.failed include the child turn's usage directly. Token counts are always present when the child reports usage; usage.costUsd is present only when the model provider reports a price.

When upgrading, replace action.started, action.completed, and action.failed handlers with their tool.call.* equivalents. Read event.toolName instead of event.name on start events. The OpenTelemetry trace schema remains version 4.

Handlers from different files run concurrently and are failure-isolated. Do not depend on execution order. Use flush to drain buffered records and shutdown to release resources.