---
title: Instrumentation
description: Configure lifecycle instrumentation, handle runtime events, and control the content each destination receives.
url: "https://eve.dev/docs/observability/instrumentation"
docs_index: /llms.txt
---

> For an index of all documentation, see [/llms.txt](/llms.txt).

An instrumentation file is a file under `agent/instrumentation/` that receives eve runtime events, such as a tool call starting or a turn ending. To export traces instead, use the [built-in OpenTelemetry APIs](./otel).

## Add instrumentation

eve names each file by its filename. A lifecycle-event file must default-export `defineInstrumentation(...)` or `disableInstrumentation()`. See [OpenTelemetry](./otel) for the OTel files.

```text
agent/instrumentation/
  audit.ts  lifecycle instrumentation
```

This file measures how long each call takes from request to finish, including any approval wait. It doesn't receive arguments or results:

```ts title="agent/instrumentation/audit.ts"
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  events: {
    "call.requested": (event, ctx) => {
      ctx.state.set({ name: event.toolName, startedAt: Date.now() });
    },
    "call.settled": (event, ctx) => {
      const started = ctx.state.get() as { name: string; startedAt: number } | undefined;
      if (started === undefined) return;

      console.log({
        tool: started.name,
        durationMs: Date.now() - started.startedAt,
        outcome: event.outcome,
      });
    },
  },
});
```

`ctx.state` stores JSON for one operation in this file, such as one tool call. It is kept while the session pauses, such as during an approval wait. eve releases it after the operation's last event.

Handlers from different files run at the same time, in no fixed order. A failure in one file doesn't affect the others.

| Field         | Purpose                                                                                                                    |
| ------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `events`      | Handlers keyed by event name. See [Lifecycle events](#lifecycle-events).                                                   |
| `tracePolicy` | Whether this file receives a trace, and with which content. See [Control inputs and outputs](#control-inputs-and-outputs). |
| `setup`       | Asynchronous initialization. Runs once at startup, before any event arrives.                                               |
| `flush`       | Sends buffered records. eve calls it before a session goes idle.                                                           |
| `shutdown`    | Releases resources when the process stops.                                                                                 |

A `setup` failure stops startup. Modules that an instrumentation file imports load before `setup`, so keep dependencies that need initialization out of those imports.

## Lifecycle events

Events that also appear in the session stream use the same names and outcomes there. Each operation emits one final event; read its `outcome` to tell completion from failure. A start event and its final event share an `idempotencyKey`, so match on it when you upsert records across replays.

| Kind             | Events                                                 | Meaning                                                                                                                                                                                                       |
| ---------------- | ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Session          | `session.started`, `session.ended`                     | Session start and end. `session.ended.outcome` is `completed` or `failed`.                                                                                                                                    |
| Turn             | `turn.started`, `turn.settled`                         | Turn work. `outcome` is `completed`, `cancelled`, or `failed`. A cancelled turn is not a failure.                                                                                                             |
| Message          | `delivery.consumed`, `delivery.settled`                | An inbound message eve recorded. The final `outcome` is `handled`, `awaiting-input`, `applied`, `ignored`, `refused`, or `failed`. `delivery.consumed` carries channel metadata when available.               |
| Model run        | `model.started`, `model.retried`, `model.settled`      | Each attempt of a model run. `model.retried` ends a failed attempt before another try. `model.settled.outcome` is `completed`, `failed`, `interrupted`, `abandoned`, or `cancelled` (the turn was cancelled). |
| Provider call    | `model.call.started`, `model.call.settled`             | One provider call; a model attempt can make several. The final `outcome` is `"completed"` or `"failed"`. Completion carries usage, finish reason, and optional `providerMetadata`.                            |
| Call             | `call.requested`, `call.settled`                       | A tool, skill, or agent call, including direct `invokeTool` calls. `capability` names its kind and name. The final `outcome` is `completed`, `failed`, `rejected`, `interrupted`, or `abandoned`.             |
| User input       | `interaction.opened`, `interaction.settled`            | A question, approval, or budget prompt, keyed by `interactionId`. The final event carries `outcome`, `responder` (who answered), and, when recorded, `answer`.                                                |
| Memory operation | `memory.operation.started`, `memory.operation.settled` | A memory search or write, with operation name, phase, and slot. The final `outcome` is `"completed"` or `"failed"`.                                                                                           |

A message that pauses for a person ends with `awaiting-input`, and the work after the answer belongs to the answer's message. OTel links a turn to a message only when the turn handled exactly one message.

Each call emits one `call.*` pair, and SDK callbacks don't add a second one. A subagent's `call.settled` includes its reported usage.

The event's `scope` field holds the session and run IDs the event belongs to. `turnId` can be absent for compaction or direct calls. Treat `idempotencyKey` as opaque. `runId` and `attempt` identify a run; runtime-context resolvers receive them as `run: { id, attempt }`.

eve rebuilds a thrown tool error from its saved record, so check `error.name`, not `instanceof`.

Use `InstrumentationEvent<"model.call.settled">` from `eve/instrumentation` to type a standalone handler. Model input content is `unknown`, because it holds the provider's request messages, not eve conversation messages. Narrow it before you read it.

The OpenTelemetry trace schema is version 4.

## Control inputs and outputs

Each file created with `defineInstrumentation(...)` has its own `tracePolicy`. It decides whether that file receives a trace and whether its events include input or output content. OpenTelemetry destinations share the process-wide policy declared by `otel(...)`.

```ts title="agent/instrumentation/audit.ts"
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  tracePolicy: ({ audience, environment }) => ({
    emit: true,
    recordInputs: audience === "public" || environment === "development",
    recordOutputs: audience === "public" || environment === "development",
  }),
  events: {
    "model.call.started": (event) => {
      console.log("input", event.input);
    },
    "model.call.settled": (event) => {
      if (event.outcome === "completed") console.log("output", event.content);
    },
  },
});
```

The function receives `agentName`, `channel`, `audience`, `environment`, and `principalType`, the kind of caller. `audience` is `"public"`, `"private"`, or `"unknown"`; `environment` is `"development"`, `"preview"`, or `"production"`.

Return `{ emit: false }` to skip the trace for this file, or `{ emit: true, recordInputs, recordOutputs }` to choose each content direction. Without a policy, or when it returns `true`, eve sends metadata for every trace and includes content in these cases:

| Environment           | Audience               | Content            |
| --------------------- | ---------------------- | ------------------ |
| Development           | Any                    | Inputs and outputs |
| Preview or production | `public`               | Inputs and outputs |
| Preview or production | `private` or `unknown` | Metadata only      |

Inputs include model prompts, tool arguments, call labels (which can contain input-derived text), channel input, and user responses. Outputs include model responses, tool results, requests for user input, model provider metadata, and error details. Content that isn't recorded is absent from the event.

The channel sets the audience once, when it creates the session. The audience controls content capture, not access. See [Audience](/docs/channels#audience) for how the channel picks `public`, `private`, or `unknown`.

## Redact fields

Events are read-only, so copy the fields you need into a payload and redact the copy before sending it:

```ts title="agent/instrumentation/audit.ts"
import { defineInstrumentation } from "eve/instrumentation";

export default defineInstrumentation({
  tracePolicy: () => ({
    emit: true,
    recordInputs: true,
    recordOutputs: false,
  }),
  events: {
    "call.requested": async (event) => {
      await sendAuditRecord({
        id: event.idempotencyKey,
        input: redactApiKey(event.input),
        kind: event.capability.kind,
        name: event.toolName,
      });
    },
  },
});

function redactApiKey(value: unknown): unknown {
  if (typeof value !== "object" || value === null || Array.isArray(value)) return value;

  const record = value as Record<string, unknown>;
  return "apiKey" in record ? { ...record, apiKey: "[redacted]" } : record;
}

async function sendAuditRecord(record: unknown): Promise<void> {
  // Send the sanitized record to your destination.
  void record;
}
```

When the destination needs no content in one direction, set `recordInputs: false` or `recordOutputs: false` instead.

## What to read next

- [OpenTelemetry](/docs/observability/otel): configure process-wide settings and third-party destinations.
- [Agent Runs](/docs/observability/agent-runs): configure Vercel's built-in trace destination.
- [Local development](../guides/dev-tui): inspect local traces in the TUI
- [Hooks](../guides/hooks): react to runtime events outside the instrumentation lifecycle

---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)