---
title: Reporters
description: Ship eval results to Braintrust, Datadog, or JUnit XML. eve runs and scores everything itself.
---

# Reporters



eve runs and grades everything itself; reporters ship the results out. The CLI prints a console summary by default (one line per eval, with failed assertions and their messages), and reporters from `eve/evals/reporters` add destinations on top.

You are responsible for ensuring any observability or eval provider is approved for the data exported to it.

Reporters attach in two places. Declare them in `evals.config.ts` to observe **every** eval in the run, the usual choice for a shared destination like one Braintrust experiment, so you don't repeat the reporter in each file. Or list them on an individual eval's `reporters` to scope a destination to that eval (or to a group of evals that share one instance).

## Braintrust

`Braintrust(...)` uploads eval results to Braintrust experiments. Put one instance in the config so it covers the whole run:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "openai/gpt-5.4-mini" },
  reporters: [Braintrust({ projectName: "weather-agent" })],
});
```

Need a destination for only some evals? Attach it per eval instead:

```ts title="evals/brooklyn-forecast.eval.ts"
import { defineEval } from "eve/evals";
import { Braintrust } from "eve/evals/reporters";

export default defineEval({
  reporters: [Braintrust({ projectName: "weather-agent" })],
  async test(t) {
    await t.send("What is the weather in Brooklyn?");
    t.succeeded();
  },
});
```

The reporter config takes an optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against. Gate assertions log as binary scores under a `gate:` prefix so experiments diff gate regressions the same way they diff soft-score regressions. Repeated names use `#2`, `#3`, and later suffixes; use `.label(name)` to give them meaningful names. Failed assertion details are stored in the Braintrust metadata under `eveFailedAssertions`; observed traces are stored under `eveTraceIds` and `eveTraceContexts`.

A reporter instance observes the evals that reference it. Share one instance across several evals (the config, a `shared.ts` export, or every entry of a dataset array) and their results land in a single experiment. Listing the same config reporter on an eval too does not double-report it.

Braintrust needs its SDK installed in the app and credentials in the environment: install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run the eval without shipping results, which also suppresses config reporters and is useful locally when iterating.

## Datadog

`Datadog(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config so it covers the whole run:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { Datadog } from "eve/evals/reporters";

export default defineEvalConfig({
  reporters: [Datadog({ projectName: "weather-agent" })],
});
```

The reporter creates one Datadog Experiment, submits one synthetic experiment span per completed eve eval, associates that eval's assertion scores as experiment metrics, and prints the Datadog Experiment URL after the run completes. Without input recording, the external Experiment uses a placeholder dataset and spans are submitted as evals finish. With `recordInputs: true`, the reporter instead creates and pushes a versioned Datadog dataset after the evals finish, adds one record per eval, starts the Experiment against that dataset version, passes each dataset record id to its corresponding experiment span, and prints both Dataset and Experiment URLs. Buffering is necessary because an imperative eval's first `t.send(...)` input is only known after that eval executes. `dd-trace` generates dataset record, experiment trace, and experiment span identifiers; the reporter does not mutate or link the agent runtime's OpenTelemetry spans.

Assertion metrics use descriptive assertion names by default: characters outside letters, numbers, underscores, and hyphens are normalized to underscores, gate labels receive a `gate_` prefix, and repeated or reserved labels receive numeric suffixes. For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Authors can use `.label("stable name")` on an assertion handle to choose a stable metric name. Because assertion names can contain authored expectations, review them before exporting sensitive evals. Pass `recordAssertionDetails: true` only when the destination is also approved for raw assertion-name tags and failed assertion messages in row metadata.

Datadog needs `dd-trace` installed in the app and Datadog credentials in the environment. The reporter is tested against `dd-trace@6.13.0`; install it with `npm install dd-trace@6.13.0` and set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate. By default, the reporter records assertion scores, eval metadata, and the target URL origin only. Pass `recordInputs: true`, `recordOutputs: true`, `recordExpectedOutputs: true`, or `recordErrors: true` if your destination is approved for eval prompts, outputs, authored expectations, or execution error messages. `recordInputs` stores the eval input in both the linked dataset record and experiment span. Input is read from the eval's first `t.send(...)` message, falling back to the eval description when no message event is available. When `recordExpectedOutputs` is enabled, expected output is read from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output` and stored in both the dataset record and experiment span; when no expected output is authored, the input-only dataset record is still valid. Those expected-output keys are excluded from the general metadata payload. Target URL credentials, paths, query parameters, and fragments are never reported.

## JUnit

`JUnit({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same thing without touching the eval file, usually the better fit because CI owns the output path, not the eval:

```bash
eve eval --strict --junit .eve/junit.xml
```

Each eval becomes one `<testcase>` named by its path-derived id; failed gates and execution errors become failures, while `t.skip(reason)` produces a JUnit `<skipped>` result.

## Custom reporters

A reporter implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same structured results the built-ins do. Every callback may return a promise for async work like a remote upload:

```ts
interface EvalReporter {
  onRunStart(evaluations: readonly EveEval[], target: EveEvalTarget): void | Promise<void>;
  onEvalStart?(event: EveEvalStartEvent): void | Promise<void>;
  onSessionStart?(event: EveEvalSessionStartEvent): void | Promise<void>;
  onEvalComplete(result: EveEvalResult, context?: EveEvalCompleteContext): void | Promise<void>;
  onRunComplete(summary: EveEvalRunSummary): void | Promise<void>;
}
```

`onRunStart` fires once before any eval runs, and `onRunComplete` fires once with the aggregated summary. Within that run:

* `onEvalStart` fires when an eval is scheduled. It includes the eval definition, target, and start time.
* `onSessionStart` fires once for each session after eve receives its first trace context. It includes `sessionId`, `primary`, and `traceContext` with `traceId`, `spanId`, and `traceFlags`.
* `onEvalComplete` fires with the checks, scores, and verdict. The runner also supplies `context`, including every distinct trace context collected across the eval's sessions. The same list is available as `result.result.traceContexts`.

An eval can create several sessions, and a long session can produce several traces, so completion exposes a list instead of a single trace id. `onEvalStart` does not include a trace because the agent session has not started yet. If the target has no tracing configured, `onSessionStart` does not fire and the completed trace list is empty.

Reporter callbacks for an eval stay ordered—eval start, traced session starts, then completion—even when several evals run concurrently. Reach for a custom reporter only when a destination isn't covered. The per-run artifacts under `.eve/evals/` retain the trace contexts alongside the result for ad-hoc inspection.

## What to read next

* [Running evals](./running): console output, `--json`, and artifacts
* [Judge](./judge): what the reported numbers mean


---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)