---
title: Reporters
description: Ship eval results to Braintrust, Datadog, or JUnit XML. eve runs and scores everything itself.
url: "https://eve.dev/docs/evals/reporters"
docs_index: /llms.txt
---

> For an index of all documentation, see [/llms.txt](/llms.txt).

A reporter sends the results eve graded to Braintrust, Datadog, a JUnit file, or your own destination. Import reporters from `eve/evals/reporters`. The CLI always prints a summary with one line per eval and each failed assertion's message.

You are responsible for ensuring any observability or eval provider is approved for the data exported to it.

Declare reporters in `evals.config.ts` to report every eval in the run, such as to one shared Braintrust experiment. List them in an eval's `reporters` to report only the evals that share that reporter instance.

## Console

The CLI adds `consoleReporter()` from `eve/evals/reporters` unless you pass
`--json`. There is no public runner API, so reporters compose only through
`evals.config.ts` and per-eval `reporters`. A custom console reporter mostly
makes sense with `--json`, for example logging to stderr so stdout stays valid
JSON:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { consoleReporter } from "eve/evals/reporters";

export default defineEvalConfig({
  reporters: [consoleReporter({ color: false, log: (line) => console.error(line) })],
});
```

Both options are optional; the default logger is `console.log` and color
follows whether stdout is a terminal. Without `--json`, another console reporter
duplicates the CLI's output.

## Braintrust

`braintrustReporter(...)` uploads eval results to Braintrust experiments. Put one instance in the config to cover the whole run:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { braintrustReporter } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "typesafe-ai/jev" },
  reporters: [braintrustReporter({ projectName: "weather-agent" })],
});
```

To report only some evals, attach the reporter per eval:

```ts title="evals/brooklyn-forecast.eval.ts"
import { defineEval } from "eve/evals";
import { braintrustReporter } from "eve/evals/reporters";

export default defineEval({
  reporters: [braintrustReporter({ projectName: "weather-agent" })],
  async test(t) {
    await t.send("What is the weather in Brooklyn?");
    t.succeeded();
  },
});
```

The reporter takes optional `projectName` and `experimentName`, plus a base experiment (by name or id) to diff against.

- Gate assertions log as scores of 0 or 1 under a `gate:` prefix, so experiment diffs show gate regressions like soft-score regressions.
- Assertions that share a name, such as several unlabeled judges, log the lowest score. The console warns when merged soft scores differ; use `.label(name)` to group or separate them.
- Errored assertions log no score.
- Metadata holds assertion scores under `eveAssertionScores`, failed assertion details under `eveFailedAssertions`, and the eval's traces under `eveTraceIds` and `eveTraceContexts`.
- In a [repeated run](./running#repeat-evals), each attempt logs its own row with the id `<id>#<attempt>` and the attempt number under `eveAttempt`.

Evals that share one reporter instance, through the config, a `shared.ts` export, or a dataset array, land in one experiment. Listing the config reporter on an eval too doesn't report it twice.

Install the `braintrust` package (`npm install braintrust`) and set `BRAINTRUST_API_KEY`. Pass `--skip-report` to run without shipping results, including from config reporters.

## Datadog

`datadogReporter(...)` uploads eval assertion scores to a Datadog LLM Observability Experiment. Put one instance in the config to cover the whole run:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { datadogReporter } from "eve/evals/reporters";

export default defineEvalConfig({
  reporters: [datadogReporter({ projectName: "weather-agent" })],
});
```

The reporter creates one Datadog Experiment per run and prints its URL afterward. Each completed eval becomes one experiment span, with its assertion scores as metrics.

With `recordInputs` (the default), the reporter syncs a versioned Datadog dataset after the evals finish and runs the Experiment against that version. Each span links to its dataset record, and the reporter prints the Dataset URL. With `recordInputs: false`, the Experiment uses a placeholder dataset, and spans are submitted as evals finish.

When an eve session's trace is sampled, the reporter adds `experimentRuntimeTraceLinks` metadata, so an Experiment row links to the agent trace that produced it.

Every run reuses the dataset named by `datasetName` (default `<projectName> evals`), so Datadog can compare experiments across runs. Records match by eval description, with `#2`, `#3` suffixes for duplicates, so give each case a unique description. In a [repeated run](./running#repeat-evals), every attempt links to the eval's one dataset record, and each span's `runIteration` is the attempt number.

Metric names come from assertion names:

- Characters other than letters, numbers, underscores, and hyphens become underscores.
- Gates get a `gate_` prefix.
- Assertions that share a name report the lowest score.
- Names that collide with built-in metrics get a numeric suffix.

For example, `succeeded` becomes `gate_succeeded` and `calledTool(get_stock_quote)` becomes `gate_calledTool_get_stock_quote`. Use `.label("stable name")` to choose a stable metric name. Assertion names can contain authored expectations, so review them before exporting sensitive evals. Row metadata keeps each assertion's name and score under `eveAssertionScores`; pass `recordAssertionDetails: false` to omit them and failed assertion messages.

Install `dd-trace` (tested against `dd-trace@6.13.0`: `npm install dd-trace@6.13.0`). Set `DD_API_KEY`, `DD_APP_KEY`, and `DD_SITE` as appropriate. Without credentials, reporting is skipped.

Like the Braintrust reporter, it records eval prompts, outputs, authored expectations, and execution error messages by default. Pass `recordInputs: false`, `recordOutputs: false`, `recordExpectedOutputs: false`, or `recordErrors: false` if your destination isn't approved for that content.

- Input is the eval's first `t.send(...)` message, or the eval description when there is none. It is stored in the dataset record and the experiment span.
- Expected output comes from eval `metadata.expectedOutput`, `metadata.expected`, or `metadata.expected_output`, and is stored in the dataset record and the experiment span. Those keys are excluded from the general metadata. A dataset record without expected output is still valid.
- Target URL credentials, paths, query parameters, and fragments are never reported.

## JUnit

`junitReporter({ filePath })` writes JUnit XML for CI annotations. The `--junit <path>` CLI flag does the same without touching eval files. The flag usually fits better because CI owns the output path:

```bash
eve eval --strict --junit .eve/junit.xml
```

Each eval becomes one `<testcase>` named by its id, or one per attempt named `<id>#<attempt>` in a repeated run. Failed gates and execution errors become failures, and `t.skip(reason)` becomes `<skipped>`.

## Custom reporters

Build a custom reporter only when no built-in reporter covers the destination. It implements the `EvalReporter` interface from `eve/evals/reporters` and receives the same results as the built-ins. Every callback can return a promise:

```ts
interface EvalReporter {
  onRunStart(evaluations: readonly EveEval[], target: EveEvalTarget): void | Promise<void>;
  onEvalStart?(event: EveEvalStartEvent): void | Promise<void>;
  onSessionStart?(event: EveEvalSessionStartEvent): void | Promise<void>;
  onEvalComplete(result: EveEvalResult, context?: EveEvalCompleteContext): void | Promise<void>;
  onRunComplete(summary: EveEvalRunSummary): void | Promise<void>;
}
```

| Callback         | Fires                                                                                                                                                                     |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `onRunStart`     | Once, before any eval runs.                                                                                                                                               |
| `onEvalStart`    | When an eval is scheduled, with the eval definition, target, and start time. No trace exists yet.                                                                         |
| `onSessionStart` | Once per session, when its first trace context arrives, with `sessionId`, `primary`, and `traceContext` (`traceId`, `spanId`, `traceFlags`). Never fires without tracing. |
| `onEvalComplete` | With the checks, scores, and verdict. `context` lists every distinct trace context across the eval's sessions, as does `result.result.traceContexts`.                     |
| `onRunComplete`  | Once, with the aggregated summary.                                                                                                                                        |

An eval's callbacks stay in that order even when evals run concurrently. In a [repeated run](./running#repeat-evals), `onEvalStart` fires once per attempt. Its event and each `onSessionStart` event carry the 1-based `attempt`, which matches `result.attempt`.

`onEvalComplete` gets a list of trace contexts because an eval can create several sessions, and a long session can produce several traces. The list is empty without tracing.

Each entry in `result.assertions` is an `AssertionResult`. `name` is the display label, `key` the identifier set by `.label(key)`, and `score` the measurement. `threshold` and `passed` describe the rule and are absent when no rule applies. When a scorer threw, `errored` is `true`, `message` carries the error, and `score` is absent. The `.eve/evals/` artifacts write the same fields, plus the trace contexts.

`result.result.derived` summarizes what happened in the eval's sessions, such as tool calls and `models`. `models` lists the distinct ids of the models the agent called, in session order. Each entry in `result.result.sessions` holds the same summary for one session, with `models` in first-use order. A reporter can use it to tell the primary session's models from other sessions'.

A subagent runs in its own session. Its models appear only when the eval attaches and reads that session with `t.target.attach(...).nextTurn()` or `watch()`. `models` includes the models that wrote compaction summaries, but not routing calls made inside a provider.

`result.result.reply` is the last turn's reply text, and `data` is its structured output, with no text fallback. `status` describes the request, while `sessionEnded` and the optional `view` describe that turn's session. The top-level `sessionId` identifies the primary session; inspect `sessions` for each session's events and view. To pick one output value, fall back to `reply` only when `data === undefined`, so structured `null` stays an output.

Braintrust and Datadog metadata use `eveStatus`, `eveAwaitingInput`, and `eveSessionEnded`. Use `eveSessionEnded` to tell a finished request from an ended session.

`derived.usage` is the token usage and cost a session recorded in its `usage.recorded` events. It includes subagents' spend and compaction summaries. On a turn, it is the session's total so far.

The eval's `usage` counts each session the eval read once, using its latest total. It skips a session that another session in the eval opened, because the opening session's total already includes it. It is absent when a counted session recorded no usage.

## What to read next

- [Running evals](./running): console output, `--json`, and artifacts
- [Judge](./judge): what the reported numbers mean

---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)