---
title: Overview
description: Define repeatable scored checks for an eve agent with defineEval and run them with eve eval.
url: "https://eve.dev/docs/evals/overview"
docs_index: /llms.txt
---

> For an index of all documentation, see [/llms.txt](/llms.txt).

An eval is a scored check that sends messages to your agent and grades what it did. Each eval lives in a `.eval.ts` file under `evals/`, and `eve eval` runs them. Use evals to catch regressions when you change a prompt or a tool.

Evals talk to a real agent server over the same HTTP API as your users, through the [TypeScript client](../guides/client/overview). A passing eval means the agent started, accepted a request, and produced the asserted result. To control what tools return, define [tool stubs](./tool-stubs) in the eval.

## `defineEval`

The `evals/` directory sits at the app root. Each `.eval.ts` file exports one eval, or an array of evals that runs once per [dataset](./cases#datasets-exporting-an-array) row. The file path is the eval's id, so `evals/weather/brooklyn-forecast.eval.ts` becomes `weather/brooklyn-forecast`. Don't add an `id` or `name`.

```text
my-agent/
├── agent/
├── evals/
│   ├── evals.config.ts
│   ├── smoke.eval.ts
│   └── weather/
│       ├── brooklyn-forecast.eval.ts
│       └── no-tools-for-greetings.eval.ts
└── package.json
```

An eval is one `async test(t)` function. It sends messages and records assertions through `t`:

```ts title="evals/weather/brooklyn-forecast.eval.ts"
import { defineEval } from "eve/evals";
import { includes } from "eve/evals/expect";

export default defineEval({
  description: "Basic message and tool-usage coverage for the weather agent.",
  async test(t) {
    const turn = await t.send("What is the weather in Brooklyn?");
    t.succeeded();
    t.calledTool("get_weather");
    t.check(turn.reply, includes("Sunny"));
  },
});
```

`test` is the only required field; `description`, `judge`, `tags`, `metadata`, `timeoutMs`, and `reporters` are optional. The init template adds `evals/**/*.ts` to `tsconfig.json`, so eval code type-checks with the app.

## `evals.config.ts`

Every `evals/` directory needs exactly one `evals.config.ts` at its root. It sets the defaults that all evals share:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { braintrustReporter } from "eve/evals/reporters";

export default defineEvalConfig({
  judge: { model: "typesafe-ai/jev" },
  reporters: [braintrustReporter({ projectName: "my-agent" })],
});
```

Every field is optional:

| Field               | Sets                                                                                                 |
| ------------------- | ---------------------------------------------------------------------------------------------------- |
| `judge`             | The model for [LLM-as-judge](./judge) assertions (`t.judge(...)`). Omit it to use `typesafe-ai/jev`. |
| `reporters`         | Reporters that observe every eval in the run, so one `braintrustReporter()` here covers all evals.   |
| `maxConcurrency`    | The concurrent eval limit.                                                                           |
| `timeoutMs`         | The per-eval timeout.                                                                                |
| `repeat`            | How many times each eval runs, to measure repeatability. See [Repeat evals](./running#repeat-evals). |
| `setup`, `teardown` | Run-wide hooks before and after the run. See [Run setup and teardown](#run-setup-and-teardown).      |

CLI flags (`--max-concurrency`, `--timeout`, `--repeat`) and per-eval values take precedence over config defaults.

### Run setup and teardown

Use `setup` to start shared resources or run migrations before eve starts the local agent server. Return shared values from `setup`, and release resources in `teardown`:

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";
import { startTestDatabase } from "./database.js";

export default defineEvalConfig({
  async setup() {
    const database = await startTestDatabase();
    return { database };
  },
  async teardown(context) {
    await context?.database.close();
  },
});
```

`context` is typed from the setup return value. To use that type in an eval file, pass the config's type to `defineEval`:

```ts title="evals/database.eval.ts"
import { defineEval } from "eve/evals";
import { equals } from "eve/evals/expect";
import type config from "./evals.config.js";

export default defineEval<typeof config>({
  async test(t) {
    const result = await t.context.database.query("SELECT 1 AS value");
    await t.require(result.rows[0]?.value, equals(1));
  },
});
```

`setup`, every eval, and `teardown` share one context object in the `eve eval` process. The agent server never sees it, so class instances and functions work as values. Evals that run at the same time share these resources, so isolate any database records they change.

`setup` runs once per `eve eval` command, before the agent server starts. `teardown` runs after the local server and its sandboxes stop, even when setup, startup, or an eval fails. A `teardown` failure fails the command. The per-eval `timeoutMs` doesn't apply to either hook.

`setup` can return nothing, and `teardown` works without `setup`. `teardown` receives `undefined` when `setup` returns nothing or throws, so `setup` must clean up anything only it can reach.

## Deterministic fixture models

`mockModel` gives a test agent fixed replies, so it runs without calling a model provider. A fixed reply is one line:

```ts title="agent/agent.ts"
import { defineAgent } from "eve";
import { mockModel } from "eve/evals";

export default defineAgent({
  model: mockModel("A deterministic reply"),
});
```

Pass a callback when the reply depends on the conversation. It receives `lastUserMessage`, `userMessages`, `userMessageCount`, the available `tools`, and prior `toolResults`:

```ts title="agent/agent.ts"
export default defineAgent({
  model: mockModel(
    ({ lastUserMessage, userMessageCount }) => `Turn ${userMessageCount}: ${lastUserMessage}`,
  ),
});
```

The callback can return `{ text, toolCalls, usage }` for deterministic tool loops or explicit token counts. Use the options form when a fixture also needs a custom model identity:

```ts title="agent/agent.ts"
model: mockModel({
  modelId: "weather-script",
  provider: "my-fixtures",
  respond: ({ toolResults }) =>
    toolResults.length === 0
      ? { toolCalls: [{ name: "get_weather", input: { city: "Brooklyn" } }] }
      : `Weather: ${JSON.stringify(toolResults[0]?.output)}`,
});
```

Omit the response to reply `"Mock response"`. `mockModel` supports streamed and non-streamed replies and estimates token usage. Its context window is 1,000,000 tokens, so eve never compacts the conversation. To test compaction, set `modelContextWindowTokens` on the agent.

The model is part of the agent definition, so use `mockModel` in a separate test agent. That agent stays mocked locally and when you deploy it as an eval target.

## The `t` context

`t` both sends messages and records assertions, so an eval is ordinary code that sends a message, then asserts on the reply.

- **Send:** `t.send(...)` starts a conversation with its first message and returns the turn. `t.session()` starts an empty conversation. Send follow-ups with `turn.session.send(...)`, and read the reply from `turn.reply`. The full list of session methods is in [the drive API](./cases#the-drive-api).
- **Assert:** use the three surfaces in the next section.

## Three assertion surfaces

| Surface                                       | Grades                                                                                                                         |
| --------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| Methods such as `t.calledTool("get_weather")` | What the agent did: on `t` for the whole run, or on one session or turn. See [Assertions](./assertions).                       |
| `t.check(value, assertion)` and `t.score(…)`  | Any value, such as `turn.reply` or parsed JSON, or a 0–1 score you compute. See [Assertions](./assertions#scores-with-tscore). |
| `t.judge("cites a source")`                   | Quality, graded by a separate judge model. See [Judge](./judge).                                                               |

`t.judge` grades the reply of the most recently finished turn. Pass `{ on: turn.session.transcript }` to grade a whole conversation.

## Gate vs soft

A failed **gate** marks the eval `failed`, and `eve eval` exits non-zero. A **soft** assertion appears in reports. A soft assertion below its threshold marks the eval `scored`, which fails the run only under `--strict`. A soft assertion with no threshold never fails.

`t.judge(...)`, `t.score(...)`, and `similarity` are soft by default; the rest are gates. Chain `.gate(threshold?)`, `.soft(threshold?)`, or `.atLeast(threshold)` to change one. See [Severity](./assertions#severity).

```ts
t.succeeded(); // gate
t.calledTool("get_weather").soft(); // record as a metric, don't gate
t.judge("cites a source"); // soft, tracked (no threshold)
t.judge("The response is consistent with output.reference.", {
  on: { response: turn.reply ?? "", reference },
}).atLeast(0.7); // soft, gated under --strict at 0.7
```

Use `await t.require(value, assertion)` for a gate that must pass before the eval continues. Call `t.skip(reason)` first to skip an eval on a target that doesn't support it.

## Run evals with eve eval

```bash
eve eval                       # run all discovered evals against a local dev server
eve eval weather               # run one eval, or every eval under evals/weather/
eve eval --url https://<app>   # target an existing server or deployment
```

Exit code `0` means every eval passed its gates. See [Running evals](./running) for the full flag list, exit codes, and CI guidance.

## A good baseline

Start with a few small smoke evals: `t.succeeded()` plus one or two content checks, with dataset files in `evals/data/`. Add a judge or Braintrust when you need fuzzy grading or a shared place to review results. In CI, run `eve eval --strict` so soft threshold misses fail the build.

## What to read next

- [Cases](./cases): single-turn evals, scripted multi-turn evals, and dataset fan-out
- [Assertions](./assertions): run-level methods and `t.check` value assertions, with matchers and severity
- [Judge](./judge): LLM-as-judge grading and the judge model
- [Targets](./targets): local vs remote targets for the same eval files
- [Reporters](./reporters): Braintrust experiments and JUnit XML
- [Running evals](./running): the `eve eval` CLI, exit codes, and artifacts
- [Tools](../tools): the surface most evals assert on

---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)