---
title: Judge
description: Grade evals with decision models using criteria, typed questions, or batches, and set thresholds on each assertion.
url: "https://eve.dev/docs/evals/judge"
docs_index: /llms.txt
---

> For an index of all documentation, see [/llms.txt](/llms.txt).

`t.judge(...)` asks a separate judge model to grade your agent's response. Use it when a fixed [assertion](./assertions) can't describe a good response. It calls [`decide` from `eve/ai`](../guides/decide#decide-inside-a-tool) and defaults to `typesafe-ai/jev`, TypeSafe AI's [Jev decision model](https://vercel.com/i/what-is-jev).

```ts
import { defineEval } from "eve/evals";

export default defineEval({
  async test(t) {
    await t.send("Explain quantum tunneling to a 10-year-old.");
    t.succeeded();
    t.judge("The response uses no math beyond arithmetic.").atLeast(0.8);
  },
});
```

The judge needs no configuration, only credentials for the judge model; see [Configuring the judge model](#configuring-the-judge-model).

## The graders

`t.judge` accepts a criteria string, one typed question, or a batch of named questions. A string becomes a boolean question asking whether the response meets those criteria. A single question returns an assertion handle; a batch returns one handle per question.

| Question                     | Assertion score                                                 |
| ---------------------------- | --------------------------------------------------------------- |
| Criteria string or `boolean` | The returned probability of true, from 0 to 1                   |
| `score`                      | The returned rubric position divided by the highest level index |
| `choice`                     | 1 when the selected option equals `expected`; otherwise 0       |

Boolean scores are model estimates, not calibrated confidence. A rubric needs at least two levels, ordered from worst to best; eve normalizes a fractional rubric position to the 0–1 threshold scale.

```ts
t.judge({
  type: "score",
  instructions: "Grade the response's clarity.",
  criteria: ["Unclear", "Mostly clear", "Clear and concise"],
})
  .label("clarity")
  .atLeast(0.75);
```

Choice questions require an `expected` option key. eve uses it for scoring and doesn't send it to the judge model. Instructions and rubric descriptions accept strings, JSON objects, or JSON arrays; descriptions can also be `null`.

## Choose the value to grade

By default, the judge model receives `{ input, output }`: the user message and reply of the most recently finished turn. Answering an approval or reading an attached session keeps that user message. For a message with a file, such as from `session.sendFile()`, `input` is its text.

Pass `on` to replace the output. JSON values stay structured:

```ts
const draft = await t.send("Draft the welcome email.");
t.judge("The response has a professional tone.", { on: draft.reply }).atLeast(0.6);
```

For a multi-turn eval, pass `session.transcript`:

```ts
const { session } = await t.send("My favorite word is marigold. Remember it.");
await session.send("What is my favorite word?");
t.judge("The assistant remembers the user's favorite word across turns.", {
  on: session.transcript,
}).atLeast(0.8);
```

The transcript contains user and assistant messages in turn order, excluding reasoning, tool calls, and tool results. See [Multi-turn evals](./cases#multi-turn-evals).

`turn.reply` is the turn's last reply. A turn can reply more than once, for example to say work has started before waiting on tasks. `session.transcript` includes every reply. To grade all of one turn's replies, filter its messages with `transcriptOf` from `eve/client`:

```ts
import { transcriptOf } from "eve/client";

const turn = await t.send("Check the deploy and tell me when it's done.");
const replies = transcriptOf(turn.events)
  .messages.filter((message) => message.role === "assistant")
  .map((message) => message.text);

t.judge("The agent reported the deploy's result.", { on: replies.join("\n\n") }).atLeast(0.8);
```

## Decide several questions together

A batch asks several questions in one request about the same value. Pass that value as `state` instead of `on`; it replaces the default `{ input, output }` entirely.

```ts
const judgments = t.judge({
  state: { response: turn.reply ?? "", reference },
  questions: {
    accurate: {
      type: "boolean",
      instructions: "Is the response consistent with the reference?",
    },
    outcome: {
      type: "choice",
      instructions: "Classify the response.",
      criteria: {
        answered: "Answers the request",
        declined: "Declines the request",
      },
      expected: "answered",
    },
  },
});
judgments.accurate.atLeast(0.9);
judgments.outcome.gate();
```

Each key names a question and its assertion handle. Every question sees the same `state`, so use separate calls for unrelated values. A batch needs at least one question, and a provider or validation failure fails every assertion in it.

## Soft scoring and thresholds

Judges are soft by default:

```ts
t.judge("The response cites a source."); // tracked, never fails
t.judge("The response cites a source.").label("citation").atLeast(0.8); // below 0.8 marks `scored`, fatal under --strict
t.judge("The response cites a source.").gate(0.9); // below 0.9 fails; omit the threshold to require 1
```

A good response's boolean probability rarely reaches exactly 1, so pass an explicit gate threshold for boolean judgments. Choice scores are binary.

A judge call copies its input and starts right away. Handles aren't awaitable; eve waits for pending judgments after `test` returns. Each call outside a batch makes its own request. An eval timeout cancels pending judgments, even if the provider ignores cancellation.

## Configuring the judge model

Model settings resolve in this order:

1. Per-call: `t.judge("…", { model, modelOptions })`.
2. Per-eval: `defineEval({ judge: { model, modelOptions }, test })`.
3. Project default: `defineEvalConfig({ judge: { model, modelOptions } })`.
4. The default model used by `decide`, currently `typesafe-ai/jev`.

A per-eval `judge` replaces the project's judge configuration, and per-call fields override both. Evals that never call `t.judge` make no judge requests.

```ts title="evals/evals.config.ts"
import { defineEvalConfig } from "eve/evals";

export default defineEvalConfig({
  judge: { model: "typesafe-ai/jev" },
});
```

String IDs resolve through the AI SDK default provider's decision API, or Vercel AI Gateway when none is configured. Gateway uses `AI_GATEWAY_API_KEY`, Vercel OIDC credentials, or eve's local Gateway connection. Gateway accepts native decision models such as `typesafe-ai/jev`. A language model ID such as `openai/gpt-6-luna` doesn't select the OpenAI adapter. For a language model, pass a provider decision model, which uses that provider's credentials:

```ts
import { openai } from "@ai-sdk/openai";

t.judge("The response addresses the request.", {
  model: openai.decisionModel("gpt-6-luna"),
  modelOptions: { providerOptions: { openai: { reasoningEffort: "high" } } },
});
```

Use the provider's `decisionModel` factory, not a language model instance. Provider support and the decision contract are experimental; see [AI SDK decision](https://ai-sdk.dev/docs/ai-sdk-core/decisions).

Missing credentials, invalid answers, unsupported question types, and provider errors fail as gates, even for soft judgments. eve never skips the eval or falls back to another model.

## Diagnostics

Reports name judge assertions `judge.boolean`, `judge.score`, and `judge.choice`, plus the question key in a batch. Failure details show the graded value, the question, and the raw answer. Assertion metadata holds the normalized score, model id, token usage, warnings, and any provider and response metadata. A batch's usage is marked as shared, not charged per question. Decision models don't promise a written rationale.

Set thresholds against your own examples. For factuality, summaries, or SQL equivalence, write an explicit question and include the reference in shared state. [Braintrust reporting](./reporters) accepts judge scores.

## What to read next

- [Assertions](./assertions): deterministic run-level and value assertions
- [Reporters](./reporters): publish scores and inspect results
- [Targets](./targets): local and remote eval targets

---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)