--- title: Assertions description: Scoped methods, value assertions, the matcher mini-language, and gate vs soft severity. url: "https://eve.dev/docs/evals/assertions" docs_index: /llms.txt --- > For an index of all documentation, see [/llms.txt](/llms.txt). An assertion grades what an eval's `test(t)` function produced. Each assertion records a result instead of throwing, so one run reports every failing assertion, not only the first. Check what the agent did with scoped methods, grade a value with `t.check`, or record your own measurement with `t.score`. For grading by a model, see [Judge](./judge). ## Scoped assertions Scoped assertions take no value and are gates by default. Call them on `t` to check the whole run after `test` finishes. Call them on a session to check that session so far, or on a turn to check one reply. A run is **parked** when it stopped to wait on an approval or question that nobody answered (human-in-the-loop, or HITL, input). | Assertion | Asserts | | ----------------------------------------------------- | ----------------------------------------------------------------------------- | | `t.succeeded()` | The run did not fail and did not park on unanswered HITL input | | `t.awaitingInput()` | The run cleanly parked on HITL input | | `t.messageIncludes(token)` | Joined assistant text contains `token` (string or RegExp) | | `turn.outputEquals(value)` / `.outputMatches(schema)` | Deep equality or Standard Schema validation of turn/session structured output | | `t.calledTool(name, opts?)` | A matching tool call completed (`input`, `output`, `status`, `count`) | | `t.calledSkill(skill, opts?)` | A load of `skill` completed (`output`, `status`, `count`) | | `t.notCalledTool(name)` | No call to `name`, whatever its status | | `t.toolOrder([...names])` | Tool requests appear in order | | `t.usedNoTools()` | No tool calls at all; skill loads don't count | | `t.maxToolCalls(n)` | At most `n` tool calls; skill loads don't count | | `t.noFailedCalls()` | No tool, subagent, or skill action reported a failure | | `t.calledAgent(name, opts?)` | A subagent delegation happened (identity, remote, output, status, count) | | `t.event(type, opts?)` / `t.notEvent(type, opts?)` | Typed event presence, data, and count matching | | `t.eventOrder([...matchers])` | Matching event groups occur in order | | `t.eventsSatisfy(label, predicate)` | Escape hatch: any predicate over the typed event stream | `succeeded()` passes when the session ended or is waiting for the next message. It fails when the run failed, an event reports an error, or the run is parked. Structured output assertions exist only on turns and sessions, where one output is unambiguous (see the [output schema guide](../guides/client/output-schema)). ```ts await t.send("What is the weather in Brooklyn?"); t.succeeded(); t.calledTool("get_weather"); ``` The same methods work on a single turn or on an attached session: ```ts const first = await t.send("Call get_weather for Brooklyn"); first.calledTool("get_weather", { count: 1 }); const attached = t.target.attach(sessionId); await attached.nextTurn(); attached.succeeded(); attached.messageIncludes("Sunny"); ``` `t.calledTool` and `t.usedNoTools` are mutually exclusive; assert one or the other in a run. A call through [`eve__tool`](../concepts/built-in-tools#eve__search-eve__tool-and-eve__skill) is reported under the name and input of the tool it reaches, such as `linear__list_issues`. Assert on that tool. `eve__search` calls count toward `t.usedNoTools()` and `t.maxToolCalls(n)`; skill loads don't. The tool-name assertions and `turn.requireToolCall` throw a `TypeError` for `eve__tool`, `eve__skill`, and built-in names such as `load_skill`. The error says what to assert instead. eve accepts a name like `load_skill` when the root agent declares a tool by that name or has a dynamic tool resolver. ## Value assertions with `t.check` `t.check(value, assertion)` grades any value, such as a turn's `.reply` or parsed JSON, with a builder from `eve/evals/expect`: ```ts import { includes, equals, matches, satisfies, similarity } from "eve/evals/expect"; t.check(turn.reply, includes(/sunny/i)); // substring or RegExp (gate) t.check(parsed, equals({ city: "Brooklyn" })); // deep structural equality (gate) t.check(parsed, matches(WeatherSchema)); // Standard Schema, e.g. Zod (gate) t.check(turn.reply, similarity("Sunny, 72F")); // fuzzy 0–1 Levenshtein (soft) t.check( latencyMs, satisfies((value) => value < 1_000, "latency under one second"), ); ``` | Builder | Scores | Default | | ---------------------- | ------------------------------------------------------- | ------- | | `includes(value)` | coerced string contains a substring or matches a RegExp | gate | | `equals(value)` | deep structural equality | gate | | `matches(schema)` | validates against a Standard Schema | gate | | `similarity(expected)` | normalized Levenshtein similarity, 1 = identical | soft | | `satisfies(fn, label)` | custom boolean predicate | gate | For example, a failed labeled equality check prints: ```text ✗ equals [status] (0% < 100%): expected {"status":"active"}; received {"status":"disabled"} ``` Pick the cheapest builder that captures "correct". Use `similarity` when an exact match is too strict, and the [judge](./judge) for nuanced grading. ## Scores with `t.score` `t.score(evaluation)` records a score from 0 to 1. Pass a number, `{ score, message?, metadata? }`, or a promise of either. Chain `.label(key)` to name it; reporters and artifacts show the label as `key`. Unlabeled scores all report as `score`. ```ts const reply = turn.reply ?? ""; t.score({ score: Math.max(0, 1 - reply.length / 400), metadata: { length: reply.length }, }).label("reply-brevity"); t.score(gradeFaithfulness(reply, reference)).label("faithfulness"); // async ``` A score without a rule doesn't affect the eval's outcome. Chain a rule when it should: ```ts t.score(gradeFaithfulness(reply, reference)).label("faithfulness").gate(0.9); // fails below 0.9 t.score(brevity).label("reply-brevity").atLeast(0.5); // marks the eval `scored`, fatal under --strict ``` Reports show the measured score: a `0.82` gated at `0.9` reports `score: 0.82`, `threshold: 0.9`, `passed: false`. An evaluator that throws or returns a score outside 0 to 1 fails the eval. The result is marked errored, with the error message and no score. [Reporters](./reporters#custom-reporters) receive the entries in `onEvalComplete` and can read them by `key`: ```ts onEvalComplete(result) { for (const { key, score, threshold, passed } of result.assertions) { if (key === undefined) continue; store.write({ evalId: result.id, key, score, threshold, passed }); } } ``` The same entries appear in the [run artifacts](./running#artifacts) under `.eve/evals/`, in `summary.json` and in one `results.jsonl` line per eval: ```json { "name": "score [reply-brevity]", "key": "reply-brevity", "score": 0.84, "severity": "soft", "errored": false } ``` `threshold` and `passed` are present only when a rule was chained. ## The matcher mini-language `t.calledTool` and `t.calledAgent` take matcher objects: | Matcher | Fields | `status` values | | ------------- | ------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | | `calledTool` | `input`, `output`, `status`, `count` | `"completed"` (default), `"pending"`, `"failed"`, `"rejected"` | | `calledAgent` | `callId`, `childSessionId`, `output`, `status`, `count` | `"completed"` (default), `"pending"`, `"failed"`, `"rejected"`, `"interrupted"`, `"abandoned"` | A numeric `count` requires that exact number of calls matching every supplied constraint. For ranges, pass a predicate that receives the number of matching calls. `calledAgent` matches each call to a subagent. A call is `pending` until it finishes, then takes its final status and, when completed, its `output`. A completed subagent keeps its session for later calls with the same `taskId`. `childSessionId` is absent when the call failed before the subagent's session opened. A subagent's approval or question doesn't change the call's status; assert it with `requireInteraction`. Matcher values accept a literal (objects partial-deep-match), a RegExp, or a predicate function that returns a boolean: ```ts t.calledTool("bash", { input: { command: /^pwd/ }, count: 1 }); t.calledTool("echo", { count: (count) => count >= 2 }); t.calledTool("echo", { output: (value) => String(value).includes(marker) }); parked.calledTool("guarded", { status: "pending", count: 1 }); t.calledTool("guarded", { output: /approved/, count: 1 }); t.calledAgent("weather", { output: /72F/, }); ``` `requireInteraction` finds an approval or question. It uses the same matchers for `input`, `prompt`, and `display`. Its `optionIds` matcher receives the option ids in order, and a literal array must match the whole list exactly: ```ts const request = session.requireInteraction({ toolName: "confirm_deploy", optionIds: ["approve", "cancel"], }); ``` ## Run state and derived facts Assertions read what happened from `session.events`: tool calls with their name, input, output, and status, subagent calls, and approvals and questions. The event assertions match by presence, absence, a numeric or predicate count, partial event data, and order: ```ts turn.notEvent("content.completed", { data: { kind: "result" } }); turn.eventOrder([ { type: "task.started", data: { name: "researcher" }, count: 2 }, { type: "task.ended", data: { outcome: "completed" }, count: 2 }, ]); ``` To check a rule that spans several events, use `eventsSatisfy`: ```ts t.eventsSatisfy("assistant reply includes the marker", (events) => events.some( (e) => e.type === "content.completed" && e.data.phase === "reply" && typeof e.data.value === "string" && e.data.value.includes(marker), ), ); ``` ### Assert and score reasoning When the model provider exposes reasoning, eve records each completed block as a `reasoning.completed` event. Assert on its text with `turn.event(...)`, or join the blocks and pass them to a judge: ```ts turn.event("reasoning.completed", { data: { reasoning: /checks? the units/i }, }); const reasoning = turn.events .filter((event) => event.type === "reasoning.completed") .map((event) => event.data.reasoning) .join("\n\n"); t.judge("The reasoning checks its assumptions before concluding.", { on: reasoning, }).atLeast(0.8); ``` A provider that doesn't expose reasoning emits no `reasoning.completed` event, so the `turn.event` gate fails and the judge grades an empty string. The default judge input and `session.transcript` exclude reasoning, so pass it with `on`. ## Required preconditions Recorded assertions aren't awaitable. When later code depends on a value, use `await t.require(value, assertion)`. It records a gate and returns the value when it passes. When it fails, it stops the test without reporting a second, duplicate error: ```ts await t.require( sessionIds, satisfies((ids) => ids.length > 0, "dispatch started a session"), ); await t.target.attach(sessionIds[0]!).nextTurn(); ``` Use the matching `require*` lookups when later code needs a tool call or a pending request: ```ts const call = turn.requireToolCall("search"); const request = session.requireInteraction({ toolName: "guarded" }); ``` ## Severity Every assertion returns a handle. Chain one of these to set its severity: - `.gate(threshold?)` is hard. A miss marks the eval `failed` and `eve eval` exits non-zero. - `.soft(threshold?)` is recorded in reports. A miss below the threshold marks the eval `scored`, which fails the run only under `--strict`. With no threshold, it never fails. - `.atLeast(threshold)` is soft with a bar (equivalent to `.soft(threshold)`). - `.label(key)` adds a stable name when an eval records several assertions of the same kind. Braintrust and Datadog keep one score per name. `eve eval` warns when soft assertions in one eval share a name but have different scores. Scoped methods and `includes`, `equals`, `matches`, and `satisfies` are gates; `similarity`, `t.judge(...)`, and `t.score(...)` are soft. Chain a severity only to change the default: ```ts t.calledTool("get_weather").soft(); // record the tool call as a metric, don't gate t.check(turn.reply, similarity("Sunny")).label("forecast wording").atLeast(0.8); t.check(turn.reply, includes("error")).soft(); // track without failing the build ``` ## What to read next - [Judge](./judge): LLM-graded assertions with thresholds - [Cases](./cases): where assertions attach - [Running evals](./running): how verdicts map to exit codes --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)