---
title: Tool Stubs
description: Define tool outcomes and outcome sequences in your evals.
url: "https://eve.dev/docs/evals/tool-stubs"
docs_index: /llms.txt
---

> For an index of all documentation, see [/llms.txt](/llms.txt).

A tool stub makes a tool return fixed JSON, or throw an error, for one eval session. Pass `stubs` to `t.session()`. Stubs work against local and deployed agents, with no mock code to deploy.

Stubs set what tools return; [assertions](./assertions) check what the agent does with it. The target must [permit tool replacement](#permit-tool-replacement) for the eval's caller.

## Author an eval

```ts title="evals/complete-task.eval.ts"
import { defineEval } from "eve/evals";

export default defineEval({
  async test(t) {
    const session = await t.session({
      stubs: [
        {
          id: "tasks",
          tool: "list_tasks",
          outcomes: [
            {
              response: {
                tasks: [
                  { id: "milk", title: "Buy milk" },
                  { id: "dog", title: "Walk dog" },
                  { id: "rent", title: "Pay rent" },
                ],
              },
            },
            {
              response: {
                tasks: [
                  { id: "dog", title: "Walk dog" },
                  { id: "rent", title: "Pay rent" },
                ],
              },
            },
          ],
        },
        {
          id: "complete-milk",
          tool: "complete_task",
          match: { task_id: { const: "milk" } },
          outcome: { response: { success: true } },
        },
      ],
    });

    const first = await session.send("What tasks do I have?");
    first.expectOk();
    first.calledTool("list_tasks", { count: 1 });
    first.messageIncludes("Buy milk");

    const second = await session.send("Complete Buy milk, then list my remaining tasks.");
    second.expectOk();
    second.calledTool("complete_task", { input: { task_id: "milk" }, count: 1 });
    second.toolOrder(["complete_task", "list_tasks"]);
    second.messageIncludes("Walk dog");
    second.messageIncludes("Pay rent");
  },
});
```

Use your agent's actual tool names and response shapes. The client accepts the same option on `client.sessions.create({ stubs })` or `client.sessions.create({ message, stubs })`.

Responses are a fixed sequence, not a simulated database: the second `list_tasks` call returns two tasks even if `complete_task` never ran. The assertions check only that the agent asked for the write. To test a real update, use the real tool.

## Permit tool replacement

eve rejects stubs by default, including in development. With [Vercel OIDC](../guides/auth-and-route-protection#verceloidc), use `eveVercelOidc` and grant the permission to the eval runner's verified subject:

```ts title="agent/channels/eve.ts"
import { vercelSubject } from "eve/channels/auth";
import { eveChannel, eveVercelOidc } from "eve/channels/eve";

export default eveChannel({
  auth: [
    eveVercelOidc({
      subjects: [
        vercelSubject({ teamSlug: "acme", projectName: "app" }),
        {
          subject: vercelSubject({ teamSlug: "acme", projectName: "eval-runner" }),
          allowToolStubs: true,
        },
      ],
    }),
  ],
});
```

Both projects can authenticate, but only the eval runner can replace tools. `vercelSubject` matches production only; pass `environment: "*"` to include all environments. Only the target's server configuration grants the permission, after token verification. A request can't grant it, and users don't inherit a project's grant.

Custom authenticators can return the same `allowToolStubs: true` after verifying the caller. See [authentication and route protection](../guides/auth-and-route-protection#tool-replacement-permission) for the grant rules and a local development example.

eve checks the permission only when it creates the session. Later requests use normal channel authentication, so anyone your channel admits to the session can use its stubs. Isolate sessions with your session-access policy, or use a separate eval deployment of the same agent code.

Checking the stubs after the eval also requires `allowToolStubs`.

## Stub fields

| Field      | Meaning                                                                                                         |
| ---------- | --------------------------------------------------------------------------------------------------------------- |
| `id`       | A rule name, unique in the session, that tracks the rule's place in its sequence. Traces don't show it.         |
| `tool`     | The existing tool's name, optionally prefixed with a local subagent path.                                       |
| `match`    | Optional constraints on input properties, written as JSON Schema. Omit it or use `{}` to match any valid input. |
| `outcome`  | One outcome returned on every matching call, such as `{ response: { tasks: [] } }`.                             |
| `outcomes` | A nonempty array of outcomes returned in order. Use exactly one of `outcome` or `outcomes`.                     |

Each outcome contains exactly one of `response`, the JSON the tool returns, or `throw`, an error [raised instead](#inject-tool-errors). The tool returns only the `response` value, not the outcome object.

Stubs are plain JSON and can be shared between evals:

```ts title="evals/fixtures/tasks.ts"
import type { ToolStub } from "eve/evals";

export const taskStubs = [
  { id: "tasks", tool: "list_tasks", outcome: { response: { tasks: [] } } },
] satisfies readonly ToolStub[];
```

Pass the fixture to `t.session({ stubs: taskStubs })`; each session keeps its own place in each sequence. Both `eve/evals` and `eve/client` export `ToolStub` and `ToolStubOutcome`. These types check the JSON shape but don't infer the response type from your tool definition.

Configuration is limited to 100 rules, 20,000 JSON nodes, nesting depth 32, and about one million characters. The `__proto__` property name is reserved.

## Match tool arguments

**eve checks rules from top to bottom. The first matching stub wins.** Put specific rules before an unconditional fallback:

```ts
const stubs = [
  {
    id: "milk",
    tool: "complete_task",
    match: { task_id: { const: "milk" } },
    outcome: { response: { success: true } },
  },
  { id: "other-tasks", tool: "complete_task", outcome: { response: { success: false } } },
];
```

A call for milk receives success; other valid calls receive the fallback. Reversed, the unconditional rule wins every time. If no rule matches, eve runs the real tool.

Every property listed in `match` must exist and satisfy its constraint. Extra top-level properties are allowed:

```json
{
  "id": "open-tasks",
  "tool": "search_tasks",
  "match": {
    "filter": {
      "type": "object",
      "properties": { "status": { "const": "open" } },
      "required": ["status"]
    },
    "query": { "type": "string", "pattern": "\\b(?i:milk)\\b" },
    "tags": { "type": "array", "contains": { "const": "urgent" } }
  },
  "outcome": { "response": { "tasks": ["Buy milk"] } }
}
```

Nested properties are optional unless listed in `required`. Matching doesn't coerce values, apply defaults, or change arguments. The real tool's input schema still validates the input first, and normal approvals and output processing still apply.

Constraints use a subset of JSON Schema 2020-12 with no references:

- Types, and scalar `const` and `enum`
- String patterns and bounds, and numeric bounds
- Array `items` and `contains`
- Object `properties`, `required`, and `additionalProperties`
- Logical and conditional schemas

`pattern` uses JavaScript regular expressions. eve validates schemas and patterns before it creates the session.

Unsupported constraints fail setup: `patternProperties`, references, formats, `multipleOf`, `minContains`, `maxContains`, `uniqueItems`, dependencies, and object/array values in `const` or `enum`. Nested `properties` and `required` can't use inherited property names such as `constructor` or `toString`. Use nested properties or array items to match structured values.

Unknown static tool names, invalid local subagent paths, and paths into remote agents fail session creation. eve checks dynamic tool names and connection operations where it can and allows the rest. After the eval, eve warns about rules that matched no calls but doesn't fail the eval. Use tool assertions when a call must happen.

A broken stub, or a failure while processing its response, fails the eval even if the agent recovers. An intentional `throw` outcome is a normal tool failure the eval can assert on. Neither falls back to the real tool. If eve can't record such a failure, it cancels the session's run. A failed or cancelled stubbed session always fails the eval.

## Sequences and session scope

`outcome` repeats on every matching call. `outcomes` moves to the next entry on each matching call, then repeats its last entry. For example, `outcomes: [{ response: "pending" }, { response: "completed" }]` returns `"pending"` once and `"completed"` after that. Only the selected rule moves forward; running out of entries never selects a later rule.

Two calls in one turn use two entries, and later turns continue the same sequence. Calls waiting for approval, or denied, don't use an entry. A replayed workflow step returns its recorded outcome without using another entry. Each session supports up to 10,000 tool calls.

Rules are fixed when the session is created. Local subagents use them through qualified tool paths, and sequence positions survive a handoff to a new deployment. A separate session starts each sequence over.

Parallel calls use responses in the order they arrive, which can vary. Use separate argument matchers when particular arguments must receive particular responses.

## Inject tool errors

Use `throw` to test how the agent handles a failed tool call. Mix errors and responses to test recovery:

```ts
const stubs = [
  {
    id: "tasks",
    tool: "list_tasks",
    outcomes: [
      { throw: { name: "TimeoutError", message: "Task service timed out" } },
      { response: { tasks: [] } },
    ],
  },
];
```

`throw.message` is required. Omit `throw.name` to use `"Error"`. eve throws a plain JavaScript `Error` with that name and message. A `TimeoutError` fails the call right away instead of waiting for a deadline.

Ordinary tools and connection operations show the model the message without the name; workflow task errors keep the name. Put details the model needs in `message`.

Use `outcome: { throw: { message: "Service unavailable" } }` to fail every matching call. In a sequence, a throwing entry is used up like a response.

An injected error follows the tool's normal error handling. Ordinary and blocking workflow calls are recorded as failed; background tasks report failure when they finish. The model can report the failure or retry. Assert the failed call explicitly:

```ts
const turn = await session.send(
  "List my tasks once. If the service fails, explain the failure and wait for me to retry.",
);
turn.expectOk();
turn.calledTool("list_tasks", { status: "failed", count: 1 });
```

`noFailedCalls()` fails on the injected error. Stub setup and response-processing failures still fail the eval.

`throw` works for ordinary tools, workflow tools, whole-subagent replacements, and qualified connection operations. An error in a persistent workflow tool ends that task, so retry with a new call instead of the failed task ID. Normal approvals apply before eve uses an outcome.

`{ response: { error: "timeout" } }` is ordinary JSON; only `throw` raises an error.

## Supported tools and subagents

Stubs replace the execution of existing ordinary, dynamic, and workflow tools. The real tool's input schema, approvals, and output processing still apply. A stub doesn't add a tool or show one to the model.

An ordinary tool with stub rules can emit an extra `call.progress` event containing the final response, even when no rule matches. Assert final results on `call.settled`, which stays correct.

### Subagents

Stubbing a subagent's tool returns the configured result without starting the subagent. Persistent tools implemented with `serve`, including the built-in `agent`, require an unconditional stub: omit `match` or use `{}`. You can't mix real and stubbed calls to the same persistent tool.

To let a local subagent run but replace its tools, prefix the tool name with the subagent path and `/`:

```ts
const stubs = [
  { id: "root-tasks", tool: "list_tasks", outcome: { response: ["Root task"] } },
  { id: "research-tasks", tool: "researcher/list_tasks", outcome: { response: ["Research task"] } },
  { id: "nested-tasks", tool: "researcher/assistant/list_tasks", outcome: { response: [] } },
];
```

`researcher` replaces the whole subagent; `researcher/list_tasks` replaces only that subagent's list tool. The built-in `agent` uses `agent/list_tasks`. The model still sees the usual tool names. A root rule never matches a subagent's tool with the same name.

Repeated sessions of a subagent at the same path share that rule's sequence. Remote agents don't receive stub rules, so replace the whole remote agent tool.

### Connections and provider tools

Use the qualified operation name, such as `linear__list_issues` or `researcher/linear__list_issues`. Put operation arguments in `match` and its JSON result in `outcome.response`, or in each sequence entry's `response`.

A stubbed connection must still be reachable, because eve lists the connection's tools to validate the operation and its arguments.

A rule names the tool itself, whether the model calls it directly or through [`eve__tool`](/docs/concepts/built-in-tools#eve__search-eve__tool-and-eve__skill). The model uses `eve__tool` for deferred tools and connection operations. A call through `eve__tool` matches by the name and `input` of the tool it reaches. `eve__search`, `eve__tool`, and `eve__skill` can't be stubbed.

Tools the model provider runs, such as provider-hosted web search, can't be stubbed. A search tool in your own agent can.

## Troubleshooting

| What you see                                                           | What to do                                                                                                          |
| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| `403` with `Tool stubbing is not permitted for this caller.`           | Grant `allowToolStubs: true` to the eval runner's subject; see [Permit tool replacement](#permit-tool-replacement). |
| `Tool stub "<id>" names an unknown or unsupported tool path "<tool>".` | Fix the tool name or subagent path. For a remote agent, stub the whole remote agent tool.                           |
| `Persistent tool "<tool>" requires an unconditional stub.`             | Remove `match` from every rule for that tool, or set it to `{}`.                                                    |
| Warning `Tool stubs did not match any calls: <ids>.`                   | Check the rule order, the `match` constraints, and the tool name. eve ran the real tool for calls no rule matched.  |
| `Tool stub session exceeded 10,000 calls. Start a new eval session.`   | Split the eval into several sessions.                                                                               |

---

For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md)

For an index of all available documentation, see [/llms.txt](/llms.txt)

For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)