--- title: Running Evals description: "The eve eval CLI: flags, filters, exit codes, artifacts, and how to wire evals into CI." url: "https://eve.dev/docs/evals/running" docs_index: /llms.txt --- > For an index of all documentation, see [/llms.txt](/llms.txt). `eve eval` runs every `.eval.ts` file under `evals/` concurrently and prints a per-eval summary. It boots a local dev server, or targets a remote one with `--url`. ```bash eve eval # run all discovered evals locally eve eval weather smoke # run selected evals (an id, or a directory prefix) eve eval --url https:// # target a remote app instead of a local host eve eval --tag fast # only evals carrying a tag eve eval --exclude-tag slow # skip evals carrying a tag eve eval --strict # soft below-threshold assertions also fail the exit code eve eval --timeout 60000 # per-eval timeout in milliseconds eve eval --max-concurrency 4 # cap concurrent eval executions (default 8) eve eval --repeat 5 # run every eval 5 times to measure repeatability eve eval --junit .eve/junit.xml # write JUnit XML eve eval --list # print discovered evals without running eve eval --verbose # stream per-eval logs and workflow run IDs eve eval --json # machine-readable output eve eval --skip-report # skip config and eval-defined reporters (e.g. Braintrust) eve eval --keep-state # keep the local server's state and Workflow data after the run ``` Positional ids match exactly or by directory prefix: `eve eval weather` runs `evals/weather.eval.ts`, every eval under `evals/weather/`, and every entry of an array-exported `weather.eval.ts`. Tag filters compose: `--tag` keeps evals carrying at least one listed tag, then `--exclude-tag` drops any eval carrying an excluded tag. A `--tag` filter that matches nothing is a configuration error (exit `2`). A run where `--exclude-tag` removes every match succeeds with nothing executed. ## Exit codes | Code | Means | | ---- | ------------------------------------------------------------------------------- | | `0` | Every non-skipped eval passed its gates (and soft thresholds, under `--strict`) | | `1` | Any eval failed (a failed gate, an execution error, or a strict threshold miss) | | `2` | Configuration error | An eval that calls `t.skip(reason)` is reported as skipped, does not count as passed or failed, and never changes the exit code. `eve eval --json` writes the complete report before exiting, including when stdout is piped to another process or redirected to a file. With `--verbose`, eve prefixes each line with its eval ID and prints `t.log()` lines as they happen. It then prints the `wrun_...` workflow run ID for every session the eval read, so you can find the runs in your workflow provider. ## Repeat evals Evals that call a real model can pass on one run and fail on the next. To measure how often an eval passes, pass `--repeat ` or set `repeat` in `evals.config.ts`. eve then runs every selected eval `n` times. The CLI flag overrides the config value. The default is `1`. ```bash eve eval weather --repeat 5 ``` With `n` greater than `1`, each attempt is a separate result with a 1-based `attempt` field: - The console and `--verbose` lines label each attempt `#`, and the summary adds a per-eval pass rate such as `3/5 passed`. The pass rate counts attempts that passed or failed their gates; scored and skipped attempts are left out. - `--json` results, JUnit test cases (named `#`), and [reporters](./reporters) receive one result per attempt. - Every attempt counts toward the run totals and the exit code, so one failed attempt fails the run. Attempts share the `--max-concurrency` limit with every other eval, so attempts of one eval can run at the same time. If an eval depends on shared state or a rate-limited provider, lower `--max-concurrency` so contention doesn't look like flakiness. ## Artifacts Each run writes artifacts under `.eve/evals//`: - `summary.json` summarizes the run. - `results.jsonl` indexes the results, one line per eval. - `evals/` holds each eval's assertion results, verdict, start and completion times, and `t.log` lines. - `evals/` also holds each session's events, with when the client sent each request and received each event. In a repeated run, each attempt's detail and event files get an `.attempt-` suffix, such as `evals/weather.attempt-2.json`. When an eval fails, check its artifact for details the console omits. ## CI In CI, run evals strict and machine-reportable: ```bash eve eval --strict --junit .eve/junit.xml ``` - `--strict` turns soft threshold misses into failures, so score regressions block the merge. - `--junit` gives the CI provider per-eval annotations; upload the `.eve/evals/` directory as a failure artifact for the full event streams. Evals run against a live model, so the CI environment must provide the model-provider credentials. Against a deployed app, add `--url`: ```bash eve eval --strict --url "$DEPLOY_URL" --junit .eve/junit.xml ``` ## What to read next - [Targets](./targets): what `--url` interacts with - [Reporters](./reporters): Braintrust and JUnit output - [CLI reference](../reference/cli): the rest of the `eve` CLI --- For a semantic overview of all documentation, see [/sitemap.md](/sitemap.md) For an index of all available documentation, see [/llms.txt](/llms.txt) For agent-facing discovery, including API and MCP surfaces, see [/agents.md](/agents.md)