Templates

Incident response

sre, an incident response agent that investigates alerts from Slack or webhooks using live Datadog, GitHub, and Vercel evidence, with read-only access by default.

View Source
  • Slack
  • Datadog
  • GitHub
  • Vercel
  • HTTP API
agent/agent.tstypescript
import type { OpenAILanguageModelResponsesOptions } from "@ai-sdk/openai";
import { defineAgent } from "eve";

/**
 * Terra is a good balance of speed, efficiency, and intelligence. A large context window is
 * used because looping through a large number of tool calls can eat a lot of context.
 *
 * `textVerbosity: "low"` is really wonderful and significantly "de-slops" the output from
 * the model, making important evidence easier to scan and understand.
 */
export default defineAgent({
  compaction: {
    thresholdPercent: 0.7,
  },
  model: "openai/gpt-5.6-terra",
  modelContextWindowTokens: 1_000_000,
  modelOptions: {
    providerOptions: {
      openai: {
        reasoningEffort: "xhigh",
        textVerbosity: "low",
      } satisfies OpenAILanguageModelResponsesOptions,
    },
  },
});

sre

sre is an eve incident response agent for Slack. It brings together the observability data you need to debug an alert or incident across Datadog, GitHub, Vercel, and other tools. Investigations are read-only by default.

Mention @sre, watch a channel for alerts, or invoke it from an external system with a webhook. The agent checks hypotheses against live signals and records each finding with a source link. Replies start with the answer, then the supporting evidence.

How it works

An investigation starts from one of three places.

  • A Slack mention or direct message.
  • A new top-level message in a watched Slack channel.
  • A POST /v1/investigate webhook call.

The agent records novel, decision-relevant findings with source links. Each investigation turn has its own evidence set. The agent records each finding-and-URL pair once, then adds recorded evidence to the Slack result.

Automated runs use webhook metadata and service-authenticated tools. They do not use per-user OAuth.

Channel watch and custom skills are optional. Both store state in Vercel Blob. The Blob store is required only for these two features.

Set up the project

Select Deploy with Vercel to clone the repository, create a Vercel project, and provision the connectors and storage.

ProvisionedSets
Slack connector with trigger path /eve/v1/slackSLACK_CONNECTOR
GitHub connector, read-only installGITHUB_CONNECTOR
Datadog connectorDATADOG_CONNECTOR
Vercel MCP connectorSRE_VERCEL_CONNECTOR
Private Vercel Blob storeVercel Blob environment variables

For a complete CLI setup with your agent, use docs/setup-for-agents.md.

After deployment:

  1. Invite the Slack app to a channel.
  2. Mention @sre and confirm that the app replies.
  3. To enable automated runs via a webhook, follow docs/automate-investigations.md.

Local development

bash
pnpm installvercel linkvercel env pullpnpm dev

Environment configuration

VariableRequiredDefaultWhat it does
SLACK_CONNECTORNoslack/sreConnector UID for the Slack channel and bot Web API.
GITHUB_CONNECTORNogithub/sreConnector UID for the GitHub tools extension.
DATADOG_CONNECTORNodatadog/sreConnector UID for the Datadog MCP tools.
SRE_VERCEL_CONNECTORNovercel/sreConnector UID for the Vercel MCP tools.
WEBHOOK_SECRETNononeShared secret for POST /v1/investigate. Use a long random value from Node crypto (randomBytes(32)).
DD_SITENodatadoghq.comDatadog site for MCP. Examples are datadoghq.eu and us5.datadoghq.com.

Copy .env.example to .env to set local environment variables.

Automation endpoint

POST /v1/investigate accepts a title, a Slack channel, an optional description, and optional metadata. It returns 202 and starts a Slack investigation.

The endpoint uses WEBHOOK_SECRET. Send the secret with x-sre-webhook-secret or Authorization: Bearer.

Customize the agent

  • Edit agent/instructions/instructions.md to change general behavior.
  • Edit the built-in skills in agent/skills/ to change investigation and handoff procedures.
  • Add tools or connections for other operational systems using Vercel Connect.
  • Integrate with agent/channels/webhook.ts to invoke the agent from any external system (see docs/automate-investigations.md).

Create runbooks in Slack

Ask @sre to create a runbook for a recurring alert or incident. Global runbooks apply to every session. Personal runbooks apply only to the requesting Slack user and override global runbooks with the same name. The agent loads a saved runbook on a later matching request, not during the request that creates it. Private Vercel Blob storage is required to save and load runbooks.

Verify changes

bash
pnpm validatepnpm test

Troubleshooting

  • If Slack mentions do not arrive, confirm that the trigger path is /eve/v1/slack.
  • If a webhook returns 401, confirm that the caller and the deployment use the same WEBHOOK_SECRET.
  • If an investigation does not start, confirm that the bot is a member of slackChannel.
  • If custom skills fail to save, confirm that the private Blob store is connected.
  • If Vercel tools request authorization, complete the Vercel Connect sign-in for the Slack user. Automated runs do not have a user identity, so they cannot use this connection.

Learn more

Explore more templates