# Alerts (Slack and email)

Alerts warn people when a test run goes wrong: a message in a Slack channel or an email, a few
seconds after the run completes. No code, no relay. For machine integrations (your own
systems, PagerDuty), use [webhooks](webhooks.md).

## Rules

A channel has up to three rules. It sends one message when **any** of them matches at the end
of a run:

| Field | Sends when | Default |
|---|---|---|
| `on_failure` | the run verdict is `fail` (a failed, errored or blocked call) | `true` |
| `on_score_below` | the run score is under this value (0 to 100) | off (`null`) |
| `on_latency_p95_above_ms` | a call of the run has a p95 response latency above this value | off (`null`) |

A channel sends at most one message per run, even when several rules match: the message says
which ones did. Canceled runs send nothing. A paused channel (`"enabled": false`) sends
nothing. Rules apply to every run of the organization, whatever its scenario, tags or
environment: to route only some runs, use [webhook filters](webhooks.md#filter-an-endpoint-by-scenario-tag-or-environment).

The latency rule uses each call's p95 response latency (the time the agent under test takes
to answer the caller), and fires on the worst call of the run.

The message gives the run number and scenario, why it fired, the counts (passed, failed,
errors, blocked), the score, the worst latency and a link to the run:

```
CallQA run #247 (Book appointment - edge case): FAIL
Why: failed, score 58 < 80
5/6 passed, 1 failed, 0 errors, 0 blocked. Score 58. Worst latency p95 1840 ms.
Report: https://app.callqa.dev/o/acme/runs/7b0c.../
```

## Slack

1. In Slack, create an incoming webhook for the channel (Apps, **Incoming WebHooks**, **Add to
   Slack**, choose the channel). Copy its URL, `https://hooks.slack.com/services/...`.
2. In CallQA, open **Organization > Alerts**, click **+ Alert channel**, choose Slack and paste
   the URL. Or with an
   `admin` key:

```bash
curl -sS -X POST https://app.callqa.dev/api/v1/alert-channels \
  -H "Authorization: Bearer $CALLQA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "#voice-alerts", "kind": "slack",
       "webhook_url": "https://hooks.slack.com/services/T000/B000/XXXX",
       "on_failure": true, "on_score_below": 80}'
```

The Slack URL is a secret (anyone holding it can post in your channel):

- it must be an `https://hooks.slack.com/` URL under `/services/` (incoming webhook),
  `/triggers/` or `/workflows/` (Slack Workflow Builder); GovSlack's `hooks.slack-gov.com`
  works too. It must resolve to a public address;
- it is never shown again in full: the API and the dashboard show
  `https://hooks.slack.com/services/...XXXX`;
- to replace it, send a new `webhook_url` with `PATCH`. Leave it out to keep the current one.

## Email

```bash
curl -sS -X POST https://app.callqa.dev/api/v1/alert-channels \
  -H "Authorization: Bearer $CALLQA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "On-call", "kind": "email", "to": ["oncall@example.com"],
       "on_latency_p95_above_ms": 1500}'
```

Up to 10 recipients per channel. Addresses are stored in lower case, without duplicates.

## Test a channel

`POST /alert-channels/{id}/test` (or **Send test** in the dashboard) sends a test message now
and tells whether it went through:

```json
{ "sent": false, "error": "AlertConfigError: Slack answered HTTP 404" }
```

Tests are limited to 5 per minute per channel.

## Delivery

Alerts are sent by the CallQA dispatcher, a few seconds after the run completes. A failure
(Slack down, mail server error) is retried after 1, 5 and 30 minutes, then given up. The **Alerts**
page shows the last five deliveries of each channel (the error of a failed one on hover) and
the 12 latest alerts of all channels; **Alerts sent** lists the latest 200, filterable by
channel. Pausing a channel also
drops its pending retries.

## API

| Request | Scope |
|---|---|
| `GET /alert-channels` | `admin` |
| `POST /alert-channels` | `admin` |
| `GET /alert-channels/{id}` | `admin` |
| `PATCH /alert-channels/{id}` | `admin` |
| `DELETE /alert-channels/{id}` | `admin` |
| `POST /alert-channels/{id}/test` | `admin` |

`PATCH` changes only the fields you send; `null` on `on_score_below` or
`on_latency_p95_above_ms` removes that rule. The `kind` of a channel cannot change. A channel
must keep at least one rule. Full schemas in the [API reference](api-reference.md#alerts).

Only admins manage alert channels, in the dashboard and through the API. Creating, changing
and deleting a channel is recorded in the audit log, without the Slack URL.