Alerts (Slack and email)

Alerts warn people when a test run goes wrong: a message in a Slack channel or an email, a few seconds after the run completes. No code, no relay. For machine integrations (your own systems, PagerDuty), use webhooks.

Rules

A channel has up to three rules. It sends one message when any of them matches at the end of a run:

A channel sends at most one message per run, even when several rules match: the message says which ones did. Canceled runs send nothing. A paused channel ("enabled": false) sends nothing. Rules apply to every run of the organization, whatever its scenario, tags or environment: to route only some runs, use webhook filters.

The latency rule uses each call's p95 response latency (the time the agent under test takes to answer the caller), and fires on the worst call of the run.

The message gives the run number and scenario, why it fired, the counts (passed, failed, errors, blocked), the score, the worst latency and a link to the run:

CallQA run #247 (Book appointment - edge case): FAIL
Why: failed, score 58 < 80
5/6 passed, 1 failed, 0 errors, 0 blocked. Score 58. Worst latency p95 1840 ms.
Report: https://app.callqa.dev/o/acme/runs/7b0c.../

Slack

1. In Slack, create an incoming webhook for the channel (Apps, Incoming WebHooks, Add to Slack, choose the channel). Copy its URL, https://hooks.slack.com/services/....
2. In CallQA, open Organization > Alerts, click + Alert channel, choose Slack and paste the URL. Or with an admin key:

curl -sS -X POST https://app.callqa.dev/api/v1/alert-channels \
  -H "Authorization: Bearer $CALLQA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "#voice-alerts", "kind": "slack",
       "webhook_url": "https://hooks.slack.com/services/T000/B000/XXXX",
       "on_failure": true, "on_score_below": 80}'

The Slack URL is a secret (anyone holding it can post in your channel):

• it must be an https://hooks.slack.com/ URL under /services/ (incoming webhook), /triggers/ or /workflows/ (Slack Workflow Builder); GovSlack's hooks.slack-gov.com works too. It must resolve to a public address;
• it is never shown again in full: the API and the dashboard show https://hooks.slack.com/services/...XXXX;
• to replace it, send a new webhook_url with PATCH. Leave it out to keep the current one.

Email

curl -sS -X POST https://app.callqa.dev/api/v1/alert-channels \
  -H "Authorization: Bearer $CALLQA_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "On-call", "kind": "email", "to": ["[email protected]"],
       "on_latency_p95_above_ms": 1500}'

Up to 10 recipients per channel. Addresses are stored in lower case, without duplicates.

Test a channel

POST /alert-channels/{id}/test (or Send test in the dashboard) sends a test message now and tells whether it went through:

{ "sent": false, "error": "AlertConfigError: Slack answered HTTP 404" }

Tests are limited to 5 per minute per channel.

Delivery

Alerts are sent by the CallQA dispatcher, a few seconds after the run completes. A failure (Slack down, mail server error) is retried after 1, 5 and 30 minutes, then given up. The Alerts page shows the last five deliveries of each channel (the error of a failed one on hover) and the 12 latest alerts of all channels; Alerts sent lists the latest 200, filterable by channel. Pausing a channel also drops its pending retries.

API

PATCH changes only the fields you send; null on on_score_below or on_latency_p95_above_ms removes that rule. The kind of a channel cannot change. A channel must keep at least one rule. Full schemas in the API reference.

Only admins manage alert channels, in the dashboard and through the API. Creating, changing and deleting a channel is recorded in the audit log, without the Slack URL.