# Scenario grammar reference

_Generated by `tools/gen_grammar_reference.py` from `callqa_spec`. Do not edit by hand._

Grammar version: **`schema_version: 1`**. This page is generated from the models the server uses to validate scenarios, and every example on it is checked with `callqa_spec.load_scenario`.

- Machine readable: [`scenario.schema.json`](scenario.schema.json) (JSON Schema, also served by `GET /api/v1/scenarios/schema`).
- How to write good scenarios, server-side checks and plan limits: [Scenario guide](../scenarios.md).
- Compatibility rules: [Versioning](../versioning.md).

## Contents

- [Scenario](#scenario)
- [Target](#target)
- [Persona](#persona)
- [Limits](#limits)
- [Variant](#variant)
- [Steps](#steps)
- [Assertions](#assertions)
- [Short form](#short-form)
- [Extension fields `x_*`](#extension-fields-x_)
- [Complete examples](#complete-examples)

## General rules

- YAML and JSON are equivalent. The API takes JSON, or YAML on `POST /scenarios/import`.
- Unknown fields are **rejected**, at every level, unless their name starts with `x_`.
- Strings are trimmed (leading and trailing spaces removed).
- `type` selects the kind of a step or an assertion.

## Scenario

The document you write. Required: `name`, `target`, `persona`, `goal`.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `schema_version` | integer | no | `1` |  | Older documents are upgraded on read; a version newer than the server is refused. |
| `name` | string | yes |  |  |  |
| `target` | [Target](#target) | yes |  |  |  |
| `persona` | [Persona](#persona) | yes |  |  |  |
| `goal` | string | yes |  |  |  |
| `test_data` | map (free keys) | no | `{}` |  |  |
| `steps` | list of [Step](#steps) | no | `[]` |  | Optional IVR navigation hints. |
| `assertions` | list of [Assertion](#assertions) | no | `[]` |  |  |
| `limits` | [Limits](#limits) | no | defaults of the object |  |  |
| `variants` | list of [Variant](#variant) | no | `[]` |  |  |
| `tags` | list of string | no | `[]` |  |  |
| `schedule` | string or null | no | null |  | Cron expression, UTC unless timezone. The API also accepts `every day at 09:00 UTC`-style text. See the scenario guide. |
| `timezone` | string or null | no | null |  | IANA time zone of `schedule` (e.g. `Europe/Paris`). Default UTC. Local wall-clock time: 8:00 stays 8:00 across daylight saving changes. Checked against the IANA time zone database. A time zone may also end the `schedule` text; if both are given they must be the same. |
| `call_window` | string or null | no | null |  | 'HH:MM-HH:MM Area/City'. Only scheduled runs wait for the window; checked by the API. |
| `environments` | map (free keys) or null | no | null |  | Per environment plan, keyed by environment slug (`staging`): whether the scenario runs there, with its own `schedule`, `timezone` and `call_window`. The top-level fields stay the plan of the default environment. |

## Target

The phone number under test and what answers it.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `phone` | string | yes |  |  | Number to call, E.164 (e.g. +15550123456). E.164 after removing spaces, dashes, dots and parentheses (`+1 (415) 555-0100` is accepted and stored as `+14155550100`). |
| `kind` | enum: `ai_agent`, `ivr`, `iva`, `human`, `unknown` | no | `unknown` |  |  |
| `platform` | string or null | no | null |  | Free text: vapi, retell, bland, livekit, twilio, genesys... |
| `language` | string | no | `en-US` |  | BCP-47 language the call is held in. |

`kind` values: `ai_agent`, `ivr`, `iva`, `human`, `unknown`.

## Persona

Who the caller is and how they behave.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `description` | string | yes |  |  | Who the caller is and how they behave. |
| `voice` | string or null | no | null |  | TTS voice id; provider default if empty. |
| `accent` | string or null | no | null |  |  |
| `speaking_rate` | number | no | `1.0` | >= 0.5, <= 2.0 |  |
| `patience` | enum: `low`, `normal`, `high` | no | `normal` |  |  |
| `interrupts` | boolean | no | false |  | If true the caller barges in while the agent speaks. |

## Limits

Hard limits for each call of this scenario.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `max_duration_s` | integer | no | `300` | > 0, <= 1800 |  |
| `silence_timeout_s` | integer | no | `20` | > 0, <= 300 |  |
| `ring_timeout_s` | integer | no | `45` | > 0, <= 120 |  |

## Variant

A test case derived from the scenario (#1..#N). Every field except `title` overrides the scenario for this case; `assertions` are appended to the scenario's assertions, `test_data` is merged over it.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `title` | string | yes |  |  |  |
| `source` | enum: `auto`, `manual` | no | `manual` |  |  |
| `persona` | [Persona](#persona) or null | no | null |  |  |
| `goal` | string or null | no | null |  |  |
| `instructions` | string or null | no | null |  | Extra guidance for this run. |
| `test_data` | map (free keys) or null | no | null |  |  |
| `assertions` | list of [Assertion](#assertions) or null | no | null |  | Appended to the base assertions. |
| `enabled` | boolean | no | true |  |  |

## Steps

Optional navigation hints, followed in order. Useful for IVR menus. An AI agent target usually needs none: the caller converses from `persona` and `goal`.

```yaml
steps:
  - type: expect
    text: "main menu options"
    timeout_s: 20
  - type: dtmf
    digits: "2"
  - type: wait
    seconds: 1.5
  - type: say
    text: "account balance"
```

### `say` (SayStep)

The caller says this text.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `type` | `"say"` | yes |  |  |  |
| `text` | string | yes |  |  |  |

### `dtmf` (DtmfStep)

The caller presses these keys. `w` is a 0.5 s pause, `W` a 1 s pause.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `type` | `"dtmf"` | yes |  |  |  |
| `digits` | string | yes |  |  | Only `0-9 * # w W`. |

### `expect` (ExpectStep)

The caller waits until the target says something with this meaning.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `type` | `"expect"` | yes |  |  |  |
| `text` | string | yes |  |  | What the target should say (semantic match, not exact). |
| `timeout_s` | number | no | `15.0` | > 0, <= 300 |  |

### `wait` (WaitStep)

The caller stays silent for this many seconds.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `type` | `"wait"` | yes |  |  |  |
| `seconds` | number | yes |  | > 0, <= 600 |  |

## Assertions

Pass/fail checks run on the call trace after the call. Every assertion has these common fields:

| Field | Type | Default | Description |
|---|---|---|---|
| `id` | string or null | null | Stable id, reused across runs for trends. |
| `severity` | enum: `fail`, `warn` | `fail` |  |

`severity: fail` makes the call fail when the check fails. `severity: warn` only marks it `warn`. Semantic assertions are judged by an LLM judge that quotes the transcript as evidence; deterministic ones are computed from the trace timings.

### `must_include` (MustInclude)

Semantic: judged on meaning by the LLM judge, not by exact words.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `text` | string | yes |  |  | Behaviour the agent must show, judged semantically. |

```yaml
assertions:
  - type: must_include
    text: "confirm date and time"
```

### `must_not` (MustNot)

Semantic: the agent must never show this behaviour.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `text` | string | yes |  |  |  |

```yaml
assertions:
  - type: must_not
    text: "hallucinate availability"
```

### `goal_reached` (GoalReached)

Semantic: did the caller reach the scenario (or variant) `goal`?

No other field.

```yaml
assertions:
  - type: goal_reached
```

### `llm_check` (LlmCheck)

Semantic: a yes/no question the judge answers from the transcript.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `question` | string | yes |  |  | Yes/no question answered by the judge from the transcript. |

```yaml
assertions:
  - type: llm_check
    question: "Did the agent offer a callback?"
```

### `max_latency_ms` (MaxLatency)

Deterministic. Latency of one turn = start of target speech minus end of caller speech. `percentile` picks p50, p95 or the max over the call.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `value` | integer | yes |  | > 0 | Per turn, end of caller speech to start of target speech. |
| `percentile` | enum: `p50`, `p95`, `max` | no | `p95` |  |  |

```yaml
assertions:
  - type: max_latency_ms
    value: 1500
    percentile: p95
```

### `max_duration_s` (MaxDuration)

Deterministic. Whole call, from dialing to the last event.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `value` | integer | yes |  | > 0 |  |

```yaml
assertions:
  - type: max_duration_s
    value: 180
    severity: warn
```

### `max_interruptions` (MaxInterruptions)

Deterministic. Counts the times the target talked over the caller.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `value` | integer | yes |  | >= 0 | Times the target talked over the caller. |

```yaml
assertions:
  - type: max_interruptions
    value: 1
```

### `transcript_regex` (TranscriptRegex)

Deterministic. Python regular expression, case insensitive, searched in each turn of `speaker`. `expect_match: false` asserts that nothing matches.

| Field | Type | Required | Default | Constraints | Description |
|---|---|---|---|---|---|
| `pattern` | string | yes |  |  | Must compile as a Python regular expression. |
| `speaker` | enum: `target`, `caller`, `any` | no | `target` |  |  |
| `expect_match` | boolean | no | true |  |  |

```yaml
assertions:
  - type: transcript_regex
    pattern: "booking (reference|number)"
    speaker: target
```

## Short form

The format shown on callqa.dev is accepted as is. On import it is folded into the typed grammar:

| Short form | Becomes |
|---|---|
| `phone: "+1..."` | `target.phone` |
| `persona: "text"` | `persona.description` |
| `must_include: [..]` | one `must_include` assertion per item |
| `must_not: [..]` | one `must_not` assertion per item |
| `max_latency_ms: N` | a `max_latency_ms` assertion (p95) |

This short document:

```yaml
name: "Book appointment - edge case"
phone: "+14155550100"
persona: "Impatient customer, speaks fast, has accent"
goal: "Book a haircut for tomorrow 2pm"
must_include:
  - "confirm date and time"
must_not:
  - "hallucinate availability"
max_latency_ms: 2000
```

is stored as (with every default filled in):

```yaml
schema_version: 1
name: Book appointment - edge case
target:
  phone: '+14155550100'
  kind: unknown
  language: en-US
persona:
  description: Impatient customer, speaks fast, has accent
  speaking_rate: 1.0
  patience: normal
  interrupts: false
goal: Book a haircut for tomorrow 2pm
test_data: {}
steps: []
assertions:
- severity: fail
  type: must_include
  text: confirm date and time
- severity: fail
  type: must_not
  text: hallucinate availability
- severity: fail
  type: max_latency_ms
  value: 2000
  percentile: p95
limits:
  max_duration_s: 300
  silence_timeout_s: 20
  ring_timeout_s: 45
variants: []
tags: []
```

## Extension fields `x_*`

Any key that starts with `x_` is accepted at any level and kept verbatim (stored, exported, returned by the API). CallQA never interprets them. Use them for your own metadata (owner, ticket, region) or to try a field before it joins the grammar.

```yaml
name: "Checkout flow"
target:
  phone: "+14155550100"
  x_region: "us-west"
persona:
  description: "Returning customer"
goal: "Buy the blue plan"
x_owner: "voice-team"
x_jira: "VOICE-142"
```

## Complete examples

Real-world scenarios from [`examples/scenarios/`](../../../examples/scenarios/). Each one is validated by the test suite.

- [`book-appointment.yaml`](../../../examples/scenarios/book-appointment.yaml): Book appointment - edge case (5 assertions, 4 variants)
- [`cancel-subscription.yaml`](../../../examples/scenarios/cancel-subscription.yaml): Cancel subscription (5 assertions, 2 variants)
- [`ivr-dtmf-navigation.yaml`](../../../examples/scenarios/ivr-dtmf-navigation.yaml): IVR - check account balance (3 assertions, 1 variant)
- [`payment-test-card.yaml`](../../../examples/scenarios/payment-test-card.yaml): Pay an invoice by card (6 assertions, 1 variant)
- [`reschedule-minimal.yaml`](../../../examples/scenarios/reschedule-minimal.yaml): Reschedule a dentist appointment (1 assertion, 0 variants)
- [`spanish-language.yaml`](../../../examples/scenarios/spanish-language.yaml): Spanish line - order status (4 assertions, 1 variant)
- [`transfer-to-human.yaml`](../../../examples/scenarios/transfer-to-human.yaml): Escalate to a human agent (5 assertions, 1 variant)