Writing scenarios

A scenario is a YAML (or JSON) document. This guide shows how to write, validate and manage them. Every field is described in the grammar reference.

In the dashboard (Scenarios, New scenario or Edit), the editor shows the same document in two synchronized views: a Human view in cards (persona, goal, IVR steps, checks, test cases) and the YAML, validated as you type. While you edit an existing scenario, a valid document is saved automatically; Save saves it at once.

The short form

The format shown on callqa.dev is valid as is:

name: "Book appointment - edge case"
phone: "+14155550100"
schedule: "every day at 09:00 UTC"
call_window: "09:00-18:00 America/New_York"
persona: "Impatient customer, speaks fast, has accent"
goal: "Book a haircut for tomorrow 2pm"
must_include:
  - "confirm date and time"
  - "provide booking reference"
must_not:
  - "hang up before confirmation"
  - "hallucinate availability"
max_latency_ms: 2000

The full form

The full form gives you typed assertions (with ids and severities), IVR steps, limits, variants and extension fields:

schema_version: 1
name: "IVR - check account balance"
target:
  phone: "+14155550103"
  kind: ivr
  platform: genesys
persona:
  description: "Existing customer who knows the menu and presses keys quickly"
  patience: high
goal: "Reach the account balance message through the IVR menu"
steps:
  - type: expect
    text: "welcome message and main menu options"
  - type: dtmf
    digits: "2"
  - type: expect
    text: "asks for the account number followed by the pound key"
  - type: dtmf
    digits: "12345678#"
assertions:
  - id: balance-read
    type: transcript_regex
    pattern: "balance (is|of)"
  - id: reached-goal
    type: goal_reached
  - id: fast-tree
    type: max_duration_s
    value: 120
    severity: warn
limits:
  max_duration_s: 180
tags: ["ivr", "dtmf"]
x_owner: "payments-team"

More complete examples in examples/scenarios/.

Writing good checks

Describe behaviour, not wording. must_include: "confirm date and time" passes if the agent confirms in any words. Use transcript_regex only when the exact words matter (a legal disclaimer, a reference format).
One idea per assertion. "Confirm the date and give a reference" is two assertions: you will know which one failed.
Give stable ids to assertions you track over time (id: confirm-date): the dashboard can follow the same check across runs and edits.
Use severity: warn for nice-to-haves (tone, empathy, speed): they lower the score and mark the call warn, without failing it.
Set a latency budget with max_latency_ms. It is measured per turn, from the end of the caller's speech to the start of the target's answer, at p95 by default.
Put test data in test_data, not in the goal: the caller uses it when asked, and the judge can detect answers that contradict it.
Use variants for edge cases: same scenario, different persona or goal, extra assertions. The base assertions apply to every variant.

Validate before saving

POST /scenarios/validate checks a JSON document without saving it. It returns the normalized document (short form folded, defaults filled in):

curl -sS -X POST https://app.callqa.dev/api/v1/scenarios/validate \
  -H "Authorization: Bearer $CALLQA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Smoke", "phone": "+14155550100", "persona": "Calm caller", "goal": "Ask the opening hours"}' \
  | jq '{valid, warnings, assertions: .normalized.assertions}'

Errors come back as 422 with one entry per problem:

{
  "detail": "invalid scenario",
  "code": "invalid_scenario",
  "errors": [
    { "loc": "assertions.0.value", "msg": "Input should be greater than 0" },
    { "loc": "persona", "msg": "Value error, unknown field(s) ['mood']; custom fields must start with 'x_'" }
  ]
}

loc is the dotted path of the field (variants.2.assertions.0.text).

What the server checks

On top of the grammar (types, required fields, unknown fields, E.164, DTMF characters, regex syntax), the server refuses a scenario when:

Check	`loc`
The number is not a valid, dialable number	`target.phone`
The number is emergency, premium rate, shared cost, personal, pager, voicemail or unclassified	`target.phone`
`schedule` is neither a cron expression nor an accepted English form	`schedule`
`timezone` is not an IANA time zone	`timezone`
`timezone` and the time zone written in `schedule` differ	`schedule`
`call_window` is not `HH:MM-HH:MM`, optionally followed by an IANA time zone	`schedule`
A schedule or call window in `environments.<slug>` is invalid	`environments.<slug>.schedule`
`environments` names an environment that does not exist	`environments`
Two variants have the same `title`	`variants`
Another non-archived scenario has the same `name`	`name`

And answers 403 with code: feature_not_in_plan (or country_not_in_plan), the plan that lifts the limit and an upgrade link, when:

the scenario or a variant uses test_data and your plan does not include custom test data injection (Growth and above);
a variant has source: auto and your plan does not include auto-derived coverage (Growth and above);
the number is in a new country and your plan's country limit is reached.

Every new content of a scenario also goes through a compliance review (acceptable use policy): it is saved either way, but only an allowed scenario can run. The verdict and its reasons are in the compliance object of the API and MCP answers and on the scenario page.

Schedules and call windows

schedule accepts a 5-field cron expression, in UTC, optionally followed by a time zone, or a plain English form. Set the time zone with the optional timezone field (an IANA name such as Europe/Paris, UTC by default) or at the end of the schedule text; if you give both they must be the same:

schedule: "0 8 * * 1-5"
timezone: "Europe/Paris"

Times are local wall-clock times: the example runs at 08:00 in Paris all year, that is 06:00 UTC in summer and 07:00 UTC in winter. On the night the clocks go forward, a time that does not exist that day (02:30 in Paris, for example) runs at the next valid minute; on the night they go back, a time that happens twice runs once (an hourly schedule still runs every real hour).

`schedule`	Runs
`"0 9 * * *"`	every day at 09:00 UTC
`"30 8 * * 1-5 Europe/Paris"`	weekdays at 08:30, Paris time
`"30 8 * * 1-5"` with `timezone: "Europe/Paris"`	the same
`"every day at 09:00 UTC"`	every day at 09:00 UTC
`"every weekday at 08:30 America/New_York"`	Monday to Friday at 08:30, New York time
`"every monday at 07:00 Europe/London"`	Mondays at 07:00, London time
`"every hour"`	every hour, on the hour

call_window: "09:00-18:00 America/New_York" keeps scheduled calls inside business hours: a scheduled call that comes due outside the window waits for the next opening. Windows can cross midnight ("22:00-06:00 UTC"). Runs you trigger by hand, by API or from CI are not held by the window.

Archiving a scenario stops its schedule.

One schedule per environment

A scenario runs on its own schedule in each environment where it is activated (see Environments). In the document, the top-level schedule, timezone and call_window are the plan of your default environment, and environments holds the plans of the other ones, keyed by slug. Each slug must be an environment of your organization (the example needs staging: Add development and staging on the Versions and environments board creates it); an unknown slug is refused with the list of your environments, never created on the fly (see Environments):

name: "Opening hours - every environment"
phone: "+14155550100"
persona: "Calm caller"
goal: "Ask the opening hours"
schedule: "every weekday at 08:30 America/New_York"
environments:
  staging:
    schedule: "every hour"
    timezone: "America/New_York"

On the scenario page, each environment has its own schedule field. There you can also type a schedule in plain words the parser does not know: CallQA proposes a reading (for example "Every day at 04:00 UTC, cron 0 4 * * *") and saves it only once you confirm it. The API accepts only cron expressions and the forms in the table above.

Create, import, update, export

Task	Request	Scope
Create from JSON	`POST /scenarios`	`write`
Create or update from YAML (upsert by `name`)	`POST /scenarios/import` with `Content-Type: application/yaml`	`write`
Replace	`PUT /scenarios/{id}`	`write`
Read	`GET /scenarios/{id}`	`read`
Export	`GET /scenarios/{id}/export?format=yaml` (or `json`)	`read`
List	`GET /scenarios?tag=booking&q=appointment&include_archived=false`	`read`
Archive	`POST /scenarios/{id}/archive`	`write`
Delete	`DELETE /scenarios/{id}`	`admin`

Export and import round-trip: keep your scenarios in Git and sync them from CI with callqa scenario import file.yaml (see the CLI).
On update (PUT or import), variants are matched by title: a variant that keeps its title keeps its id and its history. A variant removed from the document is deleted.
Prefer archive to delete. An archived scenario keeps its history, stops its schedule and frees its name. Deleting (admins only) is permanent: past runs and calls are kept, but they lose their link to the scenario and its variants.
Unknown x_* fields are stored and returned as is by GET and export.

Variants

Task	Request	Scope
List	`GET /scenarios/{id}/variants`	`read`
Add a manual variant	`POST /scenarios/{id}/variants`	`write`
Enable, disable or replace	`PATCH /variants/{variant_id}` with `{"enabled": false}` or `{"spec": {...}}`	`write`
Delete	`DELETE /variants/{variant_id}`	`admin`
Auto-derive N variants	`POST /scenarios/{id}/variants/derive` with `{"count": 4}`	`write`
From a real call	`POST /scenarios/{id}/variants/from-bundle`	`write`

curl -sS -X POST "https://app.callqa.dev/api/v1/scenarios/$SCENARIO/variants" \
  -H "Authorization: Bearer $CALLQA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "title": "Asks for Sunday 7am",
        "goal": "Ask for Sunday 7am (closed), then accept the first slot offered",
        "assertions": [{"type": "must_not", "text": "offer a slot outside opening hours"}]
      }'

Titles are unique within a scenario (422 otherwise).

Generate scenarios from a description

POST /scenarios/generate proposes scenario documents from a plain description of your service. Nothing is saved: review the proposals, then create the ones you keep.

curl -sS -X POST https://app.callqa.dev/api/v1/scenarios/generate \
  -H "Authorization: Bearer $CALLQA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"description": "Hair salon booking line: book, move, cancel.", "phone": "+14155550100", "count": 5}' \
  | jq '.scenarios[].name, .rejected'

Proposals that do not pass validation are dropped and counted in rejected. When the generator is not available the endpoint answers 503.

Edit a scenario in plain words (assistant)

In the dashboard, the scenario editor and the new scenario page have an Assistant box. Type or dictate what you want ("I want this test to pass even if the salon offers 2 pm", "add a variant where the customer is in a hurry"); the assistant writes the new scenario, checks it against the grammar and your plan, and shows a diff with the saved version. Accept and save saves it (audited as a normal edit), Reject drops it, and a follow-up request refines the pending proposal.

A session holds up to 15 requests; a dictation lasts 3 minutes at most. The audio is transcribed and not stored.
On a call that failed, warned or ended in error, Fix this test opens the assistant with the failure (checks, quotes, transcript) and starts at once: it proposes to relax a check that was too strict, or to add a test case that reproduces the bug.
Done, back to the scenario closes the session. Nothing is saved without Accept.
Limits per organization: requests and dictations per hour, and a daily budget. The assistant is not billed per request.

Build a matrix of test cases (Scale)

On a scenario page, Build a matrix of test cases creates many test cases at once: every combination of personas, patience levels, test data sets and variations (extra instructions) becomes one manual test case. Preview lists the combinations, then Create adds them. At most 50 test cases per build and 20 values per field; titles that already exist are skipped. Available on the Scale plan.