Scenario grammar reference

Generated by tools/gen_grammar_reference.py from callqa_spec. Do not edit by hand.

Grammar version: schema_version: 1. This page is generated from the models the server uses to validate scenarios, and every example on it is checked with callqa_spec.load_scenario.

• Machine readable: scenario.schema.json (JSON Schema, also served by GET /api/v1/scenarios/schema).
• How to write good scenarios, server-side checks and plan limits: Scenario guide.
• Compatibility rules: Versioning.

Contents

• Scenario
• Target
• Persona
• Limits
• Variant
• Steps
• Assertions
• Short form
• Extension fields x_*
• Complete examples

General rules

• YAML and JSON are equivalent. The API takes JSON, or YAML on POST /scenarios/import.
• Unknown fields are rejected, at every level, unless their name starts with x_.
• Strings are trimmed (leading and trailing spaces removed).
• type selects the kind of a step or an assertion.

Scenario

The document you write. Required: name, target, persona, goal.

Target

The phone number under test and what answers it.

kind values: ai_agent, ivr, iva, human, unknown.

Persona

Who the caller is and how they behave.

Limits

Hard limits for each call of this scenario.

Variant

A test case derived from the scenario (#1..#N). Every field except title overrides the scenario for this case; assertions are appended to the scenario's assertions, test_data is merged over it.

Steps

Optional navigation hints, followed in order. Useful for IVR menus. An AI agent target usually needs none: the caller converses from persona and goal.

steps:
  - type: expect
    text: "main menu options"
    timeout_s: 20
  - type: dtmf
    digits: "2"
  - type: wait
    seconds: 1.5
  - type: say
    text: "account balance"

say (SayStep)

The caller says this text.

dtmf (DtmfStep)

The caller presses these keys. w is a 0.5 s pause, W a 1 s pause.

expect (ExpectStep)

The caller waits until the target says something with this meaning.

wait (WaitStep)

The caller stays silent for this many seconds.

Assertions

Pass/fail checks run on the call trace after the call. Every assertion has these common fields:

severity: fail makes the call fail when the check fails. severity: warn only marks it warn. Semantic assertions are judged by an LLM judge that quotes the transcript as evidence; deterministic ones are computed from the trace timings.

must_include (MustInclude)

Semantic: judged on meaning by the LLM judge, not by exact words.

assertions:
  - type: must_include
    text: "confirm date and time"

must_not (MustNot)

Semantic: the agent must never show this behaviour.

assertions:
  - type: must_not
    text: "hallucinate availability"

goal_reached (GoalReached)

Semantic: did the caller reach the scenario (or variant) goal?

No other field.

assertions:
  - type: goal_reached

llm_check (LlmCheck)

Semantic: a yes/no question the judge answers from the transcript.

assertions:
  - type: llm_check
    question: "Did the agent offer a callback?"

max_latency_ms (MaxLatency)

Deterministic. Latency of one turn = start of target speech minus end of caller speech. percentile picks p50, p95 or the max over the call.

assertions:
  - type: max_latency_ms
    value: 1500
    percentile: p95

max_duration_s (MaxDuration)

Deterministic. Whole call, from dialing to the last event.

assertions:
  - type: max_duration_s
    value: 180
    severity: warn

max_interruptions (MaxInterruptions)

Deterministic. Counts the times the target talked over the caller.

assertions:
  - type: max_interruptions
    value: 1

transcript_regex (TranscriptRegex)

Deterministic. Python regular expression, case insensitive, searched in each turn of speaker. expect_match: false asserts that nothing matches.

assertions:
  - type: transcript_regex
    pattern: "booking (reference|number)"
    speaker: target

Short form

The format shown on callqa.dev is accepted as is. On import it is folded into the typed grammar:

This short document:

name: "Book appointment - edge case"
phone: "+14155550100"
persona: "Impatient customer, speaks fast, has accent"
goal: "Book a haircut for tomorrow 2pm"
must_include:
  - "confirm date and time"
must_not:
  - "hallucinate availability"
max_latency_ms: 2000

is stored as (with every default filled in):

schema_version: 1
name: Book appointment - edge case
target:
  phone: '+14155550100'
  kind: unknown
  language: en-US
persona:
  description: Impatient customer, speaks fast, has accent
  speaking_rate: 1.0
  patience: normal
  interrupts: false
goal: Book a haircut for tomorrow 2pm
test_data: {}
steps: []
assertions:
- severity: fail
  type: must_include
  text: confirm date and time
- severity: fail
  type: must_not
  text: hallucinate availability
- severity: fail
  type: max_latency_ms
  value: 2000
  percentile: p95
limits:
  max_duration_s: 300
  silence_timeout_s: 20
  ring_timeout_s: 45
variants: []
tags: []

Extension fields x_*

Any key that starts with x_ is accepted at any level and kept verbatim (stored, exported, returned by the API). CallQA never interprets them. Use them for your own metadata (owner, ticket, region) or to try a field before it joins the grammar.

name: "Checkout flow"
target:
  phone: "+14155550100"
  x_region: "us-west"
persona:
  description: "Returning customer"
goal: "Buy the blue plan"
x_owner: "voice-team"
x_jira: "VOICE-142"

Complete examples

Real-world scenarios from examples/scenarios/. Each one is validated by the test suite.

• book-appointment.yaml: Book appointment - edge case (5 assertions, 4 variants)
• cancel-subscription.yaml: Cancel subscription (5 assertions, 2 variants)
• ivr-dtmf-navigation.yaml: IVR - check account balance (3 assertions, 1 variant)
• payment-test-card.yaml: Pay an invoice by card (6 assertions, 1 variant)
• reschedule-minimal.yaml: Reschedule a dentist appointment (1 assertion, 0 variants)
• spanish-language.yaml: Spanish line - order status (4 assertions, 1 variant)
• transfer-to-human.yaml: Escalate to a human agent (5 assertions, 1 variant)