# Concepts

```
Target (phone number, verified)
  └── Scenario (persona, goal, assertions, test data)
        ├── Variant #1..#N (test cases, AUTO or MANUAL)
        └── Environment plans (where it runs, schedule and call window per environment)

Environment (development, staging, production)
  └── Version (v2.3.0, a git sha: the code deployed there)

Run #247 (one run of a scenario in one environment: manual, API, CI, schedule or replay)
  └── Call (one real phone call per variant)
        ├── Trace (append-only events: turns, DTMF, interruptions, hangup...)
        └── Result (verdict per assertion, score 0-100, metrics)
```

In the dashboard, **Scenarios** holds scenarios and their test cases, **Monitoring** the runs,
calls, phone numbers and reports, and **Dashboard** the overview and the **Versions and
environments** board.

## Target

A **target** is a phone number under test, in E.164 format (`+14155550100`). It belongs to your
organization and has a `kind` (`ai_agent`, `ivr`, `iva`, `human`, `unknown`), an optional
`platform` (free text: vapi, retell, genesys...) and a `language`.

When registered, the number is classified by country and line type. Premium-rate, shared-cost,
emergency and unclassified numbers are refused. A new number is `unverified`: **manual, scheduled
and CI runs need a verified number** (`verification_status: verified`, a code in its greeting, see
[Number verification](number-verification.md)). Other statuses: `pending` (code issued),
`expired` (after 180 days), `locked` (3 failed attempts in 24 hours), `suspended`.

The phone number of a target never changes. To test another number, register a new target.
A number used in a scenario for the first time is registered automatically as `unverified`.

API: `/targets`. See the [API reference](api-reference.md#targets).

## Scenario

A **scenario** describes one thing to test on one target, in a versioned grammar:

- who calls (`persona`) and what they want (`goal`);
- optional `test_data` the caller uses (customer id, test card, address);
- optional `steps` to navigate an IVR (say, press DTMF keys, expect, wait);
- `assertions`: the pass/fail checks;
- `limits` (max duration, silence and ring timeouts);
- optional `schedule`, `timezone` and `call_window` for recurring runs, and per environment
  plans in `environments`;
- `tags`, and its `variants`.

You write it in YAML or JSON, in the dashboard or through the API. Scenario names are unique
among the non-archived scenarios of your organization.
See [Writing scenarios](scenarios.md) and the [grammar reference](reference/scenario-grammar.md).

## Variant (test case)

A **variant** is one test case of a scenario, numbered #1..#N. It can override the persona, the
goal, add `instructions`, merge extra `test_data`, and **append** assertions. Each variant has a
stable id, kept when you edit the scenario (variants are matched by title), so its history
stays comparable across runs.

- `source: auto`: derived by CallQA from the scenario (edge cases, personas, accents, error
  paths) with `POST /scenarios/{id}/variants/derive`. Needs a plan with auto-derived coverage.
- `source: manual`: written by you, or created from a real call with
  "Add to regression suite" (`POST /scenarios/{id}/variants/from-bundle`).

A disabled variant (`enabled: false`) is skipped by runs. A scenario without variants runs as a
single call.

## Run

A **run** is one execution of a scenario: one call per enabled variant, or the
subset you pick. Runs are numbered per organization (`number`, shown as "Run #247"). Webhook
event names keep the older name `suite_run.*` (and the `suite_run_id` field): a suite run is a run.

| Field | Values |
|---|---|
| `trigger` | `manual` (dashboard), `api`, `ci`, `schedule`, `replay` |
| `status` | `queued`, `running`, `completed`, `canceled` |
| `verdict` | `null` until completed, then `pass`, `warn` or `fail` |

Counters: `total`, `passed`, `failed`, `warned`, `errored`, `blocked`, plus `score` and
`cost_cents`. The run verdict is:

- `fail` if any call failed, errored or was blocked;
- else `warn` if any call is `warn`;
- else `pass`.

The run `score` is the average of the scores of its scored calls, rounded.

A run never stays `running` forever: a call with no result after its maximum duration plus a
margin ends as `error`, and the run completes (with verdict `fail`) once all its calls are
finished. A call still waiting for its call window keeps its run open.

Canceling a run (`POST /runs/{id}/cancel`) cancels the calls not dialed yet. Calls already in
progress finish normally.

## Call

A **call** is one real phone call placed by the CallQA caller to your target. Its `status`
goes through:

```
queued -> dispatching -> dialing -> ringing -> in_progress -> judging -> completed
```

It can also end as `error` (the call failed on the way), `canceled`, or `blocked_policy` /
`blocked_insufficient_credit` when it was never placed. A blocked call has a `block_reason`.
See [call statuses and block reasons](errors.md#call-statuses-and-block-reasons).

Before dialing, CallQA checks, in this order: organization not closed nor suspended, number not
archived nor suspended, number verified (or the sandbox limits of an unverified number),
destination allowed, country included in your plan, concurrency limits, calls per hour to this
number, the call window of the scenario in the run's environment (scheduled runs only), then
reserves credit (monthly quota, then the wallet if overage is on).

While a test call is in progress, anyone in the organization can **Listen live** on the call
page: both voices (or only the caller, or only your agent) are relayed to the browser.
Listening records nothing. Key presses (DTMF) are not played in the live audio: the page shows
them as they happen ("Test caller pressed 1"), next to the button and in the conversation. The
status badge follows the call (`dialing`, `ringing`, `in_progress`, `judging`), and the page
reloads with the verdict when the call ends. **Export bundle** and **Replay this call** appear
once the call is finished.

## Trace

The **trace** is the append-only list of events of a call. It is the proof of what happened,
and it is always stored (the audio recording is optional). Metrics and verdicts are computed
from the trace only, so they can always be recomputed.

The trace, the transcript, the detailed checks and the audio are kept for your plan's
retention period. After it, or when an admin deletes the call history on the **Data and
privacy** page, only the score, the verdict, the latencies and the check names stay for the
dashboards.

Each event has `seq` (monotonic per call), `t_ms` (milliseconds since dialing), `at` (UTC wall
clock), `type` and `data`. Main types:

| `type` | `data` |
|---|---|
| `turn` | `speaker` (`caller` or `target`), `text`, `start_ms`, `end_ms` |
| `interruption` | `by`, `over`, `at_ms`, `overlap_ms` |
| `dtmf_sent` / `dtmf_received` | `digits` / `digit` |
| `silence` | `duration_ms`, `speaker` |
| `hangup` | `by` (`caller`, `target`, `system`), `reason`, `sip_status` |
| `error` | `where`, `message`, `retryable` |

Lifecycle events: `dispatched`, `dialing`, `ringing`, `answered`. Debug events:
`end_of_turn`, `tool_call`, `llm_metrics`, `stt_metrics`, `tts_metrics`, `note`.

API: `GET /calls/{id}/events` (paginated, `after_seq` to fetch only new events).

## Verdict

After the call, each assertion gets a **verdict**: `passed` (true/false), `severity`
(`fail` or `warn`), `evidence` (a quote from the transcript) and `evidence_ms` (when it was
said), plus a `detail`.

Implicit checks are added to yours: `answered` (severity `fail`) on every call, and with the
`callqa-rubric-1` judge, warn-level checks `no_premature_hangup` (the target hung up on the
caller), `max_silence` (a target silence longer than `limits.silence_timeout_s`),
`dtmf_steps_sent` (when the scenario has DTMF steps) and `no_hallucination` (claims that
contradict `test_data` or the goal). Every quote returned by the LLM judge is checked against the
transcript in code: a quote that does not exist cannot fail a call on its own.

The call verdict (`verdict` on the call) is:

- `error`: the call could not be completed or judged;
- `fail`: at least one `severity: fail` check failed;
- `warn`: only `severity: warn` checks failed;
- `pass`: everything passed.

Deterministic assertions (`max_latency_ms`, `max_duration_s`, `max_interruptions`,
`transcript_regex`) are computed from the trace timings and text. Semantic ones
(`must_include`, `must_not`, `goal_reached`, `llm_check`) are judged by an LLM judge that
must quote the transcript.

## Score

Each judged call gets a **score** from 0 to 100, computed in code from its verdicts (never
chosen by the LLM). The call result carries `rubric_version`, the version of the scoring grid:
scores are only comparable within the same rubric version. With the `callqa-rubric-1` grid:

- each check has a weight by type (for example `goal_reached` 25, `must_include` 15,
  `max_latency_ms` 15, `llm_check` 10); a `warn` check counts half;
- score = weighted share of passed checks;
- an unanswered call scores 0; a call that could not be judged has no score;
- a call with a failed `fail` check is capped at 60, so a failing call never scores higher
  than a passing one.

## Environment

An **environment** is where a version of your voice agent runs: `development`, `staging`,
`production`. Every organization has one default environment, used by runs and versions that
name none (scheduled runs of older scenarios, older CI jobs).

- A scenario is **activated** in the environments where it should run, each with its own
  schedule, time zone and call window. On the scenario page, or with
  `PUT /scenarios/{id}/environments/{env}`.
- A run belongs to one environment. A number can be reached at another number in an
  environment (the staging line of the same agent).

See [Environments](environments.md).

## Version

A **version** is the code of your voice agent deployed in one environment: a label
(`v2.3.0`, a short sha), optional git ref, sha and link, notes and tags, and the moment it went
live (`deployed_at`). Runs of the environment placed after that moment are attributed to it
until the next version, unless the trigger names a version. Versions show as markers on the
dashboard, so you can see which deployment changed the score.

Send `"version": "v2.3.0"` (with `git_sha`, `git_ref`) when you trigger a run from CI: the
version is created in the run's environment if it does not exist yet. Manage them with
`/versions` or on the **Versions and environments** board; **Promote** copies a version to the
next environment. Versions replace the former **releases**: `/releases` and the `release` field
still work as deprecated aliases until 2027-03-31.

## Tag

A **tag** is a free keyword (up to 40 characters, with a color): a code name, a customer, a
team. CallQA gives it no meaning of its own; it is a filter on the dashboards, the board, the
run lists and the webhook endpoints. Tags go on:

- scenarios, from the `tags` list of the scenario document or in place on the scenario page;
- runs, from the `tags` of the trigger request (`["ci", "nightly"]`);
- versions and phone numbers.

Missing tags are created on the fly. Tag or untag many objects at once with `POST /tags/bulk`.
Filter with `?tag=` (`GET /scenarios?tag=booking`, `/runs`, `/versions`). Rename, recolor or
delete them with `/tags/{id}` or on the **Tags** page; renaming keeps the tag everywhere.

## Organization, members and roles

Everything belongs to an **organization**. People join as members with a role:

| Role | Can |
|---|---|
| `admin` | everything, including billing, members, API keys, webhooks and hard deletes |
| `member` | create and edit scenarios, numbers, tags, versions, environments and reports; trigger runs and replays; verify numbers |
| `guest` | read only: dashboards, reports, runs, calls |

API keys belong to the organization, not to a person. See [Authentication](authentication.md).