Concepts

Target (phone number, verified)
  └── Scenario (persona, goal, assertions, test data)
        ├── Variant #1..#N (test cases, AUTO or MANUAL)
        └── Environment plans (where it runs, schedule and call window per environment)

Environment (development, staging, production)
  └── Version (v2.3.0, a git sha: the code deployed there)

Run #247 (one run of a scenario in one environment: manual, API, CI, schedule or replay)
  └── Call (one real phone call per variant)
        ├── Trace (append-only events: turns, DTMF, interruptions, hangup...)
        └── Result (verdict per assertion, score 0-100, metrics)

In the dashboard, Scenarios holds scenarios and their test cases, Monitoring the runs, calls, phone numbers and reports, and Dashboard the overview and the Versions and environments board.

Target

A target is a phone number under test, in E.164 format (+14155550100). It belongs to your organization and has a kind (ai_agent, ivr, iva, human, unknown), an optional platform (free text: vapi, retell, genesys...) and a language.

When registered, the number is classified by country and line type. Premium-rate, shared-cost, emergency and unclassified numbers are refused. A new number is unverified: manual, scheduled and CI runs need a verified number (verification_status: verified, a code in its greeting, see Number verification). Other statuses: pending (code issued), expired (after 180 days), locked (3 failed attempts in 24 hours), suspended.

The phone number of a target never changes. To test another number, register a new target. A number used in a scenario for the first time is registered automatically as unverified.

API: /targets. See the API reference.

Scenario

A scenario describes one thing to test on one target, in a versioned grammar:

• who calls (persona) and what they want (goal);
• optional test_data the caller uses (customer id, test card, address);
• optional steps to navigate an IVR (say, press DTMF keys, expect, wait);
• assertions: the pass/fail checks;
• limits (max duration, silence and ring timeouts);
• optional schedule, timezone and call_window for recurring runs, and per environment plans in environments;
• tags, and its variants.

You write it in YAML or JSON, in the dashboard or through the API. Scenario names are unique among the non-archived scenarios of your organization. See Writing scenarios and the grammar reference.

Variant (test case)

A variant is one test case of a scenario, numbered #1..#N. It can override the persona, the goal, add instructions, merge extra test_data, and append assertions. Each variant has a stable id, kept when you edit the scenario (variants are matched by title), so its history stays comparable across runs.

• source: auto: derived by CallQA from the scenario (edge cases, personas, accents, error paths) with POST /scenarios/{id}/variants/derive. Needs a plan with auto-derived coverage.
• source: manual: written by you, or created from a real call with "Add to regression suite" (POST /scenarios/{id}/variants/from-bundle).

A disabled variant (enabled: false) is skipped by runs. A scenario without variants runs as a single call.

Run

A run is one execution of a scenario: one call per enabled variant, or the subset you pick. Runs are numbered per organization (number, shown as "Run #247"). Webhook event names keep the older name suite_run.* (and the suite_run_id field): a suite run is a run.

Counters: total, passed, failed, warned, errored, blocked, plus score and cost_cents. The run verdict is:

• fail if any call failed, errored or was blocked;
• else warn if any call is warn;
• else pass.

The run score is the average of the scores of its scored calls, rounded.

A run never stays running forever: a call with no result after its maximum duration plus a margin ends as error, and the run completes (with verdict fail) once all its calls are finished. A call still waiting for its call window keeps its run open.

Canceling a run (POST /runs/{id}/cancel) cancels the calls not dialed yet. Calls already in progress finish normally.

Call

A call is one real phone call placed by the CallQA caller to your target. Its status goes through:

queued -> dispatching -> dialing -> ringing -> in_progress -> judging -> completed

It can also end as error (the call failed on the way), canceled, or blocked_policy / blocked_insufficient_credit when it was never placed. A blocked call has a block_reason. See call statuses and block reasons.

Before dialing, CallQA checks, in this order: organization not closed nor suspended, number not archived nor suspended, number verified (or the sandbox limits of an unverified number), destination allowed, country included in your plan, concurrency limits, calls per hour to this number, the call window of the scenario in the run's environment (scheduled runs only), then reserves credit (monthly quota, then the wallet if overage is on).

While a test call is in progress, anyone in the organization can Listen live on the call page: both voices (or only the caller, or only your agent) are relayed to the browser. Listening records nothing. Key presses (DTMF) are not played in the live audio: the page shows them as they happen ("Test caller pressed 1"), next to the button and in the conversation. The status badge follows the call (dialing, ringing, in_progress, judging), and the page reloads with the verdict when the call ends. Export bundle and Replay this call appear once the call is finished.

Trace

The trace is the append-only list of events of a call. It is the proof of what happened, and it is always stored (the audio recording is optional). Metrics and verdicts are computed from the trace only, so they can always be recomputed.

The trace, the transcript, the detailed checks and the audio are kept for your plan's retention period. After it, or when an admin deletes the call history on the Data and privacy page, only the score, the verdict, the latencies and the check names stay for the dashboards.

Each event has seq (monotonic per call), t_ms (milliseconds since dialing), at (UTC wall clock), type and data. Main types:

Lifecycle events: dispatched, dialing, ringing, answered. Debug events: end_of_turn, tool_call, llm_metrics, stt_metrics, tts_metrics, note.

API: GET /calls/{id}/events (paginated, after_seq to fetch only new events).

Verdict

After the call, each assertion gets a verdict: passed (true/false), severity (fail or warn), evidence (a quote from the transcript) and evidence_ms (when it was said), plus a detail.

Implicit checks are added to yours: answered (severity fail) on every call, and with the callqa-rubric-1 judge, warn-level checks no_premature_hangup (the target hung up on the caller), max_silence (a target silence longer than limits.silence_timeout_s), dtmf_steps_sent (when the scenario has DTMF steps) and no_hallucination (claims that contradict test_data or the goal). Every quote returned by the LLM judge is checked against the transcript in code: a quote that does not exist cannot fail a call on its own.

The call verdict (verdict on the call) is:

• error: the call could not be completed or judged;
• fail: at least one severity: fail check failed;
• warn: only severity: warn checks failed;
• pass: everything passed.

Deterministic assertions (max_latency_ms, max_duration_s, max_interruptions, transcript_regex) are computed from the trace timings and text. Semantic ones (must_include, must_not, goal_reached, llm_check) are judged by an LLM judge that must quote the transcript.

Score

Each judged call gets a score from 0 to 100, computed in code from its verdicts (never chosen by the LLM). The call result carries rubric_version, the version of the scoring grid: scores are only comparable within the same rubric version. With the callqa-rubric-1 grid:

• each check has a weight by type (for example goal_reached 25, must_include 15, max_latency_ms 15, llm_check 10); a warn check counts half;
• score = weighted share of passed checks;
• an unanswered call scores 0; a call that could not be judged has no score;
• a call with a failed fail check is capped at 60, so a failing call never scores higher than a passing one.

Environment

An environment is where a version of your voice agent runs: development, staging, production. Every organization has one default environment, used by runs and versions that name none (scheduled runs of older scenarios, older CI jobs).

• A scenario is activated in the environments where it should run, each with its own schedule, time zone and call window. On the scenario page, or with PUT /scenarios/{id}/environments/{env}.
• A run belongs to one environment. A number can be reached at another number in an environment (the staging line of the same agent).

See Environments.

Version

A version is the code of your voice agent deployed in one environment: a label (v2.3.0, a short sha), optional git ref, sha and link, notes and tags, and the moment it went live (deployed_at). Runs of the environment placed after that moment are attributed to it until the next version, unless the trigger names a version. Versions show as markers on the dashboard, so you can see which deployment changed the score.

Send "version": "v2.3.0" (with git_sha, git_ref) when you trigger a run from CI: the version is created in the run's environment if it does not exist yet. Manage them with /versions or on the Versions and environments board; Promote copies a version to the next environment. Versions replace the former releases: /releases and the release field still work as deprecated aliases until 2027-03-31.

Tag

A tag is a free keyword (up to 40 characters, with a color): a code name, a customer, a team. CallQA gives it no meaning of its own; it is a filter on the dashboards, the board, the run lists and the webhook endpoints. Tags go on:

• scenarios, from the tags list of the scenario document or in place on the scenario page;
• runs, from the tags of the trigger request (["ci", "nightly"]);
• versions and phone numbers.

Missing tags are created on the fly. Tag or untag many objects at once with POST /tags/bulk. Filter with ?tag= (GET /scenarios?tag=booking, /runs, /versions). Rename, recolor or delete them with /tags/{id} or on the Tags page; renaming keeps the tag everywhere.

Organization, members and roles

Everything belongs to an organization. People join as members with a role:

API keys belong to the organization, not to a person. See Authentication.