CI/CD integration

Run your voice tests on every deploy and fail the pipeline when quality drops below your threshold. There are two ways to wait for the result:

Both use the same endpoint and the same pass rule.

The API

POST /api/v1/scenarios/{scenario_id}/runs?wait=true&threshold=80&timeout=600

Scope: run. Body (all optional):

{
  "version": "v2.3.0",
  "git_sha": "9f2c1ab",
  "git_ref": "main",
  "environment": "staging",
  "tags": ["ci"],
  "variants": ["2b9d...", "8f41..."],
  "ci": true
}

Answers:

The run object has a ci block: {"threshold": 80, "passed": true}. passed is null while the run is not finished.

The pass rule

When the run is completed:

• with a threshold: passed if score >= threshold and no call has the verdict fail, ended in error or was blocked (for example by insufficient credit). One failed test case breaks the gate whatever the average score: a broken case cannot be averaged away by the passing ones. warn calls only lower the score.
• without a threshold: passed only if the run verdict is pass. A warn run fails the gate.

A canceled run never passes: it answers 412, not 202. Choose the threshold from your recent scores on the dashboard, a few points under your usual level, then raise it as the agent improves.

The API key needs the run scope to trigger the run and the read scope to read it back (the CLI and the GitHub Action list the calls of the run for their summary). write is not needed in CI.

Curl only

code=$(curl -sS -o run.json -w '%{http_code}' -X POST --max-time 960 \
  -H "Authorization: Bearer $CALLQA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"version": "'"$GIT_SHA"'", "git_sha": "'"$GIT_SHA"'", "tags": ["ci"]}' \
  "https://app.callqa.dev/api/v1/scenarios/$SCENARIO/runs?wait=true&timeout=900&threshold=80")
jq '{status, verdict, score, ci, url}' run.json
[ "$code" = "200" ] || exit 1

Do not rely on curl --fail alone: it treats 202 (still running) as a success, so a slow suite would pass the gate. Check the status code as above: only 200 passes.

The server waits asynchronously: a waiting request holds no worker and no database connection, so a timeout of 900 seconds costs nothing on our side.

Proxies and load balancers in front of your runner may cut a request that stays open for minutes. If that happens, use the CLI instead.

The CLI

examples/cli/callqa is one Python file with no dependency. Copy it into your repository (for example ci/callqa):

mkdir -p ci && curl -sSfo ci/callqa https://app.callqa.dev/docs/examples/cli/callqa

export CALLQA_API_KEY=...
python3 ci/callqa run --scenario "$SCENARIO" --threshold 80 \
  --code-version v2.3.0 --git-sha "$GIT_SHA" --environment staging --tag ci \
  --wait --summary callqa-summary.md

--code-version, --git-sha, --git-ref and --environment map to the body fields above (--release is still accepted as the former name of --code-version). It triggers the run, polls GET /runs/{id} every 10 s (--poll), retries on 429 and on network errors while polling, then writes a Markdown summary and exits with:

Other commands: callqa me, callqa status RUN_ID, callqa bundle export CALL_ID, callqa replay --call CALL_ID --wait, callqa scenario import file.yaml.

GitHub Actions

examples/ci/github-action/ is a composite action. Copy it to .github/actions/callqa/ and copy the CLI next to its action.yml (.github/actions/callqa/callqa). Then:

jobs:
  voice-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 40
    steps:
      - uses: actions/checkout@v4
      - id: callqa
        uses: ./.github/actions/callqa
        with:
          api-key: ${{ secrets.CALLQA_API_KEY }}
          scenario-id: ${{ vars.CALLQA_SCENARIO_ID }}
          threshold: "80"
          version: ${{ github.sha }}
          git-sha: ${{ github.sha }}
          environment: staging
          tags: "ci,${{ github.ref_name }}"
      - if: always()
        run: echo "Report ${{ steps.callqa.outputs.run-url }}"

Outputs: run-id, run-url, verdict, score, passed. The action writes the summary table to the job summary page.

Full workflow: examples/ci/github-workflow.yml. Curl-only workflow: examples/ci/github-workflow-curl.yml.

GitLab CI

voice-test:
  stage: voice-test
  image: python:3.12-slim
  needs: ["deploy-staging"]
  timeout: 40m
  script:
    - python3 ci/callqa run --scenario "$CALLQA_SCENARIO_ID" --threshold 80
        --code-version "$CI_COMMIT_SHORT_SHA" --git-sha "$CI_COMMIT_SHA" --tag gitlab
        --wait --summary callqa-summary.md
  artifacts:
    when: always
    paths: [callqa-summary.md]

Store CALLQA_API_KEY as a masked, protected CI/CD variable. Full file: examples/ci/gitlab-ci.yml.

Jenkins

stage('Voice tests') {
  environment { CALLQA_API_KEY = credentials('callqa-api-key') }
  steps {
    sh 'python3 ci/callqa run --scenario "$CALLQA_SCENARIO_ID" --threshold 80 --code-version "$GIT_COMMIT" --wait'
  }
}

Full file: examples/ci/Jenkinsfile.

Tips

• Run after the deploy, against the environment you just deployed: pass environment (for example staging) so the call dials that environment's number. See Environments.
• One scenario per pipeline job, or run several jobs in parallel. Calls of one run are already placed in parallel, up to your plan's concurrent call limit.
• Budget time: a call usually lasts 1 to 5 minutes, bounded by limits.max_duration_s. Set the CI job timeout above your longest call plus queueing.
• Name the version (version: v2.3.0 or the commit SHA): the Dashboard puts a marker on the score curve for each version, and Versions and environments compares them.
• Every CI run uses quota: calls beyond your monthly quota are paid from the prepaid wallet. A call the wallet cannot cover is not placed and fails the gate. See Billing.
• Subset for fast feedback: pass variants to run a smoke subset on every commit, and the full suite nightly with a schedule (one per environment, see Writing scenarios).
• Prefer being notified over polling for long suites: Webhooks.