# CI/CD integration

Run your voice tests on every deploy and **fail the pipeline when quality drops below your
threshold**. There are two ways to wait for the result:

| | Server waits (`?wait=true`) | Client polls (CLI, GitHub Action) |
|---|---|---|
| Tools | one `curl` | `examples/cli/callqa` (Python 3.9+, no dependency) |
| Longest wait | 900 s | as long as you want (`--timeout`) |
| Summary | JSON of the run | Markdown table per test case, job outputs |
| Best for | short suites, minimal setup | real suites, GitHub summaries, flaky networks |

Both use the same endpoint and the same pass rule.

## The API

```
POST /api/v1/scenarios/{scenario_id}/runs?wait=true&threshold=80&timeout=600
```

Scope: `run`. Body (all optional):

```json
{
  "version": "v2.3.0",
  "git_sha": "9f2c1ab",
  "git_ref": "main",
  "environment": "staging",
  "tags": ["ci"],
  "variants": ["2b9d...", "8f41..."],
  "ci": true
}
```

| Parameter | Where | Meaning |
|---|---|---|
| `wait` | query | `true`: hold the request until the run finishes. Default `false`. |
| `timeout` | query | Seconds to wait, default `600`, capped at `900`. |
| `threshold` | query | Minimum run score, 0 to 100. Stored on the run. |
| `version` | body | Version of your code under test (`v2.3.0`, a short sha), max 60 characters. Looked up in the run's environment, created there if new. Without it the run belongs to the version live in the environment. |
| `git_sha`, `git_ref`, `source_url` | body | Commit, branch or tag, and pipeline page of that version, stored when the version is created. |
| `environment` | body | Environment by slug, name or id (`staging`). Default: your default environment. It decides the number dialed. |
| `release` | body | **Deprecated**: the former name of `version`, read when `version` is absent. See [Versioning](versioning.md). |
| `tags` | body | Free keywords added to the run. Created if new. |
| `variants` | body | Only these enabled variants. Default: all enabled variants. |
| `ci` | body | Mark the run as triggered by CI (`trigger: "ci"`). Implied by `wait=true`. |

Answers:

| Status | When | Body |
|---|---|---|
| `201` | Without `wait`: the run is queued | the run |
| `200` | With `wait`: finished and **passed** | the run |
| `412` | With `wait`: finished and **failed** | the run |
| `202` | With `wait`: still running at `timeout` | the run: poll `GET /runs/{id}` |
| `400` | `threshold` out of range (`invalid_threshold`), unknown environment, scenario archived or no matching enabled variant (`run_not_possible`) | error |
| `404` | Unknown scenario | error |

The run object has a `ci` block: `{"threshold": 80, "passed": true}`. `passed` is `null` while
the run is not finished.

## The pass rule

When the run is `completed`:

- **with a threshold**: passed if `score >= threshold` **and** no call has the verdict `fail`,
  ended in `error` or was blocked (for example by insufficient credit). One failed test case
  breaks the gate whatever the average score: a broken case cannot be averaged away by the
  passing ones. `warn` calls only lower the score.
- **without a threshold**: passed only if the run verdict is `pass`. A `warn` run fails the gate.

A canceled run never passes: it answers `412`, not `202`. Choose the threshold from your recent
scores on the dashboard, a few points under your usual level, then raise it as the agent
improves.

The API key needs the `run` scope to trigger the run and the `read` scope to read it back (the
CLI and the GitHub Action list the calls of the run for their summary). `write` is not needed
in CI.

## Curl only

```bash
code=$(curl -sS -o run.json -w '%{http_code}' -X POST --max-time 960 \
  -H "Authorization: Bearer $CALLQA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"version": "'"$GIT_SHA"'", "git_sha": "'"$GIT_SHA"'", "tags": ["ci"]}' \
  "https://app.callqa.dev/api/v1/scenarios/$SCENARIO/runs?wait=true&timeout=900&threshold=80")
jq '{status, verdict, score, ci, url}' run.json
[ "$code" = "200" ] || exit 1
```

Do not rely on `curl --fail` alone: it treats `202` (still running) as a success, so a slow
suite would pass the gate. Check the status code as above: only `200` passes.

The server waits asynchronously: a waiting request holds no worker and no database
connection, so a `timeout` of 900 seconds costs nothing on our side.

Proxies and load balancers in front of your runner may cut a request that stays open for
minutes. If that happens, use the CLI instead.

## The CLI

[`examples/cli/callqa`](../../examples/cli/) is one Python file with no dependency. Copy it
into your repository (for example `ci/callqa`):

```bash
mkdir -p ci && curl -sSfo ci/callqa https://app.callqa.dev/docs/examples/cli/callqa
```

```bash
export CALLQA_API_KEY=...
python3 ci/callqa run --scenario "$SCENARIO" --threshold 80 \
  --code-version v2.3.0 --git-sha "$GIT_SHA" --environment staging --tag ci \
  --wait --summary callqa-summary.md
```

`--code-version`, `--git-sha`, `--git-ref` and `--environment` map to the body fields above
(`--release` is still accepted as the former name of `--code-version`). It triggers the run, polls `GET /runs/{id}` every 10 s (`--poll`), retries on `429` and on
network errors while polling, then writes a Markdown summary and exits with:

| Exit code | Meaning |
|---|---|
| `0` | passed |
| `1` | quality gate failed (score under threshold, verdict fail, error or blocked call) |
| `2` | usage error (missing key, bad argument) |
| `3` | API or network error (bad key, missing scope, unknown scenario...) |
| `4` | timed out while waiting (`--timeout`, default 1800 s); the run continues |
| `5` | the run was canceled |

Other commands: `callqa me`, `callqa status RUN_ID`, `callqa bundle export CALL_ID`,
`callqa replay --call CALL_ID --wait`, `callqa scenario import file.yaml`.

## GitHub Actions

[`examples/ci/github-action/`](../../examples/ci/github-action/) is a composite action. Copy it
to `.github/actions/callqa/` and copy the CLI next to its `action.yml`
(`.github/actions/callqa/callqa`). Then:

```yaml
jobs:
  voice-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 40
    steps:
      - uses: actions/checkout@v4
      - id: callqa
        uses: ./.github/actions/callqa
        with:
          api-key: ${{ secrets.CALLQA_API_KEY }}
          scenario-id: ${{ vars.CALLQA_SCENARIO_ID }}
          threshold: "80"
          version: ${{ github.sha }}
          git-sha: ${{ github.sha }}
          environment: staging
          tags: "ci,${{ github.ref_name }}"
      - if: always()
        run: echo "Report ${{ steps.callqa.outputs.run-url }}"
```

| Input | Default | |
|---|---|---|
| `api-key` | required | Key with `read` and `run` scopes, from a secret |
| `scenario-id` | required | Scenario to run (all its enabled variants) |
| `threshold` | `80` | Empty: pass only on verdict `pass` |
| `version` | | Version of your code under test (created in the environment if new) |
| `git-sha` | | Commit of that version |
| `environment` | | Environment slug. Empty: the default environment |
| `release` | | Deprecated: use `version` |
| `tags` | | Comma-separated |
| `variants` | | Comma-separated variant ids |
| `wait` | `true` | `false`: trigger and return |
| `timeout` | `1800` | Seconds |
| `poll-interval` | `10` | Seconds |
| `api-url` | `https://app.callqa.dev/api/v1` | |

Outputs: `run-id`, `run-url`, `verdict`, `score`, `passed`. The action writes the summary table
to the job summary page.

Full workflow: [`examples/ci/github-workflow.yml`](../../examples/ci/github-workflow.yml).
Curl-only workflow: [`examples/ci/github-workflow-curl.yml`](../../examples/ci/github-workflow-curl.yml).

## GitLab CI

```yaml
voice-test:
  stage: voice-test
  image: python:3.12-slim
  needs: ["deploy-staging"]
  timeout: 40m
  script:
    - python3 ci/callqa run --scenario "$CALLQA_SCENARIO_ID" --threshold 80
        --code-version "$CI_COMMIT_SHORT_SHA" --git-sha "$CI_COMMIT_SHA" --tag gitlab
        --wait --summary callqa-summary.md
  artifacts:
    when: always
    paths: [callqa-summary.md]
```

Store `CALLQA_API_KEY` as a masked, protected CI/CD variable. Full file:
[`examples/ci/gitlab-ci.yml`](../../examples/ci/gitlab-ci.yml).

## Jenkins

```groovy
stage('Voice tests') {
  environment { CALLQA_API_KEY = credentials('callqa-api-key') }
  steps {
    sh 'python3 ci/callqa run --scenario "$CALLQA_SCENARIO_ID" --threshold 80 --code-version "$GIT_COMMIT" --wait'
  }
}
```

Full file: [`examples/ci/Jenkinsfile`](../../examples/ci/Jenkinsfile).

## Tips

- **Run after the deploy**, against the environment you just deployed: pass `environment`
  (for example `staging`) so the call dials that environment's number. See
  [Environments](environments.md).
- **One scenario per pipeline job**, or run several jobs in parallel. Calls of one run are
  already placed in parallel, up to your plan's concurrent call limit.
- **Budget time**: a call usually lasts 1 to 5 minutes, bounded by `limits.max_duration_s`.
  Set the CI job timeout above your longest call plus queueing.
- **Name the version** (`version: v2.3.0` or the commit SHA): the Dashboard puts a marker on
  the score curve for each version, and **Versions and environments** compares them.
- **Every CI run uses quota**: calls beyond your monthly quota are paid from the prepaid wallet.
  A call the wallet cannot cover is not placed and fails the gate. See [Billing](billing.md).
- **Subset for fast feedback**: pass `variants` to run a smoke subset on every commit, and the
  full suite nightly with a schedule (one per environment, see
  [Writing scenarios](scenarios.md)).
- Prefer being notified over polling for long suites: [Webhooks](webhooks.md).