CI runner

The headless runner executes scenario and suite manifests exported from the desktop app, including recorded scenarios, and writes JSON, JUnit XML and HTML reports, from the command line, a Docker image or a GitHub Action.

Use the runner only on sites you own or are contracted to test. The full reference is in docs/CI-RUNNER.md.

Export a manifest

In the desktop app, open QA automation and use:

• Export for CI on a scenario. The manifest contains scenario, profile and optionally a matrix (engines, devices, targets, concurrency, retries).
• Export suite for CI on a suite. It includes the suite's scenarios, profiles and workspace environments.
• Export approved baselines for CI for visual comparisons, which writes a .qavb pack. Approve baselines on the same operating system and browser versions the CI uses (for example in the Linux app with the same bundled engines); otherwise comparisons fail. How baselines are reviewed and kept per browser and device is described in approved baselines in CI.

Manifests are executable configuration. Never commit real form data or credentials in them; URLs with embedded usernames or passwords are rejected.

Command line

From a clone of the repository:

npm ci
npm run build
npm run browsers:install
npm run qa -- --config /path/to/scenario.json --output qa-results

Examples:

npm run qa -- --config suite.json --environment "Staging" --output qa-results
npm run qa -- --config scenario.json --baselines approved.qavb --output qa-results

The runner uses a temporary in-memory database and fresh browser contexts. It never reads the desktop app's credential vault, loads no .env file, and does not download browsers itself. Imported baselines are temporary copies; missing or changed comparisons are test failures. Baselines are never approved by the runner.

Outputs and exit codes

The output directory gets results.json, results.xml (JUnit), results.html and an artifacts/ directory with screenshots and diffs.

Configuration errors name the failing field and the reason, for example QA configuration error: Invalid manifest: matrix.engines.0: Invalid option: …. Values from the manifest or the environment are never echoed.

Proxy credentials

Leave all proxy variables unset for a direct (unproxied) run. For profiles that use a built-in provider, pass credentials through environment variables, set from your CI secret store:

DATAIMPULSE_PROXY_HOST, DATAIMPULSE_PROXY_PORT, DATAIMPULSE_PROXY_USERNAME and DATAIMPULSE_PROXY_PASSWORD remain supported as an alias for QA_PROVIDER=dataimpulse; when any QA_PROVIDER* variable is set they are ignored with a warning. Incomplete variables, or credentials for another provider than the manifest uses, exit with code 2 naming the variables, never their values. Every credential variable is removed from the process environment before browsers launch.

Site access tokens are a desktop-only feature and are not available to the runner. Allowlist CI traffic on your site instead.

Docker image

The runner image ghcr.io/ubaidbinwaris/proxy-qa-runner is published for linux/amd64. Its only browsers are Chromium, Firefox and WebKit; it also contains Node 24, the runner and the MCP server. It is built on the official Playwright image, contains no credentials and runs as the non-root user pwuser. Installed vendor browsers (Chrome, Edge, Brave and others) are not included; use chromium, firefox or webkit.

docker run --rm --init --ipc=host \
  --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -v "$PWD:/work" \
  ghcr.io/ubaidbinwaris/proxy-qa-runner:1.5.2 \
  --config /work/suite.json --environment Staging --output /work/qa-results

• --init reaps browser processes; --ipc=host avoids Chromium running out of shared memory.
• --user "$(id -u):$(id -g)" -e HOME=/tmp makes the reports belong to you.
• Add --network host (Linux) when the site under test runs on the host's localhost.
• Paths in results.json are container paths. Mount the directory at the same path on both sides (-v "$PWD:$PWD" -w "$PWD") to keep them valid on the host.

Pass proxy settings by name, so values never appear on the command line or in shell history:

export QA_PROVIDER_USERNAME=... QA_PROVIDER_PASSWORD=...   # from your secret store
docker run --rm --init --ipc=host -e QA_PROVIDER -e QA_PROVIDER_HOST -e QA_PROVIDER_PORT \
  -e QA_PROVIDER_USERNAME -e QA_PROVIDER_PASSWORD -v "$PWD:/work" \
  ghcr.io/ubaidbinwaris/proxy-qa-runner:1.5.2 --config /work/suite.json

Pin the image to a release tag or digest. To build it locally, see docs/CI-RUNNER.md.

GitHub Action

To run browser tests in GitHub Actions, use the composite action UbaidBinWaris/proxy_browser/action. It runs the image on Linux runners with Docker (GitHub-hosted ubuntu-* runners qualify), uploads the report directory and fails the step when the exit code is not 0.

jobs:
  qa:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - uses: UbaidBinWaris/proxy_browser/action@v1.5.2
        with:
          config: qa/signup-suite.json
          environment: Staging
        env:
          QA_PROVIDER_USERNAME: ${{ secrets.QA_PROVIDER_USERNAME }}
          QA_PROVIDER_PASSWORD: ${{ secrets.QA_PROVIDER_PASSWORD }}

Outputs: exit-code (the runner's exit code; 125 means Docker could not start the container) and results-json (the absolute path of results.json). To decide in a later step, set continue-on-error: true and read steps.<id>.outputs.exit-code.

Credentials are never inputs. Put them in the step's env: from secrets.*. The action forwards every set variable in the QA_PROVIDER*, DATAIMPULSE_PROXY_* and QA_PROXY_* families by name and masks their values in the log. Paths outside the workspace are rejected.

A complete consumer workflow is in examples/ci/github-workflow.yml.

Security notes

• Secrets go only through environment variables. Never write them into manifests, with: inputs, URLs or committed files.
• Reports can contain screenshots of your forms and the test data you typed. Use synthetic data and keep artifact retention short.
• Do not run the action on pull_request_target with untrusted code.

Try the sample

The repository's examples/ci/ folder contains a static form, a direct-connection scenario manifest and the consumer workflow:

python3 -m http.server 8080 --bind 127.0.0.1 --directory examples/ci/site &
docker run --rm --init --ipc=host --network host --user "$(id -u):$(id -g)" -e HOME=/tmp \
  -v "$PWD:$PWD" -w "$PWD" ghcr.io/ubaidbinwaris/proxy-qa-runner:1.5.2 \
  --config examples/ci/scenario.json --output qa-results

Expected: passed: 1/1 cases, exit code 0, and qa-results/results.{json,xml,html}.

Common questions

How do I pass proxy credentials in CI?

Leave all proxy variables unset for a direct run. For profiles that use a built-in provider, pass the credentials as QA_PROVIDER* environment variables (or the older DATAIMPULSE_PROXY_* alias) from your CI secret store; QA_PROXY_* instead sets one custom gateway for the whole run. See Proxy credentials.