Running in CI

Visual tests in CI differ from other tests in two ways: they produce files that a human must see, and they are sensitive to the machine they run on. Both shape the pipeline.

Run in the pinned environment

The CI job should run the comparison in the same container image that produced the baselines (why). In GitHub Actions that is a container: on the job or a Docker step; the same applies elsewhere. Pin the image by tag or digest and update baselines when you update it.

Keep the images

A failed comparison leaves actual and diff images on the runner's disk, which disappears with the job. Upload them as an artifact so a reviewer can see them: actions/upload-artifact on GitHub, artifacts: in GitLab, the equivalent elsewhere. Upload on failure at least; uploading always is simpler and lets you inspect passing runs. Include the baseline too, so the three images travel together, and include the manifest if the tool writes one, so the artifact describes itself.

A good artifact is self-contained: unpack it anywhere and a viewer (the one on this site, Playwright's HTML report, BackstopJS's report) can show the run.

Shard and merge

Large suites run across several machines, and each machine produces its own images and results. The reporting step then needs to merge them. Playwright's --shard with merge-reports, Vitest's sharding with its blob reporter, Cypress parallelisation through a dashboard service all handle their own results; a visual review step on top of them has to read every shard's output. This is the case the manifest was designed for: one file per writer, named visual-regression-manifest.<label>.json, found with one glob across all artifacts and merged with mergeManifests, with a re-run attempt winning over the earlier one.

Report on the pull request

Failing the job is necessary and not sufficient. The reviewer needs to see the diffs without leaving the PR. Options, cheapest first:

1. A job summary ($GITHUB_STEP_SUMMARY on GitHub) with the failed screenshot names and the artifact link.
2. A PR comment with the counts and, where the images are reachable, the baseline, actual and diff inline. Images in comments need a URL, which means uploading them somewhere public or using the artifact's download link.
3. A check run with annotations per failed screenshot, and a button that approves. This is what the hosted services and GitHub Apps provide.
4. The service's UI, linked from a status check, for tools with service-side baselines.

If your tool writes CTRF or a manifest (which converts to CTRF), the existing CTRF GitHub Actions post summaries and comments without any custom code.

Approve from CI

With repository baselines, the approval has to end as a commit. Two patterns work well: a workflow triggered by a PR comment (/approve-visuals) or a label, which downloads the artifact, copies the approved actual images over their baselines and pushes to the PR branch; or an app that does the same from a button on the check. Both need the paths of the images relative to the repository, which is what mergeManifests computes as repoPaths, and both must refuse paths that escape the repository.

Make sure the approval commit does not trigger an infinite loop (skip CI on it, or let it run and expect it to pass), and that the token used can push to forks only if you intend that.

Keep it fast and cheap

• Run visual tests on PRs, not on every push to every branch, unless they are fast.
• Screenshot components rather than whole pages where you can: smaller images, faster diffs, more precise failures.
• Cache the browser binaries and the renderer image.
• Compare locally in the runner before uploading anything; upload only the failures' images when the artifact gets big.
• For hosted services billed per screenshot, screenshot once per PR, not once per push, and skip the run when only backend code changed.

Further reading

• Storing and sharing data from a workflow in the GitHub Actions docs.
• Playwright sharding and merging reports.
• CTRF, the Common Test Report Format, and its GitHub Actions.