-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
docs: Add Snapshots EA product documentation #17327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,33 @@ | ||
| --- | ||
| title: Snapshots | ||
| sidebar_order: 138 | ||
| description: Detect visual changes by comparing PNG snapshots against a baseline on every pull request. | ||
| new: true | ||
| --- | ||
|
|
||
| <Include name="feature-available-for-user-group-early-adopter" /> | ||
|
|
||
| Snapshots catches visual regressions before they reach users. Your CI pipeline generates PNG screenshots, uploads them to Sentry, and Sentry compares each image against a baseline from your main branch using pixel-level diffing. | ||
|
|
||
| When a snapshot changes, Sentry posts a failing GitHub Check on your pull request. You review the diff in Sentry's comparison viewer and approve or reject the changes. | ||
|
|
||
| TODO: some sort of image | ||
|
|
||
| ## How It Works | ||
|
|
||
| 1. **Generate snapshots** — Your CI job produces PNG screenshots however you like: Playwright, Paparazzi, or any tool that outputs PNGs. | ||
| 2. **Upload to Sentry** — `sentry-cli` uploads the snapshot directory along with Git metadata so Sentry can associate each upload with a commit. | ||
| 3. **Sentry diffs against the baseline** — Sentry compares uploaded snapshots against snapshots corresponding to the upload's `base_sha`. With Sentry, you do not have to manage "Golden" images and everything is determined via git metadata. | ||
| 4. **Results post to your PR** — Snapshot results post to your PR\*. You can configure settings to control status check and comment behavior. | ||
| 5. **Review and approve** — You can view snapshot results on the Sentry UI. If the Snapshot "failed" you can approve it from either the PR or the Sentry UI. | ||
|
|
||
| \*Note: Snapshots EA only supports Github | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - **Early Adopter access** — Enable the Early Adopter toggle in your [organization settings](/organization/early-adopter-features/). | ||
| - **Auth token** — A Sentry auth token with `project:write` scope (personal token) or `org:ci` scope (org-level token). | ||
| - **sentry-cli >= 3.3.4** — The `build snapshots` command requires version 3.3.4 or later. | ||
| - **GitHub integration** — Snapshots can be used with any VCS provider, but only Github is supported for status check integration and PR comments. Install the [Sentry GitHub App](/organization/integrations/source-code-mgmt/github/) and grant it access to your repository for PR integration. | ||
|
|
||
| <PageGrid /> |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| --- | ||
| title: Reviewing Snapshots | ||
| sidebar_order: 20 | ||
| description: Review snapshot diffs on your pull request and approve changes. | ||
| --- | ||
|
|
||
| <Include name="feature-available-for-user-group-early-adopter" /> | ||
|
|
||
| ## Status Checks | ||
|
|
||
| After you upload snapshots from a pull request branch, a **Snapshot Testing** GitHub Check appears on the pull request. If no snapshots changed, the check passes. If any snapshots were added, removed, or modified, the check fails and requires approval. | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we explain that the default is that added snapshots don't fail here?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. will have that in the settings section and maybe just a link to "if you want to change your failure settings..." |
||
|
|
||
| IMAGE: GitHub Check showing Snapshot Testing results on a pull request | ||
|
|
||
| ## Comparison Viewer | ||
|
|
||
| Click **Details** on the GitHub Check to open Sentry's comparison viewer. The viewer has two panels: | ||
|
|
||
| - **Sidebar** — Lists all snapshots grouped by status: modified, added, removed, renamed, and unchanged. Click a snapshot to view it. | ||
| - **Main panel** — Displays the selected snapshot's diff with three viewing modes: | ||
| - **Split** — Side-by-side view of base and current branch with a diff overlay highlighting changed pixels. | ||
| - **Wipe** — Drag a slider across the image to compare base and current branch. | ||
| - **Onion** — Overlay both images with an adjustable opacity slider. | ||
|
|
||
| IMAGE: Sentry Snapshot comparison viewer showing a modified image with split diff | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can we make this a gif of using the onion mode on an ogre ? that would make my day.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. lol we'll save that for the EA blog |
||
|
|
||
| ## Approve Changes | ||
|
|
||
| There are two ways to approve snapshot changes: | ||
|
|
||
| - Click the **Approve** button on the GitHub Check Run. | ||
| - Click **Approve** in the Sentry comparison viewer. | ||
|
|
||
| Approval resolves the failing check. If new changes are pushed to the PR, the check resets and requires approval again. | ||
|
|
||
| ## Settings | ||
|
|
||
| Configure snapshot behavior in **Project Settings > Mobile Builds > Snapshots**. | ||
|
|
||
| | Setting | Default | Description | | ||
| | ------------------------- | :-----: | ----------------------------------------------------- | | ||
| | Status Checks Enabled | On | Post GitHub Check Runs for snapshot comparisons. | | ||
| | Fail on Added Snapshots | Off | Require approval when new snapshots are added. | | ||
| | Fail on Removed Snapshots | On | Require approval when existing snapshots are removed. | | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| --- | ||
| title: Uploading Snapshots | ||
| sidebar_order: 10 | ||
| description: Structure your snapshot directory and upload from CI with sentry-cli. | ||
| --- | ||
|
|
||
| <Include name="feature-available-for-user-group-early-adopter" /> | ||
|
|
||
| Snapshots can be used with any frontend screenshot framework. We have examples in different platform documentaiton pages (TODO: EXAMPLES). This page covers the general upload structure and metadata schema. | ||
|
|
||
| ## Upload command | ||
|
|
||
| You upload snapshots with the `sentry-cli build snapshots` command, which takes a directory of images and optional JSON metadata files. | ||
|
|
||
| ```bash | ||
| sentry-cli build snapshots <output-dir> \ | ||
| --auth-token "$SENTRY_SNAPSHOTS_AUTH_TOKEN" \ | ||
| --app-id <app-id> \ | ||
| --project <project-slug> \ | ||
| --head-sha <head-commit-sha> \ | ||
| --vcs-provider github \ | ||
| --head-repo-name <owner/repo> | ||
| ``` | ||
|
|
||
| You can reference our [example workflow](#upload-with-ci) and [full flag reference](#sentry-cli-build-snapshots-reference) below. | ||
|
|
||
| ### Upload Structure | ||
|
|
||
| Each snapshot consists of two files: a `.png` image and an optional `.json` metadata file with the same base name and path. You can organize snapshots into subdirectories, but is not required and will not affect the UX. | ||
|
|
||
| ``` | ||
| snapshots/ | ||
| ├── homepage.png | ||
| ├── homepage.json | ||
| ├── settings/ | ||
| │ ├── profile.png | ||
| │ ├── profile.json | ||
| │ ├── billing.png | ||
| │ └── billing.json | ||
| ``` | ||
|
|
||
| Filenames are the identity key for each snapshot. Keep them stable across runs so Sentry can accurately diff head against base. | ||
|
|
||
| ### JSON Metadata | ||
|
|
||
| You can optionally include a JSON file to add metadata to each image: | ||
|
|
||
| ```json | ||
| { | ||
| "display_name": "Billing Page", | ||
| "group": "Settings" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we can also add other fields as extras here, they will become tags.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. didn't want to put that until they're actually used downstream (my understanding is that it's not in any way atm) |
||
| } | ||
| ``` | ||
|
|
||
| | Field | Type | Description | | ||
| | -------------- | ------ | ---------------------------------------------------------- | | ||
| | `display_name` | string | Human-readable label shown in the comparison viewer. | | ||
| | `group` | string | Groups related snapshots in the comparison viewer sidebar. | | ||
|
|
||
| TODO: IMAGE OF DISPLAY NAME AND GROUP | ||
|
|
||
| ## Upload With CI | ||
|
|
||
| To see the workflow Sentry uses on its own codebase, view the [frontend snapshots workflow](https://github.com/getsentry/sentry/blob/master/.github/workflows/frontend-snapshots.yml). | ||
|
|
||
| Below is an example GitHub Actions workflow you can use as a starting point. Replace the placeholder values with your own snapshot generation command, app ID, and Sentry project slug. | ||
|
|
||
| ```yml {filename:.github/workflows/snapshots.yml} | ||
| name: Snapshots | ||
|
|
||
| on: | ||
| push: | ||
| branches: [main] | ||
| pull_request: | ||
|
|
||
| env: | ||
| SNAPSHOT_OUTPUT_DIR: .artifacts/snapshots | ||
|
|
||
| jobs: | ||
| snapshots: | ||
| runs-on: ubuntu-24.04 | ||
| timeout-minutes: 20 | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| # Use PR head SHA, not the merge commit — avoids phantom diffs | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what is this?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. took from our sentry yml, not sure if we want in example workflow or not: https://github.com/getsentry/sentry/blob/master/.github/workflows/frontend-snapshots.yml#L42 |
||
| ref: ${{ github.event.pull_request.head.sha || github.sha }} | ||
|
|
||
| # ------------------------------------------------------- | ||
| # YOUR SNAPSHOT GENERATION STEP(S) HERE | ||
| # ------------------------------------------------------- | ||
| # Generate PNG + JSON pairs into $SNAPSHOT_OUTPUT_DIR. | ||
| # Examples: | ||
| # - Playwright screenshots | ||
| # - iOS/Android screenshots | ||
| # - Laravel Dusk screenshots | ||
| # - Any tool that produces PNGs | ||
| # ------------------------------------------------------- | ||
| - name: Generate snapshots | ||
| run: <your-snapshot-command> | ||
|
|
||
| - name: Install sentry-cli | ||
| if: ${{ !cancelled() }} | ||
| run: curl -sL https://sentry.io/get-cli/ | SENTRY_CLI_VERSION=3.3.4 sh | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 3.3.5 is the latest, but we can probably use a placeholder here.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you're saying dynamic value, can you link to how I would do that? Or are you saying literally put a placeholder value |
||
|
|
||
| - name: Upload snapshots | ||
| if: ${{ !cancelled() }} | ||
| env: | ||
| SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }} | ||
| run: | | ||
| ARGS=( | ||
| --auth-token "$SENTRY_AUTH_TOKEN" | ||
| build snapshots "${{ env.SNAPSHOT_OUTPUT_DIR }}" | ||
| --app-id <your-app-id> | ||
| --project <your-sentry-project> | ||
| --head-sha "${{ github.event.pull_request.head.sha || github.sha }}" | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. going off our workflow: https://github.com/getsentry/sentry/blob/master/.github/workflows/frontend-snapshots.yml#L86 also how much do we want to write for github actions vs. general? not sure what we do for size/distro |
||
| --vcs-provider github | ||
| --head-repo-name "${{ github.repository }}" | ||
| ) | ||
|
|
||
| if [[ "${{ github.event_name }}" == "pull_request" ]]; then | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. these will also all be auto detected so we should just rely on that. |
||
| ARGS+=( | ||
| --base-sha "${{ github.event.pull_request.base.sha }}" | ||
| --base-repo-name "${{ github.repository }}" | ||
| --head-ref "${{ github.head_ref }}" | ||
| --base-ref "${{ github.base_ref }}" | ||
| --pr-number "${{ github.event.number }}" | ||
| ) | ||
| fi | ||
|
|
||
| sentry-cli "${ARGS[@]}" | ||
| ``` | ||
|
|
||
| ## `sentry-cli build snapshots` Reference | ||
|
|
||
| ```bash | ||
| sentry-cli build snapshots [OPTIONS] --app-id <APP_ID> <PATH> | ||
| ``` | ||
|
|
||
| | Flag | Description | | ||
| | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ||
| | `<PATH>` | Directory containing `.png` images (and optional `.json`). | | ||
| | `--app-id <APP_ID>` | Identifier for the app or bundle (e.g., `web-frontend`, `ios-app`). Must be consistent across uploads — Sentry uses it to match head and base snapshots. Max 255 characters. | | ||
| | `--auth-token <TOKEN>` | Sentry auth token with `project:write` or `org:ci` scope. Can also be set via `SENTRY_AUTH_TOKEN`. | | ||
| | `-o`, `--org <ORG>` | Sentry organization slug. Can also be set via `SENTRY_ORG`. | | ||
| | `-p`, `--project <PROJECT>` | Sentry project slug. Can also be set via `SENTRY_PROJECT`. | | ||
| | `--head-sha <SHA>` | 40-character commit SHA of the snapshot build. Auto-detected in CI. | | ||
| | `--base-sha <SHA>` | Base branch commit SHA for comparison (PR only). Auto-detected in CI. | | ||
| | `--vcs-provider <PROVIDER>` | VCS provider, e.g., `github`. Auto-detected from git remote. | | ||
| | `--head-repo-name <OWNER/REPO>` | Repository in `owner/repo` format. Auto-detected from git remote. | | ||
| | `--base-repo-name <OWNER/REPO>` | Base repository in `owner/repo` format (PR only, for forks). Auto-detected in CI. | | ||
| | `--head-ref <BRANCH>` | Head branch name. Auto-detected in CI. | | ||
| | `--base-ref <BRANCH>` | Base branch name (PR only). Auto-detected in CI. | | ||
| | `--pr-number <NUMBER>` | Pull request number (PR only). Auto-detected in GitHub Actions. | | ||
| | `--diff-threshold <THRESHOLD>` | Float between 0.0 and 1.0. Sentry only reports images as changed if the percentage of changed pixels exceeds this value. For example, `0.01` ignores changes under 1%. | | ||
| | `--no-git-metadata` | Skip automatic git metadata detection. | | ||
| | `--force-git-metadata` | Force git metadata collection even outside CI. | | ||
| | `--log-level <LEVEL>` | Logging verbosity: `debug`, `info`, `warn`, or `error`. | | ||
| | `--url <URL>` | Sentry instance URL. Defaults to `https://sentry.io`. | | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we add PR comments on this page as well?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll put mention in the settings but since status check control a core workflow i think worth the callout. tbh this page is all slop rn and will get a rework