feat(test): add `it.isKnownFlake` for opt-in stress-testing flaky Jest fixes by JoshuaKGoldberg · Pull Request #111860 · getsentry/sentry

JoshuaKGoldberg · 2026-03-30T23:03:33Z

It's hard to determine when a known flaky Jest test is no longer flaky. You basically just have to keep running it repeatedly. But it takes a long time (and bogs up our GHA workflows) to re-run all tests too.

This PR adds plumbing that lets us specifically re-run known flaky tests 50x in a single run:

I added an opt-in Frontend: Rerun Flaky Tests Known flaky tests should be run many times, just to be safe. label as seen on this PR
When that label is present, the frontend workflow sets a RERUN_KNOWN_FLAKY_TESTS process env var
Tests defined with the new it.isKnownFlake will define the same test 50x with incremented counter names when that var is present

Example CI failure (search flaky rerun):

  ● EventGroupingInfo › [flaky rerun x50] fetches and renders grouping info for errors › run 2/50

I added it to the following tests that have failed >=2x on master over the last month:

File	Test	CI failures (30d)	Ticket
`eventReplay/index.spec.tsx`	render replay inline onboarding	6	REPLAY-879
`stackTrace.spec.tsx`	URL link in tooltip	5	ENG-7192
`resultsSearchQueryBuilder.spec.tsx`	has: dropdown + normal tags (2 tests)	5	ENG-7201
`metricsTab.spec.tsx`	toggle query builder sidebar	4	ENG-7202
`customerDetails.spec.tsx`	disabled without billing.admin	4	ENG-7203
`eventsSearchBar.spec.tsx`	has: dropdown	3	DAIN-1271
`trace.spec.tsx`	arrowup+shift scroll (was `it.skip`)	3	BROWSE-411
`allMonitors.spec.tsx`	select all query results	2	ENG-7204
`spansSearchBar.spec.tsx`	onSearch correct query	2	ENG-7205
`react-native/metrics.spec.tsx`	onboarding content	2	ENG-7206
`useReplaysFromIssue.spec.tsx`	fetch replay ids	2	ENG-7207
`spanEvidencePreview.spec.tsx`	error on request fail	2	ENG-7208
`groupingInfoSection.spec.tsx`	render grouping info	2	ENG-7209
`timeSince.spec.tsx`	respects timezone in tooltip	1	ENG-7211
`versionHoverCard.spec.tsx`	renders	1	ENG-7212

Made with Cursor

github-actions · 2026-03-31T13:30:46Z

🚨 Warning: This pull request contains Frontend and Backend changes!

It's discouraged to make changes to Sentry's Frontend and Backend in a single pull request. The Frontend and Backend are not atomically deployed. If the changes are interdependent of each other, they must be separated into two pull requests and be made forward or backwards compatible, such that the Backend or Frontend can be safely deployed independently.

Have questions? Please ask in the #discuss-dev-infra channel.

Adds infrastructure for validating flaky test fixes in CI: - `itRepeatsWhenFlaky()`: a test wrapper in tests/js/sentry-test/ that runs a test 50x when the RERUN_KNOWN_FLAKY_TESTS env var is set, otherwise runs once as a normal it() - CI wiring: frontend.yml sets RERUN_KNOWN_FLAKY_TESTS=true when the PR has the "Frontend: Rerun Flaky Tests" label - ESLint: configured jest/no-standalone-expect to recognize itRepeatsWhenFlaky as a test block Wraps all 13 known flaky tests (identified from 30 days of CI failures on master) with itRepeatsWhenFlaky so fixes can be stress-tested: - eventReplay/index.spec.tsx (6 occ) - stackTrace.spec.tsx (5 occ) - resultsSearchQueryBuilder.spec.tsx (5 occ, 2 tests) - metricsTab.spec.tsx (4 occ) - customerDetails.spec.tsx (4 occ) - eventsSearchBar.spec.tsx (3 occ) - trace.spec.tsx (3 occ, previously skipped) - allMonitors.spec.tsx (2 occ) - spansSearchBar.spec.tsx (2 occ) - react-native/metrics.spec.tsx (2 occ) - useReplaysFromIssue.spec.tsx (2 occ) - spanEvidencePreview.spec.tsx (2 occ) - groupingInfoSection.spec.tsx (2 occ) Made-with: Cursor

…Flake Mark flaky tests for rerun infrastructure: - TimeSince › respects timezone in tooltip (ENG-7211) - VersionHoverCard › renders (ENG-7212) Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com> Made-with: Cursor

@nikkikapadia

Previously, `groupingInfoSection.spec.tsx` was testing the `GroupingInfoSection` component: but all that component does is combine the `InterimSection` component with a lazy-loaded `GroupingInfo` component. That lazy-loading can take >=100-150ms locally, and might be the cause of flake in CI. This PR changes the test to: * Directly test `GroupingInfo`, bypassing the lazy-load * Delete the specific flaky test that was checking for the component being loaded after click (since that's covered by `FoldSection`'s unit tests, as rendered by `InterimSection`) Fixes ENG-7209 ~Note that CI Jest tests are failing because the https://github.com/getsentry/sentry/labels/Frontend%3A%20Rerun%20Flaky%20Tests label is causing _other_, still-flaky tests to be run.~ Rebased on `master` so the `it.isKnownFlake` (#111860) addition isn't here yet. Made with [Cursor](https://cursor.com) Co-authored-by: @nikkikapadia

JoshuaKGoldberg · 2026-04-03T14:53:39Z

I want to merge this PR, but the frontend / Jest tests keep showing flaky test failures. Which I suppose is good 🙂. In the latest run's failures, https://github.com/getsentry/sentry/actions/runs/23950050162/job/69855165158?pr=111860:

static/app/views/detectors/list/allMonitors.spec.tsx (fix(test): stabilize flaky allMonitors select all query results test #111901)
static/app/views/discover/results/resultsSearchQueryBuilder.spec.tsx (fix(test): stabilize flaky SpanEvidencePreview spec #111914)

I'll work on getting granular PRs rebased on master & merged before this PR.

The `ReplayInlineOnboardingPanel` component is loaded via `React.lazy()`, creating an async code-splitting boundary. Inside that lazy-loaded component, `usePrompt` calls `useApiQuery` (React Query), adding another async data-fetching boundary. The combination of these two async layers occasionally exceeded the `findByText` default timeout in CI, causing the test to flake. This PR mocks the `replayInlineOnboardingPanel` module (a pattern already used for `replayClipPreview`) and asserts on a `data-test-id` instead of the rendered text. Fixes REPLAY-879 ~Note that CI Jest tests are failing because the https://github.com/getsentry/sentry/labels/Frontend%3A%20Rerun%20Flaky%20Tests label is causing _other_, still-flaky tests to be run.~ No longer applicable now that this is targeting `master` and #111860 is not yet merged. Made with [Cursor](https://cursor.com)

JoshuaKGoldberg · 2026-04-07T18:41:01Z

Oh hey all tests are passing! Which I'm taking as a sign that:

The separate work to improve frontend stability has actually made things a lot more stable (fix(test): open overlays immediately when in test mode #112196, fix(test): Upgrade framer motion, Disable animations in tests #112270)
Rerunning 50x is not enough to tease out flakiness failures. If flakiness becomes an issue again, we'll probably want to add a way to simulate an overloaded/slow server - such as running a lot of compute in the background of the it.isKnownFlake tests.

Merging as-is for now.

The `ReplayInlineOnboardingPanel` component is loaded via `React.lazy()`, creating an async code-splitting boundary. Inside that lazy-loaded component, `usePrompt` calls `useApiQuery` (React Query), adding another async data-fetching boundary. The combination of these two async layers occasionally exceeded the `findByText` default timeout in CI, causing the test to flake. This PR mocks the `replayInlineOnboardingPanel` module (a pattern already used for `replayClipPreview`) and asserts on a `data-test-id` instead of the rendered text. Fixes REPLAY-879 ~Note that CI Jest tests are failing because the https://github.com/getsentry/sentry/labels/Frontend%3A%20Rerun%20Flaky%20Tests label is causing _other_, still-flaky tests to be run.~ No longer applicable now that this is targeting `master` and #111860 is not yet merged. Made with [Cursor](https://cursor.com)

…t fixes (#111860) It's hard to determine when a known flaky Jest test is no longer flaky. You basically just have to keep running it repeatedly. But it takes a _long_ time (and bogs up our GHA workflows) to re-run all tests too. This PR adds plumbing that lets us specifically re-run known flaky tests 50x in a single run: 1. I added an opt-in https://github.com/getsentry/sentry/labels/Frontend%3A%20Rerun%20Flaky%20Tests label as seen on this PR 2. When that label is present, the frontend workflow sets a `RERUN_KNOWN_FLAKY_TESTS` process env var 3. Tests defined with the new `it.isKnownFlake` will define the same test 50x with incremented counter names when that var is present [Example CI failure (search `flaky rerun`)](https://github.com/getsentry/sentry/actions/runs/23800263946/job/69358610839?pr=111860): ```plaintext ● EventGroupingInfo › [flaky rerun x50] fetches and renders grouping info for errors › run 2/50 ``` I added it to the following tests that have failed >=2x on `master` over the last month: | File | Test | CI failures (30d) | Ticket | |------|------|-------------------|--------| | `eventReplay/index.spec.tsx` | render replay inline onboarding | 6 | [REPLAY-879](https://linear.app/getsentry/issue/REPLAY-879) | | `stackTrace.spec.tsx` | URL link in tooltip | 5 | [ENG-7192](https://linear.app/getsentry/issue/ENG-7192) | | `resultsSearchQueryBuilder.spec.tsx` | has: dropdown + normal tags (2 tests) | 5 | [ENG-7201](https://linear.app/getsentry/issue/ENG-7201) | | `metricsTab.spec.tsx` | toggle query builder sidebar | 4 | [ENG-7202](https://linear.app/getsentry/issue/ENG-7202) | | `customerDetails.spec.tsx` | disabled without billing.admin | 4 | [ENG-7203](https://linear.app/getsentry/issue/ENG-7203) | | `eventsSearchBar.spec.tsx` | has: dropdown | 3 | [DAIN-1271](https://linear.app/getsentry/issue/DAIN-1271) | | `trace.spec.tsx` | arrowup+shift scroll (was `it.skip`) | 3 | [BROWSE-411](https://linear.app/getsentry/issue/BROWSE-411) | | `allMonitors.spec.tsx` | select all query results | 2 | [ENG-7204](https://linear.app/getsentry/issue/ENG-7204) | | `spansSearchBar.spec.tsx` | onSearch correct query | 2 | [ENG-7205](https://linear.app/getsentry/issue/ENG-7205) | | `react-native/metrics.spec.tsx` | onboarding content | 2 | [ENG-7206](https://linear.app/getsentry/issue/ENG-7206) | | `useReplaysFromIssue.spec.tsx` | fetch replay ids | 2 | [ENG-7207](https://linear.app/getsentry/issue/ENG-7207) | | `spanEvidencePreview.spec.tsx` | error on request fail | 2 | [ENG-7208](https://linear.app/getsentry/issue/ENG-7208) | | `groupingInfoSection.spec.tsx` | render grouping info | 2 | [ENG-7209](https://linear.app/getsentry/issue/ENG-7209) | | `timeSince.spec.tsx` | respects timezone in tooltip | 1 | [ENG-7211](https://linear.app/getsentry/issue/ENG-7211) | | `versionHoverCard.spec.tsx` | renders | 1 | [ENG-7212](https://linear.app/getsentry/issue/ENG-7212) | Made with [Cursor](https://cursor.com) --------- Co-authored-by: getsantry[bot] <66042841+getsantry[bot]@users.noreply.github.com> Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>

JoshuaKGoldberg added the Frontend: Rerun Flaky Tests Known flaky tests should be run many times, just to be safe. label Mar 30, 2026

github-actions bot added the Scope: Frontend Automatically applied to PRs that change frontend components label Mar 30, 2026

vercel bot deployed to Preview March 30, 2026 23:06 View deployment

JoshuaKGoldberg force-pushed the fix/flaky-test-rerun-infra branch from f42d3ea to 59dbb34 Compare March 31, 2026 13:12

vercel bot deployed to Preview March 31, 2026 13:15 View deployment

JoshuaKGoldberg changed the title ~~feat(test): Add itRepeatsWhenFlaky infra for stress-testing flaky Jest fixes~~ feat(test): Add it.knownFlake for stress-testing flaky Jest fixes Mar 31, 2026

JoshuaKGoldberg force-pushed the fix/flaky-test-rerun-infra branch from 59dbb34 to ac09db1 Compare March 31, 2026 13:30

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Mar 31, 2026

JoshuaKGoldberg force-pushed the fix/flaky-test-rerun-infra branch from ac09db1 to 93d1cbe Compare March 31, 2026 13:31

JoshuaKGoldberg requested review from a team and removed request for a team March 31, 2026 13:32

JoshuaKGoldberg force-pushed the fix/flaky-test-rerun-infra branch 2 times, most recently from 5abee83 to 206555a Compare March 31, 2026 13:34

JoshuaKGoldberg force-pushed the fix/flaky-test-rerun-infra branch from 206555a to 6515f63 Compare March 31, 2026 13:34

getsantry bot added 2 commits March 31, 2026 13:34

❄️ re-freeze requirements

cb0e98d

❄️ re-freeze requirements

d7f2bcd

vercel bot deployed to Preview March 31, 2026 13:39 View deployment

This was referenced Mar 31, 2026

Flaky test: static/app/components/events/eventReplay/index.spec.tsx #111830

Closed

Flaky test: static/app/components/stackTrace/stackTrace.spec.tsx #111831

Closed

JoshuaKGoldberg removed the Scope: Backend Automatically applied to PRs that change backend components label Mar 31, 2026

JoshuaKGoldberg changed the title ~~feat(test): Add it.knownFlake for stress-testing flaky Jest fixes~~ feat(test): add it.knownFlake for opt-in stress-testing flaky Jest fixes Mar 31, 2026

vercel bot deployed to Preview March 31, 2026 18:22 View deployment

JoshuaKGoldberg marked this pull request as ready for review April 2, 2026 17:30

JoshuaKGoldberg requested review from a team as code owners April 2, 2026 17:30

vercel bot deployed to Preview April 3, 2026 13:41 View deployment

Merge branch 'master'

c019db5

vercel bot deployed to Preview April 3, 2026 14:41 View deployment

Merge branch 'master' into fix/flaky-test-rerun-infra

38d0e8b

vercel bot deployed to Preview April 7, 2026 13:55 View deployment

JoshuaKGoldberg marked this pull request as draft April 7, 2026 14:09

JoshuaKGoldberg removed request for a team April 7, 2026 14:17

Merge branch 'master'

ecb0ff8

vercel bot deployed to Preview April 7, 2026 18:17 View deployment

JoshuaKGoldberg marked this pull request as ready for review April 7, 2026 18:35

JoshuaKGoldberg merged commit df89359 into master Apr 7, 2026
66 checks passed

JoshuaKGoldberg deleted the fix/flaky-test-rerun-infra branch April 7, 2026 18:41

This was referenced Apr 8, 2026

fix(test): stabilize flaky MetricsTabContent sidebar toggle test #111909

Closed

feat(test): add stress-testing to known flaky tests reruns #112578

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(test): add `it.isKnownFlake` for opt-in stress-testing flaky Jest fixes#111860

feat(test): add `it.isKnownFlake` for opt-in stress-testing flaky Jest fixes#111860
JoshuaKGoldberg merged 11 commits intomasterfrom
fix/flaky-test-rerun-infra

JoshuaKGoldberg commented Mar 30, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Mar 31, 2026

Uh oh!

JoshuaKGoldberg commented Apr 3, 2026

Uh oh!

JoshuaKGoldberg commented Apr 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

JoshuaKGoldberg commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 31, 2026

Uh oh!

JoshuaKGoldberg commented Apr 3, 2026

Uh oh!

JoshuaKGoldberg commented Apr 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JoshuaKGoldberg commented Mar 30, 2026 •

edited

Loading