Add cross-browser and mobile-responsive E2E test matrix #800
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,106 @@ | ||
| name: Reusable E2E Cross-Browser Matrix | ||
|
|
||
| on: | ||
| workflow_call: | ||
| inputs: | ||
| dotnet-version: | ||
| description: .NET SDK version used for E2E backend setup | ||
| required: false | ||
| default: "8.0.x" | ||
| type: string | ||
| node-version: | ||
| description: Node.js version used for E2E frontend setup | ||
| required: false | ||
| default: "24.13.1" | ||
| type: string | ||
|
|
||
| permissions: | ||
| contents: read | ||
|
|
||
| env: | ||
| NUGET_PACKAGES: ${{ github.workspace }}/.nuget/packages | ||
|
|
||
| jobs: | ||
| e2e-cross-browser: | ||
| name: E2E (${{ matrix.project }}) | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| include: | ||
| - project: chromium | ||
| browser: chromium | ||
| - project: firefox | ||
| browser: firefox | ||
| - project: webkit | ||
| browser: webkit | ||
| - project: mobile-chrome | ||
| browser: chromium | ||
| - project: mobile-safari | ||
| browser: webkit | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v6 | ||
|
|
||
| - name: Setup .NET | ||
| uses: actions/setup-dotnet@v5 | ||
| with: | ||
| dotnet-version: ${{ inputs.dotnet-version }} | ||
| cache: true | ||
| cache-dependency-path: | | ||
| backend/Taskdeck.sln | ||
| backend/**/*.csproj | ||
|
|
||
| - name: Setup Node | ||
| uses: actions/setup-node@v6 | ||
| with: | ||
| node-version: ${{ inputs.node-version }} | ||
| cache: npm | ||
| cache-dependency-path: frontend/taskdeck-web/package-lock.json | ||
|
|
||
| - name: Restore backend | ||
| run: dotnet restore backend/Taskdeck.sln | ||
|
|
||
| - name: Install frontend dependencies | ||
| working-directory: frontend/taskdeck-web | ||
| run: npm ci | ||
|
|
||
| - name: Cache Playwright browsers | ||
| uses: actions/cache@v5 | ||
| with: | ||
| path: ~/.cache/ms-playwright | ||
| key: ms-playwright-${{ runner.os }}-${{ hashFiles('frontend/taskdeck-web/package-lock.json') }} | ||
|
|
||
| - name: Install Playwright browsers | ||
| working-directory: frontend/taskdeck-web | ||
| run: npx playwright install --with-deps ${{ matrix.browser }} | ||
|
|
||
|
Comment on lines
+75
to
+78
There was a problem hiding this comment. Each matrix job installs all Playwright browsers ( |
||
| - name: Remove stale E2E database | ||
| working-directory: frontend/taskdeck-web | ||
| run: node -e "require('fs').rmSync('taskdeck.e2e.ci.db',{force:true});" | ||
|
|
||
| - name: Run Playwright tests (${{ matrix.project }}) | ||
| timeout-minutes: 15 | ||
| working-directory: frontend/taskdeck-web | ||
| env: | ||
| CI: "true" | ||
| TASKDECK_E2E_DB: taskdeck.e2e.ci.db | ||
| TASKDECK_RUN_DEMO: "0" | ||
| run: npx playwright test --project=${{ matrix.project }} --reporter=line | ||
|
|
||
| - name: Upload Playwright report | ||
| if: failure() | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: playwright-report-${{ matrix.project }} | ||
| path: frontend/taskdeck-web/playwright-report | ||
| if-no-files-found: ignore | ||
|
|
||
| - name: Upload Playwright test results | ||
| if: failure() | ||
| uses: actions/upload-artifact@v7 | ||
| with: | ||
| name: playwright-test-results-${{ matrix.project }} | ||
| path: frontend/taskdeck-web/test-results | ||
| if-no-files-found: ignore | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,125 @@ | ||||||
| # Flaky Test Policy | ||||||
|
|
||||||
| Last Updated: 2026-04-09 | ||||||
|
|
||||||
| ## Purpose | ||||||
|
|
||||||
| This document defines how flaky E2E tests are identified, quarantined, and remediated in the Taskdeck test suite. The goal is to maintain CI signal quality: a red build should always mean a real problem. | ||||||
|
|
||||||
| ## Definition | ||||||
|
|
||||||
| A test is **flaky** when it produces inconsistent pass/fail results across runs without any code change. Common causes: | ||||||
|
|
||||||
| - Timing-dependent waits or race conditions | ||||||
| - Test isolation failures (shared state between tests or browser profiles) | ||||||
| - Browser-specific rendering timing (especially cross-browser matrix) | ||||||
| - Network/server startup non-determinism | ||||||
|
|
||||||
| ## Tagging Strategy | ||||||
|
|
||||||
| Taskdeck E2E tests use Playwright tag annotations in test titles: | ||||||
|
|
||||||
| | Tag | Purpose | Runs in CI | | ||||||
| |-----|---------|------------| | ||||||
| | (no tag) | Default smoke tests | PR gate (chromium only) | | ||||||
| | `@smoke` | Explicit smoke designation | PR gate (chromium only) | | ||||||
| | `@cross-browser` | Critical journeys across all desktop browsers | Nightly + manual (`testing` label) | | ||||||
| | `@mobile` | Mobile viewport responsive tests | Nightly + manual (`testing` label) | | ||||||
| | `@quarantine` | Known flaky, excluded from CI | Never (local debug only) | | ||||||
|
|
||||||
| ### How to tag a test | ||||||
|
|
||||||
| Add the tag to the test title string: | ||||||
|
|
||||||
| ```typescript | ||||||
| test('@cross-browser board creation workflow', async ({ page }) => { | ||||||
| // ... | ||||||
| }) | ||||||
|
|
||||||
| test('@mobile card editing on small screen', async ({ page }) => { | ||||||
| // ... | ||||||
| }) | ||||||
| ``` | ||||||
|
|
||||||
| Multiple tags can be combined: | ||||||
|
|
||||||
| ```typescript | ||||||
| test('@cross-browser @mobile responsive navigation', async ({ page }) => { | ||||||
| // ... | ||||||
| }) | ||||||
| ``` | ||||||
|
|
||||||
| ## CI Matrix Strategy | ||||||
|
|
||||||
| | CI Lane | Trigger | Projects Run | Tag Filter | | ||||||
| |---------|---------|-------------|------------| | ||||||
| | `ci-required.yml` (PR gate) | Every PR/push | chromium only | All tests except `@mobile` | | ||||||
| | `ci-extended.yml` | `testing` label or manual | All 5 projects | Per-project grep (see config) | | ||||||
| | `ci-nightly.yml` | Daily 03:25 UTC | All 5 projects | Per-project grep (see config) | | ||||||
|
|
||||||
| ## Quarantine Process | ||||||
|
|
||||||
| ### Step 1: Identify | ||||||
|
|
||||||
| When a test fails intermittently (2+ inconsistent results in nightly or PR runs): | ||||||
|
|
||||||
| 1. File a GitHub issue with label `flaky-test` and link the failing test file/line | ||||||
| 2. Include failure logs, trace artifacts, and which browser(s) are affected | ||||||
|
|
||||||
| ### Step 2: Quarantine | ||||||
|
|
||||||
| Add `@quarantine` tag to the test title: | ||||||
|
|
||||||
| ```typescript | ||||||
| test('@quarantine @cross-browser flaky board reload test', async ({ page }) => { | ||||||
| // ... | ||||||
| }) | ||||||
| ``` | ||||||
|
|
||||||
| The Playwright config excludes `@quarantine` from all CI projects via a top-level `grepInvert` in `playwright.config.ts`. The test still runs locally for debugging (pass `--grep="@quarantine"` explicitly to override). | ||||||
|
|
||||||
| The top-level exclusion is already configured: | ||||||
|
There was a problem hiding this comment. In Playwright,
Suggested change
|
||||||
|
|
||||||
| ```typescript | ||||||
| // playwright.config.ts (top level) | ||||||
| grepInvert: /@quarantine/, | ||||||
|
Comment on lines
+79
to
+85
There was a problem hiding this comment. The doc suggests adding |
||||||
| ``` | ||||||
|
|
||||||
| ### Step 3: Investigate | ||||||
|
|
||||||
| The issue assignee must: | ||||||
|
|
||||||
| 1. Reproduce locally (run the specific test with `--repeat-each=5`) | ||||||
| 2. Check for timing issues (missing `waitFor`, race conditions) | ||||||
| 3. Check for test isolation issues (shared state, database leaks) | ||||||
| 4. Check for browser-specific behavior (compare across projects) | ||||||
|
|
||||||
| ### Step 4: Fix and Un-quarantine | ||||||
|
|
||||||
| 1. Fix the root cause | ||||||
| 2. Verify stability: run `npx playwright test --project=<affected> --grep="test name" --repeat-each=10` | ||||||
| 3. Remove the `@quarantine` tag | ||||||
| 4. Close the issue with a link to the fix PR | ||||||
|
|
||||||
| ## Remediation Timeline | ||||||
|
|
||||||
| | Severity | SLA | Escalation | | ||||||
| |----------|-----|------------| | ||||||
| | Blocks PR gate (chromium smoke) | Fix within 24 hours or quarantine | Immediate team notification | | ||||||
| | Nightly cross-browser failure | Fix within 1 week | Review in next standup | | ||||||
| | Nightly mobile-only failure | Fix within 2 weeks | Track in sprint backlog | | ||||||
|
|
||||||
| ## Prevention Guidelines | ||||||
|
|
||||||
| 1. **Use explicit waits**: Always `await expect(locator).toBeVisible()` before interacting | ||||||
| 2. **Avoid fixed timeouts**: Use `waitForResponse` / `waitForURL` instead of `page.waitForTimeout` | ||||||
| 3. **Isolate test state**: Each test gets a fresh user via `registerAndAttachSession` | ||||||
| 4. **Use unique names**: Include `Date.now()` in board/card/column names to prevent collisions | ||||||
| 5. **Test deterministically**: Avoid tests that depend on animation timing or CSS transitions | ||||||
| 6. **Keep browser profiles independent**: Never share cookies, localStorage, or database state across browser projects | ||||||
|
|
||||||
| ## Monitoring | ||||||
|
|
||||||
| - Nightly CI results are reviewed daily for new failures | ||||||
| - Flaky test issues are prioritized alongside regular bugs | ||||||
| - A test that has been quarantined for more than 30 days without progress should be escalated or removed | ||||||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The Playwright browser cache key includes
matrix.project, but the workflow installs the full Playwright browser set (npx playwright install --with-deps) for every job. This will create 5 separate caches with identical contents and reduce cache hit rates / waste storage. Consider removingmatrix.projectfrom the cache key (or adding a restore key) so all matrix jobs can reuse the same cached browsers.