-
Notifications
You must be signed in to change notification settings - Fork 0
Integrate e2e tests in PR checks #133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
f0ea7d5
3b2b17b
6063c49
fe24730
6374af3
cde41c8
c1ed5d4
d8bd5f7
eaf48b1
cead6a0
6d5a888
75e9bf7
27190d0
642d99e
5e18844
60c4e26
f929745
011af39
6657ae0
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,125 @@ | ||
| name: E2E Tests | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: [opened, synchronize, reopened] | ||
|
|
||
| jobs: | ||
| e2e-tests: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 20 | ||
|
|
||
| steps: | ||
| - name: Wait for Netlify Preview | ||
| id: netlify | ||
| run: | | ||
| PR_NUMBER=${{ github.event.pull_request.number }} | ||
| DEPLOY_URL="https://deploy-preview-${PR_NUMBER}--superhero-c9df18.netlify.app" | ||
| echo "Waiting for Netlify deployment at: $DEPLOY_URL" | ||
| SECONDS=0 | ||
| MAX_WAIT=600 | ||
| WAIT_INTERVAL=15 | ||
| while [ $SECONDS -lt $MAX_WAIT ]; do | ||
| HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$DEPLOY_URL" || echo "000") | ||
| if [ "$HTTP_STATUS" = "200" ]; then | ||
| echo "✅ Netlify deployment is ready!" | ||
| echo "url=$DEPLOY_URL" >> $GITHUB_OUTPUT | ||
| exit 0 | ||
| fi | ||
| echo "Deployment not ready yet (HTTP $HTTP_STATUS). Waiting ${WAIT_INTERVAL}s..." | ||
| sleep $WAIT_INTERVAL | ||
| SECONDS=$((SECONDS + WAIT_INTERVAL)) | ||
| done | ||
| echo "❌ Deployment did not become ready within ${MAX_WAIT}s" | ||
| exit 1 | ||
| - name: Clone keypair-testing repo | ||
| uses: actions/checkout@v4 | ||
| with: | ||
| repository: aeternity/keypair-testing | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Makes sense to specify the commit of "keypair-testing" explicitly, to keep the relation between repositories. Otherwise, e2e tests won't work for old commits in "superhero" because "keypair-testing" get changed. |
||
| path: keypair-testing | ||
| token: ${{ secrets.KEYPAIR_TESTING_TOKEN }} | ||
|
|
||
| - name: Setup Node.js | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '20' | ||
| cache: 'npm' | ||
| cache-dependency-path: keypair-testing/package-lock.json | ||
|
|
||
| - name: Install dependencies | ||
| working-directory: keypair-testing | ||
| run: npm ci | ||
|
|
||
| - name: Install Playwright browsers (Only Chrome) | ||
| working-directory: keypair-testing | ||
| run: npx playwright install --with-deps chromium | ||
|
|
||
| - name: Run Superhero React E2E tests | ||
| id: test_run | ||
| working-directory: keypair-testing | ||
| env: | ||
| SUPERHERO_COM_BASE_URL: ${{ steps.netlify.outputs.url }} | ||
| run: | | ||
| SUPERHERO_COM_BASE_URL=$SUPERHERO_COM_BASE_URL npx playwright test tests_superhero_react/superhero-react.spec.ts --retries=1 --config=./playwright.config.ts | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The problem with this approach is that in some cases, the test fails not because the app is broken, but because it is not compatible with the app changes. Then the developer needs to adjust the failing test, but since it is in another repository, he needs to do another PR to the test suite 🙈 The best would be to keep tests in the same repository as the app.
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do you need |
||
| - name: Extract failed tests | ||
| if: always() | ||
| id: failed_tests | ||
| working-directory: keypair-testing | ||
| run: | | ||
| if [ -f "test-output.log" ]; then | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. |
||
| # Extract failed test names from list reporter output | ||
| FAILED_TESTS=$(grep "✘" test-output.log | sed 's/^.*\[.*\]//' | sed 's/^[[:space:]]*//' | head -10) | ||
| if [ -n "$FAILED_TESTS" ]; then | ||
| echo "failed<<EOF" >> $GITHUB_OUTPUT | ||
| echo "$FAILED_TESTS" >> $GITHUB_OUTPUT | ||
| echo "EOF" >> $GITHUB_OUTPUT | ||
| else | ||
| echo "failed=" >> $GITHUB_OUTPUT | ||
| fi | ||
| else | ||
| echo "failed=" >> $GITHUB_OUTPUT | ||
| fi | ||
| - name: Upload Playwright report | ||
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: playwright-report-${{ github.event.pull_request.number }} | ||
| path: keypair-testing/playwright-report/ | ||
| retention-days: 30 | ||
| if-no-files-found: warn | ||
|
|
||
| - name: Comment PR with test results | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This step is not necessary as GitHub already displays the CI status at the end of the PR and for every commit. It also appears more compact than the comments. |
||
| if: always() | ||
| uses: actions/github-script@v7 | ||
| with: | ||
| retries: 3 | ||
| retry-exempt-status-codes: 400,401,403,404 | ||
| script: | | ||
| const jobStatus = '${{ job.status }}'; | ||
| const status = jobStatus === 'success' ? '✅ E2E Tests PASSED' : '❌ E2E Tests FAILED'; | ||
| const failedTests = `${{ steps.failed_tests.outputs.failed }}`.trim(); | ||
| let body = `## ${status}\n\n**Tested URL:** ${{ steps.netlify.outputs.url }}\n\n`; | ||
| if (failedTests) { | ||
| body += `### ❌ Failed Tests:\n\`\`\`\n${failedTests}\n\`\`\`\n\n`; | ||
| } | ||
| body += `📊 [View Full Report](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})`; | ||
| await github.rest.issues.createComment({ | ||
| issue_number: context.issue.number, | ||
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| body: body | ||
| }); | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not obvious that e2e tests should depend on deployment. Maybe it would be faster to build the app here or get the built app from the previous step? Doing so will definitely be more reliable since it won't depend on Netlify.