Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 125 additions & 0 deletions .github/workflows/e2e-tests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
name: E2E Tests

on:
pull_request:
types: [opened, synchronize, reopened]

jobs:
e2e-tests:
runs-on: ubuntu-latest
timeout-minutes: 20

steps:
- name: Wait for Netlify Preview
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not obvious that e2e tests should depend on deployment. Maybe it would be faster to build the app here or get the built app from the previous step? Doing so will definitely be more reliable since it won't depend on Netlify.

id: netlify
run: |
PR_NUMBER=${{ github.event.pull_request.number }}
DEPLOY_URL="https://deploy-preview-${PR_NUMBER}--superhero-c9df18.netlify.app"
echo "Waiting for Netlify deployment at: $DEPLOY_URL"
SECONDS=0
MAX_WAIT=600
WAIT_INTERVAL=15
while [ $SECONDS -lt $MAX_WAIT ]; do
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$DEPLOY_URL" || echo "000")
if [ "$HTTP_STATUS" = "200" ]; then
echo "✅ Netlify deployment is ready!"
echo "url=$DEPLOY_URL" >> $GITHUB_OUTPUT
exit 0
fi
echo "Deployment not ready yet (HTTP $HTTP_STATUS). Waiting ${WAIT_INTERVAL}s..."
sleep $WAIT_INTERVAL
SECONDS=$((SECONDS + WAIT_INTERVAL))
done
echo "❌ Deployment did not become ready within ${MAX_WAIT}s"
exit 1
- name: Clone keypair-testing repo
uses: actions/checkout@v4
with:
repository: aeternity/keypair-testing
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense to specify the commit of "keypair-testing" explicitly, to keep the relation between repositories. Otherwise, e2e tests won't work for old commits in "superhero" because "keypair-testing" get changed.

path: keypair-testing
token: ${{ secrets.KEYPAIR_TESTING_TOKEN }}

- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
cache-dependency-path: keypair-testing/package-lock.json

- name: Install dependencies
working-directory: keypair-testing
run: npm ci

- name: Install Playwright browsers (Only Chrome)
working-directory: keypair-testing
run: npx playwright install --with-deps chromium

- name: Run Superhero React E2E tests
id: test_run
working-directory: keypair-testing
env:
SUPERHERO_COM_BASE_URL: ${{ steps.netlify.outputs.url }}
run: |
SUPERHERO_COM_BASE_URL=$SUPERHERO_COM_BASE_URL npx playwright test tests_superhero_react/superhero-react.spec.ts --retries=1 --config=./playwright.config.ts
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with this approach is that in some cases, the test fails not because the app is broken, but because it is not compatible with the app changes. Then the developer needs to adjust the failing test, but since it is in another repository, he needs to do another PR to the test suite 🙈 The best would be to keep tests in the same repository as the app.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you need --retries=1 because of flaky tests?

- name: Extract failed tests
if: always()
id: failed_tests
working-directory: keypair-testing
run: |
if [ -f "test-output.log" ]; then
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Failed Tests Not Reported in PR Comments

The E2E test workflow attempts to extract failed test names from test-output.log, but the Playwright command doesn't save its output to this file. As a result, failed tests are never reported in the PR comment.

Fix in Cursor Fix in Web

# Extract failed test names from list reporter output
FAILED_TESTS=$(grep "✘" test-output.log | sed 's/^.*\[.*\]//' | sed 's/^[[:space:]]*//' | head -10)
if [ -n "$FAILED_TESTS" ]; then
echo "failed<<EOF" >> $GITHUB_OUTPUT
echo "$FAILED_TESTS" >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
else
echo "failed=" >> $GITHUB_OUTPUT
fi
else
echo "failed=" >> $GITHUB_OUTPUT
fi
- name: Upload Playwright report
if: always()
uses: actions/upload-artifact@v4
with:
name: playwright-report-${{ github.event.pull_request.number }}
path: keypair-testing/playwright-report/
retention-days: 30
if-no-files-found: warn

- name: Comment PR with test results
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This step is not necessary as GitHub already displays the CI status at the end of the PR and for every commit. It also appears more compact than the comments.

if: always()
uses: actions/github-script@v7
with:
retries: 3
retry-exempt-status-codes: 400,401,403,404
script: |
const jobStatus = '${{ job.status }}';
const status = jobStatus === 'success' ? '✅ E2E Tests PASSED' : '❌ E2E Tests FAILED';
const failedTests = `${{ steps.failed_tests.outputs.failed }}`.trim();
let body = `## ${status}\n\n**Tested URL:** ${{ steps.netlify.outputs.url }}\n\n`;
if (failedTests) {
body += `### ❌ Failed Tests:\n\`\`\`\n${failedTests}\n\`\`\`\n\n`;
}
body += `📊 [View Full Report](https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }})`;
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: body
});