[Task] Expand CI/CD pipeline with Playwright End-to-End frontend tests

[tejassinghbhati]

What needs work?

Following the recent establishment of the comprehensive backend pytest suite and CI pipeline, the backend API is now continuously verified. However, the WebAPP frontend, which handles critical user workflows such as case creation, parameter editing, solver execution, and CSV extraction—currently lacks automated testing.

Running the application successfully requires the UI and API to coordinate smoothly, but without a frontend testing suite, we cannot automatically catch breakages in the user interface or regressions in how the UI consumes the backend API.

Possible approach (optional)

Tooling

• Introduce Playwright for headless browser End-to-End (E2E) testing.
• Playwright is preferred because:
  • Excellent Python/JavaScript interoperability
  • Built-in trace viewers for debugging
  • Strong support for complex DOM interaction testing

Infrastructure

• Add pytest-playwright to:

  • Keep the testing ecosystem purely in Python
  • Maintain tight coupling with the backend test suite

• Configure a conftest.py fixture to:

  • Spin up the Flask API locally before executing browser tests

Core Test Coverage (Smoke Tests)

• Load App: Verify index.html loads successfully
• Case Management:
  • Create a mock case
  • Verify it appears in the active session
  • Delete the case
• Navigation & Diagnostics:
  • Ensure major tabs render properly
  • Verify equations in the ModelFile diagnostic UI load without console errors

CI Integration

• Expand .github/workflows/ci.yml to include:
  • A dedicated UI/Playwright job
  • Run tests on every PR to main

What good looks like (optional)

• The frontend has automated verification for:

  • UI rendering
  • Basic user workflows

• A new CI pipeline job (UI Tests):

  • Runs on pull requests
  • Blocks merges if the frontend is broken or out-of-sync with the API

• Existing behavior remains unchanged

• Clear documentation in README.md explaining:

  • How to run UI tests locally

Anything to keep in mind? (optional)

Workflow State

• E2E tests can be flaky if not isolated
• Tests should:
  • Dynamically create and destroy dummy cases
  • Avoid reliance on pre-existing data

Port Conflicts

• Flask server used in tests should:
  • Run on a different port (e.g., 5003)
  • Avoid conflicts with local development environments

DOM Stability

• Current frontend lacks strong data-test attributes
• Initial selectors may rely on:
  • IDs
  • Text content

Related issue, PR, or discussion (optional)

This initiative builds on the recent Backend Testing & CI foundation and will test the newly added diagnostic UIs:

• Closes:

  • [Enhancement] Establish CI/CD Pipeline with GitHub Actions for Linting, Testing, and Build Verification [Enhancement] Establish CI/CD Pipeline with GitHub Actions for Linting, Testing, and Build Verification #58

• Related to:

  • Add pytest test suite and GitHub Actions CI pipeline Add pytest test suite and GitHub Actions CI pipeline #14
  • [Bug] Fix bugs in core classes + add test suite and CI pipeline [Bug] Fix bugs in core classes + add test suite and CI pipeline #10
  • [Task] Add GitHub Actions workflow for Python linting with Ruff [Task] Add GitHub Actions workflow for Python linting with Ruff #175
  • [v5.5] Diagnostics UI: model/log viewer + ModelFile page + MathJax [v5.5] Diagnostics UI: model/log viewer + ModelFile page + MathJax #408

Proposed track

{"Track" => "Stability"}

[m13v]

for the pytest-playwright approach, the Flask fixture in conftest.py is critical to get right. the tricky part is making sure the Flask server fully starts before Playwright connects to it. we use a health check loop that polls the /health endpoint until it returns 200 before running any browser tests. without that, you get intermittent failures on cold starts especially in CI where resource contention is higher. for DOM stability without data-test attributes, using IDs is fragile since Flask-generated IDs can change between renders. text content selectors work better for initial coverage but plan to add data-testid incrementally. the separate port (5003) for tests is smart, we learned that the hard way when test workers stomped on a running dev server.

[m13v]

fwiw we built a framework that handles Playwright test discovery and generation for web apps. the element targeting with multi-strategy selector fallbacks (data-testid, role, text, css): https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs and the test generation engine that outputs standard Playwright files: https://github.com/assrt-ai/assrt-freestyle/blob/main/src/core/freestyle.ts

[tejassinghbhati]

@m13v I am very grateful for your help! That's incredibly helpful context, especially regarding the cold-start flakiness and CSS selector fragility.

I am building the foundation now and I look forward to implementing exactly what you suggested:

Port Isolation: We're wiring the conftest.py live-server fixture identically to your approach, forcing it to port 5003 to guarantee we never stomp on anyone's running 5000 dev server during local testing.
Cold-Start Polling: The fixture is wrapped in a requests polling loop that waits for a 200 from the Flask endpoints before it ever yields the URL to the Playwright workers.
DOM Stabilty: Based on your advice, we are totally bypassing standard CSS or ID locators for the initial smoke tests. We're strictly relying on Playwright's native getByRole and getByText implementations which should be immediately more resilient against upstream HTML restructures.

I also took a look at terminator and the freestyle engine—that multi-strategy fallback logic is brilliant. That's exactly the kind of structural resilience we want to aim for as the coverage expands beyond the core smoke tests.

[m13v]

this is looking really solid. port 5003 isolation and the health check polling loop are exactly the pieces that prevent those random CI failures that waste everyone's time. one thing to watch for with the cold-start polling: make sure your timeout is generous enough for CI runners, they can be slower than local. 10-15 seconds is usually safe for Flask apps.

[tejassinghbhati]

@m13v This is so true with CI runner times. I’ve just updated the conftest.py polling loop to try for up to 15 seconds (using a 30-iteration loop with 0.5s sleeps) before it times out. That should give GitHub's runners plenty of breathing room for the Flask cold-start without holding up the test suite unnecessarily.

I've also expanded the script cover a few more routes (like #/Parameters) just to be 100% sure the DOM is mounting the layout properly before pushing.

I am very thankful to you for looking at my approach. I'll be opening the PR for this shortly so you can see how it all fits together.

[m13v]

15 seconds with 0.5s sleeps is a solid buffer. way better than the tight loops that just flake on slower runners. and yeah, testing the DOM layout on routes like #/Parameters is smart since those tend to be the ones that break silently when a component restructure happens.

one small thing that helped us: if the health check passes but the DOM still isn't ready, adding a short wait for a specific element (like a known heading or nav item) before running assertions catches those edge cases where the server is up but the frontend hasn't fully hydrated yet. sounds like you're already on that path though with the layout checks.

[tejassinghbhati]

@m13v, I wanted to leave a quick note on the current state of the PR.(#428) Maybe I need some review here.

The CI pipeline itself is structured correctly, the ui-test job runs independently, the /health polling fixture works as designed, and the isolation on port 5003 is good. The linting and backend pytest jobs are green.

However, I'm still working through a subtle local server behavior issue in the Playwright fixture. Specifically, when the test-spawned Flask subprocess serves index.html, the page navigation in Playwright's headless Chromium is timing out before DOMContentLoaded fires. The /health endpoint responds immediately, which confirms the server starts correctly, the issue is isolated to how the WSGI layer handles the initial browser page load. I have a few theories (session backend initialization on first request, static file path resolution from the subprocess CWD) that I'm actively debugging locally before pushing another fix.

I want to make sure the tests are verified locally before the next push rather than iterating against CI, I'll update this PR linked to this issue once I have a clean, evidence-backed run.