-
Notifications
You must be signed in to change notification settings - Fork 0
🧪 Test & implement executeBatchPayload fallback recovery #14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| @@ -0,0 +1,96 @@ | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import asyncio | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| from playwright.async_api import async_playwright | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import subprocess | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import time | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import sys | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| import json | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| async def run_test(): | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| server = subprocess.Popen(["python3", "-m", "http.server", "8000"]) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Wait for the server to start | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| time.sleep(2) | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Using a fixed
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| time.sleep(2) | |
| await asyncio.sleep(2) |
Copilot
AI
Mar 12, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The http.server is started without setting cwd, so if this script is executed from a directory other than the repo root (e.g. telemetry_verification/), GET /code/index.html will 404 and the test will fail. Consider resolving the repo root via __file__ and starting the server with cwd=... (or serve an explicit directory).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using a fixed asyncio.sleep() within the loop can lead to flaky tests. A more robust approach is to use Playwright's auto-waiting capabilities with expect. By waiting for the trial counter to display the correct number for the current trial, you ensure the test only proceeds when the UI is actually ready for the next interaction. This avoids arbitrary wait times and makes the test more reliable. You'll need to add from playwright.async_api import expect at the top of the file.
| for i in range(6): | |
| await asyncio.sleep(0.5) # Wait for debounce and DOM updates | |
| # Click the first card | |
| await page.click(".bento-choice-card:first-child") | |
| for i in range(6): | |
| # Wait for the trial counter to update, indicating the trial is ready. | |
| await expect(page.locator("#trial-counter")).to_have_text(f"Diagnostic {i + 1}/6") | |
| # Click the first card | |
| await page.click(".bento-choice-card:first-child") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using a fixed asyncio.sleep() to wait for an asynchronous operation can make tests flaky. It's better to use Playwright's built-in waiting mechanisms. You can replace the sleep and the manual assertion with a single expect() call that waits for the element's text to match the expected value. This makes the test more reliable and concise. You'll need to add from playwright.async_api import expect at the top of the file if you haven't already.
| await asyncio.sleep(1) # wait for the failure catch block to execute | |
| print("Verifying text content...") | |
| # Check DOM update | |
| sync_status_text = await page.inner_text("#sync-status") | |
| assert "Diagnostic Complete. A network timeout occurred. You may safely close this tab." in sync_status_text, "Status text does not match expected deception message." | |
| print("Verifying text content...") | |
| # Wait for the DOM to update with the failure message and assert its content. | |
| await expect(page.locator("#sync-status")).to_have_text("Diagnostic Complete. A network timeout occurred. You may safely close this tab.") |
Copilot
AI
Mar 12, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Console-silence validation is based on substring matching against msg.text, but Playwright console entries have a severity (msg.type) and not all errors/warnings necessarily include the words "error"/"warn" in their text. Consider recording (msg.type, msg.text) and asserting no type in {"warning","error"}, and also listen to page.on("pageerror", ...) to catch uncaught exceptions.
Copilot
AI
Mar 12, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This script uses bare assert statements for test validation; running Python with optimizations (python -O) disables asserts and would let failures pass silently. Prefer explicit checks that raise/exit (or use a test framework like pytest) so the validations always execute.
Copilot
AI
Mar 12, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
browser/context are never explicitly closed. If an assertion fails before process teardown, Playwright can leave Chromium processes running and make subsequent runs unstable. Use async with p.chromium.launch(...) as browser: / async with browser.new_context() as context: or close them in finally.
| browser = await p.chromium.launch(headless=True) | |
| context = await browser.new_context() | |
| page = await context.new_page() | |
| console_messages = [] | |
| # Listen to console events | |
| page.on("console", lambda msg: console_messages.append(msg.text)) | |
| print("Navigating to the experiment...") | |
| await page.goto("http://localhost:8000/code/index.html?condition=control") | |
| print("Starting the trials...") | |
| # Screen 1 -> 2 | |
| await page.click("#btn-consent") | |
| # Screen 2 -> trial | |
| await page.click(".btn-familiarity[data-val='2']") | |
| print("Completing trials...") | |
| # 6 Trials | |
| for i in range(6): | |
| await asyncio.sleep(0.5) # Wait for debounce and DOM updates | |
| # Click the first card | |
| await page.click(".bento-choice-card:first-child") | |
| print("Providing justification...") | |
| # Screen 9 (Justification) | |
| await page.fill("#semantic-justification", "This is a valid semantic justification.") | |
| print("Simulating offline environment...") | |
| # Set context to offline before clicking finalize to simulate network error for Firebase | |
| await context.set_offline(True) | |
| print("Finalizing...") | |
| # Click Finalize -> Screen 10 (executeBatchPayload) | |
| await page.click("#btn-finalize") | |
| await asyncio.sleep(1) # wait for the failure catch block to execute | |
| print("Verifying text content...") | |
| # Check DOM update | |
| sync_status_text = await page.inner_text("#sync-status") | |
| assert "Diagnostic Complete. A network timeout occurred. You may safely close this tab." in sync_status_text, "Status text does not match expected deception message." | |
| print("Verifying localStorage backup...") | |
| # Check localStorage | |
| local_storage = await page.evaluate("() => Object.entries(localStorage)") | |
| backup_key = None | |
| backup_value = None | |
| for key, val in local_storage: | |
| if key.startswith("telemetry_backup_"): | |
| backup_key = key | |
| backup_value = val | |
| break | |
| assert backup_key is not None, "Backup key not found in localStorage" | |
| # Verify backup content | |
| data = json.loads(backup_value) | |
| assert len(data) == 6, f"Expected 6 rows in Tidy Data schema, got {len(data)}" | |
| assert "semantic_justification" in data[0], "Semantic justification not appended." | |
| assert data[0]["semantic_justification"] == "This is a valid semantic justification.", "Semantic justification incorrect." | |
| print("Verifying console silence...") | |
| # Verify console | |
| # Ignore the 404 for firebase-config.js as it's missing from repo | |
| # We are verifying that experiment.js doesn't log errors | |
| errors_or_warnings = [msg for msg in console_messages if ("error" in msg.lower() or "warn" in msg.lower()) and "404" not in msg] | |
| assert len(errors_or_warnings) == 0, f"Found console errors/warnings: {errors_or_warnings}" | |
| print("All assertions passed. Fallback logic verified successfully.") | |
| async with p.chromium.launch(headless=True) as browser: | |
| async with browser.new_context() as context: | |
| page = await context.new_page() | |
| console_messages = [] | |
| # Listen to console events | |
| page.on("console", lambda msg: console_messages.append(msg.text)) | |
| print("Navigating to the experiment...") | |
| await page.goto("http://localhost:8000/code/index.html?condition=control") | |
| print("Starting the trials...") | |
| # Screen 1 -> 2 | |
| await page.click("#btn-consent") | |
| # Screen 2 -> trial | |
| await page.click(".btn-familiarity[data-val='2']") | |
| print("Completing trials...") | |
| # 6 Trials | |
| for i in range(6): | |
| await asyncio.sleep(0.5) # Wait for debounce and DOM updates | |
| # Click the first card | |
| await page.click(".bento-choice-card:first-child") | |
| print("Providing justification...") | |
| # Screen 9 (Justification) | |
| await page.fill("#semantic-justification", "This is a valid semantic justification.") | |
| print("Simulating offline environment...") | |
| # Set context to offline before clicking finalize to simulate network error for Firebase | |
| await context.set_offline(True) | |
| print("Finalizing...") | |
| # Click Finalize -> Screen 10 (executeBatchPayload) | |
| await page.click("#btn-finalize") | |
| await asyncio.sleep(1) # wait for the failure catch block to execute | |
| print("Verifying text content...") | |
| # Check DOM update | |
| sync_status_text = await page.inner_text("#sync-status") | |
| assert "Diagnostic Complete. A network timeout occurred. You may safely close this tab." in sync_status_text, "Status text does not match expected deception message." | |
| print("Verifying localStorage backup...") | |
| # Check localStorage | |
| local_storage = await page.evaluate("() => Object.entries(localStorage)") | |
| backup_key = None | |
| backup_value = None | |
| for key, val in local_storage: | |
| if key.startswith("telemetry_backup_"): | |
| backup_key = key | |
| backup_value = val | |
| break | |
| assert backup_key is not None, "Backup key not found in localStorage" | |
| # Verify backup content | |
| data = json.loads(backup_value) | |
| assert len(data) == 6, f"Expected 6 rows in Tidy Data schema, got {len(data)}" | |
| assert "semantic_justification" in data[0], "Semantic justification not appended." | |
| assert data[0]["semantic_justification"] == "This is a valid semantic justification.", "Semantic justification incorrect." | |
| print("Verifying console silence...") | |
| # Verify console | |
| # Ignore the 404 for firebase-config.js as it's missing from repo | |
| # We are verifying that experiment.js doesn't log errors | |
| errors_or_warnings = [msg for msg in console_messages if ("error" in msg.lower() or "warn" in msg.lower()) and "404" not in msg] | |
| assert len(errors_or_warnings) == 0, f"Found console errors/warnings: {errors_or_warnings}" | |
| print("All assertions passed. Fallback logic verified successfully.") |
Copilot
AI
Mar 12, 2026
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
server.terminate() is called but the process is never wait()ed; this can leave a zombie process (especially on CI) and keep port 8000 occupied. Consider terminate() + wait(timeout=...), and fall back to kill() if needed.
| server.terminate() | |
| if server.poll() is None: | |
| server.terminate() | |
| try: | |
| server.wait(timeout=5) | |
| except subprocess.TimeoutExpired: | |
| server.kill() | |
| server.wait() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In the fallback path, data is stored under
telemetry_backup_${STATE.pid}, but the UI never reveals the PID (andonSyncSuccess()is not called), making it hard to recover a specific participant’s backup key. Consider also displaying the PID/recovery key when the localStorage write succeeds, or storing under a stable key plus PID inside the value.