Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 9 additions & 6 deletions code/experiment.js
Original file line number Diff line number Diff line change
Expand Up @@ -349,7 +349,6 @@ function init() {
executeBatchPayload();
});

console.log(`Diagnostic Engine Initialized. PID: ${STATE.pid} | Condition: ${STATE.condition}`);
}

function loadNextTrial() {
Expand Down Expand Up @@ -469,13 +468,17 @@ async function executeBatchPayload() {
await batch.commit();
onSyncSuccess();
} else {
console.warn("Firebase not detected. Payload logged to console:", STATE.results);
setTimeout(onSyncSuccess, 1500); // Simulate sync delay
// Throw error to trigger the fallback when Firebase is not available
throw new Error("Firebase not initialized");
}
} catch (error) {
console.error("Critical Sync Failure:", error);
DOM.syncStatus.innerHTML = `<span style="color:#ff453a">⚠️ Sync Failed. Error: ${error.code || 'Network'}</span>`;
// Potential fallback: Save to localStorage for later recovery
// Fallback: Save to localStorage for later recovery
try {
localStorage.setItem(`telemetry_backup_${STATE.pid}`, JSON.stringify(STATE.results));
} catch (storageError) {
// Silently fail if localStorage is unavailable
}
DOM.syncStatus.textContent = "Diagnostic Complete. A network timeout occurred. You may safely close this tab.";
}
Comment on lines +475 to 482
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the fallback path, data is stored under telemetry_backup_${STATE.pid}, but the UI never reveals the PID (and onSyncSuccess() is not called), making it hard to recover a specific participant’s backup key. Consider also displaying the PID/recovery key when the localStorage write succeeds, or storing under a stable key plus PID inside the value.

Copilot uses AI. Check for mistakes.
}

Expand Down
96 changes: 96 additions & 0 deletions telemetry_verification/verify_payload_fallback.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
import asyncio
from playwright.async_api import async_playwright
import subprocess
import time
import sys
import json

async def run_test():
server = subprocess.Popen(["python3", "-m", "http.server", "8000"])
# Wait for the server to start
time.sleep(2)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a fixed time.sleep() to wait for the server to start can lead to flaky tests. If the server takes longer to start on a slower machine or under load, the test will fail. A more robust approach is to implement a polling mechanism that repeatedly tries to connect to the server until it's responsive before proceeding with the test.

Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

time.sleep(2) is called inside an async def and blocks the event loop, which can make the Playwright run flaky/slow. Prefer await asyncio.sleep(...) or (better) actively poll for the server port to accept connections before proceeding.

Suggested change
time.sleep(2)
await asyncio.sleep(2)

Copilot uses AI. Check for mistakes.

try:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
context = await browser.new_context()
page = await context.new_page()

console_messages = []

# Listen to console events
page.on("console", lambda msg: console_messages.append(msg.text))

print("Navigating to the experiment...")
await page.goto("http://localhost:8000/code/index.html?condition=control")
Comment on lines +9 to +25
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The http.server is started without setting cwd, so if this script is executed from a directory other than the repo root (e.g. telemetry_verification/), GET /code/index.html will 404 and the test will fail. Consider resolving the repo root via __file__ and starting the server with cwd=... (or serve an explicit directory).

Copilot uses AI. Check for mistakes.

print("Starting the trials...")
# Screen 1 -> 2
await page.click("#btn-consent")

# Screen 2 -> trial
await page.click(".btn-familiarity[data-val='2']")

print("Completing trials...")
# 6 Trials
for i in range(6):
await asyncio.sleep(0.5) # Wait for debounce and DOM updates
# Click the first card
await page.click(".bento-choice-card:first-child")
Comment on lines +36 to +39

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a fixed asyncio.sleep() within the loop can lead to flaky tests. A more robust approach is to use Playwright's auto-waiting capabilities with expect. By waiting for the trial counter to display the correct number for the current trial, you ensure the test only proceeds when the UI is actually ready for the next interaction. This avoids arbitrary wait times and makes the test more reliable. You'll need to add from playwright.async_api import expect at the top of the file.

Suggested change
for i in range(6):
await asyncio.sleep(0.5) # Wait for debounce and DOM updates
# Click the first card
await page.click(".bento-choice-card:first-child")
for i in range(6):
# Wait for the trial counter to update, indicating the trial is ready.
await expect(page.locator("#trial-counter")).to_have_text(f"Diagnostic {i + 1}/6")
# Click the first card
await page.click(".bento-choice-card:first-child")


print("Providing justification...")
# Screen 9 (Justification)
await page.fill("#semantic-justification", "This is a valid semantic justification.")

print("Simulating offline environment...")
# Set context to offline before clicking finalize to simulate network error for Firebase
await context.set_offline(True)

print("Finalizing...")
# Click Finalize -> Screen 10 (executeBatchPayload)
await page.click("#btn-finalize")

await asyncio.sleep(1) # wait for the failure catch block to execute

print("Verifying text content...")
# Check DOM update
sync_status_text = await page.inner_text("#sync-status")
assert "Diagnostic Complete. A network timeout occurred. You may safely close this tab." in sync_status_text, "Status text does not match expected deception message."
Comment on lines +53 to +58

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

Using a fixed asyncio.sleep() to wait for an asynchronous operation can make tests flaky. It's better to use Playwright's built-in waiting mechanisms. You can replace the sleep and the manual assertion with a single expect() call that waits for the element's text to match the expected value. This makes the test more reliable and concise. You'll need to add from playwright.async_api import expect at the top of the file if you haven't already.

Suggested change
await asyncio.sleep(1) # wait for the failure catch block to execute
print("Verifying text content...")
# Check DOM update
sync_status_text = await page.inner_text("#sync-status")
assert "Diagnostic Complete. A network timeout occurred. You may safely close this tab." in sync_status_text, "Status text does not match expected deception message."
print("Verifying text content...")
# Wait for the DOM to update with the failure message and assert its content.
await expect(page.locator("#sync-status")).to_have_text("Diagnostic Complete. A network timeout occurred. You may safely close this tab.")


print("Verifying localStorage backup...")
# Check localStorage
local_storage = await page.evaluate("() => Object.entries(localStorage)")

backup_key = None
backup_value = None
for key, val in local_storage:
if key.startswith("telemetry_backup_"):
backup_key = key
backup_value = val
break

assert backup_key is not None, "Backup key not found in localStorage"

# Verify backup content
data = json.loads(backup_value)
assert len(data) == 6, f"Expected 6 rows in Tidy Data schema, got {len(data)}"
assert "semantic_justification" in data[0], "Semantic justification not appended."
assert data[0]["semantic_justification"] == "This is a valid semantic justification.", "Semantic justification incorrect."

print("Verifying console silence...")
# Verify console
# Ignore the 404 for firebase-config.js as it's missing from repo
# We are verifying that experiment.js doesn't log errors
errors_or_warnings = [msg for msg in console_messages if ("error" in msg.lower() or "warn" in msg.lower()) and "404" not in msg]
assert len(errors_or_warnings) == 0, f"Found console errors/warnings: {errors_or_warnings}"
Comment on lines +19 to +85
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Console-silence validation is based on substring matching against msg.text, but Playwright console entries have a severity (msg.type) and not all errors/warnings necessarily include the words "error"/"warn" in their text. Consider recording (msg.type, msg.text) and asserting no type in {"warning","error"}, and also listen to page.on("pageerror", ...) to catch uncaught exceptions.

Copilot uses AI. Check for mistakes.
Comment on lines +58 to +85
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script uses bare assert statements for test validation; running Python with optimizations (python -O) disables asserts and would let failures pass silently. Prefer explicit checks that raise/exit (or use a test framework like pytest) so the validations always execute.

Copilot uses AI. Check for mistakes.

print("All assertions passed. Fallback logic verified successfully.")
Comment on lines +15 to +87
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

browser/context are never explicitly closed. If an assertion fails before process teardown, Playwright can leave Chromium processes running and make subsequent runs unstable. Use async with p.chromium.launch(...) as browser: / async with browser.new_context() as context: or close them in finally.

Suggested change
browser = await p.chromium.launch(headless=True)
context = await browser.new_context()
page = await context.new_page()
console_messages = []
# Listen to console events
page.on("console", lambda msg: console_messages.append(msg.text))
print("Navigating to the experiment...")
await page.goto("http://localhost:8000/code/index.html?condition=control")
print("Starting the trials...")
# Screen 1 -> 2
await page.click("#btn-consent")
# Screen 2 -> trial
await page.click(".btn-familiarity[data-val='2']")
print("Completing trials...")
# 6 Trials
for i in range(6):
await asyncio.sleep(0.5) # Wait for debounce and DOM updates
# Click the first card
await page.click(".bento-choice-card:first-child")
print("Providing justification...")
# Screen 9 (Justification)
await page.fill("#semantic-justification", "This is a valid semantic justification.")
print("Simulating offline environment...")
# Set context to offline before clicking finalize to simulate network error for Firebase
await context.set_offline(True)
print("Finalizing...")
# Click Finalize -> Screen 10 (executeBatchPayload)
await page.click("#btn-finalize")
await asyncio.sleep(1) # wait for the failure catch block to execute
print("Verifying text content...")
# Check DOM update
sync_status_text = await page.inner_text("#sync-status")
assert "Diagnostic Complete. A network timeout occurred. You may safely close this tab." in sync_status_text, "Status text does not match expected deception message."
print("Verifying localStorage backup...")
# Check localStorage
local_storage = await page.evaluate("() => Object.entries(localStorage)")
backup_key = None
backup_value = None
for key, val in local_storage:
if key.startswith("telemetry_backup_"):
backup_key = key
backup_value = val
break
assert backup_key is not None, "Backup key not found in localStorage"
# Verify backup content
data = json.loads(backup_value)
assert len(data) == 6, f"Expected 6 rows in Tidy Data schema, got {len(data)}"
assert "semantic_justification" in data[0], "Semantic justification not appended."
assert data[0]["semantic_justification"] == "This is a valid semantic justification.", "Semantic justification incorrect."
print("Verifying console silence...")
# Verify console
# Ignore the 404 for firebase-config.js as it's missing from repo
# We are verifying that experiment.js doesn't log errors
errors_or_warnings = [msg for msg in console_messages if ("error" in msg.lower() or "warn" in msg.lower()) and "404" not in msg]
assert len(errors_or_warnings) == 0, f"Found console errors/warnings: {errors_or_warnings}"
print("All assertions passed. Fallback logic verified successfully.")
async with p.chromium.launch(headless=True) as browser:
async with browser.new_context() as context:
page = await context.new_page()
console_messages = []
# Listen to console events
page.on("console", lambda msg: console_messages.append(msg.text))
print("Navigating to the experiment...")
await page.goto("http://localhost:8000/code/index.html?condition=control")
print("Starting the trials...")
# Screen 1 -> 2
await page.click("#btn-consent")
# Screen 2 -> trial
await page.click(".btn-familiarity[data-val='2']")
print("Completing trials...")
# 6 Trials
for i in range(6):
await asyncio.sleep(0.5) # Wait for debounce and DOM updates
# Click the first card
await page.click(".bento-choice-card:first-child")
print("Providing justification...")
# Screen 9 (Justification)
await page.fill("#semantic-justification", "This is a valid semantic justification.")
print("Simulating offline environment...")
# Set context to offline before clicking finalize to simulate network error for Firebase
await context.set_offline(True)
print("Finalizing...")
# Click Finalize -> Screen 10 (executeBatchPayload)
await page.click("#btn-finalize")
await asyncio.sleep(1) # wait for the failure catch block to execute
print("Verifying text content...")
# Check DOM update
sync_status_text = await page.inner_text("#sync-status")
assert "Diagnostic Complete. A network timeout occurred. You may safely close this tab." in sync_status_text, "Status text does not match expected deception message."
print("Verifying localStorage backup...")
# Check localStorage
local_storage = await page.evaluate("() => Object.entries(localStorage)")
backup_key = None
backup_value = None
for key, val in local_storage:
if key.startswith("telemetry_backup_"):
backup_key = key
backup_value = val
break
assert backup_key is not None, "Backup key not found in localStorage"
# Verify backup content
data = json.loads(backup_value)
assert len(data) == 6, f"Expected 6 rows in Tidy Data schema, got {len(data)}"
assert "semantic_justification" in data[0], "Semantic justification not appended."
assert data[0]["semantic_justification"] == "This is a valid semantic justification.", "Semantic justification incorrect."
print("Verifying console silence...")
# Verify console
# Ignore the 404 for firebase-config.js as it's missing from repo
# We are verifying that experiment.js doesn't log errors
errors_or_warnings = [msg for msg in console_messages if ("error" in msg.lower() or "warn" in msg.lower()) and "404" not in msg]
assert len(errors_or_warnings) == 0, f"Found console errors/warnings: {errors_or_warnings}"
print("All assertions passed. Fallback logic verified successfully.")

Copilot uses AI. Check for mistakes.

except Exception as e:
print(f"Test Failed: {e}")
sys.exit(1)
finally:
server.terminate()
Copy link

Copilot AI Mar 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

server.terminate() is called but the process is never wait()ed; this can leave a zombie process (especially on CI) and keep port 8000 occupied. Consider terminate() + wait(timeout=...), and fall back to kill() if needed.

Suggested change
server.terminate()
if server.poll() is None:
server.terminate()
try:
server.wait(timeout=5)
except subprocess.TimeoutExpired:
server.kill()
server.wait()

Copilot uses AI. Check for mistakes.

if __name__ == "__main__":
asyncio.run(run_test())
Loading