Skip to content

[P3] Differential testing infrastructure: st8 vs d8#184

Closed
Copilot wants to merge 3 commits intomainfrom
copilot/setup-differential-testing
Closed

[P3] Differential testing infrastructure: st8 vs d8#184
Copilot wants to merge 3 commits intomainfrom
copilot/setup-differential-testing

Conversation

Copy link
Contributor

Copilot AI commented Mar 3, 2026

  • Explore repository structure and understand existing codebase
  • Add NativeFn wrapper and JsValue::NativeFunction variant to stator_core
  • Add thread-local globals (set_global, get_global, clear_globals) to interpreter
  • Handle LdaGlobal/LdaGlobalInsideTypeof/StaGlobal opcodes in interpreter
  • Handle NativeFunction in all Call* and Construct* opcode handlers
  • Implement st8 binary with file/inline execution, print() global, proper exit codes
  • Create tests/diff/ corpus (arithmetic, comparisons, control_flow, functions, closures, exceptions)
  • Create scripts/diff_test.sh (PASS/KNOWN_DIFF/BUG categorisation, corpus mode, single-file mode)
  • Create scripts/known_diffs.txt documenting string literal quote divergence
  • Create .github/workflows/diff_test.yml (PR + nightly CI jobs)
  • Fix CI: replace env: PATH: ... with echo ... >> $GITHUB_PATH step to avoid overriding system PATH (was causing chmod: command not found)
  • All 191 tests pass, clippy clean, cargo fmt applied
Original prompt

This section details on the original issue you should resolve

<issue_title>[P3] Set up differential testing: st8 vs V8 d8</issue_title>
<issue_description>Implement automated differential testing comparing Stator (st8) output against V8 (d8):

Setup:

  • Script runner that executes the same .js file in both st8 and d8
  • Compare: stdout output, return value, thrown exceptions
  • Corpus: Test262 tests + hand-written regression tests + fuzzer-generated inputs

CI integration:

  • Run differential testing on every PR (subset of corpus)
  • Nightly: full corpus run
  • Dashboard showing pass/fail/divergence rates

Divergence categories:

  1. PASS: identical output
  2. KNOWN_DIFF: documented intentional difference (e.g., error message wording)
  3. BUG: unexpected divergence (auto-files issue)

Acceptance criteria:

  • scripts/diff_test.sh <file.js> runs in both engines, reports comparison
  • CI runs subset on every PR
  • No BUG-category divergences in the existing passing tests</issue_description>

Comments on the Issue (you are @copilot in this section)


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

…orkflow

Co-authored-by: telecos <8853689+telecos@users.noreply.github.com>
Copilot AI changed the title [WIP] Set up differential testing for st8 and V8 comparison [P3] Differential testing infrastructure: st8 vs d8 Mar 3, 2026
@telecos telecos marked this pull request as ready for review March 3, 2026 08:52
@codecov
Copy link

codecov bot commented Mar 3, 2026

Codecov Report

❌ Patch coverage is 35.96838% with 162 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
crates/stator_core/src/interpreter/mod.rs 24.85% 130 Missing ⚠️
crates/st8/src/main.rs 61.76% 26 Missing ⚠️
crates/stator_core/src/objects/value.rs 50.00% 6 Missing ⚠️

📢 Thoughts on this report? Let us know!

@telecos
Copy link
Owner

telecos commented Mar 3, 2026

@copilot fix failing ci

…PATH

Co-authored-by: telecos <8853689+telecos@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Mar 3, 2026

@copilot fix failing ci

Fixed in 7b1aa36. The Differential Tests workflow was failing with chmod: command not found because env: PATH: ${{ github.workspace }}/target/release:$PATH replaced the entire system PATH (the $PATH part is a literal string in the env: block, not a shell variable expansion). Replaced it with a dedicated echo ... >> $GITHUB_PATH step, which is the correct GitHub Actions way to prepend a directory to PATH.

@telecos
Copy link
Owner

telecos commented Mar 3, 2026

@copilot the codecov ci is failing can you fix it?

@telecos telecos closed this Mar 3, 2026
Copilot stopped work on behalf of telecos due to an error March 3, 2026 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P3] Set up differential testing: st8 vs V8 d8

2 participants