Skip to content

Add benchmark CI for performance regression detection #106

@m96-chan

Description

@m96-chan

Summary

A bench.rs file exists in nxpu-e2e-tests but there is no CI integration to track performance over time or detect regressions.

Motivation

  • Optimization passes and backend changes can silently degrade compile-time performance
  • Without baseline tracking, regressions go unnoticed until users report slowness
  • Automated benchmark comparison on PRs catches issues early

Proposed Approach

  1. Benchmark framework: Migrate to criterion.rs for statistical benchmarking
  2. CI integration options:
    • bencher.dev — Free for open-source, GitHub integration
    • github-action-benchmark — Stores results in gh-pages branch
    • Custom: store baseline in artifacts, compare with critcmp
  3. Benchmark targets:
    • Parse time (each example WGSL file)
    • Optimization pass time (per pass and total pipeline)
    • Backend emit time (per backend)
    • End-to-end compile time (WGSL → output format)

Acceptance Criteria

  • criterion.rs benchmarks for key pipeline stages
  • CI job that runs benchmarks and compares against baseline
  • PR comments showing benchmark diff (or link to dashboard)
  • Alert/failure on significant regressions (e.g., >10% slowdown)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions