Skip to content

Add ROCm mismatch bypass env flag#466

Closed
jiagaoxiang wants to merge 6 commits intodevfrom
users/douglas/te-version-mismatch-bypass
Closed

Add ROCm mismatch bypass env flag#466
jiagaoxiang wants to merge 6 commits intodevfrom
users/douglas/te-version-mismatch-bypass

Conversation

@jiagaoxiang
Copy link

Summary

  • Make the ROCm runtime/build version check configurable at import time.
  • Keep strict validation as the default behavior.
  • Allow an explicit bypass via NVTE_ALLOW_ROCM_MISMATCH=1 for environments that accept mismatch risk.

Test plan

  • Build/install TransformerEngine in an environment where ROCm runtime and build versions match; verify import succeeds.
  • Simulate a mismatch and verify import fails by default.
  • Set NVTE_ALLOW_ROCM_MISMATCH=1 and verify import succeeds despite mismatch.

Made with Cursor

Mention NVTE_ALLOW_ROCM_MISMATCH=1 directly in the mismatch assertion message so users can self-serve when runtime/build ROCm versions differ.

Made-with: Cursor
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes the ROCm runtime/build version validation at import time configurable, keeping strict mismatch rejection by default while allowing an explicit opt-out via NVTE_ALLOW_ROCM_MISMATCH=1.

Changes:

  • Add NVTE_ALLOW_ROCM_MISMATCH environment flag to bypass ROCm version mismatch validation at import time.
  • Improve the mismatch error message to include bypass instructions and risk disclaimer.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Emit a warning whenever NVTE_ALLOW_ROCM_MISMATCH is set and include whether a mismatch was detected plus runtime/build versions.

Made-with: Cursor
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Emit the ROCm mismatch warning only when NVTE_ALLOW_ROCM_MISMATCH is enabled and a runtime/build mismatch is detected, avoiding noisy logs when versions already match.

Made-with: Cursor
@ROCm ROCm deleted a comment from Copilot AI Feb 28, 2026
@ROCm ROCm deleted a comment from Copilot AI Feb 28, 2026
@jiagaoxiang jiagaoxiang requested a review from Copilot February 28, 2026 08:58
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@jiagaoxiang jiagaoxiang deleted the users/douglas/te-version-mismatch-bypass branch February 28, 2026 09:18
@jiagaoxiang jiagaoxiang reopened this Feb 28, 2026
@jiagaoxiang jiagaoxiang force-pushed the users/douglas/te-version-mismatch-bypass branch from 61a57cd to e117c9d Compare February 28, 2026 09:23
@jiagaoxiang jiagaoxiang requested a review from Copilot February 28, 2026 09:23
@ROCm ROCm deleted a comment from Copilot AI Feb 28, 2026
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ipanfilo
Copy link
Collaborator

ipanfilo commented Mar 3, 2026

Close in favor of #467

@ipanfilo ipanfilo closed this Mar 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants