Skip to content

Introduce tsan pipeline #321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alekstheod wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Introduce tsan pipeline #321

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e7b36ae
Introduce tsan pipeline
alekstheod Feb 19, 2026
78e81eb
Ignore command buffer race
alekstheod Feb 19, 2026
5ce1187
Make tsan builds to be executed locally
alekstheod Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/ci-ut.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ jobs:
- {name: "py3.13", python_version: "3.13", config: ""}
- {name: "py3.14", python_version: "3.14", config: ""}
- {name: "asan", python_version: "3.11", config: "--config=asan"}
# yamllint disable-line rule:line-length
- {name: "tsan", python_version: "3.11", config: "--config=tsan --strategy=TestRunner=local"}
container:
# note this image shall match the one defined in platform/linux:tf_linux_gpu
image: rocm/tensorflow-build@sha256:7fcfbd36b7ac8f6b0805b37c4248e929e31cf5ee3af766c8409dd70d5ab65faa
Expand Down
32 changes: 21 additions & 11 deletions jax_rocm_plugin/.bazelrc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,24 +101,34 @@ build:rocm --copt=-Wno-gnu-offsetof-extensions
build:rocm --copt=-Qunused-arguments
build:rocm --action_env=TF_HIPCC_CLANG="1"

build:asan --strip=never
#############################################################################
# Sanitizer configs
#############################################################################
build:sanitizer --linkopt="-L/usr/lib/llvm-18/lib/clang/18/lib/linux"
build:sanitizer --linkopt="-Wl,-rpath,/usr/lib/llvm-18/lib/clang/18/lib/linux"
build:sanitizer --run_under=//build/rocm:sanitizer_wrapper
build:sanitizer --action_env TF_ROCM_AMDGPU_TARGETS="gfx908,gfx90a,gfx942"
build:sanitizer --strip=never
#O1 because we need to see the asans callstack
build:sanitizer --copt -O1
build:sanitizer --copt -g
build:sanitizer --copt -fno-omit-frame-pointer
build:sanitizer --linkopt -g

build:asan --config=sanitizer
build:asan --copt -fsanitize=address
build:asan --copt -DADDRESS_SANITIZER
#O1 because we need to see the asans callstack
build:asan --copt -O1
build:asan --copt -g
build:asan --copt -gsplit-dwarf
build:asan --copt -fno-omit-frame-pointer
build:asan --linkopt -fsanitize=address
build:asan --linkopt -g1

build:asan --linkopt="-L/usr/lib/llvm-18/lib/clang/18/lib/linux"
build:asan --linkopt="-Wl,-rpath,/usr/lib/llvm-18/lib/clang/18/lib/linux"
build:asan --linkopt="-lclang_rt.asan-x86_64"
build:asan --linkopt="-lclang_rt.asan_cxx-x86_64"
build:asan --//build/rocm:sanitizer=asan
build:asan --run_under=//build/rocm:sanitizer_wrapper
build:asan --action_env TF_ROCM_AMDGPU_TARGETS="gfx908,gfx90a,gfx942"

build:tsan --config=sanitizer
build:tsan --copt -fsanitize=thread
build:tsan --linkopt -fsanitize=thread
build:tsan --linkopt="-l:libclang_rt.tsan-x86_64.so"
build:tsan --//build/rocm:sanitizer=tsan

#############################################################################
# Configuration for running RBE builds and tests
Expand Down
17 changes: 12 additions & 5 deletions jax_rocm_plugin/build/rocm/sanitizer_wrapper.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,32 @@
#!/usr/bin/env bash

ASAN_RT="/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.asan-x86_64.so"

# Only set LD_PRELOAD for the test binary, not for Bazel's wrappers
export LD_PRELOAD="${ASAN_RT}"
TSAN_RT="/usr/lib/llvm-18/lib/clang/18/lib/linux/libclang_rt.tsan-x86_64.so"

# Resolve suppression files from runfiles
ASAN_SUPP="$TEST_SRCDIR/jax_rocm_plugin/build/rocm/asan_ignore_list.txt"
LSAN_SUPP="$TEST_SRCDIR/jax_rocm_plugin/build/rocm/lsan_ignore_list.txt"
TSAN_SUPP="$TEST_SRCDIR/jax_rocm_plugin/build/rocm/tsan_ignore_list.txt"

ASAN_OPTS="use_sigaltstack=0:detect_leaks=0"
LSAN_OPTS="use_sigaltstack=0"
TSAN_OPTS="history_size=7:ignore_noninstrumented_modules=1"

if [[ -f "$ASAN_SUPP" ]]; then
ASAN_OPTS="suppressions=${ASAN_SUPP}:${ASAN_OPTS}"
export LD_PRELOAD="${ASAN_RT}"
ASAN_OPTS="suppressions=${ASAN_SUPP}:${ASAN_OPTS}"
fi
if [[ -f "$LSAN_SUPP" ]]; then
LSAN_OPTS="suppressions=${LSAN_SUPP}:${LSAN_OPTS}"
export LD_PRELOAD="${ASAN_RT}"
LSAN_OPTS="suppressions=${LSAN_SUPP}:${LSAN_OPTS}"
fi
if [[ -f "$TSAN_SUPP" ]]; then
export LD_PRELOAD="${TSAN_RT}"
TSAN_OPTS="suppressions=${TSAN_SUPP}:${TSAN_OPTS}"
fi

export ASAN_OPTIONS="${ASAN_OPTS}"
export LSAN_OPTIONS="${LSAN_OPTS}"
export TSAN_OPTIONS="${TSAN_OPTS}"

exec "$@"
2 changes: 2 additions & 0 deletions jax_rocm_plugin/build/rocm/tsan_ignore_list.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ race:xla::gpu::GpuCompiler::CompileSingleModule
race:xla::LiteralBase::Piece::Storage::Storage
race:xla::LocalClient::TransferFromOutfeedLocal
race:llvm::cl::opt_storage<bool, false, false>::setValue<int>
race:stream_executor::gpu::RocmCommandBuffer::LaunchGraph

# ignore
race:xla::gpu::(anonymous namespace)::RecoverExp2Pattern::initStaticsIfNeeded*
race:lld::lldMain
race:llvm::*
Expand Down
Loading