Add git-bisect script/workflow and generic single-target build/test script#5728
Merged
alliepiper merged 4 commits intoNVIDIA:mainfrom Sep 2, 2025
Merged
Add git-bisect script/workflow and generic single-target build/test script#5728alliepiper merged 4 commits intoNVIDIA:mainfrom
alliepiper merged 4 commits intoNVIDIA:mainfrom
Conversation
816cd69 to
1335ccd
Compare
65c549f to
8fe9ce0
Compare
8fe9ce0 to
044bcf6
Compare
910ad6d to
1c1eb63
Compare
Contributor
Author
|
Most recent run of the bisect workflow: https://github.com/NVIDIA/cccl/actions/runs/17365589071/attempts/1#summary-49291821571 |
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive git bisection tooling and targeted build/test utilities to help developers identify regression commits and efficiently test specific targets. It provides both local and CI-based workflow solutions for automated regression testing.
- Unified build/test script for targeted execution of CMake, ninja, ctest, and lit targets
- Git bisection wrapper that automates regression testing across git history
- GitHub Actions workflow for remote bisection with browser-based job initiation
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
docs/cccl/development/index.rst |
Adds reference to new build and bisect tools documentation |
docs/cccl/development/build_and_bisect_tools.rst |
Comprehensive documentation for the new build/test and git bisect utilities |
ci/util/git_bisect.sh |
Main git bisection script that wraps git bisect around the build/test helper |
ci/util/build_and_test_targets.sh |
Generic script for configuring, building, and testing specific CMake/lit targets |
.github/workflows/git-bisect.yml |
GitHub Actions workflow for running bisection jobs remotely |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Contributor
🟨 CI finished in 1h 15m: Pass: 97%/210 | Total: 1d 17h | Avg: 11m 44s | Max: 38m 02s | Hits: 88%/312237
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
- build_and_test_targets.sh: reproducible single-target configure/build/test (ninja, ctest, lit) - git_bisect.sh: wrapper to automate regression testing with minimal targets - Workflow/Bisect: GitHub Actions job to run bisect remotely and generate Markdown summaries
Contributor
🟨 CI finished in 1h 10m: Pass: 97%/210 | Total: 1d 13h | Avg: 10m 47s | Max: 40m 55s | Hits: 95%/312237
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
Contributor
🟨 CI finished in 1h 12m: Pass: 99%/210 | Total: 1d 17h | Avg: 11m 43s | Max: 52m 48s | Hits: 94%/323908
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
Contributor
🟩 CI finished in 4h 14m: Pass: 100%/210 | Total: 1d 16h | Avg: 11m 42s | Max: 52m 48s | Hits: 94%/325821
|
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| CCCL Packaging | |
| libcu++ | |
| CUB | |
| Thrust | |
| CUDA Experimental | |
| stdpar | |
| python | |
| CCCL C Parallel Library | |
| Catch2Helper | |
| NVBench Helper |
Modifications in project or dependencies?
| Project | |
|---|---|
| +/- | CCCL Infrastructure |
| +/- | CCCL Packaging |
| +/- | libcu++ |
| +/- | CUB |
| +/- | Thrust |
| +/- | CUDA Experimental |
| +/- | stdpar |
| +/- | python |
| +/- | CCCL C Parallel Library |
| +/- | Catch2Helper |
| +/- | NVBench Helper |
🏃 Runner counts (total jobs: 210)
| # | Runner |
|---|---|
| 128 | linux-amd64-cpu16 |
| 23 | windows-amd64-cpu16 |
| 17 | linux-amd64-gpu-l4-latest-1 |
| 12 | linux-arm64-cpu16 |
| 11 | linux-amd64-gpu-rtx2080-latest-1 |
| 10 | linux-amd64-gpu-h100-latest-1 |
| 6 | linux-amd64-gpu-rtxa6000-latest-1 |
| 3 | linux-amd64-gpu-rtx4090-latest-1 |
miscco
approved these changes
Sep 2, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Provides a single script for targeted builds/tests, giving developers and agentic AIs a unified, efficient interface for testing changes / reproducing issues. Supports ninja, ctest, and lit targets. A single command can now configure, build, and test specific targets in a specific devcontainer in the CCCL repo. Enables developers to quickly share reproducible, single-target build/test instructions.
The new targeted build/test script is used to implement a CCCL-specific
git bisectwrapper script that automates regression testing. A single command will launch a devcontainer and startgit bisect run, automatically testing a minimal set of build/test targets across the git history.A new Github Actions workflow has been added that wraps the new
git bisectscript. This allows bisection jobs to be started from the Github Actions UI in the browser, producing a markdown summary detailing the findings.Details and examples are found below.
build_and_test_targets.shci/util/build_and_test_targets.shconfigures, builds, and tests any set of CMake / lit targets.Options
--preset <name>: use a CMake preset.--cmake-options <str>: extra flags for the preset. Ignored if--configure-overrideis given.--configure-override <cmd>: run a custom configure command instead of a preset.--build-targets <targets>: space‑separated Ninja targets. Nothing builds unless specified.--ctest-targets <regex>: space‑separated CTest-Rpatterns. Nothing runs unless specified.--lit-precompile-tests <paths>: space‑separated libcudacxx lit test paths to precompile (no run). Paths are relative tolibcudacxx/test/libcudacxx/.--lit-tests <paths>: space‑separated libcudacxx lit test paths to execute. Paths are relative tolibcudacxx/test/libcudacxx/.--custom-test-cmd <cmd>: arbitrary command executed after build/tests.--configure-overridesupersedes--presetand--cmake-options.Pair with
.devcontainer/launch.sh -dto choose a CUDA toolkit and host compiler:.devcontainer/launch.sh -d --cuda <ver> --host <compiler> -- <script>Examples
Build a single CUB test locally:
ci/util/build_and_test_targets.sh \ --preset cub-cpp20 \ --build-targets "cub.cpp20.test.iterator"Build the same test for SM90 using a CMake option:
Build the test for SM90 with a configure override:
Build and run a single CUB test locally:
Build and run a single CUB test in a devcontainer with specific CTK and host compiler:
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \ ci/util/build_and_test_targets.sh \ --preset cub-cpp20 \ --build-targets "cub.cpp20.test.iterator" \ --ctest-targets "cub.cpp20.test.iterator"Precompile the libcudacxx lit suite:
Precompile a single libcudacxx lit test (no execution):
ci/util/build_and_test_targets.sh \ --preset libcudacxx-cpp20 \ --lit-precompile-tests \ "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"Execute one or more libcudacxx lit tests:
ci/util/build_and_test_targets.sh \ --preset libcudacxx-cpp20 \ --lit-tests \ "std/algorithms/alg.nonmodifying/alg.any_of/any_of.pass.cpp"git_bisect.shci/util/git_bisect.shwrapsgit bisectaround the build/test helper.Options
All options from
build_and_test_targets.sh, plus:--good-ref <rev>: known good commit, tag, or branch.-Ndfor "N days ago".--bad-ref <rev>: known bad commitorigin/main.-Ndfor "N days ago".--summary-file <path>: write Markdown results.Examples
Local CUB bisection from last release to origin/main:
Devcontainer CUB bisection over last week:
.devcontainer/launch.sh -d --cuda 12.3 --host gcc12 --gpus all -- \ ci/util/git_bisect.sh \ --preset cub-cpp20 \ --build-targets "cub.cpp20.test.iterator" \ --ctest-targets "cub.cpp20.test.iterator" \ --good-ref -7dCompute-sanitizer example, knowing regression was 3-4 weeks ago:
.devcontainer/launch.sh -d --cuda 12.9 --host gcc13 --gpus all \ --env CCCL_TEST_MODE=compute-sanitizer-initcheck \ --env C2H_SEED_COUNT_OVERRIDE=1 \ -- ci/util/git_bisect.sh \ --preset "cub-cpp20" \ --build-targets "cub.cpp20.test.iterator" \ --ctest-targets "cub.cpp20.test.iterator" \ --good-ref -28d \ --bad-ref -21dWorkflow/Bisect
GitHub Actions offers a "Workflow/Bisect" job that runs
git_bisect.shremotely.From the repository page:
Example Output
🔎 Bisect Result
thrust::copyinthrust::uninitialized_copy[_n]in CUDA system when possible (Usethrust::copyinthrust::uninitialized_copy[_n]in CUDA system when possible #5181)thrust::copyinthrust::uninitialized_copy[_n]in CUDA system when possible #5181♻️ Reproduction Steps
.devcontainer/launch.sh \ --cuda 12.9 --host gcc13 --env CCCL_TEST_MODE=compute-sanitizer-initcheck --env C2H_SEED_COUNT_OVERRIDE=1 --gpus all \ -- \ ./ci/util/build_and_test_targets.sh \ --preset 'cub-cpp20' \ --cmake-options '-DCMAKE_CUDA_FLAGS=-lineinfo -DCMAKE_CUDA_ARCHITECTURES=86' \ --configure-override '' \ --build-targets 'cub.cpp20.test.iterator' \ --ctest-targets 'cub.cpp20.test.iterator' \ --lit-precompile-tests '' \ --lit-tests '' \ --custom-test-cmd ''ℹ️ Commit Details