Skip to content

WIP: Isolate ARMBUILD-Ubuntu-24.04-arm CI for failure diagnosis#5948

Open
hjmjohnson wants to merge 2 commits intoInsightSoftwareConsortium:mainfrom
hjmjohnson:isolate-arm-ci-failures
Open

WIP: Isolate ARMBUILD-Ubuntu-24.04-arm CI for failure diagnosis#5948
hjmjohnson wants to merge 2 commits intoInsightSoftwareConsortium:mainfrom
hjmjohnson:isolate-arm-ci-failures

Conversation

@hjmjohnson
Copy link
Member

Summary

  • Temporarily disable all CI pipelines except ARMBUILD-Ubuntu-24.04-arm to isolate and diagnose dashboard failures reported on ARM platforms
  • GitHub Actions: Remove x86_64-rosetta and Python matrix entries from arm.yml; disable PR trigger in pixi.yml
  • Azure Pipelines: Set pr: none for Linux, LinuxPython, MacOS, MacOSPython, Windows, WindowsPython

Context

CI logs show passing tests (0 failures) but CDash flags false-positive errors from test output containing error-like strings (e.g., "Test FAILED !" from itkConicShellInteriorExteriorSpatialFunctionTest, "ITK ERROR: Error" from itkMultiThreaderExceptionsTest, "FAILED: FIXME MetaImage library" from itkReadWriteImageWithDictionaryTest). This PR isolates the ARM Ubuntu build to diagnose these dashboard reporting issues.

This is a WIP commit — all CI changes will be reverted before merge.

Test plan

  • Only ARMBUILD-Ubuntu-24.04-arm CI runs on this PR
  • Verify dashboard error reporting against CI logs

🤖 Generated with Claude Code

Temporarily disable all CI pipelines except the ARM Ubuntu build
to isolate dashboard failures for debugging.

- arm.yml: Remove x86_64-rosetta and Python matrix entries
- pixi.yml: Comment out PR trigger
- Azure pipelines: Set pr: none for Linux, LinuxPython, MacOS,
  MacOSPython, Windows, WindowsPython

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions bot added type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct labels Mar 14, 2026
@hjmjohnson
Copy link
Member Author

@blowekamp

FYI: The following build warnings only show up under the linux arm builds:

└─[0] rg "warning: " 1_ARMBUILD-Ubuntu-24.04-arm.txt|fgrep -v libpng                             (isolate-arm-ci-failures|…1⚑9)
2026-03-13T23:54:34.6885757Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-13T23:54:34.6934207Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-13T23:54:34.6951147Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:06:13.5784560Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:06:56.2395410Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:08:56.5683796Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:24:19.7212980Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]
2026-03-14T00:24:19.7420150Z ../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]

and there were some changes made to EIgen yesterday.

@hjmjohnson
Copy link
Member Author

@blowekamp

A little AI evaluation of the differences for the passing and failing logs is narrowing down the issue with what look to be very plausible:

⏺ The ThirdParty patterns on lines 80-92 use a format like ".*[/\\][Mm]odules[/\\][Tt]hird[Pp]arty[/\\][Specific]..." but there's no Eigen3 entry. Lines 127 and 130 add GoogleTest and DoubleConversion entries. But the Eigen3 warnings have this path format:

../Modules/ThirdParty/Eigen3/src/itkeigen/Eigen/src/Core/...

Let me check if the existing ThirdParty warning exceptions match this. The patterns on lines 80-92 are for specific subdirectories (DICOMParser, Expat, JPEG, etc.) — there's no catch-all for all of ThirdParty.

The warning is: Eigen/src/Core/products/SelfadjointMatrixVector.h:224:51: warning: 'result' may be used uninitialized [-Wmaybe-uninitialized]

This only appears on GCC (Ubuntu ARM) and not on Clang (macOS Rosetta), which is typical for -Wmaybe-uninitialized — it's a GCC-only warning that Clang doesn't produce.

Root cause: The ARMBUILD-Ubuntu-24.04-arm CI fails because GCC on ARM produces 8 -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h, which are counted by ctest_build(NUMBER_WARNINGS build_warnings). The itk_common.cmake script (line 495-498) treats any
build_warnings > 0 as a CI failure. These Eigen3 warnings are not suppressed by CTEST_CUSTOM_WARNING_EXCEPTION because there's no pattern matching Eigen3 in the exception list — unlike other ThirdParty subdirectories (JPEG, TIFF, VNL, etc.) that each have their own exception
entry.

The Rosetta build passes because Clang doesn't emit -Wmaybe-uninitialized.

@hjmjohnson
Copy link
Member Author

@blowekamp FYI:

⏺ Based on the analysis, the GCC -Wmaybe-uninitialized warnings from Eigen3's SelfadjointMatrixVector.h have been fixed upstream through a substantial rewrite of the algorithm. The warning-triggering code no longer exists in the upstream version.

Replace the vendored SelfadjointMatrixVector.h with the upstream version
from Eigen commit 662d5c21f ("Optimize SYMV, SYR, SYR2, and TRMV product
kernels", 2026-03-01). The upstream rewrite uses a 4-col/2-col/1-col
phased algorithm with proper pzero(Packet{}) initialization, which
eliminates GCC -Wmaybe-uninitialized warnings that were causing ARM CI
dashboard failures (build_warnings > 0 treated as CI error by
itk_common.cmake).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions bot added the area:ThirdParty Issues affecting the ThirdParty module label Mar 14, 2026
@blowekamp
Copy link
Member

@seanm ref#5876

There are more warnings suppression here for the third party than I realized:
https://github.com/InsightSoftwareConsortium/ITK/blob/main/CMake/CTestCustom.cmake.in#L80

Adding Eigen3 here seem reasonable if that is the convention. I am not sure how the changed yesterday caused this warning to be emitted, maybe it was already being suppressed somehow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:ThirdParty Issues affecting the ThirdParty module type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Testing Ensure that the purpose of a class is met/the results on a wide set of test cases are correct

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants