Skip to content

transfer dependencies in Bundle easyblock and fix sanity check for module only#4037

Merged
boegel merged 4 commits intoeasybuilders:developfrom
Thyre:20260106140234_new_pr_bundle
Jan 21, 2026
Merged

transfer dependencies in Bundle easyblock and fix sanity check for module only#4037
boegel merged 4 commits intoeasybuilders:developfrom
Thyre:20260106140234_new_pr_bundle

Conversation

@Thyre
Copy link
Collaborator

@Thyre Thyre commented Jan 6, 2026

(created using eb --new-pr)

Edit:

--sanity-check-only is still broken, see #4037 (comment)

Edit 2:

Maybe solved by 5010c6f


During test builds of ROCm-LLVM, one sanity check was failing on jsc-zen3 due to libraries not being found when trying to link a test program. Investigating further, I was able to create a reproducer EasyConfig and EasyBlock (see #4009 (comment)), which mirrored the same issue.

When building a normal EasyConfig, self.toolchain.dependencies is set to include all the dependencies of the EasyConfig. For a Bundle however, this value is empty. I'm not sure if this is because we're setting most of the values in __init__. Could be.
In Framework, the dependencies are initialized during prepare_step. To correctly transfer the dependencies to components, add a prepare_step to the Bundle EasyBlock, simply transferring the dependencies as they are.

Additionally, --module-only and --sanity-check-only builds were not executing the sanity check of Bundle components. This is due to install-step being skipped in both cases. To fix this, move the logic for determining components, for which the sanity check should run, to the same prepare_step.

Resolves #4008, Resolves #4009

@Thyre Thyre added the bug fix label Jan 6, 2026
@Thyre
Copy link
Collaborator Author

Thyre commented Jan 6, 2026

@boegelbot please test @ jsc-zen3
EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-4037"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=4037 EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-4037" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_4037 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9336

Test results coming soon (I hope)...

Details

- notification for comment with ID 3714731172 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb

  • SUCCESS OpenSSL-3.eb

  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb

  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb

  • SUCCESS gfbf-2024a.eb

  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb

  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb

  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 8 out of 8 (total: 3 hours 13 mins 11 secs) (8 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.23
See https://gist.github.com/boegelbot/d13ff1203e6feee882cbccf42959b9f7 for a full test report.

bedroge
bedroge previously approved these changes Jan 12, 2026
Copy link
Contributor

@bedroge bedroge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lgtm, but maybe @boegel could check it as well.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

I'll try a module only and sanity check only build as well, just too see if we get any unexpected results...

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS foss-2025b.eb

  • SUCCESS buildenv-default-foss-2025b.eb

  • SUCCESS OpenSSL-3.eb

  • SUCCESS gfbf-2025b.eb

  • SUCCESS X11-20250608-GCCcore-14.3.0.eb

Build succeeded for 5 out of 5 (total: 9 mins 22 secs) (5 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (device id: 0x7550, gfx: gfx1201, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), Python 3.14.2
See https://gist.github.com/Thyre/b05928fcadd48a4f2b692e8bf8281320 for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

--module-only


Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS foss-2025b.eb

  • SUCCESS buildenv-default-foss-2025b.eb

  • SUCCESS OpenSSL-3.eb

  • SUCCESS gfbf-2025b.eb

  • SUCCESS X11-20250608-GCCcore-14.3.0.eb

Build succeeded for 5 out of 5 (total: 1 min 5 secs) (5 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (device id: 0x7550, gfx: gfx1201, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), Python 3.14.2
See https://gist.github.com/Thyre/81143313c8cb4403b74f1d45faa04b3a for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

--sanity-check-only


Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS foss-2025b.eb

  • FAIL buildenv-default-foss-2025b.eb (unhandled exception: expected str, bytes or os.PathLike object, not NoneType)

Traceback (most recent call last):
File "/data/EasyBuild/lib/python3.14/site-packages/easybuild/main.py", line 180, in build_and_install_software
  (ec_res['success'], app_log, err_msg, err_code) = build_and_install_one(ec, init_env)
                                                    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
File "/data/EasyBuild/lib/python3.14/site-packages/easybuild/framework/easyblock.py", line 5109, in build_and_install_one
  result = app.run_all_steps(run_test_cases=run_test_cases)
File "/data/EasyBuild/lib/python3.14/site-packages/easybuild/framework/easyblock.py", line 4922, in run_all_steps
  self.run_step(step_name, step_methods)
  ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/EasyBuild/lib/python3.14/site-packages/easybuild/framework/easyblock.py", line 4763, in run_step
  current_method()
  ~~~~~~~~~~~~~~^^
File "/tmp/eb-jq353oc9/included-easyblocks-qvyk046p/easybuild/easyblocks/generic/bundle.py", line 486, in sanity_check_step
  fake_mod_data = self.load_fake_module(purge=True)
File "/data/EasyBuild/lib/python3.14/site-packages/easybuild/framework/easyblock.py", line 1902, in load_fake_module
  fake_mod_path = self.make_module_step(fake=True)
File "/data/EasyBuild/lib/python3.14/site-packages/easybuild/easyblocks/generic/buildenv.py", line 119, in make_module_step
  wrappers_dir = os.path.join(self.rpath_wrappers_dir, RPATH_WRAPPERS_SUBDIR)
File "<frozen posixpath>", line 77, in join
TypeError: expected str, bytes or os.PathLike object, not NoneType

  • SUCCESS OpenSSL-3.eb

  • SUCCESS gfbf-2025b.eb

  • SUCCESS X11-20250608-GCCcore-14.3.0.eb

Build succeeded for 4 out of 5 (total: 4 secs) (5 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (device id: 0x7550, gfx: gfx1201, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), Python 3.14.2
See https://gist.github.com/Thyre/d3c52bb9f4d5ffcc74446981668fde3c for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

Looks like I found two (unrelated) bugs with buildenv during testing...

  >> generating module file @ /data/EasyBuild-develop/modules/all/MPI/GCC/14.3.0/OpenMPI/5.0.8/buildenv/default.lua
== comparing module file with backup /data/EasyBuild-develop/modules/all/MPI/GCC/14.3.0/OpenMPI/5.0.8/buildenv/default.bak_20260112165059_1226463; diff is:
--- /data/EasyBuild-develop/modules/all/MPI/GCC/14.3.0/OpenMPI/5.0.8/buildenv/default.bak_20260112165059_1226463
+++ /data/EasyBuild-develop/modules/all/MPI/GCC/14.3.0/OpenMPI/5.0.8/buildenv/default.lua
@@ -31,12 +31,12 @@
 depends_on("ScaLAPACK/2.2.2-fb")

 prepend_path("CMAKE_PREFIX_PATH", root)
-prepend_path("PATH", "/data/EasyBuild-develop/software/buildenv/default-foss-2025b/bin/rpath_wrappers/gcc_wrapper")
-prepend_path("PATH", "/data/EasyBuild-develop/software/buildenv/default-foss-2025b/bin/rpath_wrappers/ld.gold_wrapper")
-prepend_path("PATH", "/data/EasyBuild-develop/software/buildenv/default-foss-2025b/bin/rpath_wrappers/gxx_wrapper")
-prepend_path("PATH", "/data/EasyBuild-develop/software/buildenv/default-foss-2025b/bin/rpath_wrappers/ld.bfd_wrapper")
-prepend_path("PATH", "/data/EasyBuild-develop/software/buildenv/default-foss-2025b/bin/rpath_wrappers/ld_wrapper")
-prepend_path("PATH", "/data/EasyBuild-develop/software/buildenv/default-foss-2025b/bin/rpath_wrappers/gfortran_wrapper")
+prepend_path("PATH", "/data/EasyBuild-develop/build/buildenv/default/foss-2025b/rpath_wrappers/gcc_wrapper")
+prepend_path("PATH", "/data/EasyBuild-develop/build/buildenv/default/foss-2025b/rpath_wrappers/ld.gold_wrapper")
+prepend_path("PATH", "/data/EasyBuild-develop/build/buildenv/default/foss-2025b/rpath_wrappers/gxx_wrapper")
+prepend_path("PATH", "/data/EasyBuild-develop/build/buildenv/default/foss-2025b/rpath_wrappers/ld.bfd_wrapper")
+prepend_path("PATH", "/data/EasyBuild-develop/build/buildenv/default/foss-2025b/rpath_wrappers/ld_wrapper")
+prepend_path("PATH", "/data/EasyBuild-develop/build/buildenv/default/foss-2025b/rpath_wrappers/gfortran_wrapper")

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

--sanity-check-only still doesn't behave as expected. This PR tries to fix both issues by using prepare_step. However, it looks like this step is skipped entirely when one passes --sanity-check-only. get_steps defines prepare_step as a step that is not skippable, yet, this is ignored in skip_step. This skippable attribute comes only into play when using --skip.

I'd consider this a bug, but maybe there's a good reason for it. Unfortunately, I don't see how we can get this fixed for --sanity-check-only, as we need to be after __init__.

See also https://easybuild.slack.com/archives/C34UA1HT7/p1768235026763329

Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

Latest commit should've fixed the sanity check, tested with a manual change in the X11 EasyConfig.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

--sanity-check-only


Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 4 out of 5 (total: 4 secs) (5 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (device id: 0x7550, gfx: gfx1201, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), Python 3.14.2
See https://gist.github.com/Thyre/95b694add3497ec9cbc650dbad477f6f for a full test report.

@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

--module-only


Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS foss-2025b.eb

  • SUCCESS buildenv-default-foss-2025b.eb

  • SUCCESS OpenSSL-3.eb

  • SUCCESS gfbf-2025b.eb

  • SUCCESS X11-20250608-GCCcore-14.3.0.eb

Build succeeded for 5 out of 5 (total: 1 min 16 secs) (5 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (device id: 0x7550, gfx: gfx1201, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), Python 3.14.2
See https://gist.github.com/Thyre/ee6dcd1be44b9f49f8a835b58b57d0de for a full test report.

@boegel boegel added this to the next release (5.2.1?) milestone Jan 12, 2026
@Thyre
Copy link
Collaborator Author

Thyre commented Jan 12, 2026

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS foss-2025b.eb

  • SUCCESS buildenv-default-foss-2025b.eb

  • SUCCESS OpenSSL-3.eb

  • SUCCESS gfbf-2025b.eb

  • SUCCESS X11-20250608-GCCcore-14.3.0.eb

Build succeeded for 5 out of 5 (total: 9 mins 28 secs) (5 easyconfigs in total)
Linux - Linux Arch Linux UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x AMD Navi 48 [Radeon RX 9070/9070 XT/9070 GRE] (device id: 0x7550, gfx: gfx1201, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), 1 x AMD Raphael (device id: 0x164e, gfx: gfx1036, driver: Linuxversion6.18.4-arch1-1(linux@archlinux)(gcc(GCC)15.2.120260103,GNUld(GNUBinutils)2.45.1)#1SMPPREEMPT_DYNAMICFri,09Jan202619:43:48+0000), Python 3.14.2
See https://gist.github.com/Thyre/77d2c9ef8f1570f04bf9f3085edd5cc4 for a full test report.

@boegel
Copy link
Member

boegel commented Jan 20, 2026

@boegelbot please test @ jsc-zen3
CORE_CNT=16
EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-4037"

Copy link
Member

@boegel boegel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@boegelbot
Copy link

@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=4037 EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb --installpath=/tmp/$USER/ebpr-4037" EB_CONTAINER= EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_4037 --ntasks="16" ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 9448

Test results coming soon (I hope)...

Details

- notification for comment with ID 3774254980 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb

  • SUCCESS OpenSSL-3.eb

  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb

  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb

  • SUCCESS gfbf-2024a.eb

  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb

  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb

  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 8 out of 8 (total: 3 hours 6 mins 25 secs) (8 easyconfigs in total)
jsczen3c3.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.7, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.23
See https://gist.github.com/boegelbot/abf27000289030c773e1618fac645780 for a full test report.

@boegel
Copy link
Member

boegel commented Jan 20, 2026

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS zlib-ng-2.2.1-GCCcore-13.3.0.eb

  • SUCCESS tokenizers-0.21.4-GCCcore-13.3.0.eb

  • SUCCESS X11-20250608-GCCcore-14.3.0.eb

  • SUCCESS Xvfb-21.1.9-GCCcore-13.2.0.eb

  • SUCCESS BioPerl-1.7.8-GCCcore-13.3.0.eb

  • SUCCESS SentencePiece-0.2.0-GCC-13.2.0.eb

  • SUCCESS FlexiBLAS-3.4.5-GCC-14.3.0.eb

  • SUCCESS intel-compilers-2025.2.0-CUDA-12.9.0.eb

Build succeeded for 8 out of 8 (total: 1 hour 42 mins 19 secs) (8 easyconfigs in total)
node4247.shinx.os - Linux RHEL 9.6, x86_64, AMD EPYC 9654 96-Core Processor (zen4), Python 3.9.21
See https://gist.github.com/boegel/9a11af2ef40cce744bcfb50d4206bdc0 for a full test report.

@boegel boegel merged commit f8ee0db into easybuilders:develop Jan 21, 2026
22 checks passed
@Flamefire
Copy link
Contributor

@Thyre For super() calls you must NOT pass self explicitly as in super().prepare_step(self, *args, **kwargs) or you'll have self twice overwriting other args!

Noticed (and fixed) while resolving conflict in #3778

@Flamefire
Copy link
Contributor

Fix only in #4057

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

5 participants