Upgrade to Spack v1.1.1 and related fixes. #196

nkoukpaizan · 2026-01-29T18:50:24Z

Merge request type

New feature
Resolves bug
Documentation
Other

Relates to

This MR updates

Summary

This upgrades the Spack submodule to v1.1.1 and deprecates incompatible build configurations (deception, newell and incline) that are not being tested with the new configuration.

With the Spack upgrade, now pointing to upstream Spack and using local Spack repos for modified packages that have yet to be contributed upstream. I plan to submit appropriate PRs to spack-packages once we know this works.
Updated Frontier modules accordingly. romcc is no longer a supported compiler, and mixes compiler modules are deprecated on Frontier. The only way to get a performant Fortran compiler for coinhsl, etc., and support HIP is to load PrgEnv-gnu and rocm. So, most packages are built built withgcc unless there is HIP code.
Moved deprecated configurations to deprecated folders with corresponding README files.

Linked Issue(s)

Partially addresses Transition dependencies from submodules into spack #38 for spdlog

pelesh · 2026-01-29T21:05:29Z

CC @cameronrutherford

This reverts commit aff53c5.

nkoukpaizan · 2026-01-30T19:24:38Z

@cameronrutherford Do you know a fix for the error I'm seeing?

Error: invalid push configuration for mirror local-buildcache: environment variable `***` (secret_variable) not set

Everything is building fine, but the push to the container registry is not working.

nkoukpaizan · 2026-01-30T21:25:04Z

@cameronrutherford Do you know a fix for the error I'm seeing?
Error: invalid push configuration for mirror local-buildcache: environment variable `***` (secret_variable) not set
Everything is building fine, but the push to the container registry is not working.

Finally figured out a way to make it work!

cameronrutherford

Some NITs, some other more interesting comments.

cameronrutherford · 2026-01-31T02:08:52Z

.github/workflows/spack_cpu_build.yaml


-        #- name: Test Build
-        #  run: cd $(spack -e . location --build-dir exago@develop) && ctest -VV
+      #- name: Test Build


I can't remember why this is commented out.. perhaps there's an issue for it? Would be good to test things work before pushing.

The reason was that no tests were ever running in GH-Actions. We've been relying on manually testing on Frontier, and just checking that things build in GH-Actions.

The Spack package relied on Spack's --run-tests to set EXAGO_RUN_TESTS. If I recall correctly, we can't used the built-in testing because of the need to install first. I've just added an independent testing option in the Spack package, independent of --run-tests, which revealed a number of failing tests. I'll have to look into it.

I think we could fix easily the issue where we need to install before testing. As far as I remember, there was an issue about finding test input files and it was "fixed" by installing them first, and then accessing them in the installation directory. Also see #148.

buildsystem/clang-hip/frontier/base.sh

buildsystem/spack/load_spack.sh

buildsystem/spack/spack_repo/exago/packages/exago/__pycache__/package.cpython-311.pyc

buildsystem/spack/spack_repo/exago/packages/coinhsl/package.py

buildsystem/spack/spack_repo/exago/packages/hiop/package.py

buildsystem/spack/spack_repo/exago/packages/hiop/__pycache__/package.cpython-311.pyc

cameronrutherford · 2026-01-31T02:27:51Z

CMakeLists.txt

-  # -fPIC required for spdlog for use with python wrapper
-  set_property(TARGET spdlog PROPERTY POSITION_INDEPENDENT_CODE ON)
+  find_package(spdlog REQUIRED)
+  find_package(fmt REQUIRED)


Is the Python wrapper needing position independent code not relevant anymore?

Also, should we give a hint to the path here for the submodule?

Perhaps we only enforce that property here / in spack when Python is enabled.

I'm requiring spdlog as an external dependency here, so I wouldn't use the submodule and PIC does not apply to the external target.

cameronrutherford · 2026-01-31T02:31:56Z

buildsystem/spack/spack_repo/exago/packages/exago/package.py

+def submodules(package):
+    submodules = []
+    submodules.append("tpl/pybind11")
+    submodules.append("tpl/spdlog")


If you're making this an external dependency, then we should probably consider doing the following in pursuit of removing submodules (I think this is an existing issue / desirable goal):

Figure out a way to only clone some submodules for different versions of ExaGO. I imagine it's doable as package is passed to the function.

Add this as a dependency at the spack level (build time, link time?) for only certain versions.

Figure out how to express the dependency for PIC if it's necessary only for the Python wrapper(?)

This can also be another issue for later if you just want this merged and it works IMO.

I only made spdlog an external dependency, since that is truly optional. We can remove the submodule in a subsequent PR once the edits are propagated upstream. We will also need to add patches to older versions when we make that change. The Spack-level dependency only applies to develop at the moment.

I have not addressed pybind11 and toml11. One thing to consider it that making these external puts the burden on the user to have them installed.

I would suggest we discuss it a little more before we decide to treat toml and pybind as external packages. ExaGO already has too many dependencies and I agree with @nkoukpaizan that we need to minimize burden on the user.

buildsystem/spack/spack_repo/exago/packages/exago/package.py

… --run-tests.

cameronrutherford · 2026-02-01T22:27:49Z

LGTM if CI passes. I'm sure you'll get a better review from a maintainer about the package.py recipe anyway.

pelesh · 2026-02-01T23:25:59Z

buildsystem/spack/spack_repo/exago/packages/hiop/package.py

+    variant("sparse", default=False, description="Enable/Disable Sparse linear algebra")
+    variant(


I think HiOp should be built with sparse support by default. We are going to deprecate mixed dense-sparse module in ExaGO and PriDec and sparse HiOp modules will be pretty much all we'll need.

That is the HiOp package, so I'll argue that the default value falls under the purview of HiOp. Not ExaGO.

Actually, HiOp default is that sparse linear algebra is on. Perhaps the HiOp spack package has not been updated accordingly.

Upgrade to Spack v1.1.1 and related fixes.

145db32

nkoukpaizan self-assigned this Jan 29, 2026

nkoukpaizan added enhancement New feature or request spack-build labels Jan 29, 2026

nkoukpaizan added 2 commits January 29, 2026 15:38

Increment registry version.

9b9c105

depend on spdlog and fmt when exago@develop.

921dd73

Spack repo add in CI.

7412f32

pelesh requested review from abhyshr and pelesh January 29, 2026 22:25

nkoukpaizan added 3 commits January 30, 2026 11:21

Try ghcr.io/ornl instead of ghcr.io/pnnl.

aff53c5

Revert "Try ghcr.io/ornl instead of ghcr.io/pnnl."

22be81d

This reverts commit aff53c5.

Try setting GITHUB_TOKEN in env to push the build cache.

5c660aa

pelesh requested a review from cameronrutherford January 30, 2026 19:03

nkoukpaizan added 2 commits January 30, 2026 14:28

mirror set --> mirror add; test push to registry without building.

3f61c9d

Add GITHUB_USER in env to push buildcache.

688aee2

nkoukpaizan force-pushed the nicholson/spack-v1.1 branch from f52f32a to ea3b5fd Compare January 30, 2026 19:51

Set access_pair in mirror configuration.

021f9dd

nkoukpaizan force-pushed the nicholson/spack-v1.1 branch from ea3b5fd to 021f9dd Compare January 30, 2026 19:55

Re-activate Spack build in CI, as buildcache push seems to be working.

2f1892e

nkoukpaizan marked this pull request as ready for review January 30, 2026 21:25

cameronrutherford requested changes Jan 31, 2026

View reviewed changes

nkoukpaizan added 3 commits February 1, 2026 14:38

[skip ci] Fix typo.

883a00b

[skip ci] Remove __pycache__.

7d6f2fe

[skip ci] Fix indent.

5455ed1

nkoukpaizan mentioned this pull request Feb 1, 2026

Transition dependencies from submodules into spack #38

Open

3 tasks

nkoukpaizan added 2 commits February 1, 2026 15:13

Test build CI step.

dc2fdf6

Always freshly build ExaGO in CI.

33081cf

nkoukpaizan force-pushed the nicholson/spack-v1.1 branch from 64de6e7 to 33081cf Compare February 1, 2026 20:27

Add 'testing' option to ExaGO Spack package (independent from Spack's…

20ee3f7

… --run-tests.

pelesh reviewed Feb 1, 2026

View reviewed changes

		variant("sparse", default=False, description="Enable/Disable Sparse linear algebra")
		variant(

Upgrade to Spack v1.1.1 and related fixes. #196

Are you sure you want to change the base?

Upgrade to Spack v1.1.1 and related fixes. #196

Uh oh!

Conversation

nkoukpaizan commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelesh commented Jan 29, 2026

Uh oh!

nkoukpaizan commented Jan 30, 2026

Uh oh!

nkoukpaizan commented Jan 30, 2026

Uh oh!

cameronrutherford left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cameronrutherford Jan 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nkoukpaizan Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

cameronrutherford commented Feb 1, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nkoukpaizan commented Jan 29, 2026 •

edited

Loading

cameronrutherford Jan 31, 2026 •

edited

Loading

nkoukpaizan Feb 1, 2026 •

edited

Loading