Kokkos integration: CI testing of MIRCO built with GPU-backend Kokkos build

[PhilipOesterlePekrun]

Description

At the moment, there only exists a CI/workflow test for MIRCO that uses a Kokkos build with OpenMP backend. It is also important to continuously test changes on a GPU backend (such as CUDA), due to the fact that memory spaces are separated in a GPU build but not in an OpenMP build. Github may not provide the hardware for this, so this might not be able to happen soon.

Related Issues and Merge Requests

• Related to Kokkos integration and performance improvements #117
• Is blocked by Kokkos integration: compilation failing with Kokkos CUDA backend due to conflict between ryml and nvcc #121

[mayrmt]

@maxfirmbach do you know, how others do that? Is providing our own hardware the only solution?

[PhilipOesterlePekrun]

@mayrmt It seems GPU runners ("larger runners") are not available for free, so running on our own hardware would be best.

https://docs.github.com/en/billing/concepts/product-billing/github-actions: "Included minutes cannot be used for larger runners. These runners will always be charged for, even when used by public repositories."

[mayrmt]

Thanks for looking into this! We will have to discuss this internally. So far, we do not test on our cluster (where our GPUs are located). Not sure, if and how we want to change that.

@reginabuehler @knarfnitram, I'll bring this up when we all are back in the office.

[maxfirmbach]

@mayrmt @PhilipOesterlePekrun I think testing on GPUs also highly depends on the hardware one is looking for. There are different vendors (considering Kokkos it's mainly AMD, Nvidia, Intel), but I guess we are mainly looking at Nvidia based GPUs. I assume the easiest way would be to have a testing machine with the respective GPU and use that. The testing is also not expensive in compute so actually a cheap computer would actually already do the job. If MPI is used at some point (not necessarily in this code project, but for integration in others), things of course scale out e.g. 4 MPI ranks -> 4 GPUs.

[maxfirmbach]

I think it also makes sense to test with the serial Kokkos backend.

[PhilipOesterlePekrun]

@maxfirmbach Yes, it depends a bit on hardware, and we have only used Cuda backend so far. But the main reason I would want to test this is because writing code for a host+device (CPU+GPU) setup requires managing the data with host vs device views and doing deep_copy and such correctly, a bit like MPI. You can quite easily accidentally write code that works with a purely-host backend but not with a host+device.

Yes, we can also test with Serial backend of course--I'll add it shortly.

[mayrmt]

I am not aware of any suitable hardware at the institute at the moment. Correct me if I'm when.