#2302: Integrate PAPI in LB data export

[pierrepebay]

Fixes #2302

[github-actions]

Pipelines results

PR tests (gcc-12, ubuntu, mpich, verbose)

Build for 690613e (2024-06-12 19:26:21 UTC)

Build failed for unknown reason. Check build logs

Build log

---

PR tests (gcc-12, ubuntu, mpich, verbose, kokkos)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (gcc-9, ubuntu, mpich, zoltan, json schema test)

Build for 08045af (2024-12-05 13:50:36 UTC)

Build failed for unknown reason. Check build logs

Build log

---

PR tests (gcc-8, ubuntu, mpich, address sanitizer)

Build for 08045af (2024-12-05 13:50:36 UTC)

Build failed for unknown reason. Check build logs

Build log

---

PR tests (gcc-10, ubuntu, openmpi, no LB)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (clang-13, alpine, mpich)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (clang-9, ubuntu, mpich)

Build for eed6191 (2024-12-12 19:46:33 UTC)

Build failed for unknown reason. Check build logs

Build log

---

PR tests (clang-10, ubuntu, mpich)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Build failed for unknown reason. Check build logs

Build log

---

PR tests (clang-13, ubuntu, mpich)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (clang-12, ubuntu, mpich)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (clang-14, ubuntu, mpich, verbose)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (clang-11, ubuntu, mpich)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (nvidia cuda 11.2, gcc-9, ubuntu, mpich)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]"
          detected during:
            instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>]" 
/vt/src/vt/objgroup/proxy/proxy_objgroup.impl.h(221): here
            instantiation of "vt::objgroup::proxy::Proxy<ObjT>::PendingSendType vt::objgroup::proxy::Proxy<ObjT>::reduce<f,Op,Target,Args...>(Target, Args &&...) const [with ObjT=vt::vrt::collection::lb::GreedyLB, f=&vt::vrt::collection::lb::GreedyLB::collectHandler, Op=vt::collective::PlusOp, Target=vt::objgroup::proxy::ProxyElm<vt::vrt::collection::lb::GreedyLB>, Args=<vt::vrt::collection::lb::GreedyPayload>]" 
/vt/src/vt/vrt/collection/balance/greedylb/greedylb.cc(222): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&MyObj::handler, Target=vt::objgroup::proxy::ProxyElm<MyObj>]" 
/vt/examples/callback/callback.cc(147): here

/vt/src/vt/pipe/pipe_manager.impl.h(135): warning: missing return statement at end of non-void function "vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]"
          detected during instantiation of "auto vt::pipe::PipeManager::makeSend<f,Target>(Target) [with f=&colHan, Target=vt::vrt::collection::VrtElmProxy<MyCol, vt::Index1D>]" 
/vt/examples/callback/callback.cc(153%0D%0A%0D%0A%0D%0A ==> And there is more. Read log. <==

Build log

---

PR tests (nvidia cuda 12.2.0, gcc-9, ubuntu, mpich, verbose)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "double" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, double> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=double]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

/vt/lib/CLI/CLI/CLI11.hpp(1029): warning #2361-D: invalid narrowing conversion from "int" to "unsigned long"
          TT { std::declval<CC>() }
               ^
          detected during:
            instantiation of "vt::CLI::detail::is_direct_constructible<T, C>::test [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" based on template arguments <std::vector<std::string, std::allocator<std::string>>, int> at line 1041
            instantiation of class "vt::CLI::detail::is_direct_constructible<T, C> [with T=std::vector<std::string, std::allocator<std::string>>, C=int]" at line 5005
            instantiation of "void vt::CLI::Option::results(T &) const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 5034
            instantiation of "T vt::CLI::Option::as<T>() const [with T=std::vector<std::string, std::allocator<std::string>>]" at line 7315

Testing - passed

Build log

---

PR tests (intel icpx, ubuntu, mpich, verbose)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (gcc-11, ubuntu, mpich, trace runtime, coverage)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Compilation - successful

Testing - passed

Build log

---

PR tests (intel icpc, ubuntu, mpich)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhibited by limit max-total-size 
remark #11076: To get full report use -qopt-report=4 -qopt-report-phase ipo
remark #11074: Inlining inhibited by limit max-size 
remark #11074: Inlining inhi%0D%0A%0D%0A%0D%0A ==> And there is more. Read log. <==

Build log

---

PR tests (gcc-9, ubuntu, mpich, zoltan)

Build for 66fe44c (2024-12-12 22:07:07 UTC)

Build failed for unknown reason. Check build logs

Build log

---

[nlslatt]

This branch needs a lot more testing before it is merged. When I put more than 4 items in VT_EVENTS, I'm seeing memory corruption after the first task collects PAPI events. I'm not sure where in the software stack the problem lies.

[nlslatt]

This branch needs a lot more testing before it is merged. When I put more than 4 items in VT_EVENTS, I'm seeing memory corruption after the first task collects PAPI events. I'm not sure where in the software stack the problem lies.

I also got memory corruption after many tasks when using only 3 items (cache-misses,page-faults,fp_assist.any). I think this PR still needs work.

[nlslatt]

Also, this PR unconditionally requires PAPI. We need an option that turns on PAPI support instead of failing configure in all cases where PAPI is not found.

[nlslatt]

Before this can be merged, it needs to be refactored so that the user interface for perf is very similar to the existing PAPI interface within RunnableNew. In fact, perf and PAPI should share a single interface in RunnableNew (so please rename those member functions to not include PAPI in the name). When perf is chosen at configure time instead of PAPI, the RunnableNew functions will forward the requests to vt::thePerfData()->startTaskMeasurement() and the like. Please let me know if this is not clear.

[lifflander]

When I try to build the container manually for gcc-14 with ubuntu 24.04, I get this error:

19.95 E: Couldn't find any package by glob 'linux-tools-6.4.16-linuxkit'
19.95 E: Couldn't find any package by regex 'linux-tools-6.4.16-linuxkit'

Any ideas?

[pierrepebay]

Putting this into draft as the work containing perf measurements has been added with #2381. We can pick up the remaining work needed to integrate PAPI later whenever this is a priority.