Reduce madevent 'Fortran overhead' by restricting cudacpp helicity calculation (which scales with SIMD) to only 16 events

[valassi]

Understand if madevent 'Fortran overhead' actually includes a parallelizable c++ component

See the discussion in #943 for CMS DY+3jets

[avalassi@itgold91 gcc11/usr] /data/avalassi/GPU2024/madgraph4gpuX/epochX/cudacpp/tlau/fromgridpacks> ./parseGridpackLogs.sh pp_dy3j.mad
pp_dy3j.mad/fortran/output.txt
[GridPackCmd.launch] OVERALL TOTAL    442.6794 seconds
[madevent COUNTERS]  PROGRAM TOTAL    438.456
[madevent COUNTERS]  Fortran Overhead 131.711
[madevent COUNTERS]  Fortran MEs      306.744
--------------------------------------------------------------------------------
pp_dy3j.mad/cppnone/output.txt
[GridPackCmd.launch] OVERALL TOTAL    734.9419 seconds
[madevent COUNTERS]  PROGRAM TOTAL    730.685
[madevent COUNTERS]  Fortran Overhead 424.426
[madevent COUNTERS]  CudaCpp MEs      306.259
--------------------------------------------------------------------------------
pp_dy3j.mad/cppsse4/output.txt
[GridPackCmd.launch] OVERALL TOTAL    437.2589 seconds
[madevent COUNTERS]  PROGRAM TOTAL    433.014
[madevent COUNTERS]  Fortran Overhead 279.465
[madevent COUNTERS]  CudaCpp MEs      153.548
--------------------------------------------------------------------------------
pp_dy3j.mad/cppavx2/output.txt
[GridPackCmd.launch] OVERALL TOTAL    260.5085 seconds
[madevent COUNTERS]  PROGRAM TOTAL    256.272
[madevent COUNTERS]  Fortran Overhead 193.151
[madevent COUNTERS]  CudaCpp MEs      63.1221
--------------------------------------------------------------------------------
pp_dy3j.mad/cpp512y/output.txt
[GridPackCmd.launch] OVERALL TOTAL    253.3131 seconds
[madevent COUNTERS]  PROGRAM TOTAL    249.01
[madevent COUNTERS]  Fortran Overhead 189.606
[madevent COUNTERS]  CudaCpp MEs      59.4043
--------------------------------------------------------------------------------
pp_dy3j.mad/cpp512z/output.txt
[GridPackCmd.launch] OVERALL TOTAL    206.2917 seconds
[madevent COUNTERS]  PROGRAM TOTAL    202.013
[madevent COUNTERS]  Fortran Overhead 166.504
[madevent COUNTERS]  CudaCpp MEs      35.5097
--------------------------------------------------------------------------------

[valassi]

The post I refer to is #943 (comment)

I wonder, specifically, if there is one part of the code that is affected by C++ -march flags for vectorization.

[valassi]

Renamed #958 as "Understand if madevent 'Fortran overhead' actually includes a (c++?) component that already exploits SIMD?". What I mean is: of course we know that there ARE at least two parts that COULD be made SIMD-parallel, namely random-to-momenta mapping and pdfs, but presently I thought that these were in fortran, and were not parallel...

[valassi]

Note, issue #958 is almost exactly the same as #546 ('reasses timing split between MEs and madevent in tmad tests'). I had seen already the same issue on gg_ttggg.

One idea I had at the time was that reducing the bridge vector size from 8192 to 32 in pure c++ runs could be beneficial (I had some early indications on he juwel HPC that this was the case). I will try this on itgold91 which has no GPU anyway.

[valassi]

Just to be sure, I tried to instrument this, but not surprisingly this is NOT a source of overhead, it is totally negligible

      IF( FBRIDGE_MODE .EQ. 1 .OR. FBRIDGE_MODE .LT. 0 ) THEN ! (CppOnly=1 or BothQuiet=-1 or BothDebug=-2)
        call counters_smatrix1multi_start( 1, VECSIZE_USED ) ! copying=1
        DO IVEC=1, VECSIZE_USED
          OUT(IVEC) = OUT2(IVEC) ! use the cudacpp ME instead of the fortran ME!
          SELECTED_HEL(IVEC) = SELECTED_HEL2(IVEC) ! use the cudacpp helicity instead of the fortran helicity!
          SELECTED_COL(IVEC) = SELECTED_COL2(IVEC) ! use the cudacpp color instead of the fortran color!
        END DO
        call counters_smatrix1multi_stop( 1 ) ! copying=1
      ENDIF

[valassi]

Well ok here's the catch (or one catch): helicity calculation

        IF ( FIRST ) THEN ! exclude first pass (helicity filtering) from timers (#461)
          call counters_smatrix1multi_start( 1, VECSIZE_USED ) ! helicity=1
          CALL FBRIDGESEQUENCE_NOMULTICHANNEL( FBRIDGE_PBRIDGE, ! multi channel disabled for helicity filtering
     &      P_MULTI, ALL_G, HEL_RAND, COL_RAND, OUT2,
     &      SELECTED_HEL2, SELECTED_COL2 )
          FIRST = .FALSE.
c         ! This is a workaround for https://github.com/oliviermattelaer/mg5amc_test/issues/22 (see PR #486)
          IF( FBRIDGE_MODE .EQ. 1 ) THEN ! (CppOnly=1 : SMATRIX1 is not called at all)
            CALL RESET_CUMULATIVE_VARIABLE() ! mimic 'avoid bias of the initialization' within SMATRIX1
          ENDIF
          CALL FBRIDGEGETNGOODHEL(FBRIDGE_PBRIDGE,NGOODHEL,NTOTHEL)
          IF( NTOTHEL .NE. NCOMB ) THEN
            WRITE(6,*) 'ERROR! Cudacpp/Fortran mismatch',
     &        ' in total number of helicities', NTOTHEL, NCOMB
            STOP
          ENDIF
          WRITE (6,*) 'NGOODHEL =', NGOODHEL
          WRITE (6,*) 'NCOMB =', NCOMB
          call counters_smatrix1multi_stop( 1 ) ! helicity=1
        ENDIF

This now gives the following on gg_ttggg

CUDACPP_RUNTIME_VECSIZEUSED=256 ./build.512y_d_inl0_hrd0/madevent_cpp < /tmp/avalassi/input_ggttggg_test_cudacpp 
...
 [COUNTERS] PROGRAM TOTAL          :    4.7272s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.1395s
 [COUNTERS] CudaCpp MEs      ( 2 ) :    3.9084s for     1280 events => throughput is 3.28E+02 events/s
 [COUNTERS] GOODHEL MEs      ( 3 ) :    0.6794s for      256 events => throughput is 3.77E+02 events/s

and

CUDACPP_RUNTIME_VECSIZEUSED=256 ./build.none_d_inl0_hrd0/madevent_cpp < /tmp/avalassi/input_ggttggg_test_cudacpp 
...
 [COUNTERS] PROGRAM TOTAL          :   22.6641s
 [COUNTERS] Fortran Overhead ( 0 ) :    0.1538s
 [COUNTERS] CudaCpp MEs      ( 2 ) :   19.0328s for     1280 events => throughput is 6.73E+01 events/s
 [COUNTERS] GOODHEL MEs      ( 3 ) :    3.4775s for      256 events => throughput is 7.36E+01 events/s

where

cat /tmp/avalassi/input_ggttggg_test_cudacpp
1024 1 1 ! Number of events and max and min iterations
0.000001 ! Accuracy (ignored because max iterations = min iterations)
0 ! Grid Adjustment 0=none, 2=adjust (NB if = 0, ftn26 will still be used if present)
1 ! Suppress Amplitude 1=yes (i.e. use MadEvent single-diagram enhancement)
0 ! Helicity Sum/event 0=exact
1 ! ICONFIG number (1-N) for single-diagram enhancement multi-channel (NB used even if suppress amplitude is 0!)

I have two things to do

• make sure that the helicity calculation is restricted to a reasonable number of events when called from fortran
• possibly (only temporarely while fixing the issue?) add a counter for helicities as in the example here

[valassi]

Renaming, task on myself
"Reduce madevent 'Fortran overhead' by restricting cudacpp helicity calculation (which scales with SIMD) to only 16 events"

I think that I had simply done this wrong :-(

The idea (#461 ?) had been to remove helicit computation from the timers. But it seems that the way it is done now there is simply a real overhead in computing helicities twice, or similar. I need to look at this better next week.

Note, this is alos mildly related to the reset_cumulative_variable issue #872 (although I think that solving this will be irrelevant there).