Add OMP pragmas in modg kernel

[haraldkl]

See PR in ateles-source.

[github-actions]

Waf Unit Tests

  2 files    2 suites   0s ⏱️
104 tests 104 ✅ 0 💤 0 ❌
208 runs  208 ✅ 0 💤 0 ❌

Results for commit 104db14.

[haraldkl]

Note: for compilation with OpenMP, you need to run waf with the following flags:

--openmp --fc_delflags='-fmax-stack-var-size=131072'

[haraldkl]

3 unit tests fail when compiling with OpenMP:

• atl/leg_coeff_differentiation_test
• atl/atl_varSys_gradientOperator_test
• polynomials/ply_fpt_3D_performance_test

The performance one may simple be from running on a different machine.

  tests that fail 3/104
    atl/leg_coeff_differentiation_test
 Starting up Ateles unit test with nprocs:            1
                and nThreads pp:           12
 -------- CHECKING LEGCOEFF  DIFF FOR DIM =            1
 --------------------------- DONE
 -------- CHECKING LEGCOEFF  DIFF FOR DIM =            2
 --------------------------- DONE
 -------- CHECKING LEGCOEFF  DIFF FOR DIM =            3
 --------------------------- DONE
 FAILED error=   1399.0000000000000

    atl/atl_varSys_gradientOperator_test
 Starting up atl_varSys_gradientOperator_test with nprocs:            1
                and nThreads pp:           12
 Initialize the mesh
--------------------------------------------------------------------------------
Loading general parameters:
Using /dev/null as null device.
Will print run time info in the end
(/proc/self/status).
Size of the IO Buffer (MB): 8
Simulation Name: simulation
 +-------------------------------------------+
  - Now:
  iterations:            0
  simTime   :    0.0000000000000000
  wallClock :    2.1978500000000001E-004

  - Time Control:
 +-------------------------------------------+
Iterations between trigger checks: 1
  - Min:
  iterations:            0
  simTime   :    0.0000000000000000

  - Max:

  - Interval:
 +-------------------------------------------+

  - Abort Criteria:
  stop_file:
  steady_state:  F

  - Status:
 +-------------------------------------------+
 Using global alltoall
 Loading communication pattern:
 isend_irecv
Write timings to: timing.res
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
 Generating an internally defined mesh
 Creating a full cubical mesh without boundaries
 Got a mesh with following properties:
                 Mesh name: Generic_Cube
 This is a predefined mesh: cube
  Total number of elements: 8
  Local number of elements: 8
      Number of properties: 0
      Number of partitions: 1
             Minimum level: 1
             Maximum level: 1
       Bounding Box Origin:     0.0000    0.0000    0.0000
       Bounding Box Length:     4.0000
     Coarsest element size:     2.0000
       Finest element size:     2.0000
--------------------------------------------------------------------------------
 Initializing state
 Initializing scheme
 Init MODG scheme ...
 MODG scheme with Q basis on level            1
 ... maximal polynomial degree:            2
 Initializing projection
 Using an oversampled order of:            6
 maxpolydegree is:            2
 (Actual oversampling factor:    2.00000000     )
 Initializing FPT
 subblockingWidth =            8
 Number of Cheb. coeffs for matrix function approximation:           18
 Smallest subblock size for evaluation of M:           64
 Corrected smallest subblock size for evaluation of M:            6
 Initializing FPT
 subblockingWidth =            8
 Number of Cheb. coeffs for matrix function approximation:           18
 Smallest subblock size for evaluation of M:           64
 Corrected smallest subblock size for evaluation of M:            6
 Initializing FPT
 subblockingWidth =            8
 Number of Cheb. coeffs for matrix function approximation:           18
 Smallest subblock size for evaluation of M:           64
 Corrected smallest subblock size for evaluation of M:            6
 Initializing FPT
 subblockingWidth =            8
 Number of Cheb. coeffs for matrix function approximation:           18
 Smallest subblock size for evaluation of M:           64
 Corrected smallest subblock size for evaluation of M:            6
 Initializing FPT
 subblockingWidth =            8
 Number of Cheb. coeffs for matrix function approximation:           18
 Smallest subblock size for evaluation of M:           64
 Corrected smallest subblock size for evaluation of M:            6
 Initializing FPT
 subblockingWidth =            8
 Number of Cheb. coeffs for matrix function approximation:           18
 Smallest subblock size for evaluation of M:           64
 Corrected smallest subblock size for evaluation of M:            6
 Initializing equation
 Initializing method data
 Initializing Solver data
 Initializing variable system
 Add timer required in the operation routines
 Add the elementwise timer for writing LB weights
--------------------------------------------------------------------------------
 Doing the actual test

 Check Gradient Operator By Element

 Get element for grad_mom
 Getting gradient by element for variable grad_mom
 Failed at Dof            4  Component            0
 Expected was    5.0000000000000000        but result was    2.0000000000000000
 Failed at Dof            7  Component            0
 Expected was    8.0000000000000000        but result was    2.0000000000000000
 Failed at Dof           10  Component            0
 Expected was    11.000000000000000        but result was    2.0000000000000000
 Failed at Dof           13  Component            0
 Expected was    14.000000000000000        but result was    2.0000000000000000
 Failed at Dof           14  Component            0
 Expected was    45.000000000000000        but result was    2.0000000000000000
 Failed at Dof           16  Component            0
 Expected was    17.000000000000000        but result was    2.0000000000000000
 Failed at Dof           17  Component            0
 Expected was    54.000000000000000        but result was    2.0000000000000000
 Failed at Dof           19  Component            0
 Expected was    20.000000000000000        but result was    2.0000000000000000
 Failed at Dof           20  Component            0
 Expected was    63.000000000000000        but result was    2.0000000000000000
 Failed at Dof           22  Component            0
 Expected was    23.000000000000000        but result was    2.0000000000000000
 Failed at Dof           23  Component            0
 Expected was    72.000000000000000        but result was    2.0000000000000000
 Failed at Dof           25  Component            0
 Expected was    26.000000000000000        but result was    2.0000000000000000
 Failed at Dof           26  Component            0
 Expected was    81.000000000000000        but result was    2.0000000000000000
 Failed at Dof            4  Component            0
 Expected was    21.000000000000000        but result was    2.0000000000000000
 Failed at Dof            6  Component            0
 Expected was    27.000000000000000        but result was    2.0000000000000000
 Failed at Dof           10  Component            0
 Expected was    13.000000000000000        but result was    2.0000000000000000
 Failed at Dof           11  Component            0
 Expected was    14.000000000000000        but result was    2.0000000000000000
 Failed at Dof           12  Component            0
 Expected was    15.000000000000000        but result was    2.0000000000000000
 Failed at Dof           13  Component            0
 Expected was    48.000000000000000        but result was    2.0000000000000000
 Failed at Dof           14  Component            0
 Expected was    51.000000000000000        but result was    2.0000000000000000
 Failed at Dof           19  Component            0
 Expected was    22.000000000000000        but result was    2.0000000000000000
 Failed at Dof           20  Component            0
 Expected was    23.000000000000000        but result was    2.0000000000000000
 Failed at Dof           21  Component            0
 Expected was    24.000000000000000        but result was    2.0000000000000000
 Failed at Dof           22  Component            0
 Expected was    75.000000000000000        but result was    2.0000000000000000
 Failed at Dof           23  Component            0
 Expected was    78.000000000000000        but result was    2.0000000000000000
 Failed at Dof           24  Component            0
 Expected was    81.000000000000000        but result was    2.0000000000000000
 Failed at Dof            4  Component            0
 Expected was    13.000000000000000        but result was    2.0000000000000000
 Failed at Dof            5  Component            0
 Expected was    14.000000000000000        but result was    2.0000000000000000
 Failed at Dof            6  Component            0
 Expected was    15.000000000000000        but result was    2.0000000000000000
 Failed at Dof            7  Component            0
 Expected was    16.000000000000000        but result was    2.0000000000000000
 Failed at Dof            8  Component            0
 Expected was    17.000000000000000        but result was    2.0000000000000000
 Failed at Dof            9  Component            0
 Expected was    18.000000000000000        but result was    2.0000000000000000
 Failed at Dof           10  Component            0
 Expected was    57.000000000000000        but result was    2.0000000000000000
 Failed at Dof           11  Component            0
 Expected was    60.000000000000000        but result was    2.0000000000000000
 Failed at Dof           12  Component            0
 Expected was    63.000000000000000        but result was    2.0000000000000000
 Failed at Dof           13  Component            0
 Expected was    66.000000000000000        but result was    2.0000000000000000
 Failed at Dof           14  Component            0
 Expected was    69.000000000000000        but result was    2.0000000000000000
 Failed at Dof           17  Component            0
 Expected was    78.000000000000000        but result was    2.0000000000000000
 Failed at Dof           18  Component            0
 Expected was    81.000000000000000        but result was    2.0000000000000000
 Failed at Dof            4  Component            0
 Expected was    5.0000000000000000        but result was    2.0000000000000000
 Failed at Dof            7  Component            0
 Expected was    8.0000000000000000        but result was    2.0000000000000000
 Failed at Dof           10  Component            0
 Expected was    11.000000000000000        but result was    2.0000000000000000
 Failed at Dof           13  Component            0
 Expected was    14.000000000000000        but result was    2.0000000000000000
 Failed at Dof           14  Component            0
 Expected was    45.000000000000000        but result was    2.0000000000000000
 Failed at Dof           16  Component            0
 Expected was    17.000000000000000        but result was    2.0000000000000000
 Failed at Dof           17  Component            0
 Expected was    54.000000000000000        but result was    2.0000000000000000
 Failed at Dof           19  Component            0
 Expected was    20.000000000000000        but result was    2.0000000000000000
 Failed at Dof           20  Component            0
 Expected was    63.000000000000000        but result was    2.0000000000000000
 Failed at Dof           22  Component            0
 Expected was    23.000000000000000        but result was    2.0000000000000000
 Failed at Dof           23  Component            0
 Expected was    72.000000000000000        but result was    2.0000000000000000
 Failed at Dof           25  Component            0
 Expected was    26.000000000000000        but result was    2.0000000000000000
 Failed at Dof           26  Component            0
 Expected was    81.000000000000000        but result was    2.0000000000000000
 Failed at Dof            4  Component            0
 Expected was    21.000000000000000        but result was    2.0000000000000000
 Failed at Dof            6  Component            0
 Expected was    27.000000000000000        but result was    2.0000000000000000
 Failed at Dof           10  Component            0
 Expected was    13.000000000000000        but result was    2.0000000000000000
 Failed at Dof           11  Component            0
 Expected was    14.000000000000000        but result was    2.0000000000000000
 Failed at Dof           12  Component            0
 Expected was    15.000000000000000        but result was    2.0000000000000000
 Failed at Dof           13  Component            0
 Expected was    48.000000000000000        but result was    2.0000000000000000
 Failed at Dof           14  Component            0
 Expected was    51.000000000000000        but result was    2.0000000000000000
 Failed at Dof           19  Component            0
 Expected was    22.000000000000000        but result was    2.0000000000000000
 Failed at Dof           20  Component            0
 Expected was    23.000000000000000        but result was    2.0000000000000000
 Failed at Dof           21  Component            0
 Expected was    24.000000000000000        but result was    2.0000000000000000
 Failed at Dof           22  Component            0
 Expected was    75.000000000000000        but result was    2.0000000000000000
 Failed at Dof           23  Component            0
 Expected was    78.000000000000000        but result was    2.0000000000000000
 Failed at Dof           24  Component            0
 Expected was    81.000000000000000        but result was    2.0000000000000000
 Failed at Dof            4  Component            0
 Expected was    13.000000000000000        but result was    2.0000000000000000
 Failed at Dof            5  Component            0
 Expected was    14.000000000000000        but result was    2.0000000000000000
 Failed at Dof            6  Component            0
 Expected was    15.000000000000000        but result was    2.0000000000000000
 Failed at Dof            7  Component            0
 Expected was    16.000000000000000        but result was    2.0000000000000000
 Failed at Dof            8  Component            0
 Expected was    17.000000000000000        but result was    2.0000000000000000
 Failed at Dof            9  Component            0
 Expected was    18.000000000000000        but result was    2.0000000000000000
 Failed at Dof           10  Component            0
 Expected was    57.000000000000000        but result was    2.0000000000000000
 Failed at Dof           11  Component            0
 Expected was    60.000000000000000        but result was    2.0000000000000000
 Failed at Dof           12  Component            0
 Expected was    63.000000000000000        but result was    2.0000000000000000
 Failed at Dof           13  Component            0
 Expected was    66.000000000000000        but result was    2.0000000000000000
 Failed at Dof           14  Component            0
 Expected was    69.000000000000000        but result was    2.0000000000000000
 Failed at Dof           17  Component            0
 Expected was    78.000000000000000        but result was    2.0000000000000000
 Failed at Dof           18  Component            0
 Expected was    81.000000000000000        but result was    2.0000000000000000

    polynomials/ply_fpt_3D_performance_test
 Starting up FPT 3D Performance Test with nprocs:            1
                and nThreads pp:            2
 Time for degree            1  trafo:     0.36877456300000000
 Time for degree            1  inverse:   0.33993993299999997
 Time for degree            3  trafo:     0.29456533799999995
 Time for degree            3  inverse:   0.35299931600000001
 Time for degree            7  trafo:     0.30292761099999987
 Time for degree            7  inverse:    9.9986270000000044E-002
 Time for degree           15  trafo:      7.6449400000000445E-004
 Time for degree           15  inverse:    7.1644400000003827E-004
 Time for degree           31  trafo:     0.22255131800000028
 Time for degree           31  inverse:   0.28162462600000016
 Time for degree           63  trafo:     0.55332276899999977
 Time for degree           63  inverse:   0.32895368099999978
 Time for degree          127  trafo:      4.9555881870000000
 Time for degree          127  inverse:    3.2462657369999999
 Time for degree          255  trafo:      9.4585691020000020
 Time for degree          255  inverse:    8.3142929029999983
 Maximal deviation:   389.30765619768113

Utests FAILED!

[haraldkl]

Deactivating the OMP parallel regions in polynomials/source/fpt/ply_legFpt_module.f90 eliminates the deviating results. So something is off with the OMP parallel computation there. Possibly some working arrays within the fpt structure that are shared but shouldn't be.

[haraldkl]

Changing the whole FPT datatype to be firstprivate unfortunately does not fix the problem.