loop08 very broken

[bernecky]

Waiting for pull request on ctestsupport to get approved.

There are several problems with the LivermoreLoops. First, loop08 has several weird things going on, so
I will assign this to Bodo, as loop08 looks like his code:

         45 - Test-seq-loop08_c-mt1 (Failed)
         46 - Test-seq-loop08_c-mt2 (Failed)
         55 - Test-seq-loop08-split_c-mt1 (Failed)
         56 - Test-seq-loop08-split_c-mt2 (Failed)
         57 - Test-seq-loop08-split_naive-mt1 (Failed)
         58 - Test-seq-loop08-split_naive-mt2 (Failed)
         59 - Test-seq-loop08-split_naive2-mt1 (Failed)
         60 - Test-seq-loop08-split_naive2-mt2 (Failed)
         61 - Test-seq-loop08-split_apl-mt1 (Failed)
         62 - Test-seq-loop08-split_apl-mt2 (Failed)
        107 - Test-mt_pth-loop08_c-mt1 (Failed)
        108 - Test-mt_pth-loop08_c-mt2 (Failed)
        110 - Test-mt_pth-loop08_naive-mt2 (SEGFAULT)
        112 - Test-mt_pth-loop08_naive2-mt2 (SEGFAULT)
        114 - Test-mt_pth-loop08_apl-mt2 (SEGFAULT)
        116 - Test-mt_pth-loop08_apl2-mt2 (SEGFAULT)
        117 - Test-mt_pth-loop08-split_c-mt1 (Failed)
        118 - Test-mt_pth-loop08-split_c-mt2 (Failed)
        119 - Test-mt_pth-loop08-split_naive-mt1 (Failed)
        120 - Test-mt_pth-loop08-split_naive-mt2 (Failed)
        121 - Test-mt_pth-loop08-split_naive2-mt1 (Failed)
        122 - Test-mt_pth-loop08-split_naive2-mt2 (Failed)
        123 - Test-mt_pth-loop08-split_apl-mt1 (Failed)
        124 - Test-mt_pth-loop08-split_apl-mt2 (Failed)
Errors while running CTest
Output from these tests are in: /home/sac/sac/benchmarks/LivermoreLoopsCtest/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ 
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ cd ..
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest$ ls
build  cmake-common  CMakeLists.txt  loop01  loop02  loop03  loop04  loop05  loop06  loop07  loop08  LoopsInFortran  README.md
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest$ git status .
On branch ctestsupport
Your branch is up to date with 'origin/ctestsupport'.

1. Here, results differ, by variant.
2. In loop04, timings are horrible for some variants (apl. apl2). See below.
3. Also in loop04-mt-pth, timings GROW as thread count increases.
4. In the loop08-apl2 SEGFAULT above, running with "-check a" quickly gives us:

*** SAC runtime error
*** Tried to subsequently de-allocate heap location in arena 6

sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ time loop04-seq/loop04_c -mt 0 20000000 35; echo $?
Running Loop4_C
Number of repetitions: 20000000
Vector length        : 1001
Upper limit (n)      : 35
Element [5] of the result is: -0.073587732613706644, WANT is: -0.0735877326136999965
x[5] - WANT is -6.64746035994312479e-15
WANT - x[5] is 6.64746035994312479e-15
Wall Clock Time spent           :   0.255 secs
Computer Performance achieved   :   1.882 GFlops/s

real    0m0.257s
user    0m0.257s
sys     0m0.000s
0
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ time loop04-seq/loop04_apl -mt 0 20000000 35; echo $?
Running Loop4_APL
Number of repetitions: 20000000
Vector length        : 1001
Upper limit (n)      : 35
Element [5] of the result is: -0.073587732613706644, WANT is: -0.0735877326136999965
x[5] - WANT is -6.64746035994312479e-15
WANT - x[5] is 6.64746035994312479e-15
Wall Clock Time spent           :   4.141 secs
Computer Performance achieved   :   0.116 GFlops/s

real    0m4.143s
user    0m4.143s
sys     0m0.000s
0
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ time loop04-seq/loop04_apl2 -mt 0 20000000 35; echo $?
Running Loop4_APL2
Number of repetitions: 20000000
Vector length        : 1001
Upper limit (n)      : 35
Element [5] of the result is: -0.073587732613706644, WANT is: -0.0735877326136999965
x[5] - WANT is -6.64746035994312479e-15
WANT - x[5] is 6.64746035994312479e-15
Wall Clock Time spent           :  16.874 secs
Computer Performance achieved   :   0.028 GFlops/s

real    0m16.875s
user    0m16.875s
sys     0m0.000s
0

Also, multi-thread timings get MUCH worse as threads are added:

 time loop04-mt_pth/loop04_apl2 -mt 2 20000000 35; echo $?
Running Loop4_APL2
Number of repetitions: 20000000
Vector length        : 1001
Upper limit (n)      : 35
^C
real    0m52.424s
user    1m44.842s
sys     0m0.004s

[bernecky]

The SEGFAULT happens only in mt_pth with -mt 2:

 sac2c ../loop08/loop08.sac -O3 -check a -o crud -DVARIANT=NAIVE
warning: Option -check c implies option -ecc.
warning: Insertion of explicit conformity checks has been enabled.
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ time crud  -mt 2 1000000 ; echo $? Running Loop8_NAIVE
Number of repetitions: 1000000
Iteration space size : 100
===========================================================================

Heap Management diagnostics:

===========================================================================
calls to sbrk()  :  2
total heap size  :  9371648 Bytes (8.9 MB)
===========================================================================
calls to malloc()         :  12
calls to calloc()         :  0
calls to realloc()        :  0
calls to valloc()         :  0
calls to memalign()       :  0
calls to posix_memalign() :  0
locks acquired            :  0
===========================================================================
Arena 0 :  Arena of Arenas
  1 bin(s) totalling 1048576 Bytes (1.0 MB)
          6 allocations:             6 (100%) fixed size allocations
                    6 (100%) (100%) after splitting
          0 de-allocations:          0 (100%) fixed size de-allocations
===========================================================================
Arena 1 :  memory chunk size:  32 Bytes
  1 bin(s) totalling 8192 Bytes (0.0 MB)
          3 allocations:             0 (  0%) fixed size allocations
                    2 ( 66%) ( 66%) from wilderness
                    1 ( 33%) (100%) after extending arena
          0 de-allocations:          0 (100%) fixed size de-allocations
===========================================================================
Arena 2 :  memory chunk size:  64 Bytes
  1 bin(s) totalling 8192 Bytes (0.0 MB)
         15 allocations:             7 ( 46%) fixed size allocations
                    2 ( 13%) ( 13%) from free list
                   12 ( 80%) ( 93%) from wilderness
                    1 (  6%) (100%) after extending arena
          2 de-allocations:          2 (100%) fixed size de-allocations
===========================================================================
Arena 3 :  memory chunk size:  128 Bytes
  1 bin(s) totalling 4096 Bytes (0.0 MB)
          3 allocations:             3 (100%) fixed size allocations
                    1 ( 33%) ( 33%) from free list
                    1 ( 33%) ( 66%) from wilderness
                    1 ( 33%) (100%) after extending arena
          2 de-allocations:          0 (  0%) fixed size de-allocations
===========================================================================
Arena 4 :  memory chunk size:  256 Bytes
  1 bin(s) totalling 8192 Bytes (0.0 MB)
          2 allocations:             2 (100%) fixed size allocations
                    1 ( 50%) ( 50%) from free list
                    1 ( 50%) (100%) after extending arena
          2 de-allocations:          0 (  0%) fixed size de-allocations
===========================================================================
Arena 5 :  memory chunk size:  272 -> 2064 Bytes
  1 bin(s) totalling 32768 Bytes (0.0 MB)
          4 allocations:             3 ( 75%) fixed size allocations
                    3 ( 75%) ( 75%) after splitting
                    1 ( 25%) (100%) after extending arena
          3 de-allocations:          3 (100%) fixed size de-allocations
===========================================================================
Arena 6 :  memory chunk size:  2064 -> 16400 Bytes
  1 bin(s) totalling 131136 Bytes (0.1 MB)
          6 allocations:             6 (100%) fixed size allocations
                    5 ( 83%) ( 83%) after splitting
                    1 ( 16%) (100%) after extending arena
          2 de-allocations:          2 (100%) fixed size de-allocations
===========================================================================
Arena 8 :  memory chunk size:  131088 -> ... Bytes
  1 bin(s) totalling 8323072 Bytes (7.9 MB)
          0 allocations:             0 (100%) fixed size allocations
          0 de-allocations:          0 (100%) fixed size de-allocations
===========================================================================
Total  (without arena of arenas) :
  7 bin(s) totalling 8515648 Bytes (8.1 MB)
         33 allocations:            21 ( 63%) fixed size allocations
                    4 ( 12%) ( 12%) from free list
                    8 ( 24%) ( 36%) after splitting
                   15 ( 45%) ( 81%) from wilderness
                    6 ( 18%) (100%) after extending arena
         11 de-allocations:          7 ( 63%) fixed size de-allocations
===========================================================================

*** SAC runtime error
*** Tried to subsequently de-allocate heap location in arena 5

sac2c -V
sac2c 1.3.3-MijasCosta-782-g8e7b0
build-type: RELEASE
built-by: "sac" at 2022-02-28T13:19:55

[bernecky]

-mt doesn't matter. If -doemrl, result is wrong. If compiled with -check a, we get the above allocation fault.

sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ time crud  -mt 1 1000000; echo $?
Running Loop8_NAIVE
Number of repetitions: 1000000
Iteration space size : 100
Element [1] of the result is: 155; should be 156
Wall Clock Time spent           :   1.143 secs
Computer Performance achieved   :   6.237 GFlops/s

real    0m1.145s
user    0m1.141s
sys     0m0.004s
1
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ sac2c ../loop08/loop08.sac -O3  -o crud -DVARIANT=NAIVE -noemrl -v0
sac@medusa:~/sac/benchmarks/LivermoreLoopsCtest/build$ time crud  -mt 1 1000000; echo $?
Running Loop8_NAIVE
Number of repetitions: 1000000
Iteration space size : 100
Element [1] of the result is: 156; should be 156
Wall Clock Time spent           :   1.379 secs
Computer Performance achieved   :   5.170 GFlops/s

real    0m1.381s
user    0m1.377s
sys     0m0.004s
0

[bernecky]

I am going to take a look at this failure.

[bernecky]

Here is a shorter version of the failing code. if Loop8_NAIVE is NOT inlined, no deallocation fault occurs.
This breaks: sac2c crud.sac -doemrl -v0 -O3 -check a
This does not break: sac2c crud.sac -noemrl -v0 -O3 -check a
cat crud.breaks.sac

use StdIO: all;
use Array: all;

// If this function is NOT inlined, deallocation fault does not occur
inline
double[.,.,.], double[.,.,.]
Loop8_NAIVE( int n, double[.,.,.] u1, double[.,.,.] u2)
{
  nl1 = 0;
  nl2 = 1;
  u1 = with {
         ([1,1,1] <= [kz,ky,kx] < [2,n,3]) 
       : u1[1,1,kx] + 1.4;
       } : modarray( u1);
  return (u1,u2);
}

int main ()
{
  S = [2,101,5];
  rep = 4;
  n = 1;
  u1 = reshape( S, tod(1+iota( prod(S))));
  u2 = u1;
  for (i=0; i<rep; i++){
    u1, u2= Loop8_NAIVE( n, u1, u2);
  }
  printf("+/,u1= %.18g\n", sum(u1));
  printf("+/,u2= %.18g\n", sum(u2));
  return( 42);
}

[bernecky]

Oddity: the above generates identical output at -bopt and at -bpc, for -doemrl and for -noemrl. I are confused.

[hv15]

So there are two problems here... the first is unrelated but weird...

1. before the loop we do a reshape of an array and assign this to u1. By the time we reach pc:mmv we get the following weird looking code:

   _emal_6726__pinl_2246__icc_1649 = _reshape_VxA_( 2, 3, [ 2, 101, 5 ], _emal_6728__pinl_2201__icc_1664);
   _emal_6726__pinl_2246__icc_1649 = _noop_( _emal_6728__pinl_2201__icc_1664);

   I think the _noop_ assignment is skipped in the generated code but I have not checked this!

2. the double free error is definitely caused by EMRL, but its really tricky to spot. Basically EMRL does a memory lift out of loop --- to do this it tries to do a buffer-swap with an existing parameter. Even though the loop is passing in u1 and u2, the latter is elided at some point (DCR probably) because nothing happens with, it just gets returned. In this instance though EMRL for some reason allows an aliasing of the lifted avis and u1 to happen... the returned avis from the loop is freed, and then latter u1 is freed and boom.

AHAHAHAAHAHAAHAH 😭 @bernecky change the loop repetition from 4 to 3 (or to 5), the double free doesn't happen any more. It seems there is a fundamental flaw here in the buffer-swapping and whether the iterations is even or odd....WTF?! I guess this is why you saw the problem first with MT backend...

[hv15]

possible solution: https://gitlab.sac-home.org/sac-group/sac2c/-/merge_requests/189

[bernecky]

As of this build:

sac2c -V
sac2c 1.3.3-MijasCosta-801-gcc9dae
build-type: RELEASE
built-by: "sac" at 2022-03-27T18:06:55

No visible changes.

When I compiled with EMRL disabled, and with -check a. The segfaults went away, and I did not see
any failures from -check a (but did not look too closely).

You might want to try running the LLoops with branch origin/ctestsupport.

[hv15]

damn, I had a closer look, and a fix for one problem revealed another, lol. I'm creating an MR for a fix now.

[bernecky]

It's more brokener with your new code.

sac2c -V
sac2c 1.3.3-MijasCosta-803-g098f2
build-type: RELEASE
built-by: "sac" at 2022-03-29T11:59:54

Hans: the loop08 failures are still there, and I am now also seeing failures with loop07:

         40 - Test-seq-loop07_naive-mt2 (Failed)
         41 - Test-seq-loop07_tc-mt1 (Failed)
         42 - Test-seq-loop07_tc-mt2 (Failed)
         55 - Test-seq-loop08-split_c-mt1 (Failed)
         56 - Test-seq-loop08-split_c-mt2 (Failed)
         57 - Test-seq-loop08-split_naive-mt1 (Failed)
         58 - Test-seq-loop08-split_naive-mt2 (Failed)
         59 - Test-seq-loop08-split_naive2-mt1 (Failed)
         60 - Test-seq-loop08-split_naive2-mt2 (Failed)
         61 - Test-seq-loop08-split_apl-mt1 (Failed)
         62 - Test-seq-loop08-split_apl-mt2 (Failed)
        101 - Test-mt_pth-loop07_naive-mt1 (Failed)
        102 - Test-mt_pth-loop07_naive-mt2 (Failed)
        103 - Test-mt_pth-loop07_tc-mt1 (Failed)
        104 - Test-mt_pth-loop07_tc-mt2 (Failed)
        117 - Test-mt_pth-loop08-split_c-mt1 (Failed)
        118 - Test-mt_pth-loop08-split_c-mt2 (Failed)
        119 - Test-mt_pth-loop08-split_naive-mt1 (Failed)
        120 - Test-mt_pth-loop08-split_naive-mt2 (Failed)
        121 - Test-mt_pth-loop08-split_naive2-mt1 (Failed)
        122 - Test-mt_pth-loop08-split_naive2-mt2 (Failed)
        123 - Test-mt_pth-loop08-split_apl-mt1 (Failed)
        124 - Test-mt_pth-loop08-split_apl-mt2 (Failed)
Errors while running CTest

[hv15]

yup something is broken, but it no longer has to do with EMR, even with -noEMRCI (to disable EMR) the results is always -NaN. Using -check a shows errors occurring either. Using -enforceIEEE also makes no difference... so I have no clue at this point...