Instructions and Documentation for Power8 and XL

[petroseskinder]

In commit 13, support for power8 and the xl compiler were added. However, accompanying instructions were not added to how to build, execute and experiment with power.

For example, it's unclear whether intel_intrinsics_power8.h must be copied to $LAMMPS_DIR/src. Additionally, for building LAMMPS, I run make power8. However, this results in numerous errors. In particular, the changes in commit 13 are not propagated. For example, the changes to IntelBuffers are not included, nor is the change for the pair_tersoff_intel.cpp.

After manually making the changes in [commit13( ) into $LAMMPS_DIR/src/, and copying all of the new power8 files into $LAMMPS_DIR/src, I obtain the following errors

rs6000_secondary_reload_inner:17011, type = load
(parallel [
        (set (reg:V4SI 82 5)
            (reg:V4SI 34 2))
        (clobber (reg:DI 4 4))
    ])
../pair_tersoff_intel.cpp: In static member function ‘static void IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::kernel_step(int, int, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::iarr, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::iarr, const int*, int, typename LAMMPS_NS::IntelBuffers<flt_t, acc_t>::atom_t*, const typename LAMMPS_NS::PairTersoffIntel::ForceConst<flt_t>::c_inner_t*, const typename LAMMPS_NS::PairTersoffIntel::ForceConst<flt_t>::c_outer_t*, typename LAMMPS_NS::IntelBuffers<flt_t, acc_t>::vec3_acc_t*, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec*, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec*, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec*, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec*, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec*, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec*, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec*, int, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::iarr, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::iarr, IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::bvec) [with int EVFLAG = 1; int EFLAG = 1; flt_t = float; acc_t = float; lmp_intel::CalculationMode mic = (lmp_intel::CalculationMode)7u; bool pack_i = false; IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::iarr = int [4]; typename LAMMPS_NS::IntelBuffers<flt_t, acc_t>::atom_t = LAMMPS_NS::IntelBuffers<float, float>::atom_t; typename LAMMPS_NS::PairTersoffIntel::ForceConst<flt_t>::c_inner_t = LAMMPS_NS::PairTersoffIntel::ForceConst<float>::c_inner_t; typename LAMMPS_NS::PairTersoffIntel::ForceConst<flt_t>::c_outer_t = LAMMPS_NS::PairTersoffIntel::ForceConst<float>::c_outer_t; typename LAMMPS_NS::IntelBuffers<flt_t, acc_t>::vec3_acc_t = LAMMPS_NS::IntelBuffers<float, float>::vec3_acc_t; IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::avec = lmp_intel::altivec::fvec; IntelKernelTersoff<flt_t, acc_t, mic, pack_i>::bvec = lmp_intel::altivec::bvec]’:
../pair_tersoff_intel.cpp:886:1: internal compiler error: in rs6000_secondary_reload_fail, at config/rs6000/rs6000.c:16984
 }
 ^
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://bugzilla.redhat.com/bugzilla> for instructions.
Preprocessed source stored into /tmp/cc15wmyP.out file, please attach this to your bugreport.
make[1]: *** [pair_tersoff_intel.o] Error 1
make[1]: Leaving directory `/home/petros/tersoff/lammps-tersoff-vector/machines/lammps-10Mar16/src/Obj_power8'
make: *** [power8] Error 2

Clearly, I am making a mistake in some portion of the build process. It's unclear though, what portion.

You can obtain a repro of the issue in our fork of the repo.

[v0i0]

1. Please run make yes-USER-INTEL; make yes-USER-OMP to update the respective packages before building.
2. You might have to copy over vector_math_altivec.h and intel_intrinsics_power8.h manually---the "failed to find include file" compiler errors should not be ambiguous.
3. This is an internal compiler error, i.e. a bug in the compiler. I've encountered it as well when trying GCC 4.x.x. Upgrading to a more recent GCC, i.e. 5/6/7, or using IBM's compiler, will resolve it.

[petroseskinder]

Thank you. I was able to make the relevant fixes, and build successfully.

Would you be open to me moving the vector_math_altivec.h and intel_intrinsics_power8.h to $THIS/src/? From an end user perspective, I believe this is the appropriate location to place them, rather than in $LAMMPS_DIR/USER-INTEL/

[petroseskinder]

Additionally, from running the test.py, all of the MIXED precision tests fail. Is that intentional?
Meaning, are you intentionally not supporting OPT-M for power, currently?

I ask because in your paper, you report only the mixed precision speedups, for single node as well as for the xeon phi's. The documented reasoning for doing so being that mixed precision is "...the setting most likely to be used in practice."

If this is intentional, do you expect to add support for mixed precision on power before the competition on November 11?

[petroseskinder]

I ran into additional issues when I was running the lmp_power8 binary I built. Specifically, I received the following error

WARNING: OMP_NUM_THREADS environment is not set. (../comm.cpp:90) using 1 OpenMP thread(s) per MPI task 
ERROR on proc 0: Substitution for illegal variable (../input.cpp:531)

From my looking through input.cpp, this is an issue with substituting the input arguments. This is strange to me, as I believe the command line args I am passing are reasonable

./lmp_power8 -in in.tersoff -pk omp 1 -pk intel 1 balance 0 mode single

From what I understand of the command, I am: (1) including the OpenMP package with one thread, (2) including the intel package (this fails with or without), (3) balance and single operate as documented.

As far as my system settings, I am compiling with gcc-5.3.0 and have a link to libgomp.

Just in case this was an issue with in.tersoff, I also passing in.tersoff_bench, obtaining similar failing results.

For context, I have been able to get everything running on an intel machine. These issues came up when using power. Since there are ~4 teams using power machines, my aim is to upstream any beneficial documentation/changes that may be useful for power users.

[v0i0]

1. I've changed the install script to automatically copy the required files for POWER and ARM, regardless of platform. This was not done before since the files are only required for a few platforms. The proper place is in USER-INTEL, not in the root, which hosts core files and only gets additional files through package installs.
2. You are absolutely correct, mixed precision is not implemented, just as it is not implemented for ARM. So the test.py output is fine. I do not think this will change for the competition.
3. Please check the scripts in the benchmark folders to figure out how to invoke the files. You are missing arguments -v p vanilla in the command line. As such, the subsitution of variabe p will fail for you, and you get an error. The parameter selects between KOKKOS (GPU), and non-KOKKOS runs.
   If this issue only occurs on POWER, not on Intel, irrespective of the above command line option, I'd be very curious...

Btw. I think it's a great idea to collect relevant information for the teams using POWER, thanks. 👍

[petroseskinder]

I've changed the install script to automatically copy the required files for POWER and ARM, regardless of platform.

Thank you!

Please check the scripts in the benchmark folders to figure out how to invoke the files. You are missing arguments -v p vanilla in the command line. As such, the subsitution of variabe p will fail for you, and you get an error. The parameter selects between KOKKOS (GPU), and non-KOKKOS runs.

Yay! That was in fact successful for me. You can see the relevant commit here on our forked repo.

I used the previous command because in the Getting Started portion of the README you state that we should run the binary using the following command:

$LAMMPS_DIR/src/lmp_intel_phi -in in.tersoff -pk omp 0 \
-pk intel 1 balance $BALANCE mode $MODE -sf intel

If it makes sense, can you update that to include -v p vanilla.

You are absolutely correct, mixed precision is not implemented, just as it is not implemented for ARM. So the test.py output is fine. I do not think this will change for the competition.

I think that's fair. However, this does introduce challenges as far as ensuring reproducibility. For single threaded execution, it seemed the single precision and mixed precision versions performed roughly the same. Was this also the case for the single node experiments?

If so, we should be able to approximate the single node tests by using the Reference and OPT-S.

Btw. I think it's a great idea to collect relevant information for the teams using POWER, thanks. 👍

Within our fork of the repo, we create a ibm-power8/ directory in machines/, adding a relevant build script. Something similar to what you are doing for hpac-arm/. Would you be open to us upstreaming those changes, through a pull request?

[v0i0]

Great to hear that it worked for you. 👍

1. The README is correct, it's just a different in.tersoff file (see the preceding lines).
2. Yes. Speedups from single threaded transfer to single node (aside from the slow-down due to comm/imbalance, which is the same for both). For the competition, clearly you won't be asked to run anything that is not implemented, so don't worry about that.
3. I'd be happy to merge in your build script, if you'd like. Please name the folder so that it indicates your organization and the architecture (i.e. hpac-arm: HPAC is my research group, ARM the architecture; lrz-ib_phi: LRZ is the computing center in Munich, with Ivy Bridge and Phi nodes). The org is important because e.g. the module system changes with org.

[koallen]

Hi, I'm Siyuan from Nanyang Technological University, Singapore also participating in the competition.

I believe the intel_intrinsics_power8.h header file needs to be copied regardless of the architecture because this header is included in intel_intrinsics.h without any #ifdef macro surrounding it. Our build was broken after I pulled the latest code with Power 8 and I have to install the power8 header to make it work (I'm using Intel CPUs).