P100 answers are wrong about half of the time

[jeffhammond]

Is this a bug in your CoMD implementation, in CUB or somewhere else?

Software

This repo with 3c4bc9f290b7bd9d13e192a9f5a2e1519ba394b0, cub-1.6.4, and GCC 4.8.5.

Hardware

GPU 0: Tesla P100-PCIE-16GB (UUID: GPU-448db3f5-6256-eafe-698a-aeaa0452c2ff)
GPU BIOS Version : 	86.00.26.00.01
Current ECC Mode : 	Enabled
Current Persistence Mode: 	Enabled
Current Application Clock : 
        Graphics                    : 405 MHz
        SM                          : 405 MHz
        Memory                      : 715 MHz
Gpu Current Temperature : 27C
Current Cuda Version : 	8.0, V8.0.44

Input

for n in `seq 1 100` ; do ./bin/CoMD-cuda-mpi -e -x $n -y $n -z $n 2>&1 | tee eam.n$n.log ; done

Output

As you can see below, in 98 tests, I get an answer accurate to 3 decimals 50 times, an answer accurate to less than 3 decimals 9 times, and the nonsensical answer of zero 39 times.

The OpenMP reference implementation is accurate to at least three decimals for every one of these cases.

$ grep "Final energy" *log
eam.n100.log:  Final energy    : 0.000000000000
eam.n10.log:  Final energy    : 0.000000000000
eam.n11.log:  Final energy    : 0.000000000000
eam.n12.log:  Final energy    : 0.000000000000
eam.n13.log:  Final energy    : -3.335415333810
eam.n14.log:  Final energy    : -3.344402073878
eam.n15.log:  Final energy    : -3.352183853550
eam.n16.log:  Final energy    : -3.358988391471
eam.n17.log:  Final energy    : -3.364988589356
eam.n18.log:  Final energy    : -3.370319963769
eam.n19.log:  Final energy    : -3.375087496409
eam.n20.log:  Final energy    : -3.379376467145
eam.n21.log:  Final energy    : -3.383255004237
eam.n22.log:  Final energy    : -3.460504275628
eam.n23.log:  Final energy    : -3.460507369744
eam.n24.log:  Final energy    : -3.460505932912
eam.n25.log:  Final energy    : -3.460505614897
eam.n26.log:  Final energy    : -3.460507156488
eam.n27.log:  Final energy    : -3.460510956982
eam.n28.log:  Final energy    : -3.460511109727
eam.n29.log:  Final energy    : -3.460509116211
eam.n30.log:  Final energy    : -3.460509949100
eam.n31.log:  Final energy    : -3.460512404440
eam.n32.log:  Final energy    : -3.460517333917
eam.n33.log:  Final energy    : -3.460516969976
eam.n34.log:  Final energy    : -3.460523260102
eam.n35.log:  Final energy    : -3.460530043886
eam.n36.log:  Final energy    : -3.460530080663
eam.n37.log:  Final energy    : -3.460530076386
eam.n38.log:  Final energy    : -3.460530062114
eam.n39.log:  Final energy    : -3.460530062445
eam.n3.log:  Final energy    : 0.000000000000
eam.n40.log:  Final energy    : -3.460530026084
eam.n41.log:  Final energy    : -3.460530020319
eam.n42.log:  Final energy    : -3.460529998146
eam.n43.log:  Final energy    : -3.460530080067
eam.n44.log:  Final energy    : -3.460530017362
eam.n45.log:  Final energy    : -3.460530059833
eam.n46.log:  Final energy    : -3.460530050431
eam.n47.log:  Final energy    : -3.460530048647
eam.n48.log:  Final energy    : -3.460530052178
eam.n49.log:  Final energy    : -3.460530040624
eam.n4.log:  Final energy    : 0.000000000000
eam.n50.log:  Final energy    : -3.460530065898
eam.n51.log:  Final energy    : -3.460530046749
eam.n52.log:  Final energy    : 0.000000000000
eam.n53.log:  Final energy    : 0.000000000000
eam.n54.log:  Final energy    : 0.000000000000
eam.n55.log:  Final energy    : 0.000000000000
eam.n56.log:  Final energy    : 0.000000000000
eam.n57.log:  Final energy    : 0.000000000000
eam.n58.log:  Final energy    : 0.000000000000
eam.n59.log:  Final energy    : 0.000000000000
eam.n5.log:  Final energy    : 0.000000000000
eam.n60.log:  Final energy    : 0.000000000000
eam.n61.log:  Final energy    : 0.000000000000
eam.n62.log:  Final energy    : 0.000000000000
eam.n63.log:  Final energy    : 0.000000000000
eam.n64.log:  Final energy    : -3.460530071940
eam.n65.log:  Final energy    : -3.460530055180
eam.n66.log:  Final energy    : -3.460530078149
eam.n67.log:  Final energy    : -3.460530055283
eam.n68.log:  Final energy    : -3.460530090024
eam.n69.log:  Final energy    : -3.460530039012
eam.n6.log:  Final energy    : 0.000000000000
eam.n70.log:  Final energy    : -3.460530030326
eam.n71.log:  Final energy    : -3.460530063041
eam.n72.log:  Final energy    : -3.460530049675
eam.n73.log:  Final energy    : 0.000000000000
eam.n74.log:  Final energy    : 0.000000000000
eam.n75.log:  Final energy    : 0.000000000000
eam.n76.log:  Final energy    : 0.000000000000
eam.n77.log:  Final energy    : 0.000000000000
eam.n78.log:  Final energy    : 0.000000000000
eam.n79.log:  Final energy    : 0.000000000000
eam.n7.log:  Final energy    : 0.000000000000
eam.n80.log:  Final energy    : -3.460530042402
eam.n81.log:  Final energy    : -3.460530075259
eam.n82.log:  Final energy    : -3.460530037493
eam.n83.log:  Final energy    : -3.460530047495
eam.n84.log:  Final energy    : -3.460530042484
eam.n85.log:  Final energy    : -3.460530063053
eam.n86.log:  Final energy    : 0.000000000000
eam.n87.log:  Final energy    : 0.000000000000
eam.n88.log:  Final energy    : 0.000000000000
eam.n89.log:  Final energy    : 0.000000000000
eam.n8.log:  Final energy    : 0.000000000000
eam.n90.log:  Final energy    : 0.000000000000
eam.n91.log:  Final energy    : 0.000000000000
eam.n92.log:  Final energy    : -3.460530059056
eam.n93.log:  Final energy    : -3.460530073254
eam.n94.log:  Final energy    : -3.460530062317
eam.n95.log:  Final energy    : -3.460530042136
eam.n96.log:  Final energy    : -3.460530044903
eam.n97.log:  Final energy    : 0.000000000000
eam.n98.log:  Final energy    : 0.000000000000
eam.n99.log:  Final energy    : 0.000000000000
eam.n9.log:  Final energy    : 0.000000000000

[jeffhammond]

I should add that the performance for the incorrect case indicates that no work as done.

$ grep "Average atom rate" *log
eam.n100.log: Average atom rate:            1101.19 atoms/us
eam.n10.log: Average atom rate:              9.67 atoms/us
eam.n11.log: Average atom rate:             12.76 atoms/us
eam.n12.log: Average atom rate:             16.66 atoms/us
eam.n13.log: Average atom rate:              6.45 atoms/us
eam.n14.log: Average atom rate:              8.96 atoms/us
eam.n15.log: Average atom rate:             10.65 atoms/us
eam.n16.log: Average atom rate:             12.16 atoms/us
eam.n17.log: Average atom rate:             14.24 atoms/us
eam.n18.log: Average atom rate:             15.79 atoms/us
eam.n19.log: Average atom rate:             18.23 atoms/us
eam.n20.log: Average atom rate:             21.12 atoms/us
eam.n21.log: Average atom rate:             22.33 atoms/us
eam.n22.log: Average atom rate:             19.49 atoms/us
eam.n23.log: Average atom rate:             21.67 atoms/us
eam.n24.log: Average atom rate:             22.08 atoms/us
eam.n25.log: Average atom rate:             23.82 atoms/us
eam.n26.log: Average atom rate:             25.63 atoms/us
eam.n27.log: Average atom rate:             21.70 atoms/us
eam.n28.log: Average atom rate:             23.77 atoms/us
eam.n29.log: Average atom rate:             26.88 atoms/us
eam.n30.log: Average atom rate:             26.29 atoms/us
eam.n31.log: Average atom rate:             28.70 atoms/us
eam.n32.log: Average atom rate:             30.57 atoms/us
eam.n33.log: Average atom rate:             27.77 atoms/us
eam.n34.log: Average atom rate:             28.53 atoms/us
eam.n35.log: Average atom rate:             30.45 atoms/us
eam.n36.log: Average atom rate:             30.46 atoms/us
eam.n37.log: Average atom rate:             32.72 atoms/us
eam.n38.log: Average atom rate:             32.69 atoms/us
eam.n39.log: Average atom rate:             31.88 atoms/us
eam.n3.log: Average atom rate:              0.23 atoms/us
eam.n40.log: Average atom rate:             33.18 atoms/us
eam.n41.log: Average atom rate:             35.16 atoms/us
eam.n42.log: Average atom rate:             32.59 atoms/us
eam.n43.log: Average atom rate:             34.55 atoms/us
eam.n44.log: Average atom rate:             37.20 atoms/us
eam.n45.log: Average atom rate:             34.56 atoms/us
eam.n46.log: Average atom rate:             36.10 atoms/us
eam.n47.log: Average atom rate:             36.49 atoms/us
eam.n48.log: Average atom rate:             34.53 atoms/us
eam.n49.log: Average atom rate:             35.86 atoms/us
eam.n4.log: Average atom rate:              0.61 atoms/us
eam.n50.log: Average atom rate:             38.54 atoms/us
eam.n51.log: Average atom rate:             36.93 atoms/us
eam.n52.log: Average atom rate:            606.72 atoms/us
eam.n53.log: Average atom rate:            616.94 atoms/us
eam.n54.log: Average atom rate:            650.27 atoms/us
eam.n55.log: Average atom rate:            661.82 atoms/us
eam.n56.log: Average atom rate:            675.76 atoms/us
eam.n57.log: Average atom rate:            703.17 atoms/us
eam.n58.log: Average atom rate:            718.96 atoms/us
eam.n59.log: Average atom rate:            726.88 atoms/us
eam.n5.log: Average atom rate:              1.23 atoms/us
eam.n60.log: Average atom rate:            759.02 atoms/us
eam.n61.log: Average atom rate:            768.25 atoms/us
eam.n62.log: Average atom rate:            779.04 atoms/us
eam.n63.log: Average atom rate:            810.19 atoms/us
eam.n64.log: Average atom rate:             39.69 atoms/us
eam.n65.log: Average atom rate:             39.83 atoms/us
eam.n66.log: Average atom rate:             38.91 atoms/us
eam.n67.log: Average atom rate:             39.79 atoms/us
eam.n68.log: Average atom rate:             38.71 atoms/us
eam.n69.log: Average atom rate:             37.90 atoms/us
eam.n6.log: Average atom rate:              2.04 atoms/us
eam.n70.log: Average atom rate:             38.68 atoms/us
eam.n71.log: Average atom rate:             39.54 atoms/us
eam.n72.log: Average atom rate:             38.71 atoms/us
eam.n73.log: Average atom rate:            933.20 atoms/us
eam.n74.log: Average atom rate:            927.36 atoms/us
eam.n75.log: Average atom rate:            953.39 atoms/us
eam.n76.log: Average atom rate:            945.81 atoms/us
eam.n77.log: Average atom rate:            956.08 atoms/us
eam.n78.log: Average atom rate:            976.78 atoms/us
eam.n79.log: Average atom rate:            979.80 atoms/us
eam.n7.log: Average atom rate:              3.19 atoms/us
eam.n80.log: Average atom rate:             40.51 atoms/us
eam.n81.log: Average atom rate:             39.83 atoms/us
eam.n82.log: Average atom rate:             39.65 atoms/us
eam.n83.log: Average atom rate:             40.58 atoms/us
eam.n84.log: Average atom rate:             39.82 atoms/us
eam.n85.log: Average atom rate:             39.84 atoms/us
eam.n86.log: Average atom rate:            1008.33 atoms/us
eam.n87.log: Average atom rate:            1031.48 atoms/us
eam.n88.log: Average atom rate:            1032.76 atoms/us
eam.n89.log: Average atom rate:            1031.55 atoms/us
eam.n8.log: Average atom rate:              4.96 atoms/us
eam.n90.log: Average atom rate:            1052.90 atoms/us
eam.n91.log: Average atom rate:            1039.54 atoms/us
eam.n92.log: Average atom rate:             39.44 atoms/us
eam.n93.log: Average atom rate:             39.30 atoms/us
eam.n94.log: Average atom rate:             40.35 atoms/us
eam.n95.log: Average atom rate:             40.24 atoms/us
eam.n96.log: Average atom rate:             39.66 atoms/us
eam.n97.log: Average atom rate:            1117.49 atoms/us
eam.n98.log: Average atom rate:            1098.96 atoms/us
eam.n99.log: Average atom rate:            1109.59 atoms/us
eam.n9.log: Average atom rate:              6.87 atoms/us