Investigate the origins of the small memory transfers from Staple to Cshift

[qiUip]

Investigate the origins of the small memory transfers by systematically checking (logging / commenting out) sections of the pathways from Staple to Cshift (we suspect it is related to how CovShift calls the Cshift).

[qiUip]

The small memory movements are seem to be originating from within Cshift itself and not something specific to CovShift or Staple. They happen in the Fermion actions as well, just there Cshift is not called often.

I have a suspicion as to 'why' but it doesn't fully add up with what I understand of the memory caching. I narrowed down the transfer to happening within this part of Cshift -> Cshift_local and from there there are many calls to Copy_plane. Copy_plane is in Grid/cshift/Cshift_common.h and I think this is the part of the code where the transfers might happen are either in:

    GRID_TRACE("Copy_plane acc cshift autoView");
    autoView(rhs_v , rhs, AcceleratorRead);
    autoView(lhs_v , lhs, AcceleratorWrite);
    tracePush("copy_plane acc_for");
    accelerator_for(i,ent,vobj::Nsimd(),{
      coalescedWrite(lhs_v[table[i].first],coalescedRead(rhs_v[table[i].second]));
    });
    tracePop("copy_plane acc_for");

or the line before that auto table = MapCshiftTable(); which calls this inline function

inline std::pair<int,int> *MapCshiftTable(void)
{
  // GPU version
#ifdef ACCELERATOR_CSHIFT    
  uint64_t sz=Cshift_table.size();
  if (Cshift_table_device.size()!=sz )    {
    Cshift_table_device.resize(sz);
  }
  acceleratorCopyToDevice((void *)&Cshift_table[0],
			  (void *)&Cshift_table_device[0],
			  sizeof(Cshift_table[0])*sz);

  return &Cshift_table_device[0];
#else 
  return &Cshift_table[0];
#endif
  // CPU version use identify map
}

I will investigate further, but this is very deep within the basic functionalities of the code so I'm worries about how much it will be possible to do here.

Another observation is that I suspect the large memory transfers before / after the small Cshift ones might be related to Staple directly. These seem to come before the addition operators after the ShiftStaple such as

          Gimpl::CovShiftForward(Umu[nu], nu,
            Gimpl::CovShiftBackward(Umu[mu], mu, Gimpl::CovShiftIdentityBackward(Umu[nu], nu))), mu);

and would make sense as the data from ShiftStaple is on the GPU BUT staple itself is not. It comes from

    GaugeLinkField dSdU_mu(grid);
    std::vector<GaugeLinkField> Umu(Nd, grid);
    for (int mu = 0; mu < Nd; mu++) {
      Umu[mu] = PeekIndex<LorentzIndex>(U, mu);
    }

    for (int mu = 0; mu < Nd; mu++) {
      // Staple in direction mu
      WilsonLoops<Gimpl>::Staple(dSdU_mu, Umu, mu);
      dSdU_mu = Ta(Umu[mu] * dSdU_mu) * factor;

      PokeIndex<LorentzIndex>(dSdU, dSdU_mu, mu);
    }
    tracePop("WilsonGaugeAction_deriv");
  }

where staple is dSdU_mu (on the CPU) and Umu is on the other hand should be on the GPU after using PeekIndex.

I'm still wrapping my head around this so any help would be nice.

Profiles from local system to follow in next comment.

[ilectra]

Long overdue update: MapCshiftTable() is to blame: It re-calculates the same table per slice on CPU, then copies it to GPU every time. I have refactored this, filling the table and moving it to GPU once in Cshift_local, before the loop over slices.
This is something that Makis kind-of fixed years ago, it was merged upstream, then reverted a few months later with no explanation. So it's a legit fix, even if under-loved.
From preliminary profiles, the small memory transfers have disappeared, and we get a bit more than 10% improvement in the first call to WilsonGaugeAction_deriv for a grid of 16.16.16.32. (The comparison is with nthreads=8 before vs nthreads=1 after, I hope this doesn't make a big difference.....)

Before:

After:

PR is coming up shortly.