Amber-lio crashes with the new updates from Gonza

[diegoarmino]

Lio crashes when running PBE0 from Amber with a segmentation fault right before the link atoms managing part.
Lio also crashes when running normal PBE closed shell, but in such case it can find built in fitting basis sets.
jcordova@xg04:/tmp/jcordova_test/Test_openshell$ tail test.out
eefld_timegfh = F, eefld_timegih = F, eefld_wavelen = 0.00000000
! -- Fock Bias Potentials: -- !
fockbias_is_active = F, fockbias_is_shaped = F, fockbias_timeamp0 = 0.00000000,
fockbias_timegrow = 0.00000000, fockbias_timefall = 0.00000000,
fockbias_readfile = atombias.in

LIO initialisation.
Error: Fitting set file
/usr/local/easybuild/software/lio/1.2-libint2.6.0-fosscuda2020a/dat/basis/fitting/DZVP Coulomb Fitti
not found.

This error happens in lioamber/basis_data.f90 however there are no differences in this file with respect to the branch before the merge.

Develop before the merge does have the seg fault with PBE0 open shell but not the fitting basis set error of PBE closed shell.
This is what I know so far.

[diegoarmino]

Recompiled in my own computer. Previously there was a dumb error in the lio.in file. I wrote nump=1 instead of nunp=1. However there IS a bug. When running the following lio.in
&lio
natom=1,
charge=-1,
nunp=1,
open=t,
extern_functional=t,
functional_id=406,
int_basis = t,
basis_set = "DZVP",
&end
The system is a water box with a sulfur radical anion S-.
The bug produces the following in stdout:

  GPU Device used: GeForce GTX 1660 Ti
  Using 3 CPU Threads and 1 GPU Threads.
 --------------------------------------------------------
 Starting PBE0 functional calculation using LIO's engine 
 for PBE and LibINT for exact exchange integrals.        
 --------------------------------------------------------
  QM atoms: 1 - MM atoms: 0
  Total number of basis: 19 (s: 4 p: 3 d: 1)
  Open shell calculation - Occupied MO(UP): 8 - Occupied MO(DOWN): 9
tipo: 15 0.102116 radio: 19.7917
esfera incluye 25 capas de 40 (radio: 1.90978)
CUDA ERROR: invalid argument [CudaMatrix::operator=]

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0  0x7fae5844bd11 in ???
#1  0x7fae5844aee5 in ???
#2  0x7fae5826108f in ???
        at /build/glibc-B3wQXB/glibc-2.31/signal/../sysdeps/unix/sysv/linux/x86_64/sigaction.c:0
#3  0x7fae5826100b in __GI_raise
        at ../sysdeps/unix/sysv/linux/raise.c:51
#4  0x7fae58240858 in __GI_abort
        at /build/glibc-B3wQXB/glibc-2.31/stdlib/abort.c:79
#5  0x7fae58a3d335 in _Z17cudaAssertNoErrorPKc
        at cuda/cuda_extra.h:400
#6  0x7fae58a4e077 in _ZN3G2G10CudaMatrixIdEaSERKS1_
        at /home/diegoa/progs/lio_test/g2g/matrix.cpp:401
#7  0x7fae58a4daa7 in _ZN3G2G10CudaMatrixIdEC2ERKS1_
        at /home/diegoa/progs/lio_test/g2g/matrix.cpp:279
#8  0x7fae58a87439 in _ZN3G2G13PointGroupGPUIdEC2ERKS1_
        at /home/diegoa/progs/lio_test/g2g/partition.h:262
#9  0x7fae58a85673 in _ZN3G2G9Partition10regenerateEv
        at /home/diegoa/progs/lio_test/g2g/regenerate_partition.cpp:542
#10  0x7fae58a6c7a9 in _Z16compute_new_gridj
        at /home/diegoa/progs/lio_test/g2g/init.cpp:300
#11  0x7fae58a6ca06 in g2g_reload_atom_positions_
        at /home/diegoa/progs/lio_test/g2g/init.cpp:339
#12  0x7fae5a723b65 in scf_
        at /home/diegoa/progs/lio_test/lioamber/SCF.f90:826
#13  0x7fae5a692963 in liomain_
        at /home/diegoa/progs/lio_test/lioamber/liomain.f90:88
#14  0x7fae5a67d3c5 in scf_in_
        at /home/diegoa/progs/lio_test/lioamber/SCF_exter.f90:70
#15  0x55fb509955be in ???
#16  0x55fb509445ad in ???
#17  0x55fb509345e8 in ???
#18  0x55fb507bfd00 in ???
#19  0x55fb50860c48 in ???
#20  0x55fb5080a7b2 in ???
#21  0x55fb5080325c in ???
#22  0x55fb508032b8 in ???
#23  0x7fae58242082 in __libc_start_main
        at ../csu/libc-start.c:308
#24  0x55fb5061ff4d in ???
#25  0xffffffffffffffff in ???
Aborted (core dumped)

And in the output file:

  Natom =     1, Nsol =        0, charge =    -1, Nunp =     1, open =  T,
  basis_set = DZVP                     fitting_set = DZVP Coulomb Fitting     ,
  int_basis =  T, nMax =   250, Told =  1.00E-06, Etold =  1.00E-01,
  IExch =   9, rMax =    16.000000, rMaxS =     5.000000, IGrid =  2,
  IGrid2 =  2, initial_guess =  0DBug =  F, n_ghosts =    0
 ! -- Convergence acceleration: -- !
  conver_method =  2, Gold =    10.000000, DIIS =  T, NDIIS =  30,
  hybrid_converg =  F, good_cut =  1.00E-03, DIIS_bias =     1.050000,
  DIIS_start =  1.00E-02, bDIIS_start =  1.00E-03, level_shift =  F,
  lvl_shift_en =     0.25000000, lvl_shift_cut =     0.00500000
 ! -- Input and output options: -- !
  verbose =   3, style =  F, timers =   0, writeXYZ =  T, writeForces =  F,
  dipole =  F, mulliken =  F, lowdin =  F, fukui =  F, print_coeffs =  F,
  VCInp =  F, FRestartIn = restart.in               , restart_freq =     0,
  FRestart = restart.out              , TDRestart =  F, writeDens =  T,
  TD_rst_freq =    500, gaussian_convert =  F,
  rst_dens =   0becke =  F
 ! -- TD-DFT and external field: -- !
  timeDep =  0, NTDStep =          1, TDStep =     0.00200000, propagator =  1,
  NBCH =   10, field =  F, a0 =  1000.00000000, epsilon =     1.00000000,
  Fx =     0.05000000, Fy =     0.05000000, Fz =     0.05000000, n_fields_iso =     0,
  n_fields_aniso =     0, field_iso_file = field.in                 ,
  field_aniso_file = fieldaniso.in            , td_do_pop =     0, td_eu_step =     0
  td_do_opop =     0, td_rho_purify =   0
 ! -- Effective Core Potentials: -- !
  ECPMode =  F, ECPTypes =   0, TipeECP = NOT-DEFINED              ,
  Fock_ECP_read =  F, Fock_ECP_write =  F, cutECP =  T, cut2_0 =    35.00000000,
  cut3_0 =    45.00000000, verbose_ECP =  0, ECP_debug =  F, fullTimer_ECP =  F,
  local_nonlocal =  0, ECP_full_range_int =  F
  ZListECP =
 ! -- Minimizations and restraints: -- !
  steep =  F, minimzation_steep =     0.05000000, energy_cut =     0.00010000,
  force_cut =     0.00001000, n_min_steeps =   500, lineal_search =  T,
  n_points =     5, number_restr =     0
 ! -- CubeGen: -- !
  CubeGen_only =  F, cube_res =    40, cube_sel =     0, cube_dens =  F,
  cube_orb =  F, cube_elec =  F, cube_dens_file = dens.cube                ,
  cube_orb_file = orb.cube                 , cube_elec_file = dens.cube
 ! -- GPU Options: -- !
  energy_all_iterations =  F, assign_all_functions =  F, remove_zero_weights =  T,
  max_function_exponent =    10, free_global_memory =     0.00000000,

Which is an incomplete print of the output: It should be:

 ! -- CubeGen: -- !
  CubeGen_only =  F, cube_res =    40, cube_sel =     0, cube_dens =  F,
  cube_orb =  F, cube_elec =  F, cube_dens_file = dens.cube                ,
  cube_orb_file = orb.cube                 , cube_elec_file = dens.cube
 ! -- GPU Options: -- !
  energy_all_iterations =  F, assign_all_functions =  F, remove_zero_weights =  T,
  max_function_exponent =    10, free_global_memory =     0.00000000,
  sphere_radius =     0.60000002, little_cube_size =     8.00000000,
  min_points_per_cube =     1, gpu_level =     4
 ! -- Transport and TBDFT: -- !
  transport_calc =  F, generate_rho0 =  F, driving_rate =     0.00100000,
  gate_field =  T, pop_drive =   1, save_charge_freq =     1, TBDFT_calc =     0,
  MTB =     0, alfaTB =     0.00000000, betaTB =     0.00000000,
  gammaTB =     0.00000000, start_TDTB =     0,
  end_TDTB =     0, n_biasTB =     0, driving_rateTB=    0.00000000,
  nbias=    0, TB_q_tot =   10, TB_charge_ref=    0.00000000, TB_q_told=    0.00000100
 ! -- Ehrenfest: -- !
  ndyn_steps =      0, edyn_steps =      0, nullify_forces =  F,
  wdip_nfreq =     0, wdip_fname = liorun.dip               , rsti_loads =  F,
  rsto_saves =  F, rsto_nfreq =  F, rsti_fname = liorun.rsti              ,
  rsto_fname = liorun.rsto              , eefld_on =  F, eefld_ampx =    0.00000000,
  eefld_ampy =     0.00000000, eefld_ampz =     0.00000000,
  eefld_timeamp =     0.20000000, eefld_timepos =     1.00000000,
  eefld_timegfh =  F, eefld_timegih =  F, eefld_wavelen =     0.00000000
 ! -- Fock Bias Potentials: -- !
  fockbias_is_active =  F, fockbias_is_shaped =  F, fockbias_timeamp0 =     0.00000000,
  fockbias_timegrow =     0.00000000, fockbias_timefall =     0.00000000,
  fockbias_readfile = atombias.in
 ! -- DOS and PDOS options: -- !
  dos_calc =  F, pdos_calc =  F, pdos_allb =  F
 ! -- VdW interaction options: -- !
  dftd3 =  F
 ! -- CEED calculation: -- !
  ceed_calc =  F, ceed_td_step =    100, k_ceed =  1.00E+00
 DEBUG: init_lio_amber() after read_options

LIO initialisation.
 ---------------------------------------------

So the output printing is not completely flushed out before the onset of the error.
When the error appears, the printing of the output file finishes always in the same spot with the same values (as far as I could see).
The error appears about half of the time at random.
Checked if one of the variables immediately following the point in which the output stops printing could be the culprit. Maybe one of such variables was not properly initialized and its value changes at random, with sometimes inconsistent values making the program crash. Adding explicitly the following GPU options:

 sphere_radius=0.6, little_cube_size=8.0,
 min_points_per_cube=1, gpu_level=4,

But the system crashes in exactly the same way.