Error during writing of history file

[arthurronner]

Hi,

Hopefully this is the last one to tackle.
I am trying to run mizuRoute on the Rhône basin. When running, the program terminates when trying to write the history file for the first time. I've tried limiting my control file to include only the minimal settings (using the option 1 example). I've also tried turning the writing of some/all variables on/off - which did not solve my issue. The error message below was generated with control_file_3.txt. The listed output directory exists.

Below is the error message - including a list of the loaded modules while running on the ALICE HPC.

Currently Loaded Modules:
  1) shared                            18) PMIx/4.2.4-GCCcore-12.3.0
  2) DefaultModules                    19) UCC/1.2.0-GCCcore-12.3.0
  3) slurm/alice/23.02.7               20) OpenMPI/4.1.5-GCC-12.3.0
  4) ALICE/default                     21) gompi/2023a
  5) GCCcore/12.3.0                    22) Szip/2.1.1-GCCcore-12.3.0
  6) zlib/1.2.13-GCCcore-12.3.0        23) HDF5/1.14.0-gompi-2023a
  7) binutils/2.40-GCCcore-12.3.0      24) cURL/8.0.1-GCCcore-12.3.0
  8) GCC/12.3.0                        25) gzip/1.12-GCCcore-12.3.0
  9) numactl/2.0.16-GCCcore-12.3.0     26) lz4/1.9.4-GCCcore-12.3.0
 10) XZ/5.4.2-GCCcore-12.3.0           27) zstd/1.5.5-GCCcore-12.3.0
 11) libxml2/2.11.4-GCCcore-12.3.0     28) bzip2/1.0.8-GCCcore-12.3.0
 12) libpciaccess/0.17-GCCcore-12.3.0  29) netCDF/4.9.2-gompi-2023a
 13) hwloc/2.9.1-GCCcore-12.3.0        30) netCDF-Fortran/4.6.1-gompi-2023a
 14) OpenSSL/1.1                       31) ncurses/6.4-GCCcore-12.3.0
 15) libevent/2.1.12-GCCcore-12.3.0    32) libarchive/3.6.2-GCCcore-12.3.0
 16) UCX/1.14.1-GCCcore-12.3.0         33) CMake/3.26.3-GCCcore-12.3.0
 17) libfabric/1.18.0-GCCcore-12.3.0

 

 (t_initf) Read in prof_inparm namelist from: dummy_empty_namelist
 (t_initf) Using profile_disable= F  profile_timer=           4
 (t_initf)  profile_depth_limit=       99999  profile_detail_limit=           1
 (t_initf)  profile_barrier= F  profile_outpe_num=           0
 (t_initf)  profile_outpe_stride=           1  profile_single_file= F
 (t_initf)  profile_global_stats= T  profile_papi_enable= F

---- read control file --- 
 <ancil_dir> --> ./ancillary_data/
 <input_dir> --> ./input/
 <output_dir> --> ./output/
 <case_name> --> rhone_test
 <sim_start> --> 2001-01-01 00:00:00
 <sim_end> --> 2001-12-31 00:00:00
 <restart_write> --> never
 <restart_date> --> 2001-01-31 00:00:00
 <route_opt> --> 123
 <dt_qsim> --> 86400
 <newFileFrequency> --> daily
 <outputFrequency> --> daily
 <fname_ntopOld> --> rhone_rivertopo.nc
 <dname_sseg> --> seg
 <dname_nhru> --> hru
 <seg_outlet> --> -9999
 <fname_qsim> --> total_runoff_rhone_pcr.nc
 <vname_qsim> --> qtot
 <vname_time> --> time
 <vname_hruid> --> ID
 <dname_time> --> time
 <dname_hruid> --> ID
 <units_qsim> --> mm/s
 <dt_ro> --> 86400
 <is_remap> --> F
 <param_nml> --> param.nml
 <basRunoff> --> F
 <instRunoff> --> F
 <dlayRunoff> --> F
 <sumUpstreamRunoff> --> F
 <KWTroutedRunoff> --> F
 <IRFroutedRunoff> --> F
 <KWroutedRunoff> --> F
 <MCroutedRunoff> --> F
 <DWroutedRunoff> --> F
 <KWTvolume> --> T
 <IRFvolume> --> T
 <KWvolume> --> T
 <MCvolume> --> F
 <DWvolume> --> F
 <outputInflow> --> F

---- calendar --- 
  calendar used for simulation and history output is the same as runoff input
  calendar used in runoff input will be read from total_runoff_rhone_pcr.nc

---- time units --- 
  time_unit for history files will be the same as runoff input
  time_unit used in runoff input will be read from total_runoff_rhone_pcr.nc

---- runoff unit --- 
  runoff unit is provided as: mm/s

---- input time stamp --- 
  Input time stamp <ro_time_stamp>:  start
  The same time stamp is used for history output

---- output/simulation time steps --- 
  simulation time step <dt_qsim>:                86400.0
  history file freqeuncy <newFileFrequency>:  daily
  history output freqeuncy <outputFrequency>: daily

---- mis. routing options --- 

---- Read river network data --- 
  Reading area into structure HRU
  Reading HRUid into structure HRU2SEG
  Reading hruSegId into structure HRU2SEG
  Reading length into structure SEG
  Reading slope into structure SEG
  Reading segId into structure NTOPO
  Reading downSegId into structure NTOPO
Obtained 10 stack frames.
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7ebec8]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7ec4bc]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7efc74]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7eb07b]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x762940]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x43cf25]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x4c4b0f]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x5e43d5]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x75d8e5]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x75db61]

2001    1    1    0    0
Obtained 10 stack frames.
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7ebec8]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7ec4bc]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7efc74]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7eb07b]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x762940]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x43cf25]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x4c4b0f]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x5e43d5]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x75d8e5]
/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x75db61]
[node802:1481127] [0] func:/easybuild/software/OpenMPI/4.1.5-GCC-12.3.0/lib/libopen-pal.so.40(opal_backtrace_buffer+0x23) [0x1555540cd263]
[node802:1481127] [1] func:/easybuild/software/OpenMPI/4.1.5-GCC-12.3.0/lib/libmpi.so.40(ompi_mpi_abort+0x117) [0x155555089157]
[node802:1481127] [2] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7ec4d0]
[node802:1481127] [3] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7efc74]
[node802:1481127] [4] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x7eb07b]
[node802:1481127] [5] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x762940]
[node802:1481127] [6] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x43cf25]
[node802:1481127] [7] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x4c4b0f]
[node802:1481127] [8] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x5e43d5]
[node802:1481127] [9] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x75d8e5]
[node802:1481127] [10] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x75db61]
[node802:1481127] [11] func:/lib64/libc.so.6(+0x295d0) [0x1555548295d0]
[node802:1481127] [12] func:/lib64/libc.so.6(__libc_start_main+0x80) [0x155554829680]
[node802:1481127] [13] func:/home/ronnerac/mizuRoute/mizuRoute_code/route/bin/route_runoff() [0x409525]

I've added the addresses as proper text in this file - where it seems to link back to the creation of the output files.
Could this error be caused by not properly defining the calendar/timestep in my input file? The runoff file dimensions look like this (ncdump):

netcdf total_runoff_rhone_pcr {
dimensions:
        ID = 2114 ;
        time = UNLIMITED ; // (365 currently)
variables:
        double time(time) ;
                time:long_name = "time" ;
                time:units = "days since 1661-01-01" ;
                time:calendar = "standard" ;
                time:standard_name = "time" ;
                time:axis = "T" ;
        double latitude(ID) ;
                latitude:long_name = "latitude" ;
                latitude:units = "degrees_north" ;
                latitude:standard_name = "latitude" ;
        double longitude(ID) ;
                longitude:long_name = "longitude" ;
                longitude:units = "degrees_east" ;
                longitude:standard_name = "longitude" ;
        double ID(ID) ;
                ID:long_name = "shape ID" ;
                ID:units = "1" ;
        double qtot(time, ID) ;
                qtot:_FillValue = -9999. ;
                qtot:long_name = "runoff" ;
                qtot:units = "kg m-2 s-1" ;

// global attributes:
                :Conventions = "CF-1.6" ;
                :License = "GPLv3+; Originla license: No license from original file is detected" ;
                :History = "Created Thu Sep 18 16:41:02 2025by EASYMORE nc_remapper; original file history: " ;
                :easymore_hash = "473abf990c30f11bda2d91703ca7efa3e9738d7389be62122873fb063c01f45f" ;
                :Source = "Remapped by EASYMORE nc_remapper from original file: C:\\Users\\ronnerac\\Documents\\GeoDataBase\\total_runoff_rhone_pcr.nc" ;

[nmizukami]

Hi @arthurronner,

1. This is not a cause of the error, but I noticed you output daily values in a daily netcdf. This is lots of netcdfs you will need to handle if you run for long time. I would suggest changing to monthly or yearly.
2. I wonder if you build PIO using pnetcdf or regular netcdf. You might used regular netcdf. If this is the case, can you add <pio_netcdf_type> withnetcdf?
3. Then, maybe try increasing the stack size in linux (ulimit -s unlimited). This is the HPC things that may be beyond my knowledge (specific to your HPC etc.)

My guess is 2. Also, mizuRoute uses dynamic link, so I believe you need to load the same compiler, and libraries (netcdf, mpi etc.) when you run.

[arthurronner]

Hi! Yes! the pio_netcdf_type fixed the issue. I was indeed using netcdf - I did not know that distinction needed to be specified in the control file. It executed successfully! Thanks again for your help.

2001   12   30    0    0

2001   12   31    0    0 
----------------------
  SUCCESSFUL EXECUTION
 ----------------------

[nmizukami]

Glad to hear! All of these interactions help us to identify what is missing/confusing in the documentation.

[nmizukami]

Summary:

if a user compiles PIO without pnetcdf, <pio_netcdf_type> needs to be set to netcdf in a control file