Martin Dix's fix for pre-industrial configuration produces a corrupted restart000 file

[penguian]

Running Martin Dix's fix as per 0f769ae results in a corrupted archive/access-esm/restart000/atmosphere/restart_dump.astart file in the following sense.

I ran the configuration twice, moving the /scratch/tm70/pcl851/access-esm/archive directory to /scratch/tm70/pcl851/access-esm/archive.coecms.1 and then /scratch/tm70/pcl851/access-esm/archive.coecms.2, then ran the following commands with results as shown.

[pcl851@gadi-login-02 access-esm]$ module use /g/data/access/modules
[pcl851@gadi-login-02 access-esm]$ module load um
[pcl851@gadi-login-02 access-esm]$ module use /g/data/hh5/public/modules/
[pcl851@gadi-login-02 access-esm]$ module load conda/analysis3-23.07
[pcl851@gadi-login-02 access-esm]$ mule-cumf archive.coecms.?/access-esm/restart000/atmosphere/restart_dump.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule       : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils   : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)


/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: archive.coecms.1/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: archive.coecms.2/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
Traceback (most recent call last):
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/bin/mule-cumf", line 10, in <module>
    sys.exit(_main())
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 1385, in _main
    comparison = UMFileComparison(um_files[0], um_files[1])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 728, in __init__
    diff_field = difference_op([field_1, field_2])
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py", line 952, in __call__
    new_field = self.new_field(source, *args, **kwargs)
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/cumf.py", line 293, in new_field
    data1 = fields[0].get_data()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py", line 730, in get_data
    data = self._data_provider._data_array()
  File "/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/ff.py", line 193, in _data_array
    data = np.fromstring(data_bytes, dtype, count=count)
ValueError: string is smaller than requested size

See also https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/6

[penguian]

Looking deeper using mule-summary, and accounting for field numbers starting from 0 rather than 1:

[pcl851@gadi-login-02 access-esm]$ mule-summary archive.coecms.1/access-esm/restart000/atmosphere/restart_dump.astart|grep -C2 "1115\|1116\|1117\|4100\|4102\|5485\|5524"
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning: 
File: archive.coecms.1/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
  Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
  File            : 145 points from -90.0, spacing 1.25
  Field (Expected): 180 points from -89.5, spacing 1.25
  Field (Lookup)  : 180 points from 89.5, spacing -1.0
Field validation failures:
  Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode: 
  Field lbcode: 31320
  warnings.warn(msg)
    1113 | W COMPNT OF WIND AFTER TIMESTEP      | 8760 |    37 |   42 |     150 |   192 |   145 |  0102/01/01 00:00:01 
    1114 | W COMPNT OF WIND AFTER TIMESTEP      | 8760 |    38 |   42 |     150 |   192 |   145 |  0102/01/01 00:00:01 
    1115 | RIVER SEQUENCE                       | 8760 |  9999 | 1905 |     151 |   360 |   180 |  0102/01/01 00:00:01 
    1116 | RIVER DIRECTION                      | 8760 |  9999 | 1906 |     152 |   360 |   180 |  0102/01/01 00:00:01 
    1117 | RIVER WATER STORAGE               M2 | 8760 |  9999 | 1902 |     153 |   360 |   180 |  0102/01/01 00:00:01 
    1118 | ACCUMULATED SURFACE RUNOFF     KG/M2 | 8760 |  9999 | 1907 |     155 |   192 |   145 |  0102/01/01 00:00:01 
    1119 | ACCUMULATED SUB-SURFACE RUNOFF KG/M2 | 8760 |  9999 | 1908 |     156 |   192 |   145 |  0102/01/01 00:00:01 
--
    4098 | SEA-ICE TEMPERATURE AFTER B. LAYER   | 8760 |     0 |  209 |    3049 |   192 |   145 | 0101/12/01 00:00:335 
    4099 | FLUX OF TRACER 1 IN BL               | 8760 |     0 | 1389 |    3100 |   192 |   145 | 0101/12/01 00:00:335 
    4100 | FLUX OF TRACER 1 IN BL               | 8760 |     0 | 1389 |    3100 |    43 |    48 | 0101/12/31 00:00:365 
 # ----- | ------------------------------------ | ---- | ----- | ---- | ------- | ----- | ----- | -------------------- 
 # index | stash_name                           | lbft | lblev | lbfc | lbuser4 | lbnpt | lbrow |                   t1 
 # ----- | ------------------------------------ | ---- | ----- | ---- | ------- | ----- | ----- | -------------------- 
    4101 | FLUX OF TRACER 2 IN BL               | 8760 |     0 | 1389 |    3101 |   192 |   145 | 0101/12/01 00:00:335 
    4102 | FLUX OF TRACER 2 IN BL               | 8760 |     0 | 1389 |    3101 |    43 |    48 | 0101/12/31 00:00:365 
    4103 | UNKNOWN STASH CODE                   | 8760 |  9999 | 1510 |    3173 |   192 |   145 | 0101/12/01 00:00:335 
    4104 | UNKNOWN STASH CODE                   | 8760 |  9999 | 1510 |    3173 |   192 |   145 | 0101/12/01 00:00:335 
--
    5483 | ATM TRACER  1               AFTER TS | 8760 |    37 |  501 |   33001 |   192 |   145 | 0101/12/01 00:00:335 
    5484 | ATM TRACER  1               AFTER TS | 8760 |    38 |  501 |   33001 |   192 |   145 | 0101/12/01 00:00:335 
    5485 | ATM TRACER  1               AFTER TS | 8760 |     1 |  501 |   33001 |    57 |    48 | 0101/12/31 00:00:365 
    5486 | ATM TRACER  2               AFTER TS | 8760 |     1 |  502 |   33002 |   192 |   145 | 0101/12/01 00:00:335 
    5487 | ATM TRACER  2               AFTER TS | 8760 |     2 |  502 |   33002 |   192 |   145 | 0101/12/01 00:00:335 
--
    5522 | ATM TRACER  2               AFTER TS | 8760 |    37 |  502 |   33002 |   192 |   145 | 0101/12/01 00:00:335 
    5523 | ATM TRACER  2               AFTER TS | 8760 |    38 |  502 |   33002 |   192 |   145 | 0101/12/01 00:00:335 
    5524 | ATM TRACER  2               AFTER TS | 8760 |     1 |  502 |   33002 |    57 |    48 | 0101/12/31 00:00:365 

[penguian]

Removing all fields with lbuser4 in [151,152,153,155,156,3100,3101,33001,33002] eliminates the ValueError and the warning messages from mule-cumf. This results in the following:

[pcl851@gadi-login-02 access-esm]$ mule-cumf archive.coecms.?/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule       : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils   : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* CUMF-II Comparison Report *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

File 1: archive.coecms.1/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart
File 2: archive.coecms.2/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart
Files DO NOT compare
  * 0 differences in fixed_length_header (with 7 ignored indices)
  * 1 field differences, of which 1 are in data

Compared 5359/5359 fields, with 5358 matches

Maximum RMS diff as % of data in file 1: 0.17261724377693055 (field 4235)
Maximum RMS diff as % of data in file 2: 0.172578822112673  (field 4235)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* Field 4235/5359 - CO2 LAND SURFACE FLUX     KG/M**2/S *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Lookup compares, data DOES NOT compare
Compared 64/64 lookup values.
File_1 lookup info:
  t1(0101/12/01 00:00:335)  lblev(0)/blev(0.0)  lbproc(128)
Data differences:
  Number of point differences  : 6/27840
  Maximum absolute difference  : 157313.73429443128
  RMS difference               : 970.40634872502915
  RMS diff as % of file_1 data : 0.17261724377693055
  RMS diff as % of file_2 data : 0.172578822112673 

[penguian]

Given

[pcl851@gadi-login-02 access-esm]$ mule-summary archive.coecms.1/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart|grep "CO2 LAND SURFACE FLUX"
    4235 | CO2 LAND SURFACE FLUX     KG/M**2/S  | 8760 |     0 | 1562 |    3326 |   192 |   145 | 0101/12/01 00:00:335 
[pcl851@gadi-login-02 access-esm]$ grep "CO2 LAND SURFACE FLUX" ~access/umdir/vn7.3/ctldata/STASHmaster/STASHmaster_A
1|    1 |    3 |  326 |CO2 LAND SURFACE FLUX     KG/M**2/S |

I also eliminated fields with lbuser4==3326. This finally results in a clean comparison:

[pcl851@gadi-login-02 access-esm]$ mule-cumf archive.coecms.?/access-esm/restart000/atmosphere/fixed.restart_dump.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

mule       : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils   : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* CUMF-II Comparison Report *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

File 1: archive.coecms.1/access-esm/restart000/atmosphere/fixed.restart_dump.astart
File 2: archive.coecms.2/access-esm/restart000/atmosphere/fixed.restart_dump.astart
Files compare
  * 0 differences in fixed_length_header (with 7 ignored indices)
  * 0 field differences, of which 0 are in data

Compared 5358/5358 fields, with 5358 matches

[penguian]

See https://github.com/penguian/access-esm/blob/pre-industrial-fix-restart/bin/fix_esm15_restart.py for a Python script that will fix restart_dump.astart by removing the fields listed above.

Usage:

$ module use /g/data/access/modules
$ module load um
$ module use /g/data/hh5/public/modules/
$ module load conda/analysis3-23.07
$ cd archive/access-esm/restart000/atmosphere/
$ python $PATHTO/bin/fix_esm15_restart.py restart_dump.astart fixed.restart_dump.astart

Thanks to Scott Wales for the original Python code as per https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/5

[MartinDix]

Because of the STASH time averaging the restart file contains partial time averages of diagnostic fields as well as the real prognostics that are strictly required. Some of these problems come from the diagnostics. E.g. fields with STASH codes 3,326 and 3,327 are CO2 fluxes which are not set in this concentration driven experiment.

The river model runs on its own 1x1 degree grid and mule thinks the river fields (stash codes 151,152,153) are invalid because this doesn't match the overall file grid.

Fields with stash codes 155 & 156 are accumulated runoff. According to the STASHmaster file these are supposed to be saved on land points only.

 ~access/umdir/vn7.3/ctldata/STASHmaster/STASHmaster_A
1|    1 |    0 |  155 |ACCUMULATED SURFACE RUNOFF     KG/M2|
2|    3 |    0 |    1 |   21 |    5 |   -1 |   -1 |    0 |    0 |    0 |    0 |

The 21 in the 4th field of the second row means land points only.

However for some reason the field header in the restart file shows it as global

mule-pumf --components=lookup --headers-only --field-property lbuser4=155 restart file
%%%%%%%%%%%%%%%%%%%%
* Field 1118/3640  *
%%%%%%%%%%%%%
...
  (15) lblrec  :         10865
  (21) lbpack  :             0

where lbpack=10865 is the data size, here equal to the number of land points.

C.f. soil moisture (lbuser4=9) which shows
(21) lbpack : 120

The model itself gets the packing from the STASHmaster file so reads this field ok. Not clear why it's not written properly? It's ok in ACCESS-CM2 restart files.

An alternative way to get rid of the problem fields is
mule-select restart restart.subset --include lbproc=0 --exclude lbuser4=151,152,153,155,156
which includes only the true prognostic fields (lbproc=0) and excludes the problem ones.

[penguian]

I also created a ticket against mule-cumf: https://code.metoffice.gov.uk/trac/um/ticket/7560#ticket