Publish decadal fixes to new c3s indices

[ellesmith88]

Publish decadal fixes to c3s-roocs-fix-proposal and c3s-roocs-fix indices.

This involves understanding what has changed with date/text field for value.

Also need to add the fix for removing the coordinate attribute (in dachar and daops) - now that the xarray fix has been released in v0.19

[agstephens]

Hi @ellesmith88, I tested the fixes in daops and noted that the reftime has a default calendar (gregorian). I think that reftime.calendar should match time.calendar. Please can you take a look? Thanks

[ellesmith88]

@agstephens I agree with you and this should be straightforward to implement now that we are deriving values in the fixes. However I will have to regenerate all the fixes (which should hopefully be a lot quicker this time!). I think it must be leftover from when I was using the example file to create the fix template.

If I am regenerating the fixes, should I also remove the fill value fix as I mentioned in roocs/daops#88? As it is all done in clisops now.

[agstephens]

@ellesmith88 Thanks, I just responded to roocs/daops#88 so you should be ready to proceed with this.

[agstephens]

Awaiting new version after early 2022 fixes are updated and published.

[ellesmith88]

Updated fixes are currently being indexed 😄

[ellesmith88]

Fixes have been updated in c3s-roocs-fix to derive reftime and no longer include a fill value fix.

[agstephens]

@ellesmith88 are the fixes all in place now? Thanks

[ellesmith88]

@agstephens Yes all fixes are in the index and master of daops and dachar are up to date

[agstephens]

Hi @ellesmith88, the fixes are looking good. Great stuff.

[agstephens]

Tested with local script:

$ ROOCS_CONFIG=../dachar/MY_roocs.ini python open-decadal-is-fixed.py
2022-01-14 14:34:03,879 - /home/users/astephen/roocs/daops/daops/utils/consolidate.py - INFO - Testing 1 files in time range: ...
2022-01-14 14:34:03,879 - /home/users/astephen/roocs/daops/daops/utils/consolidate.py - INFO - Kept 1 files
2022-01-14 14:34:03,879 - /home/users/astephen/roocs/daops/daops/utils/normalise.py - INFO - Working on datasets: OrderedDict([('c3s-cmip6.DCPP.CMCC.CMCC-CM2-SR5.dcppA-hindcast.s1960-r10i1p1f1.Amon.pr.gn.v20210719', ['/badc/cmip6/data/CMIP6/DCPP/CMCC/CMCC-CM2-SR5/dcppA-hindcast/s1960-r10i1p1f1/Amon/pr/gn/v20210719/pr_Amon_CMCC-CM2-SR5_dcppA-hindcast_s1960-r10i1p1f1_gn_196011-197012.nc'])])
2022-01-14 14:34:03,900 - elasticsearch - INFO - GET https://elasticsearch.ceda.ac.uk:443/c3s-roocs-fix/_doc/8b4a57d954a432024c532e2a6b05e281 [status:200 request:0.019s]
2022-01-14 14:34:03,991 - /home/users/astephen/roocs/daops/daops/utils/core.py - INFO - Running post-processing function: edit_var_attrs
2022-01-14 14:34:03,991 - /home/users/astephen/roocs/daops/daops/utils/core.py - INFO - Running post-processing function: edit_global_attrs
2022-01-14 14:34:03,992 - /home/users/astephen/roocs/daops/daops/utils/core.py - INFO - Running post-processing function: add_scalar_coord
2022-01-14 14:34:03,994 - /home/users/astephen/roocs/daops/daops/utils/core.py - INFO - Running post-processing function: add_coord
2022-01-14 14:34:03,996 - /home/users/astephen/roocs/daops/daops/utils/core.py - INFO - Running post-processing function: add_data_var
2022-01-14 14:34:03,997 - /home/users/astephen/roocs/daops/daops/utils/core.py - INFO - Running post-processing function: remove_coord_attr
2022-01-14 14:34:04,001 - /home/users/astephen/roocs/daops/daops/processor.py - INFO - Running subset [serial]: on Dataset with args: {'area': Area to subset over:
 None, 'level': Level range to subset over
 first_level: None
 last_level: None, 'time': Time period to subset over
 start time: 1960-01-01T00:00:00
 end time: 1961-01-01T23:59:59, 'time_components': No time components specified, 'output_type': 'xarray', 'output_dir': None, 'split_method': 'time:auto', 'file_namer': 'standard'}
/home/users/astephen/roocs/clisops/clisops/ops/subset.py:124: UserWarning: "start_date" not found within input date time range. Defaulting to minimum time step in xarray object.
  result = subset_time(self.ds, **kwargs)
/home/users/astephen/roocs/clisops/clisops/ops/subset.py:124: UserWarning: "end_date" has been nudged to nearest valid time step in xarray object.
  result = subset_time(self.ds, **kwargs)
2022-01-14 14:34:04,014 - /home/users/astephen/roocs/clisops/clisops/ops/base_operation.py - INFO - Processing Subset for times: ('1960-11-16', '1960-12-16')
2022-01-14 14:34:04,014 - /home/users/astephen/roocs/clisops/clisops/utils/output_utils.py - INFO - format_writer=None, output_type=xarray
2022-01-14 14:34:04,014 - /home/users/astephen/roocs/clisops/clisops/utils/output_utils.py - INFO - Returning output as <class 'xarray.core.dataset.Dataset'>
Check leadtime units and data type
Checking global attrs

Script is:

$ cat open-decadal-is-fixed.py
from daops.ops.subset import subset

dsid = "c3s-cmip6.DCPP.CMCC.CMCC-CM2-SR5.dcppA-hindcast.s1960-r10i1p1f1.Amon.pr.gn.v20210719"

rs = subset(collection=dsid, time="1960-01-01/1961-01-01", output_type="xarray")
ds = list(rs._results.values())[0][0]


print("Check leadtime units and data type")
assert ds.leadtime.dtype == "float64"
assert ds.leadtime.units == "days"

print("Checking global attrs")
expected_globals = {
    'forcing_description': 'f1, CMIP6 historical forcings',
    'physics_description': 'physics from the standard model configuration, with no additional tuning or different parametrization',
    'initialization_description': 'hindcast initialized based on observations and using historical forcing',
    'startdate': 's196011',
    'sub_experiment_id': 's196011'
}


for key, value in expected_globals.items():
    assert ds.attrs[key] == value


assert ds.time.values[-1].strftime("%Y-%m-%d") <= "1961-01-01"
assert ds.time.values[0].strftime("%Y-%m-%d") >= "1960-01-01"
assert ds.time.encoding["calendar"] == "365_day"