cdms2 fails on opening a file with a badly encoded string attribute

[jypeter]

@jasonb5 I have come across a NetCDF file that cdms2 can't open, probably due to a variable that has a long_name attribute with badly encoded French accented characters...

I'm using cdms2 3.1.5 from CDAT 8.2.1. Can you check if you can reproduce and correct this error? Thanks!

Using a standard test file

I first tried to reproduce the error by adding a crappy attribute to one of the CDAT test files:

ncatted -a crappy_att,climseas,c,c,'résumé' tas_mo_clim.nc tas_mo_clim_v01.nc

cdms2 has no problem opening the file, reading the variable and displaying the attribute with accented characters:

>>> f = cdms2.open('tas_mo_clim_v01.nc')
>>> v = f('climseas')
>>> v.crappy_att
'résumé'

The file that cdms2 can't open

Now, if I try to open the proba_notrig.nc file, that has the following attribute (when piping the output of ncdump -h through less)

        float proba_notrig(time_counter, lat, lon) ;
                proba_notrig:long_name = "Probabilit<E9> de non-d<E9>clenchement" ;

I trigger a long (and not very helpful) traceback

>>> g = cdms2.open('proba_notrig.nc')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 10: invalid continuation byte

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/lib/python3.8/site-packages/cdms2/dataset.py", line 1275, in __init__
    _fileobj_ = Cdunif.CdunifFile(path, mode)
SystemError: <built-in function CdunifFile> returned a result with an error set

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/lib/python3.8/site-packages/cdms2/dataset.py", line 506, in openDataset
    file1 = CdmsFile(path, "r")
  File "/home/share/unix_files/cdat/miniconda3_21-02/envs/cdatm19_nompi_py3/lib/python3.8/site-packages/cdms2/dataset.py", line 1277, in __init__
    raise CDMSError('Cannot open file %s (%s)' % (path, err))
cdms2.error.CDMSError: Cannot open file /home/scratch01/jypeter/proba_notrig.nc (<built-in function CdunifFile> returned a result with an error set)