ferret 7.6 won't handle 5d data

[apcraig]

This seems to be related to recent changes #1962.

We have a large data file with

dimensions:
	d2 = 2 ;
	ni = 5148 ;
	nj = 4400 ;
	nc = 5 ;
	nkice = 7 ;
	nksnow = 1 ;
	nkbio = 9 ;
	nkaer = 11 ;
	time = UNLIMITED ; // (1 currently)
	nvertices = 4 ;
variables:
	double time(time) ;
		time:long_name = "model time" ;
		time:units = "days since 0000-01-01 00:00:00" ;
		time:calendar = "NoLeap" ;
		time:bounds = "time_bounds" ;
	double time_bounds(time, d2) ;
		time_bounds:long_name = "boundaries for time-averaging interval" ;
		time_bounds:units = "days since 0000-01-01 00:00:00" ;
	float TLON(nj, ni) ;
		TLON:long_name = "T grid center longitude" ;
		TLON:units = "degrees_east" ;
		TLON:missing_value = 1.e+30f ;
		TLON:_FillValue = 1.e+30f ;
	float TLAT(nj, ni) ;
		TLAT:long_name = "T grid center latitude" ;
		TLAT:units = "degrees_north" ;
		TLAT:missing_value = 1.e+30f ;
		TLAT:_FillValue = 1.e+30f ;
	float ULON(nj, ni) ;
		ULON:long_name = "U grid center longitude" ;
		ULON:units = "degrees_east" ;
		ULON:missing_value = 1.e+30f ;
		ULON:_FillValue = 1.e+30f ;
	float ULAT(nj, ni) ;
		ULAT:long_name = "U grid center latitude" ;
		ULAT:units = "degrees_north" ;
		ULAT:missing_value = 1.e+30f ;
		ULAT:_FillValue = 1.e+30f ;
		ULAT:comment = "Latitude of NE corner of T grid cell" ;
	float NCAT(nc) ;
		NCAT:long_name = "category maximum thickness" ;
		NCAT:units = "m" ;
	float VGRDi(nkice) ;
		VGRDi:long_name = "vertical ice levels" ;
		VGRDi:units = "1" ;
	float VGRDs(nksnow) ;
		VGRDs:long_name = "vertical snow levels" ;
		VGRDs:units = "1" ;
	float VGRDb(nkbio) ;
		VGRDb:long_name = "vertical ice-bio levels" ;
		VGRDb:units = "1" ;
	float VGRDa(nkaer) ;
		VGRDa:long_name = "vertical snow-ice-bio levels" ;
		VGRDa:units = "1" ;
	float tmask(nj, ni) ;
		tmask:long_name = "ocean grid mask" ;
		tmask:coordinates = "TLON TLAT" ;
		tmask:missing_value = 1.e+30f ;
		tmask:_FillValue = 1.e+30f ;
		tmask:comment = "0 = land, 1 = ocean" ;
	float blkmask(nj, ni) ;
		blkmask:long_name = "ice grid block mask" ;
		blkmask:coordinates = "TLON TLAT" ;
		blkmask:comment = "mytask + iblk/100" ;
		blkmask:missing_value = 1.e+30f ;
		blkmask:_FillValue = 1.e+30f ;
...
	float keffn_top_d(time, nc, nj, ni) ;
		keffn_top_d:units = "W/m^2/K" ;
		keffn_top_d:long_name = "effective thermal conductivity of the top ice layer, ca
" ;
		keffn_top_d:coordinates = "TLON TLAT NCAT  time" ;
		keffn_top_d:cell_measures = "area: tarea" ;
		keffn_top_d:missing_value = 1.e+30f ;
		keffn_top_d:_FillValue = 1.e+30f ;
		keffn_top_d:cell_methods = "time: mean" ;
		keffn_top_d:time_rep = "averaged" ;
	float Tinz_d(time, nc, nkice, nj, ni) ;
		Tinz_d:units = "C" ;
		Tinz_d:long_name = "ice internal temperatures on CICE grid" ;
		Tinz_d:coordinates = "TLON TLAT VGRDi NCAT" ;
		Tinz_d:cell_measures = "area: tarea" ;
		Tinz_d:missing_value = 1.e+30f ;
		Tinz_d:_FillValue = 1.e+30f ;
		Tinz_d:cell_methods = "time: mean" ;
		Tinz_d:time_rep = "averaged" ;
	float Sinz_d(time, nc, nkice, nj, ni) ;
		Sinz_d:units = "ppt" ;
		Sinz_d:long_name = "ice internal bulk salinity" ;
		Sinz_d:coordinates = "TLON TLAT VGRDi NCAT" ;
		Sinz_d:cell_measures = "area: tarea" ;
		Sinz_d:missing_value = 1.e+30f ;
		Sinz_d:_FillValue = 1.e+30f ;
		Sinz_d:cell_methods = "time: mean" ;
		Sinz_d:time_rep = "averaged" ;
	float Tsnz_d(time, nc, nksnow, nj, ni) ;
		Tsnz_d:units = "C" ;
		Tsnz_d:long_name = "snow internal temperatures" ;
		Tsnz_d:coordinates = "TLON TLAT VGRDs NCAT" ;
		Tsnz_d:cell_measures = "area: tarea" ;
		Tsnz_d:missing_value = 1.e+30f ;
		Tsnz_d:_FillValue = 1.e+30f ;
		Tsnz_d:cell_methods = "time: mean" ;
		Tsnz_d:time_rep = "averaged" ;

When we load it into ferret, we get,

           *** NOTE: Unspecified or unsupported ordering of axes in variable Tinz_d
           *** NOTE: Check the dimensions: can USE/ORDER= be applied?
           *** NOTE: The default ordering will be used
           *** NOTE: Unspecified or unsupported ordering of axes in variable Sinz_d
           *** NOTE: Check the dimensions: can USE/ORDER= be applied?
           *** NOTE: The default ordering will be used
           *** NOTE: Unspecified or unsupported ordering of axes in variable Tsnz_d
           *** NOTE: Check the dimensions: can USE/ORDER= be applied?
           *** NOTE: The default ordering will be used
           *** NOTE: Axis TIME marked as TI
           *** NOTE: but it has different directions in grids.
 **ERROR: inconsistent data grids: File needs axis attributes on coordinate variables indicating correct directions

and the file won't work. If we remove the three 5d variables (these are the only 5d variables in the file) with

ncks -x -v Tinz_d,Sinz_d,Tsnz_d hilat_GJRA55v1.4_TL319_t071_onyx_NR_gl.cice.h.2019-08-10.nc test.nc

then everything works fine in ferret on the new file. Did recent changes in ferret to "stop" with inconsistent grids lead to loss of some capability of higher dimension variables? We are using "FERRET v7.6 (optimized) Linux 4.12.14-197.51-default 64-bit - 07/08/21". The original file with 5d variables works fine with ferret "FERRET v7.5 (optimized) Linux 4.12.14-150.17-default 64-bit - 07/11/19".

One other thing I can report. If I extract the three 5d variables from the file alone, then ferret seems to work fine,

/p/work/apcraig/test>ferret
 	NOAA/PMEL TMAP
 	FERRET v7.6 (optimized)
 	Linux 4.12.14-197.51-default 64-bit - 07/08/21
 	19-Jul-21 13:31     

yes? use test3.nc 
           *** NOTE: Unspecified or unsupported ordering of axes in variable Sinz_d
           *** NOTE: Check the dimensions: can USE/ORDER= be applied?
           *** NOTE: The default ordering will be used
           *** NOTE: Unspecified or unsupported ordering of axes in variable Tinz_d
           *** NOTE: Check the dimensions: can USE/ORDER= be applied?
           *** NOTE: The default ordering will be used
           *** NOTE: Unspecified or unsupported ordering of axes in variable Tsnz_d
           *** NOTE: Check the dimensions: can USE/ORDER= be applied?
           *** NOTE: The default ordering will be used
yes? sho dat
     currently SET data sets:
    1> ./test3.nc  (default)
 name     title                             I         J         K         L         M         N
 NCAT     category maximum thickness       ...       ...       ...       1:5       ...       ...
 SINZ_D   ice internal bulk salinity       1:5148    1:4400    1:7       1:5       1:1       ...
 TLAT     T grid center latitude           1:5148    1:4400    ...       ...       ...       ...
 TLON     T grid center longitude          1:5148    1:4400    ...       ...       ...       ...
 TINZ_D   ice internal temperatures on CI  1:5148    1:4400    1:7       1:5       1:1       ...
 TSNZ_D   snow internal temperatures       1:5148    1:4400    1:1       1:5       1:1       ...
 VGRDI    vertical ice levels              ...       ...       1:7       ...       ...       ...
 VGRDS    vertical snow levels             ...       ...       1:1       ...       ...       ...
 TAREA    area of T grid cells             1:5148    1:4400    ...       ...       ...       ...

So it may be more complicated than just those variables alone. It may be how other variables are also defined in the same file and the ordering of the axis in them. Any ideas?

I'm happy to provide the file or the full ncdump of it if that's helpful. The file is 19 Gb.

[pourbiazar]

Is this problem fixed? I have a similar problem. I have a WRF output that works fine with the older version of Ferret but doesn't work with the recent release. It is perhaps due to changes made on Jan. 7, 2020 (#1962). It seems that the problem is due to having variables with mixed dimensions and order. I guess an option to avoid the default behavior of Ferret (by avoiding the error check) and allowing the user to use i/j/k/l/m... would be desirable.

Here is the header for the original file:
original.txt

When I remove the problematic variables using ncks:
ncks -O -x -v ZNU,ZNW,ZS,C1F,C1H,C2F,C2H,DZS,C3F,C3H,C4F,C4H,DN,DNW,FNM,FNP,RDN,RDNW orig.nc corrected.nc

Then I will be able to use Ferret.

[ACManke]

• The fix for 1962 had to do with axis names that correspond to pseudo-variable names such as X, Y, I, J, so it may help you work with your files by means of changing dimension names. Here are some other discussions of the general topic:
  Impossible to read wrfout* with PyFerret v7.64 PyFerret#134
  How to read and plot WRF output ? PyFerret#94

Grid definition is not explicitly defined by the CF standard, and so pyferret works through the variables, trying to determine the direction of the dimensions. For a file like this, the dimensions do not have associated coordinate variables, and so an attempt is made to use dimension names. More description of this process is here, https://www.pmel.noaa.gov/maillists/tmap/ferret_users/fu_2019/msg01013.html

I think that if you rename the dimensions in your files using ncks and the list above, you should be able to open the dataset. If you are running PyFerret v7.63 then all of the above names should indicate a direction, so you could use X for "west_east" and I for "west_east_stag"; Z for "bottom_top", and perhaps layer for "soil_layers_stag", and so forth.

[apcraig]

I continue to run into problems with ferret with the axes. This example,

https://cfconventions.org/cf-conventions/cf-conventions.html#bounds-lat-lon

does not work in ferret. I get the dreaded

**ERROR: inconsistent data grids: File needs axis attributes on coordinate variables indicating correct directions

because nv is the first dimension in the bounds variables. It seems like ferret now requires all of the axis in a file to be aligned in the same position in all variables or it fails. This is a huge problem. I have tried to add axis attributes to dimensions and that helps a bit but does not help in this case. If I make the nv variable the third dimension for the bounds, then ferret reads the file.

For the cfconventions example above, what attributes (are there any) would you recommend adding to variables to make the file readable by ferret?

[apcraig]

FYI, we still have

FERRET v7.6 (optimized)
Linux 5.14.21-150400.24.46-default 64-bi - 11/27/23

as the default version. Would a newer version of ferret help?

[pourbiazar]

• The fix for 1962 had to do with axis names that correspond to pseudo-variable names such as X, Y, I, J, so it may help you work with your files by means of changing dimension names. Here are some other discussions of the general topic:
  Impossible to read wrfout* with PyFerret v7.64 PyFerret#134
  How to read and plot WRF output ? PyFerret#94

Grid definition is not explicitly defined by the CF standard, and so pyferret works through the variables, trying to determine the direction of the dimensions. For a file like this, the dimensions do not have associated coordinate variables, and so an attempt is made to use dimension names. More description of this process is here, https://www.pmel.noaa.gov/maillists/tmap/ferret_users/fu_2019/msg01013.html

I think that if you rename the dimensions in your files using ncks and the list above, you should be able to open the dataset. If you are running PyFerret v7.63 then all of the above names should indicate a direction, so you could use X for "west_east" and I for "west_east_stag"; Z for "bottom_top", and perhaps layer for "soil_layers_stag", and so forth.

This is making it hard to work with WRF output files and many other non-standard files. The problem is that Ferret cannot be used for even one problematic variable in the file. Why not make it a "warning" (as in previous versions) so we can still use Ferret without doing all this additional work to modify the file?