Impossible to read wrfout* with PyFerret v7.64

[ArtyWRF]

Hello,

I'd like to use Ferret on my PC to check for wrfout files : unfortunately, it seems there's a problem reading vertical coordinate ZNU. Please see below.

> ferret
 	NOAA/PMEL TMAP
 	PyFerret v7.64 (optimized)
 	Linux 5.11.0-1021-azure - 12/10/21
 	19-Apr-23 13:25     

yes? use wrfout_d02_2005-02-02.nc
           *** NOTE:
           *** NOTE: Could not adjust grid for variable ZNU
 **ERROR: inconsistent data grids: File needs axis attributes on coordinate variables indicating correct directions
yes?

Is there anything I can do ?

I got the same error with a previous version on the supercomputer I'm using for work :

> ferret
 	NOAA/PMEL TMAP
 	FERRET v7.44 (optimized)
 	Linux 3.10.0-957.1.3.el7.x86_64 64-bit - 12/07/18
 	19-Apr-23 23:31     

yes? use wrfout_d01_2005-01-01.nc
           *** NOTE:
           *** NOTE: Could not adjust grid for variable ZNU
           *** NOTE: Axes in grids may be inconsistent.
           *** NOTE:
           *** NOTE: Could not adjust grid for variable ZNW
           *** NOTE: Axes in grids may be inconsistent.
yes? 

[ACManke]

Could you show the NetCDF header from an ncdump of the file?

[ACManke]

I have looked into this some more. It seems that the format for wrf files is not consistent with the COARDS and CF netCDF standards used by PyFerret https://www.ncl.ucar.edu/Applications/wrfnetcdf.shtml

The error messages that you are seeing are discussed in this thread on the Ferret Users List, where there was a different sort of file but the same error, https://www.pmel.noaa.gov/maillists/tmap/ferret_users/fu_2019/msg01013.html

I expect that this problem will be resolved with updates that are in the code base but not yet in a PyFerret release. I do not know what the plans are for the next release.

However, I have a suggestion, which I will also post to the Ferret List. I will show it using a small example file. This is a file from a previous question, which I think shows the behavior that is seen in the wrf files. Its netCDF header looks like this:

netcdf sample_wrf {
dimensions:
	Time = UNLIMITED ; // (13 currently)
	bottom_top = 2 ;
	south_north = 6 ;
	west_east = 7 ;
variables:
	float FNM(Time, bottom_top) ;
		FNM:FieldType = 104 ;
		FNM:MemoryOrder = "Z  " ;
		FNM:description = "upper weight for vertical stretching" ;
		FNM:units = "" ;
		FNM:stagger = "" ;
	float P(Time, bottom_top, south_north, west_east) ;
		P:FieldType = 104 ;
		P:MemoryOrder = "XYZ" ;
		P:description = "perturbation pressure" ;
		P:units = "Pa" ;
		P:stagger = "" ;

// global attributes:
		:TITLE = "Grids as in WRF model" ;
		:history = "Downloaded orig file from link at https://www.riinu.me/2014/05/saving-subset/" ;
}

The note you are seeing about axis coordinates suggests that PyFerret would use attributes on coordinate variables to determine the directions of the dimensions when it is defining grids. The file with coordinate variables would look like this, when listed using ncdump -c

netcdf sample_wrf {
dimensions:
	Time = UNLIMITED ; // (13 currently)
	bottom_top = 2 ;
	south_north = 6 ;
	west_east = 7 ;
variables:
	double Time(Time) ;
		Time:axis = "T" ;
	double bottom_top(bottom_top) ;
		bottom_top:axis = "Z" ;
	double south_north(south_north) ;
		south_north:axis = "Y" ;
	double west_east(west_east) ;
		west_east:axis = "X" ;
	float FNM(Time, bottom_top) ;
		FNM:FieldType = 104 ;
		FNM:MemoryOrder = "Z  " ;
		FNM:description = "upper weight for vertical stretching" ;
		FNM:units = "" ;
		FNM:stagger = "" ;
	float P(Time, bottom_top, south_north, west_east) ;
		P:FieldType = 104 ;
		P:MemoryOrder = "XYZ" ;
		P:description = "perturbation pressure" ;
		P:units = "Pa" ;
		P:stagger = "" ;

// global attributes:
		:TITLE = "Grids as in WRF model" ;
		:history = "Downloaded orig file from link at https://www.riinu.me/2014/05/saving-subset/" ;
data:

 Time = 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ;

 bottom_top = 1, 2 ;

 south_north = 1, 2, 3, 4, 5, 6 ;

 west_east = 1, 2, 3, 4, 5, 6, 7 ;
}

We can create a new, separate netCDF file that contains the desired coordinate variables. Then use the netcdf utility ncks to add the coordinate variables to the wrf file.

Here is a Ferret script to write a dummy file with the desired "axis" attributes. The appropriate axis attributes are automatically written by modern Ferret and PyFerret.

! Define the coordinate axes, the dimension names and lengths from the wrf file.  Keep 
! the same upper-case and lower-case spellings as in the file. 

define axis/t=1:13:1 Time
define axis/z=1:2:1 bottom_top
define axis/y=1:6:1 south_north
define axis/x=1:7:1 west_east

! Define variables on these axes to write to a new netCDF file.

let/title=Time tcoord = t[gt=Time]
let/title=bottom_top zcoord = z[gz=bottom_top]
let/title=south_north ycoord = y[gy=south_north]
let/title=west_east xcoord = x[gx=west_east]

! Write the dummy variables. We will get the coordinate axes from this file.

cancel mode upcase
save/clobber/file=wrf_axes.nc tcoord, zcoord, ycoord, xcoord

Now, use the Unix ncks utility to copy the coordinate variables into the wrf file. (make a copy of the file if you want to keep the original)

> ncks -A -h -v Time wrf_axes.nc sample_wrf.nc
> ncks -A -h -v bottom_top wrf_axes.nc sample_wrf.nc
> ncks -A -h -v south_north wrf_axes.nc sample_wrf.nc
> ncks -A -h -v west_east wrf_axes.nc sample_wrf.nc

Now, sample_wrf.nc will open.

[ArtyWRF]

Could you show the NetCDF header from an ncdump of the file?

header.txt

[wgustafson]

I would be highly in support of adding some sort of hack into Ferret to handle WRF files with the more recent versions of Ferret/PyFerret. If the only need is to add fake index arrays for the coordinates (my understanding of the info above in this thread), maybe a flag can be added to the file opening command to automatically add indexed values to any coordinate that has no values already assigned to it. On the surface at least, that sounds like an easy fix and would prevent having to post-process all the files to add the data with ncks. WRF has been around for almost 20 years now, so I have no expectation of the WRF developers bowing to pressure to change the file format--it has many oversights in terms of simple details for basic compliance, e.g., the handling of time, which would be very easy to fix, but has not been.

As noted in other threads in the past, WRF files used to work fine in Ferret, but have progressively become more problematic due to evolution of Ferret/PyFerret. It is to the point now that I have to avoid using Ferret and resort to Python instead, which is unfortunate given the simplicity and speed of plotting in Ferret.

[ACManke]

Well, when initializing a dataset, PyFerret (and Ferret) do internally create an array of index values to use for coordinates if there is only a dimension and not a coordinate variable in the file. That's a pretty common situation. The trouble here is that it is ambiguous which direction the coordinates should have in the grids. One way around that, which I suggested in the earlier comment, is if the file has coordinate variables with attributes that give the correct direction. Previous versions of Ferret just assigned directions in each variable's grid, as it processed them during file initialization, but that resulted in files where the same axis had different directions depending on which variable one was looking at. This gave tangled logic with some files, hence the updates in more recent versions.

There has been some more work on assigning grids on opening netCDF files since the last release. The current development version of the code opens the file wrfout_v2_Lambert.nc from this page: https://www.unidata.ucar.edu/software/netcdf/examples/files.html

Is there a good source for more example files? I would be happy to do more testing to see if the current code changes are robust.

[wgustafson]

Hi, WRF's files have not really changed too much over the years outside of a new way to handle the vertical grid. However, the way the info is encoded in the output files is still the same. As a check, I can easily send you a file from v3.4.1 of WRF, so a bit newer than the older v2 file you mention. I'm guessing the v3.4.1 wrfout file will work if the v2 file worked. When compressed, it is still too big to attache here though, about 208 MB. If you contact me directly, we can find a way to get the file to you, william.gustafson@pnnl.gov.

[ACManke]

I have checked this additional example file, and the current development code in my fork opens these WRF netCDF files correctly.

See Ferret issues 1046 and 1962 - When we have run into inconsistent ordering, if there are dimensions w/o coordinate variables, or without helpful attributes, check if a variable using such a dimension is a 2-D longitude or latitude coordinate variable. If so then its dimensions are in the X and Y directions.

The updates are in routines cd_consistent_axis_orient, cd_get_generic_grids and there is a test, bench/v7jnls/err765_ordering.jnl

[wgustafson]

Thanks @ACManke! We all appreciate the effort your put into Ferret!

[ryofurue]

WRF has been around for almost 20 years now, so I have no expectation of the WRF developers bowing to pressure to change the file format

On the other hand, the CF convention seems to be steadily gaining more influence. We have been in the age of "big data" for some time and the amount of data our simulation models generate has been explosively increasing. Developers of data-related software are working hard to catch up. In this context, I keep hearing more about the CF convention. I guess this is because we need standards to develop software tools about. Software developers don't want to spend extra time to be able to handle nonstandard datasets.

Now, WRF is just one of the many simulation models. By not adopting the CF convention, WRF may eventually lose popularity. So, I guess it's in the interest of WRF users to apply pressure to the developers.