Bad Ungrib Output From CMake Build

[scrasmussen]

ISSUE: running ungrib.exe gives bad results when using the new CMake build.

• An error first appears when running ./real.exe

$ tail rsl.error.0000
Max map factor in domain 1 =  1.00. Scale the dt in the model accordingly.
Using sfcprs3 to compute psfc
d01 2013-09-09_18:00:00 Old data, no inland lake information
 Assume Noah LSM input
d01 2013-09-09_18:00:00 forcing artificial silty clay loam at    7 points, out of   1681
-------------- FATAL CALLED ---------------
grid%tsk unreasonable
-------------------------------------------

$ tail rsl.out.0000
 i,j=          36           8
 grid%landmask=   1.00000000
 grid%tsk, grid%sst, grid%tmn=   162768112.       0.00000000       281.150146
 error in the grid%tsk
 i,j=          37           8
 grid%landmask=   0.00000000
 grid%tsk, grid%sst, grid%tmn=   162768112.       161704032.       162768112.
-------------- FATAL CALLED ---------------
grid%tsk unreasonable
-------------------------------------------

• But it gets introduced earlier, which is apparent when looking at met_em.d01.2013-09-09_18:00:00.nc output produced by a CMake built ungrib and met

• If we run ungrib with one build with the old configure method or /glade/work/wrfhelp/derecho_pre_compiled_code/wpsv4.6.0_jb/ungrib.exe and the rerun ./metgrid.exe, the output looks better

INSTRUCTIONS TO REPRODUCE ERROR:

• I'm working through the A2. Coupled WRF | WRF-Hydro Test Case User Guide in appendices.pdf. (I've attached a PDF because the readthedocs is slightly out of date and the PR with updates hasn't been merged yet)
• In the PDF there are instructions for the CMake and older GNU Make methods.
• I'm building on Derecho and using the GNU modules, as listed in the appendix instructions

$ module purge
$ module load ncarenv gcc ncarcompilers cray-mpich craype netcdf cmake

• Note, in namelist.wps I use

geog_data_path = '/glade/work/wrfhelp/WPS_GEOG/',

and followed the PDF instructions

[islas]

@scrasmussen Do you have a particular namelist and set of grib data you are using that you could point me to? I tried using some GFS grib data, making sure to use the .NAM vtable, but didn't get any issues visible in the metgrid netcdf output.

[scrasmussen]

@islas I've setup the directory in /glade/derecho/scratch/soren/src/wrf/ungrib_bug with everything needed to reproduce the issue.

The coupled case we use is in this tarball: https://github.com/NCAR/wrf_hydro_nwm_public/releases/download/v5.4.0/front_range_CO_example_testcase_coupled.tar.gz

In the README.md I have the instructions to build and reproduce the error in the run/WPS directory. That Makefile is setup to easily reproduce the error with a $ make broken and $ make working

$ cat run/WPS/Makefile
all: setup

setup:
        cp ../../WPS/install/bin/geogrid.exe .
        cp ../../WPS/geogrid/GEOGRID.TBL .
        cp ../../WPS/metgrid/METGRID.TBL .
        cp ../../example_case_coupled/namelist.wps .
        sed -i.bak "s|'geog',|'/glade/work/wrfhelp/WPS_GEOG',|" namelist.wps

        cp ../../WPS/install/bin/ungrib.exe ungrib.exe.broken
        cp ../../old_configure/WPS/ungrib.exe ungrib.exe.working

        cp ../../WPS/install/bin/ungrib.exe ungrib.exe.cmake
        cp ../../old_configure/WPS/ungrib.exe ungrib.exe.make

        cp ../../WPS/install/bin/metgrid.exe metgrid.exe.cmake
        cp ../../old_configure/WPS/metgrid.exe metgrid.exe.make

        cp ../../WPS/link_grib.csh .
        cp ../../WPS/ungrib/Variable_Tables/Vtable.NAM Vtable
        ./link_grib.csh ../../example_case_coupled/WRF_FORCING/*

geogrid:
        ./geogrid.exe

broken: geogrid
        ./ungrib.exe.cmake
        ./metgrid.exe.cmake
        cp met_em.d01.2013-09-09_18:00:00.nc met_em.d01.2013-09-09_18:00:00.nc.cmake
        cp met_em.d01.2013-09-09_18:00:00.nc met_em.d01.2013-09-09_18:00:00.nc.broken

working: geogrid
        ./ungrib.exe.make
        ./metgrid.exe.make
        cp met_em.d01.2013-09-09_18:00:00.nc met_em.d01.2013-09-09_18:00:00.nc.make
        cp met_em.d01.2013-09-09_18:00:00.nc met_em.d01.2013-09-09_18:00:00.nc.working

I use the following in jupyterhub to visualize the issue

import matplotlib.pyplot as plt
import xarray as xr
dir = '/glade/derecho/scratch/soren/src/wrf/ungrib_bug/run/WPS/'
f1 = dir + 'met_em.d01.2013-09-09_18:00:00.nc.working'
f2 = dir + 'met_em.d01.2013-09-09_18:00:00.nc.broken'
ds1 = xr.open_dataset(f1)
ds2 = xr.open_dataset(f2)
var = 'SKINTEMP'
ds1[var].plot(); plt.show()
ds2[var].plot(); plt.show()

[jimbresch]

A couple of comments about this issue:

1. To find a problem with ungrib the intermediate format files need to be compared not the metgrid output.
2. There's a known issue on derecho that I discovered a while back. When CISL changes the environment on derecho the support libraries need to be recompiled before compiling WPS. You shouldn't link a new compile with an old support library (such as wrfhelp). Doing that causes some fields in some grib files to not be decoded properly. It doesn't throw an error, just bad or missing output.

The precompiled code works because WPS and the support libraries were compiled with the same environment.

Ideally, wrfhelp would have a different library for each ncarenv and update it when CISL updates derecho.

Your comments don't indicate which jasper/png/z lib you are using.

[scrasmussen]

1. To find a problem with ungrib the intermediate format files need to be compared not the metgrid output.

Right, makes sense. I usually only use jupyterhub to visualize results and haven't figure a way to look at intermediate format files with that. But thanks for the clue, here is /glade/derecho/scratch/soren/src/wrf/ungrib_bug/run/WPS/NAM:2013-09-09_18.broken

and /glade/derecho/scratch/soren/src/wrf/ungrib_bug/run/WPS/NAM:2013-09-09_18.working

Your comments don't indicate which jasper/png/z lib you are using.

Yeah, sorry about that, its a bit hidden. In the appendix pdf I attached and in /glade/derecho/scratch/soren/src/wrf/ungrib_bug/README.md (I was trying to create a directory to easily test this) it is

 export Jasper_ROOT=/glade/u/home/wrfhelp/UNGRIB_LIBRARIES/

So the issue is the new CMake configure is pointing to the /glade/u/home/wrfhelp/UNGRIB_LIBRARIES/ for the Jasper libraries? And that those Jasper libraries are presumably built with the old configure Make method?

[scrasmussen]

I used a fresh environment so no Jasper_ROOT environment variable. Here are new results of having WPS build the jasper/png/z libraries with the following command

$ ./configure_new -x -p gfortran -- -DBUILD_EXTERNALS=ON
$ ./compile_new -j 4

[islas]

@scrasmussen Can you try editing the top-level CMakeLists.txt 164-165 from

add_compile_definitions( 
                        ${WPS_DEFINITIONS_LIST}
                        ${WPS_UNDEFINITIONS_LIST}
                        $<$<COMPILE_LANGUAGE:Fortran>:USE_JPEG2000>
                        $<$<COMPILE_LANGUAGE:Fortran>:USE_PNG>
                        )

to:

add_compile_definitions( 
                        ${WPS_DEFINITIONS_LIST}
                        ${WPS_UNDEFINITIONS_LIST}
                        USE_JPEG2000
                        USE_PNG
                        )

If that does work for you as well and resolve the issue (I'm pretty sure it will), I'll go ahead and make a PR for WPS to fix this. The issue was that dec_jpeg2000.c was effectively stubbed out by missing these defines. Similar issues may have arisen from dec_png.c also missing the analogous define.

[scrasmussen]

@islas Thanks for the fix! This seems to have done it, running ./real.exe now works!