Auto tripyview

[SimonPorr]

Automatic usage of Tripyview in ESM_tools FESOM 2.6 model, including setting up the necessary environments and the installation of Tripyview tool.

[pgierz]

Looks great. I'll try this out and see how it goes, if there are no strange errors, I'll approve.

We might consider also adding a section in the handbook. What do you think, @mandresm?

[JanStreffing]

looks good from my side

[JanStreffing]

Can we go ahead with this one? @SimonPorr has also done more work for OpenIFS automatic plots, which should also be merged.

[mandresm]

Just so @JanStreffing knows, I have this in my radar, to take care of it soon.

[JanStreffing]

It would be nice to have this one merged for testing the fesom 2.7 release candidate today and tomorrow.

[pgierz]

Back-merge done, this just needs a test. I'll see if I have something on albedo which can run quickly...

[pgierz]

Unfortunately, CI tests are not passing. The failing tests all relate to AWICM3 :( I would vote to just ignore it for now, but I also really do not want to introduce a default failing state (@mandresm: any opinion?)

On the positive side, my new test for AWIESM-2.6 seems to run fine. One gets populated Jupyter notebooks and plots like so:

$ tree /albedo/work/user/pgierz/SciComp/Projects/esm-tools-auto-tripyview/experiments/atpv-new-002/analysis/fesom
/albedo/work/user/pgierz/SciComp/Projects/esm-tools-auto-tripyview/experiments/atpv-new-002/analysis/fesom
├── atpv-new-002_hmesh_None_nears.ipynb
├── atpv-new-002_hmesh_nresol_ortho.ipynb
├── atpv-new-002_hovm_clim_salt_North_Atlantic_Ocean.ipynb
├── atpv-new-002_hovm_clim_salt_Southern_Ocean.ipynb
├── atpv-new-002_hovm_clim_temp_North_Atlantic_Ocean.ipynb
├── atpv-new-002_hovm_clim_temp_Southern_Ocean.ipynb
├── atpv-new-002_hovm_salt_Arctic_Basin.ipynb
├── atpv-new-002_hovm_salt_Greenland_Sea.ipynb
├── atpv-new-002_hovm_salt_Irminger_Sea.ipynb
├── atpv-new-002_hovm_salt_Labrador_Sea.ipynb
├── atpv-new-002_hovm_salt_North_Atlantic_Ocean.ipynb
├── atpv-new-002_hovm_salt_North_Pacific_Ocean.ipynb
├── atpv-new-002_hovm_salt_Norwegian_Sea.ipynb
├── atpv-new-002_hovm_salt_Southern_Ocean.ipynb
├── atpv-new-002_hovm_salt_South_Pacific_Ocean.ipynb
├── atpv-new-002_hovm_salt_Western_Weddell_Sea.ipynb
├── atpv-new-002_hovm_temp_Arctic_Basin.ipynb
├── atpv-new-002_hovm_temp_Greenland_Sea.ipynb
├── atpv-new-002_hovm_temp_Irminger_Sea.ipynb
├── atpv-new-002_hovm_temp_Labrador_Sea.ipynb
├── atpv-new-002_hovm_temp_North_Atlantic_Ocean.ipynb
├── atpv-new-002_hovm_temp_North_Pacific_Ocean.ipynb
├── atpv-new-002_hovm_temp_Norwegian_Sea.ipynb
├── atpv-new-002_hovm_temp_Southern_Ocean.ipynb
├── atpv-new-002_hovm_temp_South_Pacific_Ocean.ipynb
├── atpv-new-002_hovm_temp_Western_Weddell_Sea.ipynb
├── atpv-new-002_hslice_clim_salt_rob_z1000.ipynb
├── atpv-new-002_hslice_clim_salt_rob_z100.ipynb
├── atpv-new-002_hslice_clim_temp_rob_z1000.ipynb
├── atpv-new-002_hslice_clim_temp_rob_z100.ipynb
├── atpv-new-002_hslice_Kv_rob_z1000.ipynb
├── atpv-new-002_hslice_Kv_rob_z100.ipynb
├── atpv-new-002_hslice_MLD2_rob.ipynb
├── atpv-new-002_hslice_np_a_ice_nps_m3.ipynb
├── atpv-new-002_hslice_np_a_ice_nps_m9.ipynb
├── atpv-new-002_hslice_np_m_ice_nps_m3.ipynb
├── atpv-new-002_hslice_np_m_ice_nps_m9.ipynb
├── atpv-new-002_hslice_np_MLD2_nps_m3.ipynb
├── atpv-new-002_hslice_np_MLD2_nps_m9.ipynb
├── atpv-new-002_hslice_salt_rob_z1000.ipynb
├── atpv-new-002_hslice_salt_rob_z100.ipynb
├── atpv-new-002_hslice_sp_a_ice_sps_m3.ipynb
├── atpv-new-002_hslice_sp_a_ice_sps_m9.ipynb
├── atpv-new-002_hslice_sp_m_ice_sps_m3.ipynb
├── atpv-new-002_hslice_sp_m_ice_sps_m9.ipynb
├── atpv-new-002_hslice_sp_MLD2_sps_m3.ipynb
├── atpv-new-002_hslice_sp_MLD2_sps_m9.ipynb
├── atpv-new-002_hslice_ssh_rob.ipynb
├── atpv-new-002_hslice_temp_rob_z1000.ipynb
├── atpv-new-002_hslice_temp_rob_z100.ipynb
├── atpv-new-002_transect_clim_salt_170W_80S-90N.ipynb
├── atpv-new-002_transect_clim_salt_30W_80S-90N.ipynb
├── atpv-new-002_transect_clim_salt_Drake_Passage.ipynb
├── atpv-new-002_transect_clim_sigma2_170W_80S-90N.ipynb
├── atpv-new-002_transect_clim_sigma2_30W_80S-90N.ipynb
├── atpv-new-002_transect_clim_sigma2_Drake_Passage.ipynb
├── atpv-new-002_transect_clim_temp_170W_80S-90N.ipynb
├── atpv-new-002_transect_clim_temp_30W_80S-90N.ipynb
├── atpv-new-002_transect_clim_temp_Drake_Passage.ipynb
├── atpv-new-002_transect_Kv_170W_80S-90N.ipynb
├── atpv-new-002_transect_Kv_30W_80S-90N.ipynb
├── atpv-new-002_transect_Kv_Drake_Passage.ipynb
├── atpv-new-002_transect_salt_170W_80S-90N.ipynb
├── atpv-new-002_transect_salt_30W_80S-90N.ipynb
├── atpv-new-002_transect_salt_Drake_Passage.ipynb
├── atpv-new-002_transect_temp_170W_80S-90N.ipynb
├── atpv-new-002_transect_temp_30W_80S-90N.ipynb
├── atpv-new-002_transect_temp_Drake_Passage.ipynb
├── atpv-new-002_transect_transp_Drake_Passage.ipynb
├── atpv-new-002_transect_transp_t_Drake_Passage.ipynb
├── atpv-new-002_transect_zmean_clim_salt_Atlantic_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_salt_global.ipynb
├── atpv-new-002_transect_zmean_clim_salt_Indian_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_salt_Pacific_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_sigma2_Atlantic_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_sigma2_global.ipynb
├── atpv-new-002_transect_zmean_clim_sigma2_Indian_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_sigma2_Pacific_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_temp_Atlantic_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_temp_global.ipynb
├── atpv-new-002_transect_zmean_clim_temp_Indian_Basin.ipynb
├── atpv-new-002_transect_zmean_clim_temp_Pacific_Basin.ipynb
├── atpv-new-002_vprofile_clim_salt.ipynb
├── atpv-new-002_vprofile_clim_temp.ipynb
├── atpv-new-002_vprofile_Kv.ipynb
├── atpv-new-002_vprofile_salt.ipynb
├── atpv-new-002_vprofile_temp.ipynb
├── atpv-new-002_zmoc_amoc.ipynb
├── atpv-new-002_zmoc_gmoc.ipynb
├── atpv-new-002_zmoc_ipmoc.ipynb
├── atpv-new-002_zmoc_t_amoc26.ipynb
├── atpv-new-002_zmoc_t_amoc40.ipynb
└── atpv-new-002_zmoc_t_amoc['max'].ipynb

0 directories, 93 files

❯ tree /albedo/work/user/pgierz/SciComp/Projects/esm-tools-auto-tripyview/experiments/atpv-new-002/viz/fesom
/albedo/work/user/pgierz/SciComp/Projects/esm-tools-auto-tripyview/experiments/atpv-new-002/viz/fesom
├── atpv-new-002_hmesh_None_nears.png
├── atpv-new-002_hmesh_nresol_ortho.png
├── atpv-new-002_hslice_Kv_rob_z1000.png
├── atpv-new-002_hslice_Kv_rob_z100.png
├── atpv-new-002_hslice_MLD2_rob.png
├── atpv-new-002_hslice_np_a_ice_nps_m3.png
├── atpv-new-002_hslice_np_a_ice_nps_m9.png
├── atpv-new-002_hslice_np_m_ice_nps_m3.png
├── atpv-new-002_hslice_np_m_ice_nps_m9.png
├── atpv-new-002_hslice_np_MLD2_nps_m3.png
├── atpv-new-002_hslice_np_MLD2_nps_m9.png
├── atpv-new-002_hslice_salt_rob_z1000.png
├── atpv-new-002_hslice_salt_rob_z100.png
├── atpv-new-002_hslice_sp_a_ice_sps_m3.png
├── atpv-new-002_hslice_sp_a_ice_sps_m9.png
├── atpv-new-002_hslice_sp_m_ice_sps_m3.png
├── atpv-new-002_hslice_sp_m_ice_sps_m9.png
├── atpv-new-002_hslice_sp_MLD2_sps_m3.png
├── atpv-new-002_hslice_sp_MLD2_sps_m9.png
├── atpv-new-002_hslice_ssh_rob.png
├── atpv-new-002_hslice_temp_rob_z1000.png
├── atpv-new-002_hslice_temp_rob_z100.png
├── atpv-new-002.html
├── atpv-new-002.json
└── atpv-new-002_vprofile_Kv.png

0 directories, 25 files

Note that I needed to swap out the module load python for conda in the main albedo.yaml. I do not like this particularly, and it would be better if the workflow sub-job defined somehow which modules it needed, or at least on the fesom level.

[pgierz]

"Almost" looks good. I needed to change some of the href paths in the generated HTML file for the graphics to show up. View in browser by opening in Jupyterhub, going to HTML file, right click, and then "Open in new browser tab".

I'll see about getting the paths automatically set correctly

[pgierz]

See PR in TriPyView for better relative paths: FESOM/tripyview#215

[pgierz]

@JanStreffing and @ackerlar: on order for this to work cleanly, one needs to modify the output file naming pattern created when tidying up the work folder after the run. In FESOM-2.5, we were appending a datestamp in the form 0101.01 or something like this.

I have reverted this to maintain the names however they come out of the model. Please check if that makes sense and that it doesn't clash with any other downstream tools.

[JanStreffing]

@JanStreffing and @ackerlar: on order for this to work cleanly, one needs to modify the output file naming pattern created when tidying up the work folder after the run. In FESOM-2.5, we were appending a datestamp in the form 0101.01 or something like this.

I have reverted this to maintain the names however they come out of the model. Please check if that makes sense and that it doesn't clash with any other downstream tools.

Okay from my side.

[pgierz]

Lars also agrees, so I am now just waiting for esm-tests to go through

[mandresm]

Hi @SimonPorr and @pgierz

Massive thanks to @SimonPorr for your work! Also thanks to @pgierz for giving it the final touches!

I have made the review and I only have 2 suggestions for back compatibility (sorry @pgierz I know I said it was okay to make those changes in the names of the output files, but @JanStreffing thinks it is better to not do it for versions 2.1 and 2.5)

[pgierz]

OK, ESM Tests (with real runs):

awiesm3:
    develop:
        awiesm3-develop-levante-TL255L91-CORE2_1d:
            levante:    comp files not found    run failed
        awiesm3-develop-levante-TL255L91-CORE2_1y:
            levante:    comp files not found    run failed
awiesm:
    2.1:
        pi-wiso:
            levante:    compiles        run failed
        pico:
            levante:    compiles        run failed
        pi:
            levante:    compiles        runs
        awiesm-2.1_icebergs:
            levante:    compiles        run failed
        bootstrap:
            levante:    compiles        run failed
    2.1-recom:
        all_awiesm-2.1-recom:
            levante:    compiles        run failed
    2.1-wiso:
        PI_ctrl_awiesm-2.1-wiso:
            levante:    compiles        run failed
awicm3:
    v3.1.1:
        awicm3-v3.1-TCO95L91-CORE2_initial:
            levante:    compiles        runs
    v3.2:
        awicm3-v3.2-TCO95L91-CORE2_initial:
            levante:    compiles        runs
    v3.3.0:
        awicm3-v3.3-TCO95L91-CORE2_initial:
            levante:    compiles        runs
    v3.0:
        awicm3-v3.0-TCO159L91-CORE2_initial:
            levante:    compiles        runs
    frontiers-xios:
        awicm3-frontiers-xios-TCO159L91-CORE2_initial:
            levante:    compiles        run failed
    develop:
        awicm3-v3.4-TCO95L91-CORE2_initial:
            levante:    comp files not found    run failed
fesom:
    1.4:
        fesom1.4-initial-monthly:
            levante:    compiles        runs
    2.1:
        fesom2.1-initial-monthly:
            levante:    compiles        runs
    2.6:
        fesom2.6-initial-monthly:
            levante:    compiles        run failed
    2.0:
        fesom2.0-initial-monthly:
            levante:    compiles        run failed
awicm:
    2.0:
        awicm2-initial-monthly:
            levante:    compiles        run failed
    1.0-recom:
        awicm1_recom_pictrl_initial:
            levante:    compiles        runs
        awicm1_recom_hist_initial:
            levante:    compiles        runs
        awicm1_recom_ssp585:
            levante:    compiles        runs
    CMIP6:
        awicm1-CMIP6-initial-monthly:
            levante:    compiles        runs

For awi-esm3: I do not have repo access
For awi-esm2: Many of those tests have not worked for quite some time now, I am happy that the PI case works. Getting the others back to a "green" state should not block this PR, in my opinion
For awicm3: @JanStreffing could you give an opinion? I would be happy, and I have no idea how up-to-date the frontiers or develop tests are.
For fesom: I'd like to have a look at the what exactly is happening there. It should be working, but again, maybe that is not for this PR.
For awicm+recom: @chrisdane or @christian-stepanek need to make a statement. As with awicm3, I would be happy with those tests. I do not know any users who actually are using awicm2 (unless I have misunderstood something, that should be fixed vegetation in jsbach)

So, from my side, this is complete. I could still add in the transfer to the public modvis.dkrz.de webserver, rerun the tests, and then commit that state as a new "truth".

@mandresm could you look over the code? Sorry, I just reloaded the page and saw your comments. Long story short for naming: OK.

[pgierz]

For FESOM 2.0: I can't be sure, but it looks like the path is getting truncated? I can try to repeat that in a test folder that is not so deeply nested.

286: forrtl: severe (29): file not found, unit 99, file /work/ab0246/a270077/SciComp/Projects/esm-tools-auto-tripyview/esm-tests-runs/run/fesom//fesom2.0-infesom.clock
286: Image              PC                Routine            Line        Source
286: fesom              0000000000581BB8  for__io_return        Unknown  Unknown
286: fesom              00000000005955C2  for_open              Unknown  Unknown
286: fesom              0000000000446271  Unknown               Unknown  Unknown
286: fesom              000000000040E152  Unknown               Unknown  Unknown
286: fesom              000000000040E0E2  Unknown               Unknown  Unknown
286: libc-2.28.so       0000155549BB07E5  __libc_start_main     Unknown  Unknown
286: fesom              000000000040DFEE  Unknown               Unknown  Unknown

[pgierz]

For FESOM-2.6: The model crashes numerically (instantly)

0:  ============================================
  0:  current_nsteps, steps_per_ib_step, icb_outfreq :          48           8           0
  0:  FESOM start iteration before the barrier...
  0:  FESOM start iteration after the barrier...
  0:
  0:  ESC[32m____________________________________________________________ESC[0m
  0:  ESC[7;32m --> FESOM STARTS TIME LOOP                                 ESC[0m
  0:  Updating SSS restoring data for month                1
  0:   --check opt_visc--> Mean Ratio Resol/Rrossby =    4.26141761850143
282:   --> subroutine vert_vel_ale --> found Nan in Wvel after cumulativ summation(...)
282:   mype =         282
282:   node =           1
282:   Wvel(nzmin:nzmax, n)=                     NaN                     NaN                     NaN                     NaN
              NaN                     NaN                     NaN                     NaN                     NaN
   NaN                     NaN                     NaN                     NaN                     NaN                     NaN
                NaN
282:   --> subroutine vert_vel_ale --> found Nan in Wvel after cumulativ summation(...)
282:   mype =         282
282:   node =           2
 73:   --> subroutine vert_vel_ale --> found Nan in Wvel after cumulativ summation(...)
 73:   mype =          73
 73:   node =           1
 73:   Wvel(nzmin:nzmax, n)=                     NaN                     NaN                     NaN                     NaN
              NaN                     NaN                     NaN                     NaN                     NaN
   NaN                     NaN

I'd like to double check if that also happens on another machine to ensure it isn't something completely silly like different NetCDF libraries, but that looks like an error stemming ultimately from a bad namelist setting .

[mandresm]

For awicm3: @JanStreffing could you give an opinion? I would be happy, and I have no idea how up-to-date the frontiers or develop tests are.

I think frontiers could go in some point soon, it's been failing forever, but that's ultimately up to @JanStreffing.

For awicm+recom: @chrisdane or @christian-stepanek need to make a statement.

Except for awicm2.1-recom all the awicm1 tests run successfully. awicm2.1-recom has been failing for a while. That uses a dev version of fesom (branch) from @a270105 and has not been working for a while in the tests (I know it works for @a270105). My approach there has been to keep it there for the purpose of seeing diffs on the GitHub tests to ensure no accidental modifications on our side. Eventually that branch will be merged into FESOM mains and then we can set a new test that actually works.

I do not know any users who actually are using awicm2 (unless I have misunderstood something, that should be fixed vegetation in jsbach)

I think we should remove awicm2 tests.

@pgierz, my only concern are the FESOM tests

[mandresm]

for FESOM 2.6 test, I have opened the issue #1418