adding path to the further_reading in runscripts

[denizural]

This pull request (must be used together with the correspondig PR in the esm_runscripts esm-tools/esm_runscripts#162)
solves the issue when further_reading contains a relative path. Normally, on the release branch this does not work in a user runscript:

fesom:
    ...
    further_reading: "output/fesom_output_control.yaml"

where fesom_output_control.yaml is inside the output directory under where the runscript resides.

Unfortunately, initially the further_reading is not very robustly implemented in the esm_tools. Check these. These are the all occurences of further_reading in all YAML files

./configs/machines/glogin.yaml
./configs/machines/mistral.yaml
further_reading:
    - batch_system/slurm.yaml

./configs/components/hdmodel/hdmodel.yaml
further_reading:
    - hdmodel/hdmodel.inherit.yaml

./configs/components/mpiom/mpiom.yaml
further_reading:
    - mpiom/mpiom.compiletime_env_changes.yaml

./configs/components/jsbach/jsbach.yaml
further_reading:
    - jsbach/jsbach.inherit.yaml
    - jsbach/jsbach.datasets.yaml

./configs/components/echam/echam.yaml
further_reading:
    - echam/echam.datasets.yaml
    - echam/echam.postprocessing.yaml

./configs/components/pism/pism.yaml
further_reading:
    - pism/pism.domains.yaml
    - pism/pism.compiletime_env_changes.yaml
    - pism/pism.input_and_restart_files.yaml
    - pism/pism.outputs.yaml
    - pism/pism.calendar.yaml
    - pism/pism.recipes.yaml

These are neither absolute nor relative paths. For this reason the functions have strange interfaces:

def look_for_file(model, item, all_config=None) but it is called as look_for_file(path, attach_value, all_config=all_config)

Although I have a more robust idea to solve this, just for the sake of time I implemented an easier solution.

Solution:

Determine the type of the path expression. Is it a path like in the

• config yaml files (eg. echam, pism, mistral, ...)
• relative path (as defined in the user runscript)
• plain path (eg. further_reading: foo.yaml)

It then contructs the correct path and passes to the look_for_file.

A separate change should also be implemented to the esm_runscripts since after resolving the sources now we also need to update the targets. No matter where the further_reading file resides in the home directory, they should always go to

• <EXPDIR>/run_20010101-20011231/scripts/fesom_output_control.yaml
• <EXPDIR>./scripts/fesom_output_control.yaml

I tested these cases

test name	branch	further_reading format	Note	Result
test 1	release	fesom_output.yaml	reference case
test 2	hotfix	fesom_output.yaml	same as test 1	✔️
test 3	hotfix	output/fesom_output.yaml	moved to output	✔️
test 4	hotfix	../fesom_output.yaml	moved one directory up	✔️

run command: esm_runscripts fesom2-ollie-initial-yearly.yaml -e further_reading_test_1 --check

I compared the test 2,3,4 against test_1 (reference) with both diff and diff tree. Eg.

1. diff -qr further_reading_test_1 further_reading_test_4
2. • tree further_reading_test_4 > tree_further_reading_test_4
   • vimdiff tree_further_reading_test_1 tree_further_reading_test_4

The output of the comparison 1 is something like

Only in further_reading_test_1/log: further_reading_test_1_esm_runscripts_20010101-20011231.log
Only in further_reading_test_1/log: further_reading_test_1_fesom.log
Only in further_reading_test_3/log: further_reading_test_3_esm_runscripts_20010101-20011231.log
Only in further_reading_test_3/log: further_reading_test_3_fesom.log
Only in further_reading_test_1/run_20010101-20011231/config/fesom: further_reading_test_1_filelist_20010101-20011231
Only in further_reading_test_3/run_20010101-20011231/config/fesom: further_reading_test_3_filelist_20010101-20011231
Files further_reading_test_1/run_20010101-20011231/config/fesom/namelist.config and further_reading_test_3/run_20010101-20011231/config/fesom/namelist.config differ
Only in further_reading_test_1/run_20010101-20011231/config: further_reading_test_1_filelist_20010101-20011231
Only in further_reading_test_1/run_20010101-20011231/config: further_reading_test_1_finished_config.yaml
Only in further_reading_test_3/run_20010101-20011231/config: further_reading_test_3_filelist_20010101-20011231
Only in further_reading_test_3/run_20010101-20011231/config: further_reading_test_3_finished_config.yaml
Files further_reading_test_1/run_20010101-20011231/scripts/env.sh and further_reading_test_3/run_20010101-20011231/scripts/env.sh differ
Files further_reading_test_1/run_20010101-20011231/scripts/fesom2-ollie-initial-yearly.yaml and further_reading_test_3/run_20010101-20011231/scripts/fesom2-ollie-initial-yearly.yaml differ
Only in further_reading_test_1/run_20010101-20011231/scripts: further_reading_test_1_compute_20010101-20011231.sad
Only in further_reading_test_3/run_20010101-20011231/scripts: further_reading_test_3_compute_20010101-20011231.sad
Files further_reading_test_1/run_20010101-20011231/work/namelist.config and further_reading_test_3/run_20010101-20011231/work/namelist.config differ
Files further_reading_test_1/scripts/fesom2-ollie-initial-yearly.yaml and further_reading_test_3/scripts/fesom2-ollie-initial-yearly.yaml differ
Files further_reading_test_1/.top_of_exp_tree and further_reading_test_3/.top_of_exp_tree differ

Output of the comparison 2 is something like

1c1
< further_reading_test_1
---
> further_reading_test_4
16,17c16,17
< │   ├── further_reading_test_1_esm_runscripts_20010101-20011231.log
< │   └── further_reading_test_1_fesom.log
---
> │   ├── further_reading_test_4_esm_runscripts_20010101-20011231.log
> │   └── further_reading_test_4_fesom.log
33c33
< │   │   │   ├── further_reading_test_1_filelist_20010101-20011231
---
> │   │   │   ├── further_reading_test_4_filelist_20010101-20011231
39,40c39,40
< │   │   ├── further_reading_test_1_filelist_20010101-20011231
< │   │   ├── further_reading_test_1_finished_config.yaml
---
> │   │   ├── further_reading_test_4_filelist_20010101-20011231
> │   │   ├── further_reading_test_4_finished_config.yaml
60c60
< │   │   ├── further_reading_test_1_compute_20010101-20011231.sad
---
> │   │   ├── further_reading_test_4_compute_20010101-20011231.sad

As you can see that the experiment tree structure is completely preserved and there are not differences in the copied files.

[mandresm]

I don't think I will have the time to review the PRs today, but I'll do it over the weekend. I think it is important to test here for example AWICM1 so that we are sure that those further readings in jsbach.yaml are reached. Can you do that?

AWICM3 uses also further readings to load environments so I'll do the testing for that.