Complete runpy package

[alecandido]

Remaining (and non-blocking) tasks for python runner:

• move pineappl to madgraph (for a yadism run, python package should be sufficient)
• merge_grids.sh -> merge 06c56d8
  • this and the former depend on Python interface improvements pineappl#74
• update README Reorganize README and docs #111
• documenting Reorganize README and docs #111
• allow for running in a generic folder (dynamically determine paths.root) Support config files #127
  • it will also allow to install the package (with pip install) and depend on it
  • slightly depends on N3PDF/fktables#4
• redirect global stderr (i.e. at a very high level) 75bc9a2
• non-python dependency list (see following comment) Reorganize README and docs #111
• add rr list subcommand
• add rr install subcommand 092979c

Follows the initial setup of #104

[alecandido]

Just to mention, we have a few non-trivial dependency (still very basic and available) that might deserve a mention somewhere

• curl
• pip
• pkg-config
• libssl-dev (on Debian), we need openssl.pc to compile cargo-c
• wget (through madgraph)
• gfortran (through madgraph)

(I found them because they were not available in the official image of Debian Buster, even though not even python, python3, nor git were available)

Note: consider that in a minimal Debian installing python3-pip we'll have a lot of dependencies, including g++ for example, so many of them might be hidden

[cschwan]

I've just started a run of all processes using the new Python version using low statistics; if there are any missing custom cuts, etc. we'll see that soon.

[alecandido]

Good, I was about to ask you if you could do something similar, or if you could give us a subset of processes for which we should do it... but you anticipated me :D

[cschwan]

Here are the first errors, for ATLAS_2JET_7TEV_R06_1520, but all slices are similarly affected:

INFO: Use Fortran compiler gfortran
INFO: Use c++ compiler g++
INFO: Generate jpeg diagrams
INFO: Generate web pages
Type "launch" to generate events from this process, or see
/home/cschwan/projects/runcards/ATLAS_2JET_7TEV_R06_1520-20211011165525/ATLAS_2JET_7TEV_R06_1520/README
Run "open index.html" to see more information about this process.
quit
quit

patching file SubProcesses/fastjetfortran_madfks_core.cc
patching file SubProcesses/fastjetfortran_madfks_full.cc
patching file SubProcesses/setscales.f
Hunk #1 succeeded at 535 (offset 8 lines).
Hunk #2 succeeded at 593 (offset 8 lines).
patch: **** Can't change to directory "'/home/cschwan/projects/runcards/ATLAS_2JET_7TEV_R06_1520-20211011165525/ATLAS_2JET_7TEV_R06_1520'" : No such file or directory
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-Ij8RpZx_-py3.7/lib/python3.7/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-Ij8RpZx_-py3.7/lib/python3.7/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-Ij8RpZx_-py3.7/lib/python3.7/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-Ij8RpZx_-py3.7/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-Ij8RpZx_-py3.7/lib/python3.7/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/home/cschwan/projects/runcards/runcardsrunner/runm.py", line 32, in subcommand
    main(dataset, theory_card, pdf)
  File "/home/cschwan/projects/runcards/runcardsrunner/runm.py", line 70, in main
    run_dataset(runner)
  File "/home/cschwan/projects/runcards/runcardsrunner/runm.py", line 114, in run_dataset
    runner.run()
  File "/home/cschwan/projects/runcards/runcardsrunner/external/mg5.py", line 123, in run
    f"Patch '{patch}' requested, but does not exist in patches folder"
ValueError: Patch '760.0' requested, but does not exist in patches folder
ATLAS_2JET_7TEV_R06_2025

[alecandido]

Too much copy-pasting, I was using the wrong pattern, fixed in c564804.

This also means that this feature is not tested by TEST_RUN_SH, should we add a test?

[cschwan]

That would be a great regression test. Let's add an invariant mass cut, let's say Mll < 120 GeV, for which we a custom cut already. I'll add one tomorrow if you don't do it in the meantime.

[cschwan]

What I forget to say above: can you please check that both stderr and stdout are redirected into the .log files? Because I had to get the error message from the terminal and there was no launch.log.

[alecandido]

That would be a great regression test. Let's add an invariant mass cut, let's say Mll < 120 GeV, for which we a custom cut already. I'll add one tomorrow if you don't do it in the meantime.

I never implemented cuts myself (of course I can do it looking to the existing ones), but for this one most likely you'll be faster.

What I forget to say above: can you please check that both stderr and stdout are redirected into the .log files? Because I had to get the error message from the terminal and there was no launch.log.

Here I would say that there was no launch.log because it failed in before:
https://github.com/NNPDF/runcards/blob/c5648040abc47ac5790859d480fe30f0edaa2ec7/runcardsrunner/external/mg5.py#L112-L113
but it would have done it for launch itself:
https://github.com/NNPDF/runcards/blob/c5648040abc47ac5790859d480fe30f0edaa2ec7/runcardsrunner/external/mg5.py#L132-L135

Actually what I can do is to catch stderr at a very high level, and redirect to a single file. If the error happens inside something that is already redirected (usually, if not already always, managed by tools.run_subprocess()) I can write to global error.log where to look (e.g. launch.log).

[cschwan]

That would be a great regression test. Let's add an invariant mass cut, let's say Mll < 120 GeV, for which we a custom cut already. I'll add one tomorrow if you don't do it in the meantime.

I never implemented cuts myself (of course I can do it looking to the existing ones), but for this one most likely you'll be faster.

OK, I'll take care of the cuts.

What I forget to say above: can you please check that both stderr and stdout are redirected into the .log files? Because I had to get the error message from the terminal and there was no launch.log.

Here I would say that there was no launch.log because it failed in before:

https://github.com/NNPDF/runcards/blob/c5648040abc47ac5790859d480fe30f0edaa2ec7/runcardsrunner/external/mg5.py#L112-L113

Oh, I see; in that case I suppose the problem is that when this step fails, the whole runner should stop.

but it would have done it for launch itself:

https://github.com/NNPDF/runcards/blob/c5648040abc47ac5790859d480fe30f0edaa2ec7/runcardsrunner/external/mg5.py#L132-L135

Actually what I can do is to catch stderr at a very high level, and redirect to a single file. If the error happens inside something that is already redirected (usually, if not already always, managed by tools.run_subprocess()) I can write to global error.log where to look (e.g. launch.log).

That's a good idea, can you take care of that? It should be copied, though, so that the terminal still shows stderr.

[alecandido]

That's a good idea, can you take care of that? It should be copied, though, so that the terminal still shows stderr.

I'm going to do it today, it should not be so difficult. Essentially I'll make a context manager based on this: https://stackoverflow.com/questions/616645/how-to-duplicate-sys-stdout-to-a-log-file

Such that you can use it more or less like this:

with tee("stdout.log", "stderr.log"):
    do_what_you_want()

catching even errors like

raise WhileRedirectedError("launch.log")

to write a line like Error occurred while stderr was redirected to '/path/to/launch.log'.

[cschwan]

@alecandido another problem with datasets that have a postrun.sh:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-5CFT75gq-py3.9/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-5CFT75gq-py3.9/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-5CFT75gq-py3.9/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-5CFT75gq-py3.9/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/cschwan/.cache/pypoetry/virtualenvs/runcardsrunner-5CFT75gq-py3.9/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/cschwan/runcards/runcardsrunner/runm.py", line 32, in subcommand
    main(dataset, theory_card, pdf)
  File "/home/cschwan/runcards/runcardsrunner/runm.py", line 70, in main
    run_dataset(runner)
  File "/home/cschwan/runcards/runcardsrunner/runm.py", line 122, in run_dataset
    runner.postprocess()
  File "/home/cschwan/runcards/runcardsrunner/external/mg5.py", line 234, in postprocess
    subprocess.run("./postrun.sh", cwd=self.dest)
  File "/usr/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: './postrun.sh'

[alecandido]

Same problem once more: no postrun.sh in TEST_RUN_SH, and it never got properly tested...

I'm immediately going to debug.

[cschwan]

@alecandido thanks!

[alecandido]

Looks like it's not a problem of rr, but actually of some postrun.sh files themselves:

❯ fd postrun.sh | xargs head -n 1 | rg '#/' -B 1
==> nnpdf31_proc/CMS_DY_13TEV_28FB_0015_0060/postrun.sh <==
#/bin/bash
--
==> nnpdf31_proc/CMS_DY_13TEV_28FB_0060_0120/postrun.sh <==
#/bin/bash
--
==> nnpdf31_proc/CMS_DY_13TEV_28FB_0120_0200/postrun.sh <==
#/bin/bash
--
==> nnpdf31_proc/CMS_DY_13TEV_28FB_0200_0510/postrun.sh <==
#/bin/bash
--
==> nnpdf31_proc/CMS_DY_13TEV_28FB_0510_3000/postrun.sh <==
#/bin/bash
--
==> nnpdf31_proc/NNPDF_DY_14TEV_40_PHENO_0040_0120/postrun.sh <==
#/bin/bash
--
==> nnpdf31_proc/NNPDF_DY_14TEV_40_PHENO_0120_3000/postrun.sh <==
#/bin/bash