explain what reproduce.sh is in the REMARK documentation

[sbenthall]

The REMARK standard (this repository, README) refers to a reproduce.sh file but this reference is incomplete as there is more assumed in the design of reproduce.sh than is described.

Either what reproduce.sh should be described better -- how to create it, the conditions under which it is used -- or the REMARK standard should include something else besides it.

[sbenthall]

The relevant text of the current standard is:

A reproduce.sh script that
Installs the requirements
Runs and reproduces all the results

"Runs and reproduces all the results" is totally ambiguous.
This either needs to be more firmly nailed down, or it needs to be resolved as part of an editorial process, see #105

[sbenthall]

CDC on call: the person has to define what the results are.

[sbenthall]

Use case: Suppose you want to build off the paper/code in 5 years. Your steps are:

• Get the REMARK, and run the code in its specified environment to see how it works. In principle, you could continue to work in this environment. In practice, you would...
• Try to get it working in a updated environment (new Python, new dependencies)

For step 2, you might want a test suite.
Really reproduce.sh is about having a test suite.

[sbenthall]

So maybe reproduce.sh should include some asserts?

[llorracc]

It occurs to me that we could start on defining reproducible figures by using the existing makefig utility in utilities.py. At a minimum probably it could figure out how to save the fig file without ephemeral metadata like time stamps.