SnakeMake integration

[celefthe]

Provide an example use case (+ example workflow script) for using drim2p with SnakeMake.

I think this is a bit more important in the preliminary steps (i.e. before ROI drawing) since those are a bit more computationally intensive. Signal extraction from ROIs should be pretty light, but might as well document it for completeness

[olivierdelree]

As mentioned yesterday, it would be nice to a better organisation of the various stages of preprocessing.

Unfortunately, we run into a problem related to SnakeMake on that. It'd be nice for portability to have a single HDF5 file per chunk that includes the RAW data and any preprocessing we've done + QA down the line. However, SnakeMake essentially relies on workflow steps to have a file as input and another as output. We should be able to get around that using flag files but it does reduce readability of the workflow a bit.

So there are three main options:

1. Generate separate intermediate files and only bring them together at the end of the preprocessing (knowing that we'll need to transfer things around for the ROI drawing in this intermediate stage).
2. Generate separate intermediate files, bring them together before ROI drawing, and then merge new ones at the very end again (and we might still need flag files for the gathering steps).
3. Use flag files exclusively and append to the same files all throughout.

Of those, 1. seems the easiest to work with SnakeMake but does not play nicely with Eddie, 2. adds some unnecessary complexity to the code (end user would most likely not notice much of a difference), and 3. adds a bit of boilerplate to the workflow files.

I think 3. is the nicer option as we'll be writing the workflow(s) once and then run it(them) as-is many times, but I'm fine with either.

[celefthe]

Yes I agree, let's go with option 3 and use flag files exclusively 👍 1 and 2 just sound like a bit of a pain.

[olivierdelree]

Very artificial examples for reference:

• A workflow that uses intermediate files:

rule all:
    input:
        expand("file{index}_finished.md", index=list(range(1, 4)))


rule z:
    output:
        "file{index}.txt"
    shell:
        "echo lorem ipsum > {output}"


rule a:
    input:
        "file{index}.txt"
    output:
        "file{index,[0-9]+}.md"
    shell:
        "cat {input} > {output}"


rule b:
    input:
        "file{index}.md"
    output:
        "file{index}_working.md"
    shell:
        "echo doing work >> {output}"


rule c:
    input:
        "file{index}_working.md"
    output:
        "file{index}_finished.md"
    shell:
        "echo finished >> {output}"

• A workflow that uses flag files:

rule all:
    input:
        expand("file{index}.md.c.done", index=list(range(1, 4)))


rule z:
    output:
        "file{index}.txt"
    shell:
        "echo lorem ipsum > {output}"


rule a:
    input:
        "file{index}.txt"
    output:
        touch("file{index}.md.a.done")
    shell:
        "cat {input} > file{wildcards.index}.md"


rule b:
    input:
        rules.a.output
    output:
        touch("file{index}.md.b.done")
    shell:
        "echo doing work >> file{wildcards.index}.md"


rule c:
    input:
        rules.b.output
    output:
        touch("file{index}.md.c.done")
    shell:
        "echo finished >> file{wildcards.index}.md"