Save logs for repeated task executions

[ryanrichholt]

Looking for thoughts on saving logs for all executions of a task. i.e. if a task fails and gets run again, both logs would be saved. This could be accomplished with an encoded timestamp. The benefit would be that it could be determined without checking the disk:

<project>/jetstream/logs/taskA_16e857f4ae5.log
<project>/jetstream/logs/taskA_16e85823fd6.log

Alternatively, the disk could be checked for existing logs, and auto-increment :

<project>/jetstream/logs/taskA.log
<project>/jetstream/logs/taskA_1.log

But, I think the second option is more confusing. Imagine you had a task named: copy_fastq_1, now the logs folder looks like this:

<project>/jetstream/logs/copy_fastq_1.log
<project>/jetstream/logs/copy_fastq_1_1.log
<project>/jetstream/logs/taskA.log
<project>/jetstream/logs/taskA_1.log

[PedalheadPHX]

See the value in this as failed logs are lost on restarts and it would be nice to have around. Question is how do you default to most recent version.