Notes and stuff

[necrolyte2]

Test Pipeline Benchmarks

https://github.com/VDBWRAIR/bioframework/blob/necrolyte2-notes/pipeline-example.script.pipes.sh
https://github.com/VDBWRAIR/bioframework/blob/necrolyte2-notes/pipeline-example.script.sh

fastq filesize	Pipe	No-Pipe	speedup
25m	11.52s	13s	1.128
250m	1m 29s	2m 6.03s	1.416
2.5G	14m 41.52s	20m 43.28s	1.409

[averagehat]

I was wrong doit can depend on tasks, using the 'tasks_depends' key, see example

[necrolyte2]

This is probably completely unrelated, but I did start this download so we had a massive dataset to test...

PacBio dataset stuff:
https://github.com/PacificBiosciences/Bioinformatics-Training/wiki/Large-Genome-Assembly-with-PacBio-Long-Reads
https://github.com/PacificBiosciences/DevNet/wiki/H_sapiens_54x_release
http://datasets.pacb.com/2014/Human54x/fastq.html

Download all pacbio dataset

wget http://datasets.pacb.com/2014/Human54x/fastq.html
awk -F'="' '/a href/ {split($2,a,"\""); print a[1]}' fastq.html | grep 'datasets' | xargs -L 1 wget

[necrolyte2]

Both Input and Output parameters already support named pipes
http://common-workflow-language.github.io/draft-3/Workflow.html#InputParameter

streamable boolean False

Currently only applies if type is File. A value of true indicates that the file is read or written sequentially without seeking. An implementation may use this flag to indicate whether it is valid to stream file contents using a named pipe. Default: false.

[necrolyte2]

This looks super promising http://pyjip.readthedocs.org/