Is there still a use for drake in "hasty" mode?

[wlandau]

I came across your article recently, and I was delighted to see more work on DAG-based data processing pipelines in R. I am sorry drake's functionality was inadequate at the time. Working to reduce overhead is a long term effort, and our latest attempt is a new "hasty" mode (PR, documentation). It sacrifices some of drake's core reproducibility features, but it makes data processing much faster.

cache <- storr::storr_environment() # for slightly faster preprocessing
make(plan, cache = cache, parallelism = "hasty", jobs = 8)

[schnorr]

Hey @guilhermealles, can you give another try with drake using the so-called "hasty" mode? You can use the drake-integration branch as is. Let me know if you need large traces.

[wlandau]

Thanks! Please let me know if there is anything I can do to help.

I forgot to mention that parallelism and load balancing in hasty mode are controlled by the clustermq package. So to run locally, you would call options(clustermq.scheduler = "multicore") before make(parallelism = "hasty").

[wlandau]

Update: I am planning major improvements to cache speed: ropensci/drake#971 (comment). If all goes well, you will be able to tell drake to use write_fst() and read_fst() and not have to go through file_out() or file_in(). I will keep you posted.