ROBUSTNESS: Write RDS files atomically

[HenrikBengtsson]

Issue

When writing an RDS file, it might fail half way through, resulting in an incomplete, corrupt file. Similarly, if one process attempts to read a file that is currently being written by another process, the read might occur on an incomplete file.

Suggestion

Always write RDS file atomically. This can be achieved by writing to *.rds.tmp and then rename the file to *.rds after saveRDS() completes. This way the target *.rds is guaranteed to be complete.

By writing to *.rds.tmp instead of tempfile() guarantees that we write to the same drive, avoiding cross-drive moves and copies.

Patch

See PR #316 for a patch. I updated writeRDS() to write atomically and I've added wait = 300, which can optionally be set to wait = 0. The latter is used to replace two remaining saveRDS() uses.

[HenrikBengtsson]

While testing my initial PR, I noticed that some of the Registry related functions were too liberal when scanning for files; if there were non-*.rds files, those functions tried to read also those as RDS files. At times, an *.rds.tmp would be read in my Slurm tests. That means that those functions would also read incomplete RDS files, because there was nothing making sure the files were complete. I guess that's one reason for try(readRDS(...)) being used, e.g.

batchtools/R/syncRegistry.R

Line 31 in 73922a4

x = try(readRDS(fn), silent = TRUE)

I've made a second commit to the PR that limits those functions to look for *.rds files, cf. c18e6f9