Using local files as source in get_vpts?

[YohanSassi]

I would like to iteratively load data from Aloft using getRad::get_vpts, for specific radar and time intervals, do some filtering / cleaning and save a local version (in "csv.gz") of the clean data frame. Because I need to go through several radars / years it takes a lot of time to iteratively load the data I want to work with from Aloft (and sometimes fail - see #134). Could it be possible to add the option of using a local directory containing the data previously downloaded (e.g., from Zenodo) as source in the get_vpts function?

This would drastically reduce the processing time and allow to specify radars / time intervals.

[bart1]

Hi @YohanSassi thank you for your remark. The goal of getRad is mostly to open up access to online data. Up to now we have stayed away from managing a local copy. For vpts data is it indeed sometimes possible to create a local copy without the disk usage exploding. If you do a lot of repeated calls you indeed might be better of storing the data locally.

A quick overview of ideas for solutions:

• You could use getRad to store data as rds files in a script and just store them in a logical structure yourself
• If you do frequently do exactly the same call (say combinations of radar and date) you could use memoise, then you create a version of a function that reads the data from a cache if you do exactly the same call. The advantage of this approach being that you can limit the cache size and do not have to think before hand what data you want nor do you have to do file management
• For the time being bioRad::download_vpfiles still work and would help you create a local copy
• Using 'utils::download.file' could also help creating a local copy. For these last two options read_vpts could help reading the files

In short for speed nothing beats a local copy however this is some overhead in management involved. If you can make sure the calls are the same every time memoise is a very nice solution.

@PietrH anything to add?

[PietrH]

I agree it makes more sense to me to store a local copy of the output of get_vpts() and to work from that. It might also be worth splitting up your query in multiple smaller calls.

Memoise/cachem are options. But personally I'd keep it simple and write out the files as rds, feather or simple csv.gz.

[YohanSassi]

Thanks for your answers!

Problem is that if I want to loop to iteratively access data for specific radar/time intervals using get_vpts() and store a local copy of its output to then work on it I end up having the loop crashing because of #134. It is basically to avoid this problem that I am using a local copy of the data (provided by @peterdesmet). But then, tricks are needed to access the data for specific radar / time intervals from the local copy because of the monthly organisation of the data, embedded into yearly folders, and radar names.

As long as the files are organised the same way in local repository and Aloft, then being able to use get_vpts() with source = "local repository name" could be very helpful to still be able to specify the radar and time interval we want to look at. I know that #134 will be resolved at some point so being able to save a local copy should be feasible, still the option of using local copy in the function would help to speed up the processing time and easily select the data of interest with the radar / datetime options.

I am not sure I am really clear, sorry, and I can understand also that you do not want to implement such option.

[bart1]

I think the option to specify a local repository or an url would be really nice, also for experimental datasets. Appending urls or file paths to the correct specification should not be very hard. vroom can read many sources without too much problems. The main challenge I see is that we are currently using a coverage file which locally might not exits. So then we should just assume files that can not be read do not exist.

@PietrH & @peterdesmet for flexibility I would see a few options that would be nice:

• Use a path as a source that gets appended in the aloft style, this would facilitate a local copy directly from aloft
• Use a glue string as a path for sources so people are more flexible in specifying the formatting of file paths
• Have an option to specify urls, as vroom reads from urls the above options could already work for this
• Have an option to read from repositories that have authentication, maybe one can specify a httr2 request that gets appended?

[peterdesmet]

• Providing a path in source (e.g. ~/Downloads/aloft-subset/) seems like the simplest solution that would help @YohanSassi. I don't think we need to support further customization atm.
• The provided path should be a directory with files organized as radar/yyyy/monthly.csv.gz (as in Aloft). That is also how files are organized on Zenodo. So downloading and upzipping e.g. ch.tgz, you'd be able to provide source = "~/Downloads/cz" to read the data locally. Want to combine countries? Then move all the radar directories to one directory.
• I'm not sure how we can work around the coverage file? Build a local list of all files (easier locally than online)?

[bart1]

* I'm not sure how we can work around the coverage file? Build a local list of all files (easier locally than online)?

Maybe your right that recursively listing files in a directory is quick enough locally with most people having an SSD nowadays. It would make the user experience simple and intuitive.

[bart1]

system.time(a<-fs::dir_ls("/home/bart/R",recurse=T, type="file"))
#>    user  system elapsed 
#>   0.082   0.050   0.131
system.time(b<-list.files("/home/bart/R", recursive=T))
#>    user  system elapsed 
#>   0.837   0.146   0.985
length(a)
#> [1] 44874
length(b)
#> [1] 44874

so listing 40000 files is anywhere from 0.15 to 1 seconds. That would be for 10 years of monthly files for 370 radars. So with caching that is perfectly fine and even without it might be good enough

[bart1]

@YohanSassi I now implemented this feature if you want test is you can try out the 135-using-local-files-as-source-in-get_vpts branch