how to deal with large number of files to download

[floswald]

hi there,

I have this command for the downloader script

./dv_downloader.sh https://dataverse.harvard.edu doi:10.7910/DVN/GQBPPO --apikey=$JPE_DV

but it takes a long time. there are 1399 files - the majority are quite small, but establishing the connection for each of them takes its toll. Is this the way you suggest to download in such a case? thanks

[pdurbin]

Hmm, I guess the alternative is to have the Dataverse server create a zip on the fly, either through the UI (screenshot below) or the equivalent API. The problem is that creating the zip is computationally expensive. That's why we allow admins to configure a size limit for how big these created-on-the-fly zips can be. https://doi.org/10.7910/DVN/GQBPPO happens to be under that limit so I guess you could try it. You could try seeing which is faster, I mean.

It looks like you aren't the author of the dataset but perhaps we could encourage them to have fewer files in the dataset by zipping up some of them. We have a nifty zip previewer/downloader that can download individual files from zips.

p.s. @qqmyers is the author of the script and @landreev probably has opinions on this as well.

[floswald]

I really like the idea that this goes directly to S3 if available. I was cooking up a translation of what you did as a Julia application, which my windows user can execute (no WSL dependency). It's quite easy to parallelize this Julia as async jobs using threads. What's your gut reaction to that? One would have to be careful not to overload the server with too many concurrent calls but I was hoping that this being S3, there would be a lot of bandwidth on the server side. Curious to hear your thoughts!

[qqmyers]

Fwiw: Harvard currently rate limits to 400 calls in 5 minutes. Downloading a given file means contacting the Dataverse server to get the signed url to the s3 store. Parallelization the s3 downloads could help if the bottleneck is in the s3 transfers rather than dealing with the rate limits and any delays in Dataverse sending the s3 urls. We don't currently have a way to request a whole list of s3 urls for multiple files at once - that would probably be a useful add.

[floswald]

Ok very helpful thanks.

[pdurbin]

We don't currently have a way to request a whole list of s3 urls for multiple files at once - that would probably be a useful add.

Related (thank for opening this, @qqmyers):

• Feature Request:Bulk download support IQSS/dataverse#11587