Full dataset takes up more storage than Google drive can store

[nabilLearns]

The full set of images provided by the NIH Clinical Center is 45.6GB, of which Google Drive only has space for 15GB. Perhaps I can make do with a smaller dataset, but this will likely result in poor classification accuracy.

TODO:

• find a way to work with full dataset

Perhaps I can compress/resize images to a smaller size before uploading them to drive. This way, there is a tradeoff between number of images I can store and the quality of images, but I'm not sure how much I need to worry about losing image quality by resizing.

[nabilLearns]

One option is to use Amazon S3 for image-storage. However, managing the image-data on S3 is only free for 5GB of storage (for 12-months). One possible approach that I can take to minimize cost is to:

1. encode images into bytes,
2. compress the byte-representations with zarr, and then
3. upload the compressed representations of images to S3 instead of the original images.

I am not sure how much memory I can expect to save with this approach. Perhaps this is something to investigate for the future.