GCS support for FdReader used in (downstream) TFDS

[kentslaney]

Following up on google/array_record/issues/120

The following code fails when it shouldn't

import tensorflow_datasets as tfds
ds = tfds.data_source("ref_coco", data_dir="gs://ref_coco", try_gcs=True)
next(iter(ds['train']))

Colab link

The path gets handed over to a Riegeli FdReader (source) and the array_record maintainers have pointed me upstream. I can't find GCS support mentioned in the docs, but given TFDS provides a try_gcs argument, it seems like GCS buckets should be supported somewhere along the line.

[QrczakMK]

Internally at Google Riegeli supports GCS (via riegeli::GcsReader), but this is currently not open sourced.

I will see how feasible that would be.

[carlthome]

What's the reasoning for keeping it closed source? As a heavy user of Google Cloud Dataflow for large datasets processing, I'm surprised to get stuck on TensorFlow Datasets code that stopped working when writing TFRecord files to Google Cloud Storage (GCS), and was hoping that migrating to array_record would alleviate those pain points. Surprised to read that it's not clear that TFDS will get GCS support with array_record.

[seongmoon729]

Hi! Any updates here?
I'm currently using gcsfuse instead, but I hope this library will provide native GCS support, which would be much better.