Support for Parquet GEOMETRY data type

[paleolimbot]

Feature description

Hi! The Parquet spec now contains GEOMETRY with basic CRS and bbox support 🙂 . The main advantage is that there are now row group-level bounding boxes in files written naively (e.g., by pyarrow) for those who don't know about or whose tools don't support the bbox covering or non-WKB encodings. It would be great to be able to read and/or write that from OGR at some point!

Additional context

The PR for adding this into Parquet C++ was recently merged ( apache/arrow#45459 ), and there are test files available in the parquet-testing repo ( https://github.com/apache/parquet-testing/tree/master/data/geospatial ) and for every item in the (freshly revamped) geoarrow-data repo ( https://github.com/geoarrow/geoarrow-data ). This should be in Arrow 21 (~late June ish), and there's an example of writing an arbitrary GeoPandas data frame + examining statistics here: https://gist.github.com/paleolimbot/1fc8e3de99d11d968e7ac01e0298c297

I imagine the DuckDB mechanism to read GeoParquet will work with little modification when DuckDB supports this...Arrow C++ will hopefully not be too onerous since the concepts in theory align with GeoParquet. The Arrow reader should give you geoarrow.wkb output (with CRS) if you set ArrowReaderProperties::arrow_extensions_enabled(true) AND somebody has registered the GeoArrow extension type, the column will be read as Binary WKB otherwise and the logical type annotation + statistics are still available from the Parquet schema and Column metadata, respectively. There's talk of having arrow extensions enabled by default and to have geoarrow.wkb added as a "canonical extension type" (which would mean it's registered all the time), although I don't know if those will be resolved before Arrow 21 is released.

cc @cholmes, who put me up to this!

[rouault]

@paleolimbot Something that is probably missing into libarrow/libparquet is a way to do spatial filtering using the GeoStatistics in the arrow::Dataset mode where the user doesn't have direct access to Parquet files and row groups. For single file reading, the OGR Parquet driver creates a batch reader that selects row groups that intersects the GeoStatistics BBOX. For arrow:Dataset / multi-Parquet file reading, with the covering bbox column of GeoParquet 1.1, the OGR Parquet driver can build an expression using it and pass it to the ScannerBuilder::Filter(), but with only GeoStatistics, I don't see you could do that currently.

[paleolimbot]

Yes, I think that is not possible with the current level of support...we'd need the type to exist in Arrow C++ as a canonical extension type and we'd need to add a compute function like intersects_box and we'd need to add the row group pruning logic. That would be an Arrow 22 project but it might be deemed as out of scope.

It's possible to expand a Dataset into its Fragments, which give you access to the file metadata (which would let you pick row groups and assemble a potentially smaller dataset):

https://github.com/apache/arrow/blob/9bd4cc8dbb8cdffa1003f8ffd9cf1a3bfa846f08/cpp/src/arrow/dataset/file_parquet.h#L158

...however, that's a drastic expansion in logic and it's totally understandable to not want to take that on.