Sharing the data in geoparquet format?

[onnimonni]

Hey,

I'm really loving that you have made the whole planet pbf file available for everyone and this is the easiest way to get my hands into the OSM dataset.

I really enjoy working with duckdb and it's very easy to query all sorts of things as guided in https://overturemaps.org.

However the full openstreetmaps data is not included in the overturemaps geoparquet files because they will do all sorts of cleanup. Because of that it's not possible to query for OSM nodes with tags like amenity=sanitary_dump_station which are very useful for people who want to travel with campers.

Here's very nice guide on how duckdb can be used with openstreetmaps data:
https://towardsdatascience.com/how-to-read-osm-data-with-duckdb-ffeb15197390/

The author of the blog post has also released tool called quackosm which can convert the openstreetmaps data to geoparquet.

Would you be interested if I would do a PR into openplanetdata to run the pbf -> parquet conversion command:

$ quackosm planet.osm.pbf

And after it's ready the script would upload the results to your R2 bucket as planet.osm.parquet.

[laurentpellegrino]

Thanks for your PR. The self-hosted runner we are using has only 64GB memory and it seems quackosm requires more for a full planet. I will try running quackosm on my local workstation which has more memory when I have time to confirm.

[laurentpellegrino]

I confirm quackosm is really greedy and requires more than 128GB of memory to convert a full OSM planet to geoparquet.

@onnimonni Do you know any option to reduce memory consumption without altering the result? in the meantime I will check if GDAL can be used as an alternative.

[laurentpellegrino]

I’ve opened a discussion here: kraina-ai/quackosm#239.

The workflow now uses ohsome‑planet instead of quackosm. Processing the full planet PBF peaks at about 46 GB of RAM, and converting it to Geoparquet takes roughly 2 h 30. The resulting file is around 330 GB and is available at http://download.openplanetdata.com/osm/planet/geoparquet/planet-latest.osm.parquet.

@onnimonni Please let me know if this meets your needs; if it does, I'll reference it on openplanetdata.com.