Is there a way to do spatial partitioning like we can with RDD using DataFrame API?

[adaros92]

Docs say we need to broadcast the data to get spatial indexing with broadcast index join but this is a problem when the data is too large to fit in one node https://sedona.apache.org/latest/api/sql/Optimizer/?h=broadcast#broadcast-index-join

Any other way to trigger spatial indexing on the join? We can do this with RDD spatial partitioning and it works, but would like to preserve data frame API without converting back and forth from/to RDD.

[alexeyegorov]

I would also be interested in this question but with SQL code. We use Sedona with DBT and rely on H3 grid to improve the performance. Having the ability to create an index like described for RDDs could probably bring an improvement on some big data applicaitons.

[jiayuasu]

@adaros92 @alexeyegorov

Apache Sedona Spatial Join via Spatial SQL uses distributed partitioning and spatial index automatically. Users don't need to specify anything.

[adaros92]

Yeah that's what I thought but I've been using version 1.7 and doing a simple st_intersect between a points dataset and waterbodies dataset. When I look at explain plan I don't see spatial index. Only way is to broadcast one of the datasets, and then I do see the index like in the image below. We are just reading regular parquet with wkb geoms, which we convert with geomfromwkb. In order to get the spatial index and partitioning I need to convert to RDD and then back.

lakes_rdd.spatialPartitioning(GridType.QUADTREE, 200)
waypoints_rdd.spatialPartitioning(lakes_rdd.getPartitioner())

lakes_rdd.buildIndex(IndexType.QUADTREE, True)

[alexeyegorov]

Motivated by this and other discussions, I spent few hours yesterday and found a solution to perform a LEFT JOIN while still using Spatial join of Sedona.
Here is the post in the other discussion thread: #2520 (reply in thread)

@jiayuasu Btw, if I understand this correctly, the spatial index is only built on the broadcasted dataset? It is not possible to build and save an index similar to Postgis GIS index? I saw in some Wherobots presentation that they build an index, but this is not open source.