Support update using UpdateMode::RewriteColumns

[fangbo]

This PR #79 implements UPDATE using MOR. The update mode is delete-insert. The updated rows are deleted and new rows are inserted.

While the PR lance-format/lance#4715 support fragment-level update columns using UpdateMode::RewriteColumns. The updated columns are written to new data files.

Each approach has its strengths: delete-insert works well when only a few rows need to be changed like:

UPDATE table_x SET col_a = xx, col_b = xx, col_c = xx WHERE id=xx

Whereas UpdateMode::RewriteColumns is better suited for bulk updates of a single column like:

UPDATE table_x SET col_a = xx

So, I plan to support two update modes in Spark. We can use a parameter spark.sql.catalog.lance.update_rewrite_columns=true/false to specify update mode. The default value is true means that using UpdateMode::RewriteColumns to do update. When the value is false, using MOR to do update.

@jackye1995 @jiaoew1991 @majin1102 What do you think about this idea ?

[majin1102]

+1 for exposing the option of rewriting columns.

We can use a parameter spark.sql.catalog.lance.update_rewrite_columns=true/false to specify update mode

Wondering if we could make it work with MERGE INTO. Then this might be spark.sql.catalog.lance.rewrite_columns. What do you think @fangbo

[fangbo]

Wondering if we could make it work with MERGE INTO

Spark's MERGE INTO uses DeltaWriter's delete, update, insert . I think merge into can use this feature.

[zhangyue19921010]

+1 for exposing the option of rewriting columns.

We can use a parameter spark.sql.catalog.lance.update_rewrite_columns=true/false to specify update mode

Wondering if we could make it work with MERGE INTO. Then this might be spark.sql.catalog.lance.rewrite_columns. What do you think @fangbo

+1 Actually I am working on Spark Merge into command for lance. and will take this optimization into account :)