Gluten Release 1.6.0

[zhztheplayer]

The thread is for tracking release 1.6.0.

The release will be cut by #11592, which is expected to merge on Feb 12.

If there's any unmerged PR that requires to be included in the release, please leave a comment here.

Important Note:

1. The release 1.6.0 will support Spark versions 3.3 / 3.4 / 3.5 / 4.0 (preview). Support for Spark 3.2 is deprecated in this release.
2. The release 1.6.0 will include a preview support for Apache 4.0. The preview support will only work with vanilla Spark 4.0, without the support for Celeborn, Uniffle, Paimon, Iceberg, Delta, Hudi, Kafka, etc., since these feature haven't been reliably tested with Gluten 1.6.0 yet. We will add the supports for Spark 4.0 in a future release of Apache Gluten once it's been tested and validated.

[PHILO-HE]

I feel the following PR should be included in 1.6 to ensure thorough removal for Spark 3.2.

• [CORE] Remove legacy Spark 3.2 compatibility code #11495

BTW, I note branch 1.6 has been created, but the code seems not frozen actually. Is it planned that way?

[zhztheplayer]

BTW, I note branch 1.6 has been created, but the code seems not frozen actually. Is it planned that way?

Nah, I just pushed it using a random commit. It's to be updated while 1.6 is cut, I'll take care of that branch.

[zhztheplayer]

@PHILO-HE @QCLyu

What do you think about the impact if this patch is not in 1.6? I assume 3.2 support is not compelete anyway and #11495 still haven't removed all the 3.2 support code. If it's a long process we may kick off the releasing process alongside.

[PHILO-HE]

@zhztheplayer, please go ahead. Maybe, we can back port it to 1.6 if timely or just keep it in 1.7.

[QCLyu]

Thanks @PHILO-HE and @zhztheplayer I'll try to get it out as soon as possible.

[FelixYBW]

Thank you, @zhztheplayer for your effort!

[zhztheplayer]

Main branch is cut for 1.7 release. Branch branch-1.6 is open at https://github.com/apache/incubator-gluten/tree/branch-1.6.

[Surbhi-Vijay]

@zhztheplayer is the Gluten-1.6.0 release finalized? Are there any known issues or pending work items before moving to 1.6.0 version?

[zhztheplayer]

@Surbhi-Vijay we don't have blockers now. I am working on finalizing the release.

[zhztheplayer]

cc @apache/gluten-committers

The RC0 vote is open at https://lists.apache.org/thread/vh1yyng3hnfw05lct4qb5w34s2w77g2d. Please take some time to review, verify, and vote if you are willing. Thanks for helping with the release.