December 26, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The Gluten community has been highly active over the past 7 days, with 30+ PRs merged or opened. Key themes include Spark 4.0 compatibility improvements, daily Velox version updates, infrastructure enhancements, and the introduction of new backends like Bolt. The project is clearly preparing for a major release with extensive code quality and build system improvements.

Key Ongoing Projects

• Spark 4.0 Compatibility: Major effort led by @zhouyuan with [GLUTEN-11316][VL] Enable JDK-21 for Spark-4.0 #11304 (JDK-21 support) and [GLUTEN-11088][VL] Adding SQLQueryTestSuite for Spark-4.0 #11136 (SQLQueryTestSuite) to ensure full Spark 4.0 compatibility
• Daily Velox Integration: @zhouyuan and @rui-mo continue daily Velox version updates ([GLUTEN-6887][VL] Daily Update Velox Version (2025_12_24) #11327, [GLUTEN-6887][VL] Daily Update Velox Version (2025_12_22) #11323, [GLUTEN-6887][VL] Daily Update Velox Version (2025_12_19) #11311) keeping Gluten synchronized with upstream performance improvements
• New Bolt Backend: @WangGuangxin introduced a new Bolt backend (WIP: add bolt backend in gluten #11261) - a Velox fork from ByteDance, representing a significant expansion of backend options
• Code Quality Tools: @xinghuayu007 added IWYU ([GLUTEN-11287][VL] Use IWYU tool to check code format #11287) and clang-tidy ([GLUTEN-11027][VL][CI] Add clang-tidy to check cpp code #11120) checks to improve C++ code quality
• Flink Ecosystem Expansion: @KevinyhZou implemented filesystem sink support ([GLUTEN-10064][FLINK] Support filesystem sink #11300) for Flink integration

Priority Items

• Memory Management Fix: [GLUTEN-3034][CORE] Fix allocating negative buffer size #3035 by @jackylee-ch addresses critical negative buffer size allocation issue with 14 comments - needs review
• S3 Connection Timeout: [GLUTEN-10950][VL][FOLLOWUP] Fix missing time unit in hadoop-aws time configuration (pre-3.4 versions) #11329 by @xumanbu and [GLUTEN-10950][VL] Add time unit to 'spark.hadoop.fs.s3a.connection.timeout' #10951 by @zml1206 fix time unit requirements for S3 connection timeouts
• Array/Map Null Handling: [GLUTEN-11330][VL] Make PartialProject support array and map with null values #11331 by @jiangjiangtian resolves IllegalStateException when accessing null values in partial project operations
• ClickHouse Version Update: [GLUTEN-10727][CH]Daily Update Clickhouse Version (20250916) #11299 by @zzcclp updates ClickHouse to v25.8.3.66-lts with significant changes (131 additions, 102 deletions)

Notable Discussions

• Support KylinOS? #11333: Community discussion about KylinOS support for domestic Chinese operating systems
• December 19, 2025: Weekly Status Update in Gluten #11315: Weekly status update highlighting production readiness concerns and EMR deployment guides
• [VL][ARM] Centos8/ARM velox build fails with gcc-11 when run on graviton that supports neoverse-v2 #9858: ARM build issues on CentOS8 with gcc-11 on Graviton processors supporting neoverse-v2
• [VL] OOM on TPCDS q67 and q95 #1359: OOM issues on TPCDS q67 and q95 requiring memory configuration optimization

Emerging Trends

• Multi-Engine Strategy: Strong focus on expanding beyond Spark to support Flink and potentially other engines
• Production Hardening: Emphasis on memory management, stability fixes, and CI/CD improvements
• Backend Diversification: Multiple backend options (Velox, ClickHouse, Bolt) indicating vendor-specific optimization needs
• Release Preparation: Extensive cleanup activities (license updates, documentation, build system improvements) suggest upcoming stable release

Good First Issues

• [VL] deprecated Spark-3.2 unit tests #8960: Deprecate Spark-3.2 unit tests - straightforward cleanup task requiring basic testing knowledge
• [VL] document the gap on unsupported case for complicated functions #10275: Document unsupported cases for complex functions - documentation task needing function understanding
• [VL] Adding intergration tests for S3/HDFS/GCS/ABFS #9184: Add integration tests for S3/HDFS/GCS/ABFS - testing task with clear requirements and existing build checks as reference