December 12, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The Gluten project has been highly active over the past 7 days with 55 merged PRs and 21 open PRs. The main focus areas include Spark 4.0 compatibility improvements, Velox backend optimizations, and infrastructure enhancements. Daily Velox version updates continue to be a regular occurrence, indicating active upstream integration.

Key Ongoing Projects

• Spark 4.0 Compatibility: Major effort led by @zhouyuan and @baibaichen to address failing unit tests ([VL] Track on Spark-4.0 failed unit tests #11088). Multiple PRs have been merged including SQLQueryTestSuite support ([GLUTEN-11088][VL] Adding SQLQueryTestSuite for Spark-4.0 #11136) and Parquet IO compatibility fixes ([GLUTEN-11088][VL] Fix GlutenParquetIOSuite compatibility issues for Spark 4.0 #11281).
• Velox Backend Optimization: Daily version updates by @GlutenPerfBot continue, with recent improvements in memory management ([GLUTEN-9456][VL] Release Velox memory manager when the reservedBytes become empty #11249) and aggregation node handling ([VL] Correct the conversion from AggregateRel.groupings to AggregationNode.groupingKeys #11264).
• Code Quality Improvements: Extensive cleanup efforts by @xinghuayu007 including IWYU tool integration ([GLUTEN-11287][VL] Use IWYU tool to check code format #11287) and parameter removal ([VL] Remove unused parameters in shuffle reader #11285).
• New Backend Development: Work in progress on Bolt backend (WIP: add bolt backend in gluten #11261) by @WangGuangxin, representing a Velox fork from ByteDance.

Priority Items

• Spark 4.0 Test Failures: [VL] Track on Spark-4.0 failed unit tests #11088 needs continued attention with remaining test suites like ArrowEvalPythonExecSuite and JsonFunctionsValidateSuite still failing.
• Memory Management: [GLUTEN-9456][VL] Release Velox memory manager when the reservedBytes become empty #11249 by @rui-mo addresses critical Velox memory manager lifecycle issues that could impact production stability.
• Compression Optimization: [GLUTEN-11283][VL] Respect spark default zstd compression level #11284 by @wecharyu fixes ZSTD compression level mismatch between Spark and Velox, potentially significant for storage costs.
• Spill Level Issues: Spill Level Exceeds Maximum in HashBuild Operator #10845 reports HashBuild operator spill level exceeding maximum, affecting query execution reliability.

Notable Discussions

• EMR Deployment Guide: Deployment Guide for Spark on EMR #11279 highlights community need for better AWS EMR deployment documentation, with @ammarchalifah seeking guidance on production deployment.
• Library Linking Strategy: Dynamic vs static linking of libstdc++ and libgcc for Gluten libraries #11282 discusses critical production issue with static vs dynamic linking of libstdc++ and libgcc, potentially affecting stability.
• Delta Lake Support: Gluten + Velox fallback when reading Delta Lake (Spark 3.5.3 + Delta 3.3.2) #11254 addresses fallback issues when reading Delta Lake tables, indicating ongoing compatibility challenges.

Emerging Trends

• Multi-Backend Strategy: Growing interest in alternative backends with both Omni (Add a new backend: Omni #10188) and Bolt (WIP: add bolt backend in gluten #11261) proposals, suggesting community desire for more options.
• Production Deployment Focus: Increased discussions around deployment guides, library conflicts, and production stability indicate maturing adoption.
• Performance Optimization: Multiple PRs focused on memory management, compression, and query plan optimization suggest production performance tuning is a priority.

Good First Issues

• [CH] support expression MakeYMInterval #6814: Support MakeYMInterval expression for ClickHouse backend - Good entry point for learning expression implementation
• [CH] support function date_from_unix_date  #4730: Implement date_from_unix_date function for ClickHouse - Simple function addition with clear requirements
• [CH] support split_part function #6807: Add split_part function support for ClickHouse - String manipulation function, well-defined scope
• [CH] support function SparkPartitionID  #6812: Implement SparkPartitionID function for ClickHouse - Useful for partitioning scenarios, straightforward implementation
• [CH] support expression MapZipWith #6815: Support MapZipWith expression for ClickHouse - More advanced but well-documented function with clear specifications

These issues are excellent starting points as they involve implementing specific functions/expressions with clear requirements, allowing new contributors to understand the codebase while making meaningful contributions. Most require basic C++ knowledge and understanding of ClickHouse backend architecture.