December 19, 2025: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The Gluten community merged 25 PRs and opened 9 new ones over the past 7 days. Major themes include Spark 4.0 compatibility fixes, daily Velox version bumps, infrastructure clean-ups, and new Flink connector work. Contributors are actively polishing the code base ahead of the next release.

Key Ongoing Projects

• Spark 4.0 compatibility – @baibaichen, @zhztheplayer and @zhouyuan continue to burn down failing UTs ([VL] Track on Spark-4.0 failed unit tests #11088); recent fixes cover Parquet IO, JSON functions, Arrow Python and Delta update commands
• Daily Velox integration – @GlutenPerfBot lands fresh Velox commits every day ([VL] Rebase with Meta/Velox  #6887); this keeps Gluten in sync with upstream performance and function improvements
• Flink ecosystem – @KevinyhZou added Kafka source support ([VL][FLINK] Support kafka connector for data source #9553, [GLUTEN-9553][FLINK] Support kafka connector for data source #11312) and filesystem sink ([FLINK] Support FileSystem Sink for nexmark q10 and q13 #10064, [GLUTEN-10064][FLINK] Support filesystem sink #11300), expanding Gluten beyond Spark
• Code-quality & build – @xinghuayu007 introduced IWYU ([GLUTEN-11287][VL] Use IWYU tool to check code format #11287) and clang-tidy ([GLUTEN-11027][VL][CI] Add clang-tidy to check cpp code #11120) checks; @PHILO-HE cleaned legacy scripts ([MINOR] Fix script name typos for TPC-H/DS data generation #11305) and directory names ([MINOR] Refactor directory/profile names to velox-backend/clickhouse-backend #10219)
• New backends – @WangGuangxin posted an early "Bolt" backend PoC (WIP: add bolt backend in gluten #11261), a Velox fork from ByteDance

Priority Items

• Memory manager stability – [GLUTEN-9456][VL] Release Velox memory manager when the reservedBytes become empty #11249 by @rui-mo (merged) fixes task-level race between Velox destructor and async I/O; critical for production
• Z-standard compression – [GLUTEN-11283][VL] Respect spark default zstd compression level #11284 by @wecharyu (merged) aligns Velox level with Spark default (3) to avoid bigger Parquet files
• Spark 4.0 test failures – [VL] Track on Spark-4.0 failed unit tests #11088 still blocks release; suites ArrowEvalPython, JsonFunctions, CSV, Hive need owners
• GPU build broken – [VL] GPU build failure due to recent CUDF upgrade #11302 (closed) required gcc-14 + cuda-toolkit-13.1; monitor follow-up PR [GLUTEN-11302][VL] Fix gpu build by bumping to cuda-13.1 #11275 by @zhouyuan
• Parquet metadata validation – disabled by default ([VL] Disable Parquet metadata validation by default due to performance regression #11233, [VL] Disable parquet file metadata validation by default #11307) until performance regression is solved

Notable Discussions

• December 12, 2025: Weekly Status Update in Gluten #11290 (Weekly status) – community call for EMR deployment guide and help with static/dynamic linking of libstdc++
• Deployment Guide for Spark on EMR #11279 – @ammarchalifah asks for AWS EMR best-practice documentation; good place to contribute docs
• Dynamic vs static linking of libstdc++ and libgcc for Gluten libraries #11282 – production alert on linking strategy; maintainers weighing static vs dynamic libgcc/libstdc++

Emerging Trends

1. Multi-engine support: Flink Kafka/filesystem sinks show Gluten positioning as a universal native accelerator
2. Release readiness: flurry of minor clean-ups (TPP.txt removal, script typos, dead config flags) signals prep for stable branch
3. Backend diversification: Omni and Bolt experiments indicate demand for vendor-specific optimizations
4. Memory & stability focus: recent fixes for OOM-GC interaction, broadcast spill, and memory-manager teardown highlight production hardening

Good First Issues

• [CH] support expression MakeYMInterval #6814 – Implement MakeYMInterval expression for ClickHouse; straightforward function mapping
• [CH] support function date_from_unix_date  #4730 – Add date_from_unix_date function for ClickHouse; already prototyped in [GLUTEN-4730][CH] feat: Support date_from_unix_date function: Add ClickHouse backend imp… #10026 by @soupam05—needs review
• [CH] support split_part function #6807 – Support split_part function for ClickHouse; string manipulation with clear spec
• [CH] support function SparkPartitionID  #6812 – Add SparkPartitionID function for ClickHouse; useful for partition-aware queries
• [CH] support expression MapZipWith #6815 – Support MapZipWith expression for ClickHouse; slightly more advanced but well-documented

All issues above are self-contained, require basic C++ and ClickHouse knowledge, and come with existing examples in the codebase—perfect for first-time contributors to learn Gluten's function registration flow.