January 02, 2026: Weekly Status Update in Gluten

[GlutenPerfBot]

This weekly update is generated by LLMs. You're welcome to join our Github for in-depth discussions.

Overall Activity Summary

The Gluten community closed 7 PRs and advanced 19 open ones, with heavy focus on Spark 4.1 readiness, Velox daily bumps, and CI/build hygiene. Spark-3.2 deprecation and ARM/neoverse-v2 build fixes are now tracked issues.

Key Ongoing Projects

• Spark 4.1 support – @baibaichen opened the shim layer ([GLUTEN-11346][CORE][VL] Add Spark 4.1 Shim Layer #11347) and compatibility mega-fix ([GLUTEN-11340][CORE][VL][CH] Fix Compatibility issues addressed in Spark 4.1 #11313, now merged) covering geospatial types, commons-collections 4.x, and test-environment alignment.
• Velox version cadence – daily bumps by @jinchengchenghh ([GLUTEN-6887][VL] Daily Update Velox Version (2025_12_30) #11348, [GLUTEN-6887][VL] Daily Update Velox Version (2025_12_29) #11337, [GLUTEN-6887][VL] Daily Update Velox Version (2026_01_02) #11349) keep Gluten pinned to latest IBM/velox commits; all merged within hours.
• New backend “Bolt” – @WangGuangxin posted an early 260 k-line draft (WIP: add bolt backend in gluten #11261) introducing a fourth execution engine.
• SVE performance – @chiranmoyh’s draft ([CH] Improve SparkFloorFunction performance with SVE #11045) shows 2× speed-up for SparkFloorFunction on Graviton3.

Priority Items

• Memory leak in ColumnarPartialProject ([VL] Memory leak for ColumnarPartialProject #11336) – reported by @liujiayi771; blocks deployments using the popular partial-project flag.
• HashJoin hang on TPC-DS 10 TB (When executing complex queries on large datasets, the HashJoin operation hangs #11335) – @xiaojie19852006 sees 3× slowdown vs Vanilla Spark; needs profiler attention.
• CSV/Parquet test failures under Spark 4.0 – umbrella issue [VL] Track on Spark-4.0 failed unit tests #11088 still has ~10 suites unclaimed; help wanted.
• ARM build failure on CentOS 8 ([VL][ARM] Centos8/ARM velox build fails with gcc-11 when run on graviton that supports neoverse-v2 #9858) – GCC-11 lacks neoverse-v2 flag; awaits volunteer to patch Velox detection script.

Notable Discussions

• PME (Parquet Modular Encryption) support (Does Apache Gluten support PME (Parquet Modular Encryption) format? #11338) – user @Dormant7 asks whether encrypted Parquet files can be read; no answer yet; security-savvy contributors welcome.

Emerging Trends

1. Spark-version velocity: 4.0 tests nearly complete, 4.1 shim landed within days of upstream release.
2. Daily Velox pin-and-bump process is now zero-touch via @GlutenPerfBot and @jinchengchenghh.
3. Build/CI hygiene trending: clang-tidy ([GLUTEN-11027][VL][CI] Add clang-tidy to check cpp code #11120), IWYU ([GLUTEN-11287][VL] Use IWYU tool to check code format #11287), Maven wrapper adoption (Use build/mvn wrapper in velox_backend_x86 workflow #11332) to cut environment drift.
4. Performance work shifting to vectorization (SVE) and memory-leak hunting rather than new operators.

Good First Issues

• [VL] deprecated Spark-3.2 unit tests #8960 Remove Spark-3.2 unit-test leftovers – grep and delete stale suites; no C++/Java changes required.
• [VL] document the gap on unsupported case for complicated functions #10275 Document gaps in from_json/to_json support – update docs when functions fall back to Spark; good for doc/QA contributors.
• [VL] Adding intergration tests for S3/HDFS/GCS/ABFS #9184 Add S3/HDFS/GCS/ABFS integration tests – extend existing GHA build checks with mini-cluster/docker tests; infra skills helpful.
• [CH] support expression MakeYMInterval #6814 Implement MakeYMInterval expression for ClickHouse backend – pure C++ registration and unit test; mirrors existing date functions.