[GLUTEN-11559][VL] Add incremental C++ build script for fast development iteration by baibaichen · Pull Request #11595 · apache/gluten

baibaichen · 2026-02-10T08:49:15Z

What changes were proposed in this pull request?

Gluten's incremental C++ builds take ~3 minutes even for no-op builds (make-based workflow), making the AI-driven edit → compile → test → fix loop impractically slow. With this new dev/inc-build-cpp.sh script using Ninja and smart vcpkg timestamp preservation, incremental builds now complete in under 30 seconds and zero-change builds in 0.27 seconds.

Performance Improvements:

Switch from Make to Ninja: 4m53s → 0.1s no-op builds (1400x faster)
Skip redundant cmake configure steps (let Ninja handle regeneration)
Default skip vcpkg check (use --update_vcpkg when needed)
Preserve vcpkg timestamps to avoid unnecessary relinking
Result: 3m14s → 0.27s total no-op time

Features:

Auto-detects BUILD_TYPE from velox _build/debug or _build/release directory
Auto-detects SCALA_VERSION from backends-velox/target/scala-X.XX directory
Auto-extracts vcpkg options (BUILD_TESTS, ENABLE_S3/GCS/ABFS) from CMakeCache.txt
Validates CMakeCache.txt exists (required for incremental build)
Self-contained: no dependency on build-helper-functions.sh for easy maintenance

Usage:

./dev/builddep-veloxbe-inc.sh [--build_type=Debug|Release] [--scala_version=2.13] [--update_vcpkg]

Why are the changes needed?

AI-assisted development requires fast iteration cycles. The existing dev/package-vcpkg.sh is designed for full builds and includes overhead from Velox source reset, OS dependency setup, and Maven builds. This script focuses purely on incremental C++ compilation for the edit → compile → test → fix loop.

Does this PR introduce any user-facing change?

No, this is a developer tool for faster C++ incremental builds.

How was this patch tested?

Syntax validation: bash -n dev/inc-build-cpp.sh
No-op build performance verified: 0.27s (down from 3m14s)
Incremental build after single-file change: ~25s
All lines ≤ 100 characters for code readability

Related Issue

Fixes #11559 (PR 2 of 2)
Previous PR: #11560

🤖 AI-Assisted Development: This PR was created with Claude Code to optimize the AI-driven development workflow.

baibaichen · 2026-02-10T10:23:10Z

@PHILO-HE @marin-ma

PHILO-HE

Thanks for proposing this. Some comments. Please check if they make sense.

PHILO-HE · 2026-02-10T15:57:42Z

dev/builddep-veloxbe-inc.sh

+for arg in "$@"; do
+    case $arg in
+        --build_type=*)    BUILD_TYPE="${arg#*=}" ;;
+        --scala_version=*) SCALA_VERSION="${arg#*=}" ;;


Given that this script is for supporting incremental build for C++ code. It seems strange to include scala version as build option.

scala_version is not for C++ compilation. It is used to determine the target directory when copying native libraries, since the output path differs between Scala 2.12 and 2.13. We assume 2.12 and 2.13 outputs are mutually exclusive, and a full build has already been done, so we can auto-detect it by checking whether scala-2.12 or scala-2.13 exists under the target directory.

PHILO-HE · 2026-02-10T16:01:53Z

dev/builddep-veloxbe-inc.sh

+fi
+
+TARGET_DIR="$GLUTEN_DIR/backends-velox/target/scala-${SCALA_VERSION}"
+TARGET_DIR="$TARGET_DIR/classes/$(platform)/$(arch)"


It seems in the maven build phase, libgluten.so and libvelox.so will be copied to the target directory according to the code below. If so, it may be unnecessary to do the copy here.

https://github.com/apache/incubator-gluten/blob/ae2fc5b5ebeb5bb2b0d60a3c2588f4274597a29c/backends-velox/pom.xml#L229

This script targets the C++-only incremental iteration workflow: we build C++ and immediately place the artifacts where Maven/Scala expects them. If Scala code changes, a subsequent mvn run can skip steps due to timestamp behavior.

PHILO-HE · 2026-02-10T16:46:44Z

dev/inc-build-cpp.sh

+
+# Step 2: Build Velox
+step 2 "Building Velox (incremental)"
+export simdjson_SOURCE=AUTO Arrow_SOURCE=AUTO


This setting may be unnecessary now. Could you confirm it in local build?

PHILO-HE · 2026-02-10T17:08:21Z

dev/inc-build-cpp.sh

+
+    source "$GLUTEN_DIR/dev/vcpkg/env.sh" $BUILD_OPTIONS
+
+    # Restore timestamps for unchanged files


Nit: suggest making it more readable.

** unchanged binary files.

PHILO-HE · 2026-02-10T17:24:21Z

dev/builddep-veloxbe-inc.sh

+BUILD_OPTIONS="$BUILD_OPTIONS --enable_gcs=$(extract_opt ENABLE_GCS)"
+BUILD_OPTIONS="$BUILD_OPTIONS --enable_abfs=$(extract_opt ENABLE_ABFS OFF)"
+
+if [ "$UPDATE_VCPKG" = "ON" ]; then


RUN_VCPKG_INSTALL seems more clear to developers?

I use UPDATE_VCPKG because if vcpkg.json changes, we need to refresh/update vcpkg dependencies, and that can happen in practice. RUN_VCPKG_INSTALL sounds like we always just run an install, which is less accurate.

PHILO-HE · 2026-02-10T17:25:22Z

dev/builddep-veloxbe-inc.sh

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Incremental C++ build for Gluten + Velox (no source reset, no Maven, no OS setup)


Could you add a comment for clarification?

Only applicable when vcpkg is enabled for static linking.

FelixYBW · 2026-02-11T07:25:50Z

Is ./dev/inc-build-cpp.sh going to relace builddep-veloxbe.sh? If so we may overwrite builddep-veloxbe.sh. If we need both, we may change the name to ./dev/builddep-veloxbe-inc.sh to align with existing one.

Add dev/inc-build-cpp.sh for fast incremental C++ builds with auto-detection: Performance improvements: - Switch from Make to Ninja: 4m53s -> 0.1s no-op (1400x faster) - Skip redundant cmake configure steps (use 'cmake --build' directly) - Default skip vcpkg check (use --update_vcpkg when needed) - Preserve vcpkg timestamps to avoid unnecessary relinking - Result: 3m14s -> 0.27s total no-op time Auto-detection: - BUILD_TYPE from velox _build/debug or _build/release directory - SCALA_VERSION from backends-velox/target/scala-X.XX directory - Vcpkg options (BUILD_TESTS, ENABLE_S3/GCS/ABFS) from CMakeCache.txt - Validate CMakeCache.txt exists (required for incremental build) Helper functions extracted to build-helper-functions.sh: - initialize_num_threads, get_platform, get_maven_arch - log_build_step, validate_directory - setup_vcpkg_environment, copy_gluten_libraries Usage: ./dev/inc-build-cpp.sh [--build_type=Debug|Release] [--scala_version=2.13] [--update_vcpkg]

--update_vcpkg implies dependencies have changed, so preserving timestamps to avoid Ninja relinking was wrong — it would hide real library updates and produce incorrect incremental builds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…build-cpp.sh These variables default to AUTO in Velox's CMakeLists.txt and are only read during cmake configure, not during cmake --build. The export had no effect on incremental builds.

Align naming with existing builddep-veloxbe.sh convention.

PHILO-HE

Looks good. Thanks!

github-actions bot added the BUILD label Feb 10, 2026

baibaichen force-pushed the fix/incremental-build-script branch from d52add0 to 1041287 Compare February 10, 2026 11:11

PHILO-HE reviewed Feb 10, 2026

View reviewed changes

baibaichen and others added 5 commits February 25, 2026 05:55

Remove unnecessary simdjson_SOURCE/Arrow_SOURCE env exports from inc-…

76dd73e

…build-cpp.sh These variables default to AUTO in Velox's CMakeLists.txt and are only read during cmake configure, not during cmake --build. The export had no effect on incremental builds.

Rename inc-build-cpp.sh to builddep-veloxbe-inc.sh

7a02e48

Align naming with existing builddep-veloxbe.sh convention.

Update comment: clarify vibe coding purpose and fix usage path

b66f9f3

baibaichen force-pushed the fix/incremental-build-script branch from 1041287 to b66f9f3 Compare February 25, 2026 05:56

baibaichen added the AI AI Related issues and PR label Feb 26, 2026

PHILO-HE approved these changes Feb 26, 2026

View reviewed changes

baibaichen merged commit 9d34339 into apache:main Feb 26, 2026
60 of 61 checks passed

baibaichen deleted the fix/incremental-build-script branch February 26, 2026 03:28


		source "$GLUTEN_DIR/dev/vcpkg/env.sh" $BUILD_OPTIONS

		# Restore timestamps for unchanged files

Conversation

baibaichen commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Related Issue

Uh oh!

baibaichen commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PHILO-HE left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

baibaichen Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FelixYBW commented Feb 11, 2026

Uh oh!

PHILO-HE left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

baibaichen commented Feb 10, 2026 •

edited

Loading

baibaichen commented Feb 10, 2026 •

edited

Loading

baibaichen Feb 25, 2026 •

edited

Loading