[GLUTEN-11658][CORE] Restore scala.recompile.mode default to 'all' and introduce fast-build profile#11659
Merged
baibaichen merged 2 commits intomainfrom Feb 27, 2026
Merged
Conversation
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
GLUTEN-11658-restore-scala-recompile-mode
branch
from
c92398b to
02b3461
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
GLUTEN-11658-restore-scala-recompile-mode
branch
from
02b3461 to
b9e300e
Compare
|
Run Gluten Clickhouse CI on x86 |
baibaichen
force-pushed
the
GLUTEN-11658-restore-scala-recompile-mode
branch
from
b9e300e to
addef18
Compare
|
Run Gluten Clickhouse CI on x86 |
|
Run Gluten Clickhouse CI on x86 |
…d introduce fast-build profile - Restore scala.recompile.mode default from 'incremental' back to 'all' to fix CI/clean build failures caused by Zinc's incremental compiler not tracking transitive Spark dependencies correctly - Rename the 'bloop' Maven profile (added in #11645) to 'fast-build' and add scala.recompile.mode=incremental inside it; used by bloop-setup.sh and run-scala-test.sh where Zinc analysis is warm and reliable - Simplify run-scala-test.sh: replace 5 redundant -D skip flags with -Pfast-build profile activation - Add pathing JAR support in run-scala-test.sh to avoid OS command-line length limits when classpath is very long Closes #11658 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Problem: scala-maven-plugin 4.9.2 changed behavior for Scala >= 2.12 builds. When maven.compiler.target is set (e.g. "1.8"), it now passes the value to computeBytecodeVersionOptions() [1] which injects '-release 1.8' into scalac args. The scalac '-release' flag is only supported on Java 9+, so building under JDK 8 with recompileMode=all (restored by the parent commit) fails with: scalac error: -release is only supported on Java 9 and higher Root cause: PR #11645 (which introduced scala-maven-plugin 4.9.2) also set recompileMode=incremental, which routes through Zinc's SbtIncrementalCompiler and bypasses computeBytecodeVersionOptions(). Restoring recompileMode=all exposes the 4.9.2 regression for Java 8 builds. Fix: in the java-8 profile, pin scala.compiler.version to 4.8.0, which does not have this regression. Java 9+ builds continue to use 4.9.2 unaffected. [1] https://github.com/davidB/scala-maven-plugin/blob/4.9.2/src/main/java/scala_maven/ScalaMojoSupport.java#L620-L648 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
baibaichen
force-pushed
the
GLUTEN-11658-restore-scala-recompile-mode
branch
from
a94455c to
9adfadf
Compare
|
Run Gluten Clickhouse CI on x86 |
zhouyuan
approved these changes
Feb 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #11658
PR #11560 changed
scala.recompile.modefromalltoincremental, which breaks CI/clean builds because Zinc's incremental compiler does not reliably track transitive Spark dependencies, causing errors likeSymbol 'type org.apache.spark.sql.internal.SqlApiConf' is missing from the classpath.Changes
pom.xml: Restorescala.recompile.modedefault toall(safe for CI/clean builds)pom.xml: Rename thebloopprofile (added in [CORE] Add Bloop integration for faster Scala incremental compilation #11645) tofast-buildand addscala.recompile.mode=incrementalinside it — used bybloop-setup.shandrun-scala-test.shwhere Zinc analysis is warmdev/bloop-setup.sh: Update-Pbloop→-Pfast-builddev/run-scala-test.sh: Replace 5 redundant-Dskip flags with-Pfast-build; also add pathing JAR support to avoid OS command-line length limitsdocs/developers/bloop-integration.md: Update profile name referenceEffect
all(safe)./dev/bloop-setup.shincremental(via-Pfast-build)./dev/run-scala-test.shincremental(via-Pfast-build)-Dscala.recompile.mode=incrementalTest plan
help:evaluateconfirms default isall,fast-buildprofile overrides toincrementalmvn clean test-compilewith Spark 4.1: BUILD SUCCESS in ~5 min, no classpath errorsrun-scala-test.sh GlutenDeprecatedDatasetAggregatorSuite: 5/5 tests passed🤖 Generated with Claude Code