Skip to content

[GLUTEN-10903][VL] Build cudf always#11691

Merged
zhouyuan merged 1 commit intoapache:mainfrom
jinchengchenghh:cudf_build
Mar 3, 2026
Merged

[GLUTEN-10903][VL] Build cudf always#11691
zhouyuan merged 1 commit intoapache:mainfrom
jinchengchenghh:cudf_build

Conversation

@jinchengchenghh
Copy link
Contributor

@jinchengchenghh jinchengchenghh commented Mar 3, 2026

Because the base image apache/gluten:centos-9-jdk8-cudf is updated every week while cudf maybe updated in place, so we need to build cudf every time to avoid API compatibility.

Related issue: #10903

Fixes: #11693

@jinchengchenghh
Copy link
Contributor Author

Because the old cudf exists in Docker image apache/gluten:centos-9-jdk8-cudf directory /usr/local/include/cudf, so velox does not build cudf in the expected path, so the build failed, I'm verifying if the build Dockerfile can be successful.

@jinchengchenghh
Copy link
Contributor Author

After verified successfully, I will re-trigger Docker image push Actions

@jinchengchenghh
Copy link
Contributor Author

I have verified with this Dockerfile

FROM ghcr.io/facebookincubator/velox-dev:adapters
ENV CUDA_ARCHITECTURES=70
ENV LD_LIBRARY_PATH=/opt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/curl-build/lib:$LD_LIBRARY_PATH


RUN yum install -y sudo patch perl && \
    dnf remove -y cuda-toolkit-12* && dnf install -y cuda-toolkit-13-1; \
    dnf autoremove -y && dnf clean all; \
    rm -rf /opt/rh/gcc-toolset-12 && ln -s /opt/rh/gcc-toolset-14 /opt/rh/gcc-toolset-12; \
    ln -sf /usr/local/bin/cmake /usr/bin && \
    git clone --depth=1 https://github.com/apache/incubator-gluten /opt/gluten && \
    cd /opt/gluten && \
    git config --global user.email "you@example.com" && \
    git config --global user.name "Your Name" && \
    git fetch origin pull/11691/head && \
    git cherry-pick 03f7e9647d0f0a68acc79d6131aded1e7a4914d0 && \
    source /opt/rh/gcc-toolset-14/enable && \
    bash ./dev/buildbundle-veloxbe.sh --run_setup_script=OFF --build_arrow=ON --spark_version=3.4 --build_tests=ON --build_benchmarks=ON --enable_gpu=ON && \
    rm -rf /opt/gluten && \
    rm -rf /root/.cache/ccache

@jinchengchenghh jinchengchenghh requested a review from zhouyuan March 3, 2026 14:05
@zhouyuan
Copy link
Member

zhouyuan commented Mar 3, 2026

@zhouyuan zhouyuan merged commit 4accc0f into apache:main Mar 3, 2026
59 of 61 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[VL] GPU CI failed due to missing header

2 participants