Skip to content

Conversation

@mohit7705
Copy link

@mohit7705 mohit7705 commented Jan 8, 2026

What does this PR do?

This PR fixes an issue where Substrait scalar expressions could cause
duplicate registration of the same function URI during serialization.

The function reference was being encoded multiple times while building
nested expressions (e.g. large OR chains), leading to exponential growth
in extension URIs and serialized plan size.

What was changed?

  • Ensure EncodeFunction(call.id()) is invoked exactly once per
    ScalarFunction encoding.
  • Avoid repeated URI registration while serializing nested expressions.

Why is this needed?

Without this fix, expressions with many nested logical operators
(e.g. OR conditions) cause the Substrait plan size to grow exponentially,
which can severely impact memory usage and performance.

Testing

  • Reproduced using a Python script that serializes expressions with
    increasing OR conditions.
  • Verified that serialization still succeeds and the fix does not
    introduce regressions.

Related issue

Fixes #48761

@mohit7705 mohit7705 requested a review from westonpace as a code owner January 8, 2026 20:53
@github-actions
Copy link

github-actions bot commented Jan 8, 2026

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@mohit7705 mohit7705 changed the title [C++] Fix duplicate Substrait function URI registration GH-48761: [C++] Fix duplicate Substrait function URI registration Jan 8, 2026
@HyukjinKwon
Copy link
Member

Just my two cents:

Reproduced using a Python script that serializes expressions with
increasing OR conditions.
Verified that serialization still succeeds and the fix does not
introduce regressions.

I think we should probably add a test to prevent the regression. If it's difficult to add one, would be best to describe why and how you tested 👍

Comment on lines 1252 to 1256

auto scalar_fn =
std::make_unique<substrait::Expression::ScalarFunction>();

// Encode function ONCE per scalar function (FIX)
Copy link

@ahsanabbas123 ahsanabbas123 Jan 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please explain how does constructing the scalar function before encoding the call help avoid the repeated URIs as the rest of the changes seem to be just formatting?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review and for pointing this out. @ahsanabbas123 , @HyukjinKwon

I want to clarify that the earlier discussion can be ignored — I’ve updated the implementation since then.

What changed:
The fix now ensures that EncodeFunction(call.id()) is invoked exactly once per ScalarFunction, by constructing the ScalarFunction object before recursively encoding arguments. This prevents repeated registration of the same function URI when serializing nested expressions (e.g. large OR chains), which was the root cause of the exponential growth.

Why this fixes the issue:
Previously, function encoding could occur during recursive argument serialization, causing duplicate URI registrations. With this change, the function reference is encoded once per scalar function, regardless of nesting depth.

Testing:
I reproduced the original issue using a Python script that serializes expressions with increasing OR conditions and verified that:

serialization still succeeds

the number of registered extension URIs no longer grows exponentially

no regressions were observed

Please let me know if you’d like me to add a regression test for this, or if the explanation can be improved.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to update the PR description. I would want to follow how you tested and verify the PR 🙂

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I verified this change by building Arrow C++ from source after the fix (make -j$(nproc)) and confirmed the Substrait code compiles successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[C++] Substrait serialised expression size increases exponentially due to extension URIs being repeated

3 participants