Skip to content

[VL] Delta write / Spark 3.2, 3.3 write: Fix too-small partition number limit in dynamic partition writer#11420

Merged
zhztheplayer merged 1 commit intoapache:mainfrom
zhztheplayer:wip-delta-write-partition
Jan 15, 2026
Merged

[VL] Delta write / Spark 3.2, 3.3 write: Fix too-small partition number limit in dynamic partition writer#11420
zhztheplayer merged 1 commit intoapache:mainfrom
zhztheplayer:wip-delta-write-partition

Conversation

@zhztheplayer
Copy link
Member

Increases the partition number limit from 128 to 65536 to fix the following error:

Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: (128 vs. 128) Exceeded limit of 128 distinct partitions.
Retriable: False
Expression: nextPartitionId < maxPartitions_
Function: run
File: /opt/code/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/PartitionIdGenerator.cpp
Line: 78
Stack trace:
# 0  std::shared_ptr<facebook::velox::VeloxException::State const> facebook::velox::VeloxException::State::make<facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1}>(facebook::velox::VeloxException::Type, facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1})
# 1  facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)
# 2  facebook::velox::VeloxUserError::VeloxUserError(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, std::basic_string_view<char, std::char_traits<char> >)
# 3  void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxUserError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 4  facebook::velox::connector::hive::PartitionIdGenerator::run(std::shared_ptr<facebook::velox::RowVector> const&, facebook::velox::raw_vector<unsigned long>&)
# 5  Java_org_apache_gluten_datasource_VeloxDataSourceJniWrapper_splitBlockByPartitionAndBucket
# 6  0x0000f9d0048689ab
# 7  0x0000f9d0048689aa

…er limit in dynamic partition writer

Increases the partition number limit from 128 to 65536 to fix the following error:

```
Error Source: USER
Error Code: INVALID_ARGUMENT
Reason: (128 vs. 128) Exceeded limit of 128 distinct partitions.
Retriable: False
Expression: nextPartitionId < maxPartitions_
Function: run
File: /opt/code/incubator-gluten/ep/build-velox/build/velox_ep/velox/connectors/hive/PartitionIdGenerator.cpp
Line: 78
Stack trace:
# 0  std::shared_ptr<facebook::velox::VeloxException::State const> facebook::velox::VeloxException::State::make<facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1}>(facebook::velox::VeloxException::Type, facebook::velox::VeloxException::make(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)::{lambda(auto:1&)#1})
# 1  facebook::velox::VeloxException::VeloxException(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, facebook::velox::VeloxException::Type, std::basic_string_view<char, std::char_traits<char> >)
# 2  facebook::velox::VeloxUserError::VeloxUserError(char const*, unsigned long, char const*, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, std::basic_string_view<char, std::char_traits<char> >, bool, std::basic_string_view<char, std::char_traits<char> >)
# 3  void facebook::velox::detail::veloxCheckFail<facebook::velox::VeloxUserError, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&>(facebook::velox::detail::VeloxCheckFailArgs const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
# 4  facebook::velox::connector::hive::PartitionIdGenerator::run(std::shared_ptr<facebook::velox::RowVector> const&, facebook::velox::raw_vector<unsigned long>&)
# 5  Java_org_apache_gluten_datasource_VeloxDataSourceJniWrapper_splitBlockByPartitionAndBucket
# 6  0x0000f9d0048689ab
# 7  0x0000f9d0048689aa
```
@github-actions github-actions bot added the VELOX label Jan 14, 2026
@zhztheplayer zhztheplayer marked this pull request as ready for review January 15, 2026 10:00
@zhztheplayer zhztheplayer merged commit e4924b6 into apache:main Jan 15, 2026
64 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants