Skip to content

[VL] Add task barrier support for NativePlanEvaluator to make Velox task reusable #11514

@zhztheplayer

Description

@zhztheplayer

Description

Currently ColumnarBatchOutIterator returned by NativePlanEvaluator#createKernelWithBatchIterator is disposable. It could only serve one input stream and output the result data one single time.

We'd add support for Velox's task barrier to make it reusable.

Fixing this will significantly improve performance of the ongoing native Delta statistics writer PR where we are creating a new Velox task for every single written partition, which can become super slow when there are thousands of partitions to write.

Reference, Velox task barrier:

https://facebookincubator.github.io/velox/develop/task-barrier.html

Gluten version

Main

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions