Add a new backend: Omni

[wjunLu]

What is Omni?

Our team has developed a series of deeply optimized acceleration libraries on ARM platforms. As part of these works, we forked the Gluten project and added a new backend called omni.

This forked repository follows the examples of the Velox and ClickHouse backends by adding an omni backend adaptation backends-omni and a native adaptation cpp-omni to integrate with OmniOperator acceleration library (currently in the open-source process).

How to maintain this feature?

If the Gluten community is willing to merge the omni adaptation, our team has a dedicated group that can take responsibility for maintaining this feature, ensuring its long-term evolution, and providing corresponding ARM CI resources for build validation.

Performance

• The OmniOperator achieves an average performance improvement of over 70% in Spark, on TPC-DS 99 benchmark test cases.
• The OmniOperator achieves an average performance improvement of over 20% in Hive, on TPC-DS 99 benchmark test cases.

GPIP Doc

I have submitted the GPIP doc for details, please access:

• https://docs.google.com/document/d/1gFTNq0Zh4xb5B4T2UrhBC9N61QqV6VTqkJbyPdQoi2I/edit?usp=sharing

OmniOperator

• https://github.com/openeuler-mirror/OmniOperator

Omni-Backend adaption draft

• https://github.com/openeuler-mirror/Gluten

[FelixYBW]

Is omni library open source?

[wjunLu]

OmniOperator is still in open-source progress, with a planned official release date around mid-September.

The backend adaptation and native adaptation for omni (which were opened on forked repo ) are refactoring based on the previous discussions with community.

[FelixYBW]

Will omni contribute to Apache or some fundatation? What's the governance model? Does it accept external committers?

[wjunLu]

Yes, it will be contributed to some foundation(which is not decided finally)

[FelixYBW]

Can Omni input/output Arrow data?

[wjunLu]

No, it doesn't support Arrow data

[zhztheplayer]

What I am curious about is the architecture of the Omni library - can we consider it as a fork of Velox?

[wjunLu]

Performance comparison on ARM64:

• 20% improvement over Velox in a single-node TPCDS 1TB scenario
• 24% improvement over Velox in a 15-node multi-concurrency TPCDS 10TB scenario

[wjunLu]

If you want to know more performance results, please leave requirements here, I can try to test them.

[wjunLu]

BTW. I have finished the GPIP doc, but where should I submit it upon?

[FelixYBW]

You may copy to google doc and paste the link here. We have JIRA but rarely use it.

[wjunLu]

Done, please see the description.

[zhouyuan]

This forked repository follows the examples

it seems the repo is not public available now, would you please confirm?

[wjunLu]

Sorry, since the forked repo is currently hosted on a temporary platform, so it may not be very stable.

I just tried accessing it and didn't encounter any issues.

Please re-check this, and see backends-omni directory.

[FelixYBW]

Can we fork to github?

[wjunLu]

Of course, yes!

But this is not the final version, we are still refactoring that as an independent gluten-omni directory under incubator-gluten repo, which will like gluten-flink.

[zjuwangg]

Is omni used to accelerate other data process system, such as presto? What's omni long term plan?

[wjunLu]

Not yet to presto.

As for the long term plan, we can discuss more if you see the requirements.

[FelixYBW]

@wjunLu Since Omni supports other aarch64 CPUs, now the key is Omni's open source plan, ideally it can follow Apache's governonce mode. We don't have committer access to velox which leads to lots of pain to us.

[wjunLu]

The current plan is to open source OmniOperator in the openEuler community by September 20th (openEuler is a project of the OpenAtom Foundation).

[wjunLu]

Hi, @ALL!

OmniOperator has been open-sourced now! Please feel free to check, @zhouyuan @zjuwangg @zhztheplayer @FelixYBW :

• https://gitee.com/openeuler/OmniOperator

And the related backend adaption, which is still being refactored in accordance with flink's approach, is here for your reference:

• https://gitee.com/openeuler/Gluten

[wjunLu]

Yes, it's the mirror on github

Let me check if above two repos have been mirrored.

[wjunLu]

All repositories have been synchronized to github.com/openeuler-mirror , please take a look.

In this time, the master branch of https://github.com/openeuler-mirror/Gluten adapts omni-backend only on v1.3.0, we are bumping this.

[zhztheplayer]

Are you able to join the next community meeting? https://lists.apache.org/thread/yxnzgw3jzyd21p7jvlb28cov6d8v466l

Let's discuss this in that meeting if it works for you.

[wjunLu]

Of course yes!
What should I prepare before this meeting? Since my English is not such good...

[zhztheplayer]

No worries. Personally I was more curious about the background of the project, e.g., history of the project, the relationship between the project and others (e.g., Velox), and probably, whether it's to be donated to a foundation in the future, etc.

Other community members may have other casual questions, I am not sure. Of course we can continue the discussions only on GitHub either, if that works better for you.

[FelixYBW]

@wjunLu Heard from Frank Omni will be integrated into Bolt Engine. If so let's start a new thread to talk about Bolt engine instead.

[wjunLu]

I got this news, I'll start a new one

[wjunLu]

I found this discussion about Bolt #10929, but didn't see that integrated with omni

[FelixYBW]

Can you sync internally? What I heard is Omni will be merged into Bolt. It's much easier to be added as single backend.