How to dynamically switch between native Spark and Gluten backend based on the execution plan?

[M1zukiqwq]

I'm currently exploring the possibility of dynamically selecting the execution engine (either the native Spark engine or one of Gluten's backends) based on the characteristics of the query's execution plan.
Could anyone point me to the relevant sections of the Gluten codebase that I would need to modify to implement this kind of dynamic, plan-based engine selection logic? I would appreciate any guidance or suggestions on where to start looking.

[zjuwangg]

#7143 #5177
RAS may be a good choice to begin.
cc @zhztheplayer

[M1zukiqwq]

#7143 #5177 RAS may be a good choice to begin. cc @zhztheplayer

I want implement whole stage fallback, not single stagefallback.

[zjuwangg]

#7143 #5177 RAS may be a good choice to begin. cc @zhztheplayer

I want implement whole stage fallback, not single stagefallback.

What's the meaning of whole stage fallback, you mean all plan ?

[M1zukiqwq]

#7143 #5177 RAS may be a good choice to begin. cc @zhztheplayer

I want implement whole stage fallback, not single stagefallback.

What's the meaning of whole stage fallback, you mean all plan ?

It means if the execution plan meets certain conditions, the entire task will still be executed in the native Spark engine even if the gluten JAR package is loaded.

[zhztheplayer]

It means if the execution plan meets certain conditions.

Do you have the concrete conditions, or a rough strategy in mind already?

[M1zukiqwq]

It means if the execution plan meets certain conditions.

Do you have the concrete conditions, or a rough strategy in mind already?

I use a decider to decide use gluten or not, and I integrate this decider by use Catalyst Extension.Therefore, I want to use this decorator's judgment to decide whether to use gluten to execute the query.

[PHILO-HE]

@M1zukiqwq, could you share more details about your decider? I'm curious whether it's tailored to your specific workload or designed to be generalizable for common use cases .If it's workload-specific, we might consider exposing APIs that allow users to inject their own custom deciders into Gluten. Otherwise, it may be better to contribute it directly to the Gluten project.

[PHILO-HE]

@M1zukiqwq, "based on the characteristics of the query's execution plan", does these characteristics include whether an operator is supported by Gluten?

[FelixYBW]

Do you mean fallback whole queries or stages? to queries, you can set spark.gluten.enabled=false for the query. To stages we currently can support whole stage fallback once there is one operator in the stage needs fallback by setting spark.gluten.sql.columnar.wholeStage.fallback.threshold=1.

If you need to fallback any whole stage you want, how do you define the stage? you can easily add some config like spark.gluten.fallbackstage=5. But you would like to fallback based on some rules, RAS is a good choice.