[VL] Orc native scan can't read the table created with Hive

[beliefer]

Backend

VL (Velox)

Bug description

We created the table with Hive client.

CREATE TABLE default.test_orc_table_hive_gluten
(
    id int,
    name string
)
PARTITIONED BY (dt string)
STORED AS ORC;

insert into test_orc_table_hive_gluten partition(dt='20240728') values (1, 'a'),(2,'b');

And query this table with Spark SQL.
select * from test_orc_table_hive_gluten where dt = '20240728';
We expected the output should be

1	a	20240728
2	b	20240728

The output show below.

NULL	NULL	20240728
NULL	NULL	20240728

Gluten version

1.3

Spark version

Spark 3.2.2 ~ 3.5.2

Spark configurations

No response

System information

No response

Relevant logs

[beliefer]

ping @zhouyuan @rui-mo @taiyang-li @ccat3z @kevinwilfong

[rui-mo]

@beliefer Thanks. Let me have a look if I can reproduce.

[rui-mo]

@beliefer I tested above on Gluten main branch, but cannot reproduce the problem.

1	a	20240728
2	b	20240728
Time taken: 0.428 seconds, Fetched 2 row(s)

The plan is as below.

[beliefer]

Is the test table created with Hive client ? The version of Hive is 1.X

[rui-mo]

@beliefer No, I submitted this query through spark-sql, but I assume the Velox plan should be similar. Would you like to provide the ORC file you generated so our tests are consistent?

[beliefer]

Yeah. It is OK if we create the table and insert data with Spark SQL. But it is surely bad if you create the table with Hive.

[beliefer]

Let me show the schema.
desc formatted test_orc_table_hive_gluten;
The output is:

id                  	int                 	                    
name                	string              	                    
dt                  	string              	                    
# Partition Information	                    	                    
# col_name          	data_type           	comment             
dt                  	string              	                    
                    	                    	                    
# Detailed Table Information	                    	                    
Database            	default             	                    
Table               	test_orc_table_hive_gluten	                    
Owner               	hadoop              	                    
Created Time        	Mon Jul 29 11:06:19 HKT 2024	                    
Last Access         	UNKNOWN             	                    
Created By          	Spark 2.2 or prior  	                    
Type                	MANAGED             	                    
Provider            	hive                	                    
Table Properties    	[transient_lastDdlTime=1762163867]	                    
Location            	hdfs://mypath/warehouse/test_orc_table_hive_gluten	                    
Serde Library       	org.apache.hadoop.hive.ql.io.orc.OrcSerde	                    
InputFormat         	org.apache.hadoop.hive.ql.io.orc.OrcInputFormat	                    
OutputFormat        	org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat	                    
Storage Properties  	[serialization.format=1]	                    
Partition Provider  	Catalog

The HDFS info:
hdfs dfs -ls hdfs://mypath/warehouse/test_orc_table_hive_gluten/dt=20240728
The output is:
drwxr-x--- - hadoop hadoop 0 2024-07-29 11:06 hdfs://mypath/warehouse/test_orc_table_hive_gluten/dt=20240728

hdfs dfs -ls hdfs://mypath/warehouse/test_orc_table_hive_gluten/dt=20240728
The output is:
-rw-r----- 3 hadoop hadoop 297 2024-07-29 11:06 hdfs://mypath/warehouse/test_orc_table_hive_gluten/dt=20240728/000000_0

[beliefer]

000000_0.txt

@rui-mo I can't upload the 000000_0 due to the GitHub limit. Please download the 000000_0.txt and rename it to 000000_0

[rui-mo]

@beliefer I tested with the ORC file provided above but still did not reproduce. Would you please try reading this file in your end and paste the plan to this issue?

select * from tmp where dt = '20240728';
25/11/04 16:39:55 WARN GlutenFallbackReporter: Validation failed for plan: Project[QueryId=1], due to: 
 - Validation failed with exception from: ProjectExecTransformer, reason: Not supported to map spark function name to substrait function name: toprettystring(_col0#0, Some(GMT+0)), class name: ToPrettyString.
+-----+-----+--------+
|_col0|_col1|dt      |
+-----+-----+--------+
|1    |a    |20240728|
|2    |b    |20240728|
+-----+-----+--------+

[beliefer]

@rui-mo Thank you very much! I tested with latest Gluten. The issue fixed.
But I manual pick code from #10697 to Gluten 1.3, the issue occurred.
Is there some other code cause this different ? Could you tell me if you know.

[rui-mo]

@beliefer I believe the issue you're encountering is unrelated to the problem addressed in #10697. That pull request added support for schema evolution based on field index matching. However, in the scenario described here, there is no schema renaming involved, so schema evolution does not apply.

[beliefer]

@rui-mo Thank you! I will investigate the root cause. Thank you again!