Regression on main (58): Parquet argument error: Parquet error: Required field type_ is missing

[alamb]

Describe the bug

While testing the unreleased arrow version with the latest DataFusion, some of the tests begin failing like this when reading some parquet files

Error: ParquetError(ArrowError("Parquet argument error: Parquet error: Required field type_ is missing"))

To Reproduce

See the DataFusion upgrade PR here

• Upgrade DataFusion to arrow-rs/parquet 58.0.0 / object_store 0.13.0 datafusion#19728

To run

cargo test --profile=ci  --test core_integration -- path_partition

---- sql::path_partition::parquet_distinct_partition_col stdout ----
Error: ParquetError(ArrowError("Parquet argument error: Parquet error: Required field type_ is missing"))

The error appears to come from here:

arrow-rs/parquet/src/file/metadata/thrift/mod.rs

Line 1260 in 5695bb3

return Err(general_err!("Required field type_ is missing"));

However, that came in via the thrift remodel in c94698c which has been released.

This PR looks like it could be related:

• Add options to skip decoding Statistics and SizeStatistics in Parquet metadata #9008

Expected behavior
Tests should pass

Additional context
*

[alamb]

When I reverted #9008 / 86b6617 locally

git revert 86b6617

The DataFusion tests still fail

[alamb]

I also teseted reverting #9051 / 67e04e7

git revert 67e04e7

And the issue still happens

[jhorstmann]

One observation, this error could also trigger if the code tries to read a page from a wrong offset. If there is a zero byte at that offset, that could be interpreted as an empty thrift struct, and the type_ field would be the first required field that would trigger the error. Maybe the error could then be related to the object_store update?

[alamb]

One observation, this error could also trigger if the code tries to read a page from a wrong offset. If there is a zero byte at that offset, that could be interpreted as an empty thrift struct, and the type_ field would be the first required field that would trigger the error. Maybe the error could then be related to the object_store update?

Thank you @jhorstmann -- that was a great idea

I investigated more and it turns out the issue was with a bug in my upgrade of the test. Here is the fix apache/datafusion@6762cb7 in case you care.