Skip to content

Conversation

@steveloughran
Copy link
Contributor

  • Downgrades the hadoop version
  • Adds a comment to libs.versions saying why it needs to be kept down and hence when it can be upgraded.

Fixes #15100

- Downgrades the hadoop version
- Adds a comment to libs.versions saying why it needs to be kept down and hence when it
  can be upgraded.

Fixes apache#15100
@steveloughran steveloughran marked this pull request as draft January 27, 2026 13:55
@manuzhang
Copy link
Member

I'm not sure this is worth it given

  1. We've released iceberg with hadoop 3.4 and Spark 3.4 support since 1.7
  2. Spark 3.4 support is already deprecated and we will remove after next release.

@steveloughran
Copy link
Contributor Author

I'd like to get the bulk delete work of #10233 in, with the resulting 1000x speedup in performance in deleting files, but can't get the direct-API-invocation in without breaking spark 3.4.

Forcing the downgrade not only means I can return to the reflection-based binding.

There's also that risk of accidental overloads, enum/option use etc with any change made using hadoop apis stop working, even if previous releases were all OK

@manuzhang
Copy link
Member

I'd like to get the bulk delete work of #10233 in, with the resulting 1000x speedup in performance in deleting files, but can't get the direct-API-invocation in without breaking spark 3.4.

Can we hold it till dropping Spark 3.4 support?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build: iceberg core is built against a later version of hadoop libs than spark 3.x releases

2 participants