Support ZSTD compression level

[leekeiabstraction]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

We found that column pruning breaks because compression was not implemented on rust client.

This was fixed by adding compression support (with default ZSTD compression level), now we should add the missing compression level support: #108 (comment)

Solution

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[leekeiabstraction]

@luoyuxia Please assign this to me

[leekeiabstraction]

On Java side, we use lower level arrow classes e.g. VectorUnloader instead of using ArrowStreamWriter.

This was documented in apache/fluss#187

Currently, Fluss still need to support Java8, and that's why Fluss sticks on arrow-15.0 (the latest is v18.0). Arrow 15.0 Java client already introduced the CompressionCodec interfaces for ArrowStreamWriter, but doesn't natively support LZ4 and ZSTD compressions (implemented in later versions). But it should work if we learn the implementation by extending our own CompressionCodec.Factory: https://github.com/apache/arrow-java/commits/main/compression/src/main/java/org/apache/arrow/compression

[leekeiabstraction]

arrow-rs' CompressionContext currently does not support changing of ZSTD compression level and always default to 3.

We have two/three paths here,

1. Take Java side approach by implementing some lower level code onto fluss-rust to support ZSTD compression level
2. Contribute upstream to arrow-rs, wait for change to be released (monthly cadence)
3. Both of the above.

[luoyuxia]

arrow-rs' CompressionContext currently does not support changing of ZSTD compression level and always default to 3.

We have two/three paths here,

1. Take Java side approach by implementing some lower level code onto fluss-rust to support ZSTD compression level
2. Contribute upstream to arrow-rs, wait for change to be released (monthly cadence)
3. Both of the above.

Thanks for the investgation. I think we can pend this issue since modify level I believe is a rare case.