Skip to content

[Java] Add experimental Arrow Flight support#141

Open
danilonajkov-db wants to merge 7 commits intomainfrom
java-arrow
Open

[Java] Add experimental Arrow Flight support#141
danilonajkov-db wants to merge 7 commits intomainfrom
java-arrow

Conversation

@danilonajkov-db
Copy link
Contributor

@danilonajkov-db danilonajkov-db commented Mar 12, 2026

What changes are proposed in this pull request?

Adds Arrow Flight ingestion to the Java SDK. The Java side accepts VectorSchemaRoot directly, serializes to IPC bytes internally, and passes them across the JNI boundary where the Rust SDK handles encoding, framing, and transmission over the Arrow Flight gRPC protocol.

  • New ZerobusArrowStream with ingestBatch(), waitForOffset(), flush(), close(), getUnackedBatches()
  • New ArrowStreamConfigurationOptions with builder pattern
  • New SDK methods: createArrowStream(), recreateArrowStream()
  • Opt-in: add arrow-vector and arrow-memory-netty as dependencies (provided scope, >= 15.0.0)
  • Integration tests and example in examples/arrow/

How is this tested?

Added arrow flight integration tests

@danilonajkov-db danilonajkov-db changed the title [WIP][Java] Apache Arrow support [WIP][Java] Add experimental Arrow Flight support Mar 12, 2026
@danilonajkov-db danilonajkov-db changed the title [WIP][Java] Add experimental Arrow Flight support [Java] Add experimental Arrow Flight support Mar 13, 2026
@danilonajkov-db danilonajkov-db marked this pull request as ready for review March 13, 2026 11:04
*/
public long ingestBatch(VectorSchemaRoot batch) throws ZerobusException {
if (batch == null) {
throw new ZerobusException("Batch must not be null");
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how should we handle this case? I think throwing an exception makes sense, but we can make it a no-op too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the regular gRPC stream the return type is Optional<Long> and we return an empty option, so I think matching that would be okay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants