Investigate porting Eclipse Store AFS Kafka adapter using Confluent.Kafka (.NET)

[hadv]

Summary

Investigate whether we can use the Confluent.Kafka .NET library to port the Eclipse Store AFS adapter for Apache Kafka to NebulaStore.

• Upstream reference: https://github.com/eclipse-store/store/tree/main/afs/kafka
• Objective: Determine feasibility, scope, and a minimal viable path to integrate a Kafka-backed AFS adapter in the NebulaStore architecture, aligning with the Eclipse Store design while staying within our repo’s .NET 9.0 implementation.

Goals

• Understand feature parity and constraints between Eclipse Store’s AFS Kafka adapter and what’s feasible with Confluent.Kafka on .NET.
• Define an adapter design that fits our module structure and naming preferences (NebulaStore.Storage.Embedded*), without over-engineering beyond Eclipse Store scope.
• Produce a recommendation (and optionally a small spike/POC) to validate produce/consume paths and basic configuration.

Deliverables / Acceptance Criteria

• Brief design note outlining the mapping from Eclipse Store AFS Kafka components to NebulaStore (.NET) counterparts
• Compatibility and semantics review (e.g., delivery guarantees we can reasonably support with Confluent.Kafka; at-least-once vs exactly-once trade-offs)
• Minimal API surface proposal consistent with our storage module structure and naming
• POC (optional but preferred): a small console or test harness that can produce and consume messages using Confluent.Kafka with our proposed abstraction
• Risk and limitation assessment (operational risks, dependencies, performance considerations)
• Clear go/no-go recommendation with next steps if we proceed (phased PR plan)

Proposed Approach

1. Review the upstream Java adapter to identify responsibilities (configuration, producer/consumer lifecycle, message formats, error handling, retries, partition/offset mgmt).
2. Map those responsibilities to Confluent.Kafka APIs and .NET idioms, noting any gaps.
3. Outline an adapter that respects NebulaStore’s module layout and keeps scope aligned to Eclipse Store’s features.
4. If feasible within a short timebox, implement a small spike to validate:
   • Basic producer/consumer lifecycle
   • Config plumbing (bootstrap servers, topic naming, serializers)
   • Simple integration point that could evolve into a proper module
5. Summarize findings, risks, and a suggested incremental implementation plan (PRs per phase, per project preference).

Risks / Considerations

• Exactly-once semantics are complex; we may target at-least-once initially.
• Dependency surface area: Confluent.Kafka NuGet package and transitive native dependencies (librdkafka).
• Operational requirements (Kafka availability, security configs: SASL/SSL).
• Performance characteristics vs. expectations.

Open Questions

• Target module/namespace placement within our current structure (e.g., NebulaStore.Storage.Embedded or a sibling module for AFS adapters)?
• Minimal config set we want to expose initially.
• Acceptance tests strategy (local Kafka via Docker for CI, or mock-based?).

Next Steps (if greenlit)

• Create a dedicated branch and initial scaffolding
• Introduce Confluent.Kafka via package manager (NuGet)
• Add basic producer/consumer wrappers + configuration
• Add tests and example usage, then open a PR for review

/assign @hadv

[hadv]

#34