[improve][io][kca] support fully-qualified topic names in source records#10
Closed
[improve][io][kca] support fully-qualified topic names in source records#10
Conversation
5b35aaa to
6afa94c
Compare
6afa94c to
fa9a291
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The current implementation of the Kafka Connect adaptor in Pulsar IO does not support fully-qualified Pulsar topic names in source records. Instead, it forcefully prepends the default
persistent://<tenant>/<namespace>/prefix to all destination topics. This behavior can be problematic in multi-tenant environments where dynamic topic routing is required.For example, consider a setup with:
tenant1,tenant2)tenant1,tenant2)global(shared) tenantUsing Kafka Connect transformations, users may want to route records to tenant-specific topics based on the schema:
With this configuration, changes to
tenant1.ordersshould go topersistent://tenant1/procurement/orders, andtenant2.orderstopersistent://tenant2/procurement/orders.However, the current implementation prepends
persistent://global/procurement/to the already fully-qualified topic, resulting in invalid topic names likepersistent://global/procurement/persistent://tenant1/procurement/orders. This causes runtime exceptions and connector failure loops.By supporting fully-qualified topic names, this change enables more flexible and tenant-aware architectures without requiring additional processing layers.
Modifications
This change enhances
AbstractKafkaConnectSourceto correctly handle fully-qualified topic names by using theorg.apache.pulsar.common.naming.TopicNameutility. If the topic is valid and fully-qualified, it is respected as-is. Otherwise, the adaptor falls back to the existing behavior, ensuring backward compatibility.Verifying this change
This change added tests and can be verified as follows:
KafkaConnectSourceTesttest to verify that transforms can be initialized and appliedDocumentation
docdoc-requireddoc-not-neededdoc-complete