Issues to execute tests on MSK Serveless Clusters - Topic are not being created from Kubernetes pods

[EderRoger]

Hi,

I'm trying without sucess perform tests on our MSK IAM Serverless clusters using Kubernetes Openmessaging pods
As per MSK documentation we created all the necessary IAM roles and permissions but when we are running the perf tests the topic creation step stuck in a loop. Look the stack below

Nov 09, 2023 3:29:29 PM com.amazonaws.http.AmazonHttpClient addProxyConfig INFO: Configuring Proxy. Proxy Host: Proxy Port: 4438 15:29:29.408 [qtp1390301622-20] INFO AbstractLogin - Successfully logged in. 15:29:29.451 [qtp1390301622-20] WARN AdminClientConfig - The configuration 'debug' was supplied but isn't a known config. 15:29:29.453 [qtp1390301622-20] INFO AppInfoParser - Kafka version: 3.2.1 15:29:29.454 [qtp1390301622-20] INFO AppInfoParser - Kafka commitId: -- 15:29:29.454 [qtp1390301622-20] INFO AppInfoParser - Kafka startTimeMs: --- 15:29:29.499 [qtp1390301622-23] INFO WorkerHandler - Received create topics request for topics: { "numberOfTopics" : 1, "numberOfPartitionsPerTopic" : 10 } 15:29:39.518 [pool-2-thread-1] INFO KafkaTopicCreator -
Created topics 0/1 ^C

We are stuck on this loop "Created topics 0/1 " even with the connection successfully with the cluster

our configs:

commonConfig: | bootstrap.servers=<msk_serverless_url>.amazonaws.com:9098 security.protocol=SASL_SSL sasl.mechanism=AWS_MSK_IAM debug=true sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn="arn:aws:iam::" awsDebugCreds=true awsRoleSessionName="msk-perf-test"; sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Our IAM policy has opened all permissions on Kafka to test
We did try using serviceaccount and run the perf tests but face same issue.

As another test, we used the same serviceaccount to the benchmark pods.
From the pod I did the download for Kafka binaries and execute the topic creation and did work.
./kafka-topics.sh --bootstrap-server :9098 --create --topic test-topic --partitions 1 --replication-factor 3 --command-config client.properties

./kafka-topics.sh --bootstrap-server <url>:9098 --create --topic test-topic --partitions 1 --replication-factor 3 --command-config client.properties [2023-11-13 15:23:06,550] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2023-11-13 15:23:06,550] WARN The configuration 'sasl.client.callback.handler.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) Created topic test-topic.
./kafka-topics.sh --list --bootstrap-server <url>:9098 --command-config client.properties [2023-11-13 15:26:16,255] WARN The configuration 'sasl.jaas.config' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) [2023-11-13 15:26:16,256] WARN The configuration 'sasl.client.callback.handler.class' was supplied but isn't a known config. (org.apache.kafka.clients.admin.AdminClientConfig) test-topic

Looks like the issue is using the KafkaTopicCreatror

any idea what we can do to make this happen?