Skip to content

Some Pulsar cluster pods not starting up #21

@jyoges

Description

@jyoges

Hi I tried to install pulsar but some pods are failing to start as below. I have attached the error logs as well.
Please advise. Thanks.

CR file:

kind: PulsarCluster
metadata:
  name: example-pulsarcluster
spec:
  zookeeper:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  autoRecovery:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
  bookie:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  broker:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  proxy:
    image:
      repository: apachepulsar/pulsar-all
      tag: 2.6.3
      pullPolicy: IfNotPresent
    size: 3
  manager:
    enable: true
    host: manager.pulsar.com
    annotations:
      kubernetes.io/ingress.class: "nginx"

Failing pods

 oc get pods
NAME                                                              READY   STATUS             RESTARTS   AGE
example-pulsarcluster-bookie-autorecovery-deployment-5f9bdkqkkl   0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-autorecovery-deployment-5f9bdstzp6   0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-autorecovery-deployment-5f9bdww48g   0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-statefulset-0                        0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-bookie-statefulset-1                        0/1     CrashLoopBackOff   8          16m
example-pulsarcluster-broker-deployment-76478578c-9tgmm           0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-broker-deployment-76478578c-ssxdv           0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-broker-deployment-76478578c-ttkkr           0/1     CrashLoopBackOff   8          19m
example-pulsarcluster-init-cluster-metadata-job-j65ll             0/1     Completed          0          20m
example-pulsarcluster-manager-deployment-9f8c49d99-rwbgz          1/1     Running            0          19m
example-pulsarcluster-proxy-deployment-85cb9f7666-2f9m6           1/1     Running            0          19m
example-pulsarcluster-proxy-deployment-85cb9f7666-8lnwz           1/1     Running            0          19m
example-pulsarcluster-proxy-deployment-85cb9f7666-mnfsd           1/1     Running            0          19m
example-pulsarcluster-zookeeper-statefulset-0                     1/1     Running            0          25m
example-pulsarcluster-zookeeper-statefulset-1                     1/1     Running            0          23m
example-pulsarcluster-zookeeper-statefulset-2                     1/1     Running            0          22m
pulsar-operator-64c69f896f-5mmms                                  1/1     Running            0          46m

Error log
example-pulsarcluster-bookie-autorecovery-deployment-5f9bdkqkkl:

 Applying config zkServers = example-pulsarcluster-zookeeper-service
JMX enabled by default
Error: Could not find or load main class "

example-pulsarcluster-bookie-statefulset-0

 Applying config zkServers = example-pulsarcluster-zookeeper-service
Error: Could not find or load main class "

example-pulsarcluster-broker-deployment-76478578c-9tgmm

000
06:41:36.389 [pulsar-ordered-OrderedExecutor-3-0-EventThread] INFO  org.apache.pulsar.broker.namespace.OwnershipCache - Successfully acquired ownership of /namespace/public/functions/0x40000000_0x80000000
06:41:36.403 [pulsar-ordered-OrderedExecutor-7-0] INFO  org.apache.pulsar.broker.PulsarService - No ledger offloader configured, using NULL instance
06:41:36.430 [pulsar-ordered-OrderedExecutor-7-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Opening managed ledger public/functions/persistent/assignments
06:41:36.456 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/assignments] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[112, 117, 98, 108, 105, 99, 47, 102, 117, 110, 99, 116, 105, 111, 110, 115, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 97, 115, 115, 105, 103, 110, 109, 101, 110, 116, 115], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds
06:41:36.463 [bookkeeper-ml-workers-OrderedExecutor-0-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [], allBookies [].
06:41:36.470 [bookkeeper-ml-workers-OrderedExecutor-0-0] WARN  org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [], allBookies [].
06:41:36.470 [bookkeeper-ml-workers-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.client.LedgerCreateOp - Not enough bookies to create ledger
06:41:36.471 [bookkeeper-ml-workers-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl - [public/functions/persistent/assignments] Failed to initialize managed ledger: Not enough non-faulty bookies available
06:41:36.472 [bookkeeper-ml-workers-OrderedExecutor-0-0] INFO  org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/assignments] Closing managed ledger
06:41:36.473 [bookkeeper-ml-workers-OrderedExecutor-0-0] WARN  org.apache.pulsar.broker.service.BrokerService - Failed to create topic persistent://public/functions/assignments
org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
06:41:36.477 [pulsar-web-44-5] ERROR org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to create non-partitioned topic persistent://public/functions/assignments
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
        at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_282]
        at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990) ~[?:1.8.0_282]
        at org.apache.pulsar.broker.service.BrokerService$3.openLedgerFailed(BrokerService.java:1086) ~[org.apache.pulsar-pulsar-

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions