Failed to upgrade from vertica-operator v25.2.0 to v25.3.0

[alita1991]

After upgrading the verticadb-operator from version 23.2.0 to 23.3.0, I am unable to create or upgrade VerticaDB instances. What I did was to update the deprecated field nmaTLSSecret with httpsNMATLSSecret.

Warning HTTPServerNotSetup 2m48s (x108 over 27m) verticadb-operator The httpsNMATLS.secret must be set when running with vclusterops deployment (unexpected behaviour)

Additionally, while attempting to create or upgrade VerticaDB instances, the operator generated over 17,000 Kubernetes secrets, which is not the expected behavior.

vertica            vertica-single-https-tls-rdzwl           kubernetes.io/tls                3      19m
vertica            vertica-single-https-tls-rfxwg           kubernetes.io/tls                3      3m29s
vertica            vertica-single-https-tls-rj4wd           kubernetes.io/tls                3      9m2s
vertica            vertica-single-https-tls-rt9zj           kubernetes.io/tls                3      5m56s
vertica            vertica-single-https-tls-rzgpt           kubernetes.io/tls                3      13m
vertica            vertica-single-https-tls-s79sd           kubernetes.io/tls                3      23m

The VerticaDB instance manifest used for deployment is the following:

apiVersion: vertica.com/v1
kind: VerticaDB
metadata:
  name: vertica-single
  annotations:
    vertica.com/k-safety: "0"
    vertica.com/nma-resources-requests-cpu: "250m"
    vertica.com/nma-resources-requests-memory: "256Mi"
    vertica.com/nma-resources-limits-cpu: "1"
    vertica.com/nma-resources-limits-memory: "1Gi"
    argocd.argoproj.io/sync-wave: "1"
spec:
  httpsNMATLSSecret: vertica-nma-tls
  image: opentext/vertica-k8s:25.2.0-3-minimal
  imagePullPolicy: Always
  imagePullSecrets:
    - name: vertica-pull-secret
  securityContext:
    allowPrivilegeEscalation: false
    readOnlyRootFilesystem: true
    capabilities:
      drop:
        - ALL
  podSecurityContext:
    fsGroup: 10001
    runAsUser: 10001
    runAsGroup: 10001
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  communal:
    path: "s3://vertica/vertica-single"
    endpoint: http://minio.minio.svc:80
    credentialSecret: minio-s3-creds
  local:
    requestSize: 50Gi
  volumes:
    - name: tmp
      emptyDir: {}
  volumeMounts:
    - name: tmp
      mountPath: /tmp
  subclusters:
    - name: defaultsubcluster
      size: 1
      resources:
        requests:
          cpu: 250m
          memory: 512Mi
          ephemeral-storage: 64Mi
        limits:
          cpu: 4
          memory: 4Gi
          ephemeral-storage: 128Mi
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: vertica-single
              topologyKey: kubernetes.io/hostname
  shardCount: 1

[laurentiusoica]

Same here

[qindotguan]

As we removed v1beta1 in the 25.3.0 release, there is an extra step to patch the existing CRD before applying the new CRD. Please follow the steps to upgrade the operator:

# 1. patch existing CRD
kubectl patch crd verticadbs.vertica.com --type merge -p '{"spec":{"conversion":null}}'
# 2. apply new CRD
kubectl apply --server-side=true --force-conflicts -f https://github.com/vertica/vertica-kubernetes/releases/latest/download/crds.yaml
# 3. upgrade
helm upgrade operator-name --wait vertica-charts/verticadb-operator

Details:
https://docs.vertica.com/25.3.x/en/containerized/db-operator/upgrading-db-operator/#helm-charts

[alita1991]

Hi @qindotguan,

I have similar issues if I do a scratch to provision VerticaDB. The instances are not created at all with the config from this ticket.

Also, I'm not able to understand why the operator creates so many secrets that are not deleted at all (over 17k on a long-lived environment), all related to nma tls secret, even if I provide myself the secret.

[cchen-vertica]

@alita1991 This is a bug. We will take a look. Could you share the operator log with me? On your side, you can do two things to bypass this issue:

1. set httpsNMATLS.secret to vertica-nma-tls rather than "httpsNMATLSSecret".
2. set metadata.annotaitons.vertica.com/remove-tls-secret-on-vdb-delete to "true", then after you remove vdb, all auto-generated secrets will be removed.

[alita1991]

Hi @cchen-vertica,

I’m unable to reproduce the issue with the secrets. The only thing I still see is the manifest diff that ArgoCD detected.

I uploaded a screenshot from ArgoCD for a multi-instance deployment, with the diff I see on the manifest, which does not allow the resource to be fully reconciled. (for a single replica vertica, just the httpsNMATLSSecret is part of the diff)

Please note that webhooks are disabled, and the migration to httpsNMATLSSecret has been completed in preparation for the upgrade. If there is anything I need to do other than the httpsNMATLSSecret change in the new version of the manifest, please let me know.

[cchen-vertica]

@alita1991 You just need to set metadata.annotaitons.vertica.com/remove-tls-secret-on-vdb-delete to "true" so the auto-generated secrets can be removed when vdb is removed.

[alita1991]

@qindotguan For the CRD patch to set spec.conversion to null, I’m considering an alternative using kubectl replace, which also has parity in ArgoCD through argocd.argoproj.io/sync-options: Replace=true. Do you see any issues with handling the migration this way, or should it be fine?

Worth noting: webhooks are disabled in my setup, which is why I’m getting the error:

[spec.conversion.webhookClientConfig: Forbidden: should not be set when strategy is not set to Webhook, 
 spec.conversion.conversionReviewVersions: Forbidden: should not be set when strategy is not set to Webhook]. 
 Retrying attempt #4 at 2:01PM.

If you have a better approach in mind, I’d be happy to hear it, thank you