Operator fails reconciliation of existing vdb after upgrade from 25.3.0 to 25.3.1 (due to missing "vertica.com/enable-tls-auth" annotation?)

[bzakhar]

After upgrading the operator to 25.3.1, the following reconciliation failure is logged and the operator can't make progress:

2025-10-16T15:43:49.340Z INFO controllers.VerticaDB starting actor {"verticadb": {"name":"vertdb","namespace":"test"}, "reconcile-uuid": "f006fa6b-32bb-497d-b10b-ad88660d66c4", "name": "*vdb.TLSServerCertGenReconciler"}
2025-10-16T15:43:52.338Z INFO controllers.VerticaDB.TLSServerCertGenReconciler created certificate and secret vertdb-nma-tls-d55bc for NMATLSSecret {"verticadb": {"name":"vertdb","namespace":"test"}, "reconcile-uuid": "f006fa6b-32bb-497d-b10b-ad88660d66c4"}
2025-10-16T15:43:52.343Z INFO verticadb-resource default {"name": "vertdb", "GroupVersion": "vertica.com/v1"}
2025-10-16T15:43:52.349Z INFO verticadb-resource validate update {"name": "vertdb", "GroupVersion": "vertica.com/v1"}
2025-10-16T15:43:52.353Z ERROR controllers.VerticaDB.TLSServerCertGenReconciler failed to reconcile secret for NMATLSSecret {"verticadb": {"name":"vertdb","namespace":"test"}, "reconcile-uuid": "f006fa6b-32bb-497d-b10b-ad88660d66c4", "error": "admission webhook "vverticadb.v1.kb.io" denied the request: VerticaDB.vertica.com "vertdb" is invalid: [spec.httpsNMATLS: Forbidden: cannot set httpsNMATLS when vertica.com/enable-tls-auth is set to false, spec.clientServerTLS: Forbidden: cannot set clientServerTLS when vertica.com/enable-tls-auth is set to false]"}
github.com/vertica/vertica-kubernetes/pkg/controllers/vdb.(*TLSServerCertGenReconciler).reconcileSecrets
/workspace/pkg/controllers/vdb/tlsservercertgen_reconciler.go:113
github.com/vertica/vertica-kubernetes/pkg/controllers/vdb.(*TLSServerCertGenReconciler).Reconcile
/workspace/pkg/controllers/vdb/tlsservercertgen_reconciler.go:75
github.com/vertica/vertica-kubernetes/pkg/controllers/vdb.(*VerticaDBReconciler).Reconcile
/workspace/pkg/controllers/vdb/verticadb_controller.go:176
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:114
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:311
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222
2025-10-16T15:43:52.353Z INFO controllers.VerticaDB aborting reconcile of VerticaDB {"verticadb": {"name":"vertdb","namespace":"test"}, "reconcile-uuid": "f006fa6b-32bb-497d-b10b-ad88660d66c4", "result": {"Requeue":false,"RequeueAfter":0}, "err": "admission webhook "vverticadb.v1.kb.io" denied the request: VerticaDB.vertica.com "vertdb" is invalid: [spec.httpsNMATLS: Forbidden: cannot set httpsNMATLS when vertica.com/enable-tls-auth is set to false, spec.clientServerTLS: Forbidden: cannot set clientServerTLS when vertica.com/enable-tls-auth is set to false]"}
2025-10-16T15:43:52.353Z ERROR Reconciler error {"controller": "verticadb", "controllerGroup": "vertica.com", "controllerKind": "VerticaDB", "VerticaDB": {"name":"vertdb","namespace":"test"}, "namespace": "test", "name": "vertdb", "reconcileID": "79f86de0-8c64-4d20-8f44-162801ceb6a3", "error": "admission webhook "vverticadb.v1.kb.io" denied the request: VerticaDB.vertica.com "vertdb" is invalid: [spec.httpsNMATLS: Forbidden: cannot set httpsNMATLS when vertica.com/enable-tls-auth is set to false, spec.clientServerTLS: Forbidden: cannot set clientServerTLS when vertica.com/enable-tls-auth is set to false]"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:261
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.4/pkg/internal/controller/controller.go:222`

CR is not annotated with vertica.com/enable-tls-auth but has

httpsNMATLS:
mode: TRY_VERIFY
secret: vertdb-https-tls-gxphd

[roypaulin]

Hey @bzakhar, there is a webhook rule in 25.3.1 that is too strict and prevents any progress after upgrade. We will backport the fix in 25.3.1 and let you know so you can pull the image and retry.
Also, that annotation will be deprecated in next release (25.4.0), it's just not a good way to control tls, we will now use the spec. But there won't be any issues when you upgrade from 25.3 to 25.4

[roypaulin]

Hey @bzakhar,The fixed has been backported to 25.3.1. Make sure to pull 25.3.1 image again and retry.