RHACM CRDs left behind after deleting the ACM operator cause a panic in cert-manager

[rbmateescu]

I had RHACM ACM installed in my OCP cluster and after a while I deleted the multiclusterhub and the ACM operator. All RHACM CRDs were left behind .

cert-manager is now crashing , logging a panic like this :

{"level":"info","ts":1629120252.4404705,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"certmanager-controller"}
I0816 13:24:14.843192       1 trace.go:116] Trace[1458323237]: "Reflector ListAndWatch" name:pkg/mod/k8s.io/client-go@v0.0.0-20191016111102-bec269661e48/tools/cache/reflector.go:96 (started: 2021-08-16 13:24:02.594366184 +0000 UTC m=+68.246585093) (total time: 12.24858301s):
Trace[1458323237]: [12.246642649s] [12.246642649s] Objects listed
{"level":"info","ts":1629120254.9407926,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"certmanager-controller","worker count":1}
{"level":"info","ts":1629120254.940956,"logger":"controller_certmanager","msg":"Reconciling CertManager","Request.Namespace":"","Request.Name":"default"}
{"level":"info","ts":1629120254.9410067,"logger":"controller_certmanager","msg":"The namespace","ns":"ibm-common-services"}
{"level":"debug","ts":1629120254.9412458,"logger":"controller-runtime.manager.events","msg":"Normal","object":{"kind":"CertManager","name":"default","uid":"ac691bdd-66f5-47e7-bd1c-2da6101e26c3","apiVersion":"operator.ibm.com/v1alpha1","resourceVersion":"464281930"},"reason":"Initializing","message":"Instance found"}
E0816 13:24:14.946034       1 runtime.go:78] Observed a panic: runtime.boundsError{x:0, y:0, signed:true, code:0x0} (runtime error: index out of range [0] with length 0)
goroutine 5255 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x15d5700, 0xc001fd83d8)
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/runtime/runtime.go:74 +0xa6
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/runtime/runtime.go:48 +0x86
panic(0x15d5700, 0xc001fd83d8)
	/usr/local/go/src/runtime/panic.go:965 +0x1b9
github.com/ibm/ibm-cert-manager-operator/pkg/controller/certmanager.checkRhacm(0x1948290, 0xc00070c840, 0xc003b9a000, 0x16a555d, 0x6, 0x16aa126, 0xc, 0x16abf71)
	/home/prow/go/src/github.com/IBM/ibm-cert-manager-operator/pkg/controller/certmanager/prereqs.go:220 +0x4bf
github.com/ibm/ibm-cert-manager-operator/pkg/controller/certmanager.(*ReconcileCertManager).Reconcile(0xc0002fb740, 0x0, 0x0, 0xc001138c14, 0x7, 0x12c3b90dc0, 0xc000714990, 0xc0003086c8, 0xc0003086c0)
	/home/prow/go/src/github.com/IBM/ibm-cert-manager-operator/pkg/controller/certmanager/certmanager_controller.go:252 +0x765
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0007692c0, 0x1557be0, 0xc0006aa9c0, 0xffffffffffffff00)
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256 +0x166
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0007692c0, 0xc00132e100)
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232 +0xb0
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc00293ffa0)
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152 +0x5f
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc00293ffa0, 0x3b9aca00, 0x0, 0x1, 0xc0000aa2a0)
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153 +0x105
k8s.io/apimachinery/pkg/util/wait.Until(0xc00293ffa0, 0x3b9aca00, 0xc0000aa2a0)
	/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:193 +0x32d
panic: runtime error: index out of range [0] with length 0 [recovered]
	panic: runtime error: index out of range [0] with length 0

Removing the RHACM CRDs fixed the problem . The following error message is logged, but the pod is not crashing anymore :

{"level":"error","ts":1629121282.84336,"logger":"controller_certmanager","msg":"Error trying to check for RHACM","error":"no matches for kind \"MultiClusterHub\" in version \"operator.open-cluster-management.io/v1\"","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/prow/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/ibm/ibm-cert-manager-operator/pkg/controller/certmanager.(*ReconcileCertManager).Reconcile\n\t/home/prow/go/src/github.com/IBM/ibm-cert-manager-operator/pkg/controller/certmanager/certmanager_controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/prow/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.4.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/prow/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004115801-a2eda9f80ab8/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1629121282.8435316,"logger":"controller_certmanager","msg":"RHACM does not exist"}