Reconcilliation Issues Causes Log Flooding #3125
Description
Doing a fresh install (0.78.1) using the Helm Chart (or the operator directly) causes our logs to be flooded with reconciliation errors.
There is a slack conversation related to this @ https://tektoncd.slack.com/archives/CKZQUT3MH/p1765846498548289
After talking with everyone it seems this is a Knative race condition (and we believe we see this as well in other non-Tekton Knative deploys) and it is benign so we are filtering it our using Fluent on our end (maybe related to this discussion question i filed quite a while back - knative/serving#15442)
kubectl logs -n tekton-operator tekton-operator-fb67b66f7-zt8v8 -c tekton-operator-lifecycle
{"severity":"INFO","timestamp":"2025-12-18T16:11:27.893988113Z","logger":"tekton-operator-lifecycle","caller":"tektonconfig/tektonconfig.go:186","message":"TektonPruner is disabled. Shutting down event based pruner","commit":"e2e528b","knative.dev/pod":"tekton-operator-fb67b66f7-zt8v8","knative.dev/controller":"github.com.tektoncd.operator.pkg.reconciler.shared.tektonconfig.Reconciler","knative.dev/kind":"operator.tekton.dev.TektonConfig","knative.dev/traceid":"0b28c7cf-1bca-4233-81ad-e6b711afd7e6","knative.dev/key":"config","tektonconfig":"config"}
{"severity":"ERROR","timestamp":"2025-12-18T16:11:27.905035349Z","logger":"tekton-operator-lifecycle","caller":"tektonconfig/tektonconfig.go:296","message":"Post-reconcile hook failed","commit":"e2e528b","knative.dev/pod":"tekton-operator-fb67b66f7-zt8v8","knative.dev/controller":"github.com.tektoncd.operator.pkg.reconciler.shared.tektonconfig.Reconciler","knative.dev/kind":"operator.tekton.dev.TektonConfig","knative.dev/traceid":"0b28c7cf-1bca-4233-81ad-e6b711afd7e6","knative.dev/key":"config","tektonconfig":"config","error":"requeue after: 10s","stacktrace":"github.com/tektoncd/operator/pkg/reconciler/shared/tektonconfig.(*Reconciler).ReconcileKind\n\tgithub.com/tektoncd/operator/pkg/reconciler/shared/tektonconfig/tektonconfig.go:296\ngithub.com/tektoncd/operator/pkg/client/injection/reconciler/operator/v1alpha1/tektonconfig.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/operator/pkg/client/injection/reconciler/operator/v1alpha1/tektonconfig/reconciler.go:236\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250915135827-db4c336acdbe/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250915135827-db4c336acdbe/controller/controller.go:489"}
{"severity":"INFO","timestamp":"2025-12-18T16:11:35.604305302Z","logger":"tekton-operator-lifecycle","caller":"tektonconfig/tektonconfig.go:186","message":"TektonPruner is disabled. Shutting down event based pruner","commit":"e2e528b","knative.dev/pod":"tekton-operator-fb67b66f7-zt8v8","knative.dev/controller":"github.com.tektoncd.operator.pkg.reconciler.shared.tektonconfig.Reconciler","knative.dev/kind":"operator.tekton.dev.TektonConfig","knative.dev/traceid":"46083482-8cd8-4461-8e3b-ed2721ce9d85","knative.dev/key":"config","tektonconfig":"config"}
{"severity":"ERROR","timestamp":"2025-12-18T16:11:35.616765407Z","logger":"tekton-operator-lifecycle","caller":"tektonconfig/tektonconfig.go:296","message":"Post-reconcile hook failed","commit":"e2e528b","knative.dev/pod":"tekton-operator-fb67b66f7-zt8v8","knative.dev/controller":"github.com.tektoncd.operator.pkg.reconciler.shared.tektonconfig.Reconciler","knative.dev/kind":"operator.tekton.dev.TektonConfig","knative.dev/traceid":"46083482-8cd8-4461-8e3b-ed2721ce9d85","knative.dev/key":"config","tektonconfig":"config","error":"requeue after: 10s","stacktrace":"github.com/tektoncd/operator/pkg/reconciler/shared/tektonconfig.(*Reconciler).ReconcileKind\n\tgithub.com/tektoncd/operator/pkg/reconciler/shared/tektonconfig/tektonconfig.go:296\ngithub.com/tektoncd/operator/pkg/client/injection/reconciler/operator/v1alpha1/tektonconfig.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/operator/pkg/client/injection/reconciler/operator/v1alpha1/tektonconfig/reconciler.go:236\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250915135827-db4c336acdbe/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250915135827-db4c336acdbe/controller/controller.go:489"}
{"severity":"INFO","timestamp":"2025-12-18T16:11:45.623840331Z","logger":"tekton-operator-lifecycle","caller":"tektonconfig/tektonconfig.go:186","message":"TektonPruner is disabled. Shutting down event based pruner","commit":"e2e528b","knative.dev/pod":"tekton-operator-fb67b66f7-zt8v8","knative.dev/controller":"github.com.tektoncd.operator.pkg.reconciler.shared.tektonconfig.Reconciler","knative.dev/kind":"operator.tekton.dev.TektonConfig","knative.dev/traceid":"53de5612-10f2-4f6e-a2a2-baee766c7394","knative.dev/key":"config","tektonconfig":"config"}
{"severity":"ERROR","timestamp":"2025-12-18T16:11:45.63498045Z","logger":"tekton-operator-lifecycle","caller":"tektonconfig/tektonconfig.go:296","message":"Post-reconcile hook failed","commit":"e2e528b","knative.dev/pod":"tekton-operator-fb67b66f7-zt8v8","knative.dev/controller":"github.com.tektoncd.operator.pkg.reconciler.shared.tektonconfig.Reconciler","knative.dev/kind":"operator.tekton.dev.TektonConfig","knative.dev/traceid":"53de5612-10f2-4f6e-a2a2-baee766c7394","knative.dev/key":"config","tektonconfig":"config","error":"requeue after: 10s","stacktrace":"github.com/tektoncd/operator/pkg/reconciler/shared/tektonconfig.(*Reconciler).ReconcileKind\n\tgithub.com/tektoncd/operator/pkg/reconciler/shared/tektonconfig/tektonconfig.go:296\ngithub.com/tektoncd/operator/pkg/client/injection/reconciler/operator/v1alpha1/tektonconfig.(*reconcilerImpl).Reconcile\n\tgithub.com/tektoncd/operator/pkg/client/injection/reconciler/operator/v1alpha1/tektonconfig/reconciler.go:236\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20250915135827-db4c336acdbe/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20250915135827-db4c336acdbe/controller/controller.go:489"}