SREP-2636: MUO Should Not Block Upgrade If SL Sent Failed

[karanjitsingh01]

What type of PR is this?

(bug/feature/cleanup/documentation/test/refactor)

What this PR does / why we need it?

If MUO failed to send SL, the upgrade was blocked and stuck in a loop, this fix continues the upgrade process and set "upgrade_notification_failed" metric so that an alert can be raised and SRE can manually update SL

Which Jira/Github issue(s) this PR fixes?

https://issues.redhat.com/browse/SREP-2636

Fixes #
Remove return error and instead log and continue

Special notes for your reviewer:

Pre-checks (if applicable):

• [Y ] Tested latest changes against a cluster
• Included documentation changes with PR

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: karanjitsingh01
Once this PR has been reviewed and has the lgtm label, please assign iamkirkbater for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci-robot]

@karanjitsingh01: This pull request references SREP-2636 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.22.0" version, but no target version was set.

Details

In response to this:

What type of PR is this?

(bug/feature/cleanup/documentation/test/refactor)

What this PR does / why we need it?

If MUO failed to send SL, the upgrade was blocked and stuck in a loop, this fix continues the upgrade process and set "upgrade_notification_failed" metric so that an alert can be raised and SRE can manually update SL

Which Jira/Github issue(s) this PR fixes?

https://issues.redhat.com/browse/SREP-2636

Fixes #
Remove return error and instead log and continue

Special notes for your reviewer:

Pre-checks (if applicable):

• [Y ] Tested latest changes against a cluster
• Included documentation changes with PR

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 62.50000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.41%. Comparing base (a59f964) to head (2c7ebd7).
⚠️ Report is 14 commits behind head on master.

Files with missing lines	Patch %	Lines
pkg/notifier/ocmnotifier.go	26.66%	9 Missing and 2 partials ⚠️
pkg/eventmanager/eventmanager.go	94.11%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #566      +/-   ##
==========================================
+ Coverage   54.35%   54.41%   +0.06%     
==========================================
  Files         123      123              
  Lines        6123     6154      +31     
==========================================
+ Hits         3328     3349      +21     
- Misses       2592     2601       +9     
- Partials      203      204       +1     

Files with missing lines	Coverage Δ
pkg/eventmanager/eventmanager.go	54.60% <94.11%> (+5.71%)	⬆️
pkg/notifier/ocmnotifier.go	29.55% <26.66%> (-0.10%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rbhilare]

/hold

[openshift-ci]

@karanjitsingh01: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.