Hot Reload Failure Test - New Instance Kill

[ric-pro]

Tried to address the issue: #1962

The test following test passed:

• perform a regular hot reload scenario
• write variants of this test that kills the new shotover immediately after the new shotover emits these logs:
  • log for gradual shutdown
• create a 3rd shotover instance with the same hot reload args as the first 2 instances.
• assert that the 3rd instance successfully startsup.

But the following test failed:

• perform a regular hot reload scenario
• write variants of this test that kills the new shotover immediately after the new shotover emits these logs:
  • log for received FD's
• create a 3rd shotover instance with the same hot reload args as the first 2 instances.
• assert that the 3rd instance successfully startsup.

With the failed test, what I was trying to do was:

1. Old shotover is running and serving connections
2. New shotover starts and request for FD
3. Old instance sends FD to new instance
4. New Instance is killed before it receives the FD
   This will simulate a situation where an error occurs in between the hot reload process and the new shotover terminates. I was trying to test if a new shotover can be spinned up to cover for the termination of old shotover. The third instance will have same args as the first 2 instances. Ideally, the new shotover (third) should be able to take over from where the 2nd shotover terminated.

[codspeed-hq]

CodSpeed Performance Report

Merging #1966 will not alter performance

_{Comparing ric-pro:test1 (c210d98) with main (2b6910b)}

Summary

✅ 38 untouched

Was testing whether my account has approver rights in shotover