Run liveness checks on uncrowded 2112 port

[loosebazooka]

• Readiness should still run on 3000

Summary

This might help with liveness checks that are being blocked by an overloaded port 3000 if we end up with a high volume of regular requests and the pod because starved.

Also move off the default global servermux and use an explicit one for utils (metrics, ping)

Readiness and Liveness still both hit ping just on different ports. It's possible they should have different names to prevent confusion.

Locally testing this, there are no (but practically probably just fewer) healthz related resets compared to running on 3000, when overloading the local instance running with docker compose.

Release Note

Documentation

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 26.18%. Comparing base (488eb97) to head (02a97d8).
⚠️ Report is 650 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2782       +/-   ##
===========================================
- Coverage   66.46%   26.18%   -40.28%     
===========================================
  Files          92      191       +99     
  Lines        9258    20125    +10867     
===========================================
- Hits         6153     5270      -883     
- Misses       2359    14027    +11668     
- Partials      746      828       +82     

Flag	Coverage Δ
e2etests	49.58% <100.00%> (+2.02%)	⬆️
unittests	16.71% <0.00%> (-30.98%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Hayden-IO]

Isn't the issue that the server is processing too many requests? Will the port change help?

[loosebazooka]

It's likely I misunderstood how gc assist debt works. It sounds like there might be a very minuscule improvement here due to a lighter middleware stack on port 3000, but not as much as claimed. Each connection is in its own go routine so gc assist debt isn't isolated to http server instances (an assumption that fueled a lot of this experiment). It appears that this may have a little effect and the real reason docker-compose tests were working better was because the compound effect of a few mitigation strategies in pgi#3612

I think I'll just close this for now.