Stuck project health spinner for projects with deployment that's being torn down

[mikeb26]

Summary:

While testing I encountered another hung spinner condition for project health.

Steps to reproduce:

1. create & deploy project
2. deactivate project (bopmatic project deactivate)
3. while the project is being deactivated, click on the console's project page
4. observe spinner is stuck and web console shows "something went wrong calling the API" error
5. the backend GetMetricSamples api fails with a "no such service" error because at some point during the deactivation it becomes no longer present.

Workaround:

shift-refresh

Suggested fix:

I suspect there's an inherent race condition where there exists a window of time between the last time the console called ProjectDescribe() & cached the result and GetMetricsSamples(). Within that time window, the project's state may have changed (e.g. because the user deactivated it, or did a new deployment that deleted a preexisting service). Whenever GetMetricsSamples fails for any reason, refresh whatever cached state the console is depending upon (e.g. ProjectDescribe()/ServiceDescribe() etc) and re-evaluate the decision on whether to display project health (and then call GetMetricsSamples() if the answer is still yes).

Screenshot attached.