Investigate metrics scaling needs in cf-for-k8s

[kkburr]

Is your feature request related to a problem? Please describe.

According to metrics-server requirements, metrics-server can scale to support up to 5000 nodes in a cluster. However, multiple teams have seen it crash with fewer nodes than 5000.

Describe the solution you'd like

Determine scaling needs of metrics-server and metric-proxy. Answer the following questions:

1. What are the scaling recommendations for metrics-server? How many nodes can each instance metrics-server handle?
2. Is metric-proxy getting load-balanced? Confirm service would work with multiple metric-proxies.
3. How many requests can each instance of metric-proxy handle?
4. What vertical scaling limits (resource contrains) should we put in place?

Additional context

• releng's scaling cf-for-k8s doc
• Logging performance story (mentions need to support 2000 AIs): Tracker #4535069
• SAP load testing of cf-for-k8s version 0.1.0
• Slack threads:
  https://vmware.slack.com/archives/CP4FVPLTE/p1600380022018100
  https://vmware.slack.com/archives/CTSTYENR0/p1602882070496600

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.