Prefer connected + HRW peer selection strategy

[prashantv]

HRW hashing works well for load distribution, but if the preferred peer is down, we'll still keep trying to connect to it on every single RPC. This is fine if we have a low number of RPCs, but can have high overhead on high RPS calls.

I think the behaviour we want is:

• Have a background goroutine that connects to the most available preferred peer
• On a request, try to find the most preferred peer that is connected
• If there are not connected peers, then return the top preferred peer (even if it's unconnected)

The background goroutine should ensure that we have connections to the top N peers.