Make local k8s setup work for mmless configuration

[uds5501]

Gets the pods to contact to each other locally. Without these changes, the pods were using the entire pod name to be able to connect with each other.

However, in k8s, the dns records for pod names doesn't exist, however they exist for specific nodes. Kindly see the example

DNS record search by pod name

bash-5.1# nslookup druid-tiny-cluster-coordinators-56d5865467-2hmqc
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find druid-tiny-cluster-coordinators-56d5865467-2hmqc.cluster.local: NXDOMAIN

** server can't find druid-tiny-cluster-coordinators-56d5865467-2hmqc.cluster.local: NXDOMAIN

** server can't find druid-tiny-cluster-coordinators-56d5865467-2hmqc.druid.svc.cluster.local: NXDOMAIN

** server can't find druid-tiny-cluster-coordinators-56d5865467-2hmqc.svc.cluster.local: NXDOMAIN

** server can't find druid-tiny-cluster-coordinators-56d5865467-2hmqc.svc.cluster.local: NXDOMAIN

** server can't find druid-tiny-cluster-coordinators-56d5865467-2hmqc.druid.svc.cluster.local: NXDOMAIN

DNS record by nodename.

bash-5.1# nslookup druid-tiny-cluster-coordinators                 
Server:         10.96.0.10
Address:        10.96.0.10:53

** server can't find druid-tiny-cluster-coordinators.cluster.local: NXDOMAIN

Name:   druid-tiny-cluster-coordinators.druid.svc.cluster.local
Address: 10.244.0.104

** server can't find druid-tiny-cluster-coordinators.cluster.local: NXDOMAIN

** server can't find druid-tiny-cluster-coordinators.svc.cluster.local: NXDOMAIN

** server can't find druid-tiny-cluster-coordinators.svc.cluster.local: NXDOMAIN

This PR has:

• been tested on a real K8S cluster to ensure creation of a brand new Druid cluster works.
• been tested for backward compatibility on a real K*S cluster by applying the changes introduced here on an existing Druid cluster. If there are any backward incompatible changes then they have been noted in the PR description.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

---

Key changed/added files in this PR

• tiny-cluster-mmless.yaml

[cryptoe]

The changes seem cool to me.
@AdheipSingh It would be great if you could take a look as well.

[cryptoe]

Thanks for the review @AdheipSingh !!

[SatyaKuppam]

I think the problem you are having here is using deployments instead of statefulsets which is the canonical way to route to pods in K8s. You can use headless services for all statefulsets for communicating with the pods. Another thing to keep in mind is setting druid.host will override the Druid default behaviour which is getCanonicalName(), this is similar to runnings hostname --fqdn` that will provide entire pod address.