Overlord should call peons with announced host instead of podIP in mm-less setup

[Sh1ftry]

We are trying to use mm-less in a druid cluster which nodes are configured to communicate via mTLS.

druid_client_https_certAlias: certificate
druid_client_https_keyStorePath: /etc/druid-tls/keystore.jks
druid_client_https_keyStoreType: jks
druid_client_https_trustStorePath: /etc/druid-tls/truststore.jks
druid_server_https_certAlias: certificate
druid_server_https_keyStorePath: /etc/druid-tls/keystore.jks
druid_server_https_keyStoreType: jks
druid_server_https_requireClientCertificate: "true"
druid_server_https_trustStorePath: /etc/druid-tls/truststore.jks

Peons are setup to annouce themselves under <ip with dots replaced by dashes>.druid.pod host. This host is one of the dnsNames in a cretificate used for mTLS communication.

2023-12-20T09:15:21,222 INFO [task-runner-0-priority-0] org.apache.druid.server.coordination.CuratorDataSegmentServerAnnouncer - Announcing self[DruidServerMetadata{name='100-96-0-60.druid.pod:8091', hostAndPort='null', hostAndTlsPort='100-96-0-60.druid.pod:8091', maxSize=0, tier='_default_tier', type=indexer-executor, priority=0}] at [/druid/announcements/100-96-0-60.druid.pod:8091]

The confguration doesn't work after enabling mm-less setup, because overlord is trying to reach the peons using their pod's IP instead of the annouced host.

2023-12-20T09:15:57,385 INFO [ServiceClientFactory-3] org.apache.druid.rpc.ServiceClientImpl - Service [index_kafka_otterbots_dev_722af0a188685a7_icaccadn] request [GET https://100.96.0.60:8091/druid/worker/v1/chat/index_kafka_otterbots_dev_722af0a188685a7_icaccadn/time/start] encountered exception on attempt #8; retrying in 10,000 ms

Disabling hostname verification helps.

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.