TLS validation fails in connection from searchhead to indexpeers in cluster scenarios?

[DATEVeG]

Hello,

we tried to enable TLS validation with Splunk 9.0.2 as described in the Splunk documentation. Unfortunately, this caused our distributed Splunk system consisting of index cluster and searchhead cluster to stop working. Specifically, searches could no longer be performed.
We discovered the following messages in the search head log:

11-06-2022 23:59:59.839 +0100 ERROR DistributedPeerManagerHeartbeat [95344 DistributedPeerMonitorThread] - Send failure while pushing public key to search peer = https://10.10.10. 10:8089 , error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.

This makes us wonder, because actually the connection for the TLS connection should be established over fqdn and not over an ip address. When establishing a connection over the IP, it is also logical that the TLS name check against the host name fails, since only the FQDN and no IP address is stored in the TLS certificate.

We therefore wondered why the search head addresses our index peer 10.10.10.10 via the IP and not the fqdn. The Splunk documentation states that the search head gets the list of index peers from the cluster manager. However, in the cluster manager, our index peers also report by name, at least that is what the output of the cluster master suggests:

>> splunk/bin/splunk show cluster-status
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Replication factor met
Search factor met
All data is searchable
Indexing Ready YES
HA Mode: Disabled

indexpeer01.bla.fasel 132BC25B-7774-40D9-AAED-22F9795C8E3F site2
Searchable YES
Status Up
Bucket Count=2412

indexpeer02.bla.fasel 9B5E23F6-3D53-4AAF-805F-DEDF3ACF9D87 site2
Searchable YES
Status Up
Bucket Count=2467

indexpeer03.bla.fasel CCBC4D24-025E-45FB-A68D-9C1A14219D3F site1
Searchable YES
Status Up
Bucket Count=2384

indexpeer04.bla.fasel D649A1C7-9E86-4E48-9B47-144C70029C15 site1
Searchable YES
Status Up
Bucket Count=2475

On our search heads, our search peers are listed by IP-Adress in the attribute PeerURI instead of the fqdn.
How can we change it to the fqdn?
Do other users have the problem as well? Or does TLS validation work even though IP addresses are listed there?

Thanks!

[FlorianScho]

We have the same issue. 
Any news on this how to fix?

[SplunkingKnight]

Hi @FlorianScho,

as a workaround, we have added the IP addresses of our Splunk instances in the TLS certificates as SAN and also included them in the server.conf in the sslAltNameToCheck parameter.

Best regards

[SplunkingKnight]

Hi @DATEVeG,

we have exactly the same problem. Have you found a way to force the use of the FQDN instead of the ip addresses?

Best regards