we tried to enable TLS validation with Splunk 9.0.2 as described in the Splunk documentation. Unfortunately, this caused our distributed Splunk system consisting of index cluster and searchhead cluster to stop working. Specifically, searches could no longer be performed.
We discovered the following messages in the search head log:
11-06-2022 23:59:59.839 +0100 ERROR DistributedPeerManagerHeartbeat [95344 DistributedPeerMonitorThread] - Send failure while pushing public key to search peer = https://10.10.10. 10:8089 , error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed - please check the output of the `openssl verify` command for the certificates involved; note that if certificate verification is enabled (requireClientCert or sslVerifyServerCert set to "true"), the CA certificate and the server certificate should not have the same Common Name.
This makes us wonder, because actually the connection for the TLS connection should be established over fqdn and not over an ip address. When establishing a connection over the IP, it is also logical that the TLS name check against the host name fails, since only the FQDN and no IP address is stored in the TLS certificate.
We therefore wondered why the search head addresses our index peer 10.10.10.10 via the IP and not the fqdn. The Splunk documentation states that the search head gets the list of index peers from the cluster manager. However, in the cluster manager, our index peers also report by name, at least that is what the output of the cluster master suggests:
>> splunk/bin/splunk show cluster-status
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.
Replication factor met
Search factor met
All data is searchable
Indexing Ready YES
HA Mode: Disabled
indexpeer01.bla.fasel 132BC25B-7774-40D9-AAED-22F9795C8E3F site2
indexpeer02.bla.fasel 9B5E23F6-3D53-4AAF-805F-DEDF3ACF9D87 site2
indexpeer03.bla.fasel CCBC4D24-025E-45FB-A68D-9C1A14219D3F site1
indexpeer04.bla.fasel D649A1C7-9E86-4E48-9B47-144C70029C15 site1
On our search heads, our search peers are listed by IP-Adress in the attribute PeerURI instead of the fqdn.
How can we change it to the fqdn?
Do other users have the problem as well? Or does TLS validation work even though IP addresses are listed there?