We already had custom certificates in place for all inter-Splunk communications. We are starting to enable the aditional verify-settings and in that process we started with the Deployment Server/SH Deployer.
On that machine we created a server.conf with the following content:
The server certificate is valid and contains the hostname and domainname of the server (CN=<host>.<domain>)
If we start Splunk, the following lines appear in splunkd.log:
ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=304029ms
ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed.
ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details..
ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details.
INFO MongodRunner [45938 MongodLogThread] - mongod exited normally (exit code 0, status: PID 45939 exited with code 0).
The Mongo log only contains logging saying it received a signal 15 (Terminated) and that it will shutdown accordingly.
If we remove (or disable) the 'sslVerifyServerName' setting under the kvstore-stanza, everything works fine. Is there anything we missed in this setup? What additional steps of checks are required to enable TLS Hostname verification for the KVStore?
In the meantime, I've also enabled verbose logging for the KVstore. This however, does nog provide much more help. The mongo logfile now contains many lines saying: Session from <ipadress> encountered a network error during SourceMessage: SocketException: sslv3 alert bad certificate
Running a curl-command from the host to the KVstore shows a valid SSL-chain: curl -v --cacert etc/auth/customcerts/splunk_ca.pem https://localhost:8191