When enabling sslVerifyServerName for kvstore, the KVStore will not start?

[HumanPrinter]

We are trying to implement the guidelines for enabling TLS Hostname verification (Configure TLS certificate host name validation - Splunk Documentation). This is working for most stanzas, but we are facing an issue with enabling the 'sslVerifyServerName' setting for kvstore.

We already had custom certificates in place for all inter-Splunk communications. We are starting to enable the aditional verify-settings and in that process we started with the Deployment Server/SH Deployer.

On that machine we created a server.conf with the following content:

[sslConfig]
serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem
sslPassword = [REDACTED]
sslRootCAPath = /opt/splunk/etc/auth/customcerts/splunk_ca.pem
sslVerifyServerCert = true
sslVerifyServerName = true
cliVerifyServerName = true

[kvstore]
serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem
sslPassword = [REDACTED]
sslVerifyServerCert = true
sslVerifyServerName = true

[search_state]
sslVerifyServerCert = true
sslVerifyServerName = true

[pythonSslClientConfig]
sslVerifyServerCert = true
sslVerifyServerName = true

The server certificate is valid and contains the hostname and domainname of the server (CN=<host>.<domain>)

If we start Splunk, the following lines appear in splunkd.log:

ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=304029ms
ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed.
ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details..
ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details.
INFO  MongodRunner [45938 MongodLogThread] - mongod exited normally (exit code 0, status: PID 45939 exited with code 0).

The Mongo log only contains logging saying it received a signal 15 (Terminated) and that it will shutdown accordingly.

If we remove (or disable) the 'sslVerifyServerName' setting under the kvstore-stanza, everything works fine. Is there anything we missed in this setup? What additional steps of checks are required to enable TLS Hostname verification for the KVStore?

Thanks in advance,
Oscar

[jkat54]

Check this server.conf setting that you may be  missing:

goes under sslOptions

sslCommonNameToCheck = <commonName1>, <commonName2>, ...
* If set, and 'sslVerifyServerCert' is set to "true",
  splunkd limits most outbound HTTPS connections to hosts which
  use a certificate with one of the listed common names.

[HumanPrinter]

In the meantime, I've also enabled verbose logging for the KVstore. This however, does nog provide much more help. The mongo logfile now contains many lines saying:
Session from <ipadress> encountered a network error during SourceMessage: SocketException: sslv3 alert bad certificate

Running a curl-command from the host to the KVstore shows a valid SSL-chain:
curl -v --cacert etc/auth/customcerts/splunk_ca.pem https://localhost:8191