When enabling sslVerifyServerName for kvstore, the KVStore will not start?

HumanPrinter — Tue, 09 May 2023 07:37:26 GMT

We are trying to implement the guidelines for enabling TLS Hostname verification (Configure TLS certificate host name validation - Splunk Documentation). This is working for most stanzas, but we are facing an issue with enabling the 'sslVerifyServerName' setting for kvstore.

We already had custom certificates in place for all inter-Splunk communications. We are starting to enable the aditional verify-settings and in that process we started with the Deployment Server/SH Deployer.

On that machine we created a server.conf with the following content:

[sslConfig] serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem sslPassword = [REDACTED] sslRootCAPath = /opt/splunk/etc/auth/customcerts/splunk_ca.pem sslVerifyServerCert = true sslVerifyServerName = true cliVerifyServerName = true [kvstore] serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem sslPassword = [REDACTED] sslVerifyServerCert = true sslVerifyServerName = true [search_state] sslVerifyServerCert = true sslVerifyServerName = true [pythonSslClientConfig] sslVerifyServerCert = true sslVerifyServerName = true

The server certificate is valid and contains the hostname and domainname of the server (CN=<host>.<domain>)

If we start Splunk, the following lines appear in splunkd.log:

ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=304029ms ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed. ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details.. ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details. INFO MongodRunner [45938 MongodLogThread] - mongod exited normally (exit code 0, status: PID 45939 exited with code 0).

The Mongo log only contains logging saying it received a signal 15 (Terminated) and that it will shutdown accordingly.

If we remove (or disable) the 'sslVerifyServerName' setting under the kvstore-stanza, everything works fine. Is there anything we missed in this setup? What additional steps of checks are required to enable TLS Hostname verification for the KVStore?

Thanks in advance,
Oscar

Re: When enabling sslVerifyServerName for kvstore, the KVStore will not start?

HumanPrinter — Tue, 09 May 2023 08:01:38 GMT

In the meantime, I've also enabled verbose logging for the KVstore. This however, does nog provide much more help. The mongo logfile now contains many lines saying:
Session from <ipadress> encountered a network error during SourceMessage: SocketException: sslv3 alert bad certificate

Running a curl-command from the host to the KVstore shows a valid SSL-chain:
curl -v --cacert etc/auth/customcerts/splunk_ca.pem https://localhost:8191

Re: When enabling sslVerifyServerName for kvstore, the KVStore will not start?

jkat54 — Thu, 26 Oct 2023 10:15:07 GMT

Check this server.conf setting that you may be missing:

goes under sslOptions

sslCommonNameToCheck = <commonName1>, <commonName2>, ...
* If set, and 'sslVerifyServerCert' is set to "true",
  splunkd limits most outbound HTTPS connections to hosts which
  use a certificate with one of the listed common names.

topic Re: When enabling sslVerifyServerName for kvstore, the KVStore will not start? in Security

When enabling sslVerifyServerName for kvstore, the KVStore will not start?

Re: When enabling sslVerifyServerName for kvstore, the KVStore will not start?

Re: When enabling sslVerifyServerName for kvstore, the KVStore will not start?