Security

When enabling sslVerifyServerName for kvstore, the KVStore will not start?

HumanPrinter
Explorer

We are trying to implement the guidelines for enabling TLS Hostname verification (Configure TLS certificate host name validation - Splunk Documentation). This is working for most stanzas, but we are facing an issue with enabling the 'sslVerifyServerName' setting for kvstore.

We already had custom certificates in place for all inter-Splunk communications. We are starting to enable the aditional verify-settings and in that process we started with the Deployment Server/SH Deployer.

On that machine we created a server.conf with the following content:

 

 

[sslConfig]
serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem
sslPassword = [REDACTED]
sslRootCAPath = /opt/splunk/etc/auth/customcerts/splunk_ca.pem
sslVerifyServerCert = true
sslVerifyServerName = true
cliVerifyServerName = true

[kvstore]
serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem
sslPassword = [REDACTED]
sslVerifyServerCert = true
sslVerifyServerName = true

[search_state]
sslVerifyServerCert = true
sslVerifyServerName = true

[pythonSslClientConfig]
sslVerifyServerCert = true
sslVerifyServerName = true

 

 

The server certificate is valid and contains the hostname and domainname of the server (CN=<host>.<domain>)

If we start Splunk, the following lines appear in splunkd.log:

 

 

ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=304029ms
ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed.
ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details..
ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details.
INFO  MongodRunner [45938 MongodLogThread] - mongod exited normally (exit code 0, status: PID 45939 exited with code 0).

 

 

The Mongo log only contains logging saying it received a signal 15 (Terminated) and that it will shutdown accordingly.

 

If we remove (or disable) the 'sslVerifyServerName' setting under the kvstore-stanza, everything works fine. Is there anything we missed in this setup? What additional steps of checks are required to enable TLS Hostname verification for the KVStore?

Thanks in advance,
Oscar

Labels (3)
Tags (4)
0 Karma

HumanPrinter
Explorer

In the meantime, I've also enabled verbose logging for the KVstore. This however, does nog provide much more help. The mongo logfile now contains many lines saying:
Session from <ipadress> encountered a network error during SourceMessage: SocketException: sslv3 alert bad certificate

Running a curl-command from the host to the KVstore shows a valid SSL-chain:
curl -v --cacert etc/auth/customcerts/splunk_ca.pem https://localhost:8191

0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...