We are trying to implement the guidelines for enabling TLS Hostname verification (Configure TLS certificate host name validation - Splunk Documentation). This is working for most stanzas, but we are facing an issue with enabling the 'sslVerifyServerName' setting for kvstore. We already had custom certificates in place for all inter-Splunk communications. We are starting to enable the aditional verify-settings and in that process we started with the Deployment Server/SH Deployer. On that machine we created a server.conf with the following content:     [sslConfig] serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem sslPassword = [REDACTED] sslRootCAPath = /opt/splunk/etc/auth/customcerts/splunk_ca.pem sslVerifyServerCert = true sslVerifyServerName = true cliVerifyServerName = true [kvstore] serverCert = /opt/splunk/etc/auth/customcerts/azsplunkdep1d.management.dev_chained.pem sslPassword = [REDACTED] sslVerifyServerCert = true sslVerifyServerName = true [search_state] sslVerifyServerCert = true sslVerifyServerName = true [pythonSslClientConfig] sslVerifyServerCert = true sslVerifyServerName = true     The server certificate is valid and contains the hostname and domainname of the server (CN=<host>.<domain>) If we start Splunk, the following lines appear in splunkd.log:     ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=Failed to receive response from kvstore error=, service not ready after waiting for timeout=304029ms ERROR KVStoreConfigurationProvider [45937 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed. ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - KV Store changed status to failed. Failed to start KV Store process. See mongod.log and splunkd.log for details.. ERROR KVStoreBulletinBoardManager [45937 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details. INFO MongodRunner [45938 MongodLogThread] - mongod exited normally (exit code 0, status: PID 45939 exited with code 0).     The Mongo log only contains logging saying it received a signal 15 (Terminated) and that it will shutdown accordingly.   If we remove (or disable) the 'sslVerifyServerName' setting under the kvstore-stanza, everything works fine. Is there anything we missed in this setup? What additional steps of checks are required to enable TLS Hostname verification for the KVStore? Thanks in advance, Oscar ... View more

We have a Splunk cluster running which consists of search heads, indexers, heavy forwarders and other Splunk instances (e.g. deployment server, cluster master, ...) and many Universal Forwarders We want to encrypt all inter-Splunk communications (both inside the Splunk-cluster and between Universal Forwarders and Heavy Forwarders) with custom certificates which are signed by a custom root CA. Initially, this should not be a problem since there is plenty of documentation on this subject. However, we cannot find any documentation for the scenario in which the root CA needs to be renewed. How can this be done without any downtime (or at least a minimum downtime)? All the scenario's we have seen soo far require a big bang approach in which the cluster and Universal Forwarders will not operate properly untill all the servers and clients have the new root CA. Does anyone have any thoughts on this subject? Thanks ... View more

We are running a Universal Forwarder on our Windows servers which host several of our application. Each application logs to the same Windows Event Logbook, but use different sources to be able to determine the source of the logging. We have configured the UF to forward all messages from the logbook to the heavy forwarders according to the inputs.conf-snippet below: [WinEventLog://ServerApps] disabled = 0 sourcetype = ServerAppLogs source = ServerApp renderXml = false The problem is, whenever a new application is deployed, it will register a new source with the Windows Event logbook (using Powershell: [System.Diagnostics.EventLog]::CreateEventSource($eventSourceName, $eventLogName) ). But the UF does not pick up this new source untill it is restarted even though we did not specify any whitelisting or blacklisting. Is it possible to make the UF listen to new sources in the Windows Event Log without having to restart it? PS. We are using version 7.3.1 of the Universal Forwarder ... View more