We are running a Splunk cluster (version 8.1.2) and trying to secure the forwarding from the Universal Forwarders (also version 8.1.2) to the Heavy Forwarders in our cluster. I've followed the documentation to accomplish this using custom certificates and we have succeeded to secure the traffic between the Universal Forwarders running on Linux and our Heavy Forwarders (also running on Linux). However, the Universal Forwarders on Windows fail to successfully sent their data. Our configuration is as follows: We have created a root CA that is shared by all Splunk nodes We have created a server certificate signed by the root CA that is shared by the Heavy Forwarders We have created a certificate signed by the root CA that is shared by the Universal Forwarders The Universal Forwarders contain an app with a outputs.conf with the following content     [tcpout]     defaultGroup  = ufw_group     [tcpout:ufw_group]      server  = splunkhf1d:9997     clientCert  = C:\Program Files\SplunkUniversalForwarder\etc\apps\ufw_base\local\splunkUfd_chained.pem     sslPassword  = $7$1x1tBdfWOZKofTNvhO1BD2/EJqF6yzM6fyiGVpqdDWEFQdm8Y1J+SGrN   Note that the sslPassword was pasted in plain text and was encrypted by Splunk upon restart.   The log of the Universal Forwarder shows:     ERROR  AesGcm - Text decryption - error in finalizing: No errors in queue     ERROR  AesGcm - AES-GCM Decryption failed!     ERROR  Crypto - Decryption operation failed: AES-GCM Decryption failed!     WARN   ConfigEncryptor - Decryption operation failed: AES-GCM Decryption failed!   I have also tried to specify the path to the root CA in the server.conf but this did not help either. Finally, I have tried to install the Universal Forwarder using the graphical user interface and to specify the certificates in the installation wizard. The strange thing is that the certificate options do not show up in any of the configuration files after the installation is complete and forwarding also does not work.   Has anyone successfully configured forwarding over SSL/TLS from a Windows host or is this only supported on Linux hosts? ... View more

Last week we upgraded our Splunk-cluster from version 7.3.5 to 7.3.6. Since that moment, alerts that are triggered no longer are able to send mail. The _internal index shows an event stating " ERROR sendemail:461 - ' rootCAPath ' while sending mail to : xxx@xx " From other posts it seems to be required to add the list_settings capability to our user roles. However, prior to the update we have had no problems with alert mails without adding this capability to user roles. The release notes for version 7.3.6 don't mention any fix or change in this regard. Since the documentation is not quite clear about the impact of adding this capability to a user role (what additional possibilities are available to users with this capability) and this didn't seem to be required up until version 7.3.5 we would like to be sure this capability won't harm our setup ... View more

We have a Splunk cluster running which consists of search heads, indexers, heavy forwarders and other Splunk instances (e.g. deployment server, cluster master, ...) and many Universal Forwarders We want to encrypt all inter-Splunk communications (both inside the Splunk-cluster and between Universal Forwarders and Heavy Forwarders) with custom certificates which are signed by a custom root CA. Initially, this should not be a problem since there is plenty of documentation on this subject. However, we cannot find any documentation for the scenario in which the root CA needs to be renewed. How can this be done without any downtime (or at least a minimum downtime)? All the scenario's we have seen soo far require a big bang approach in which the cluster and Universal Forwarders will not operate properly untill all the servers and clients have the new root CA. Does anyone have any thoughts on this subject? Thanks ... View more

We are running a Universal Forwarder on our Windows servers which host several of our application. Each application logs to the same Windows Event Logbook, but use different sources to be able to determine the source of the logging. We have configured the UF to forward all messages from the logbook to the heavy forwarders according to the inputs.conf-snippet below: [WinEventLog://ServerApps] disabled = 0 sourcetype = ServerAppLogs source = ServerApp renderXml = false The problem is, whenever a new application is deployed, it will register a new source with the Windows Event logbook (using Powershell: [System.Diagnostics.EventLog]::CreateEventSource($eventSourceName, $eventLogName) ). But the UF does not pick up this new source untill it is restarted even though we did not specify any whitelisting or blacklisting. Is it possible to make the UF listen to new sources in the Windows Event Log without having to restart it? PS. We are using version 7.3.1 of the Universal Forwarder ... View more