We are running a Splunk cluster (version 8.1.2) and trying to secure the forwarding from the Universal Forwarders (also version 8.1.2) to the Heavy Forwarders in our cluster. I've followed the documentation to accomplish this using custom certificates and we have succeeded to secure the traffic between the Universal Forwarders running on Linux and our Heavy Forwarders (also running on Linux). However, the Universal Forwarders on Windows fail to successfully sent their data. Our configuration is as follows: We have created a root CA that is shared by all Splunk nodes We have created a server certificate signed by the root CA that is shared by the Heavy Forwarders We have created a certificate signed by the root CA that is shared by the Universal Forwarders The Universal Forwarders contain an app with a outputs.conf with the following content     [tcpout]     defaultGroup  = ufw_group     [tcpout:ufw_group]      server  = splunkhf1d:9997     clientCert  = C:\Program Files\SplunkUniversalForwarder\etc\apps\ufw_base\local\splunkUfd_chained.pem     sslPassword  = $7$1x1tBdfWOZKofTNvhO1BD2/EJqF6yzM6fyiGVpqdDWEFQdm8Y1J+SGrN   Note that the sslPassword was pasted in plain text and was encrypted by Splunk upon restart.   The log of the Universal Forwarder shows:     ERROR  AesGcm - Text decryption - error in finalizing: No errors in queue     ERROR  AesGcm - AES-GCM Decryption failed!     ERROR  Crypto - Decryption operation failed: AES-GCM Decryption failed!     WARN   ConfigEncryptor - Decryption operation failed: AES-GCM Decryption failed!   I have also tried to specify the path to the root CA in the server.conf but this did not help either. Finally, I have tried to install the Universal Forwarder using the graphical user interface and to specify the certificates in the installation wizard. The strange thing is that the certificate options do not show up in any of the configuration files after the installation is complete and forwarding also does not work.   Has anyone successfully configured forwarding over SSL/TLS from a Windows host or is this only supported on Linux hosts? ... View more

Last week we upgraded our Splunk-cluster from version 7.3.5 to 7.3.6. Since that moment, alerts that are triggered no longer are able to send mail. The _internal index shows an event stating " ERROR sendemail:461 - ' rootCAPath ' while sending mail to : xxx@xx " From other posts it seems to be required to add the list_settings capability to our user roles. However, prior to the update we have had no problems with alert mails without adding this capability to user roles. The release notes for version 7.3.6 don't mention any fix or change in this regard. Since the documentation is not quite clear about the impact of adding this capability to a user role (what additional possibilities are available to users with this capability) and this didn't seem to be required up until version 7.3.5 we would like to be sure this capability won't harm our setup ... View more

I am trying to configure an NGINX-proxy that will forward all requests from a Universal Forwarder to the Deploymentserver in my Splunk-cluster. I have tried to follow the examples that are available on the internet as wel as on Splunkbase. However, I run into an error which I can't seem to solve. My setup is as follows: A windows machine running the Universal Forwarder (version 7.1.2 x64) A CentOS 7.5 machine running Splunk as a deploymentserver (version 7.2.0) A CentOS 7.5 machine running NGINX (version 1.12.2) When I connect the Universal Forwarder directly to the deploymentserver, everything works fine. But when I connect with the NGINX-proxy, the communication breaks down. NGINX reports a '502 Bad Gateway' and the deploymentserver reports 'WARN HttpListener - Socket error from 10.101.0.243:40076 while idling: error:1408F10B:SSL routines:SSL3_GET_RECORD:wrong version number' My NGINX config looks as follows: user nginx nginx; worker_processes auto; pid /var/run/nginx.pid; error_log /var/log/nginx/error.log; events { worker_connections 1024; use epoll; } http { log_format main_http '$remote_addr - $remote_user [$time_local] $ssl_protocol/$ssl_cipher "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; upstream splunk_deploymentserver { ip_hash; server 10.101.0.237:8089; } server { listen 8089 ssl; ssl_certificate ssl/server.pem; ssl_certificate_key ssl/server.key; access_log /var/log/nginx/access_splunkdeployment.log main_http; location / { proxy_pass http://splunk_deploymentserver; } } } The SSL-certificate that is served by NGINX, is a copy of the SSL-certificate of the deployment server. Any help on solving this problem is very much appreciated. Oscar ... View more