Configure Splunk forwarding on Windows hosts to use your own certificates

[HumanPrinter]

We are running a Splunk cluster (version 8.1.2) and trying to secure the forwarding from the Universal Forwarders (also version 8.1.2) to the Heavy Forwarders in our cluster.

I've followed the documentation to accomplish this using custom certificates and we have succeeded to secure the traffic between the Universal Forwarders running on Linux and our Heavy Forwarders (also running on Linux). However, the Universal Forwarders on Windows fail to successfully sent their data.

Our configuration is as follows:

We have created a root CA that is shared by all Splunk nodes
We have created a server certificate signed by the root CA that is shared by the Heavy Forwarders
We have created a certificate signed by the root CA that is shared by the Universal Forwarders

The Universal Forwarders contain an app with a outputs.conf with the following content

    [tcpout]

    defaultGroup = ufw_group

    [tcpout:ufw_group]

    server = splunkhf1d:9997

    clientCert = C:\Program Files\SplunkUniversalForwarder\etc\apps\ufw_base\local\splunkUfd_chained.pem

    sslPassword = $7$1x1tBdfWOZKofTNvhO1BD2/EJqF6yzM6fyiGVpqdDWEFQdm8Y1J+SGrN

Note that the sslPassword was pasted in plain text and was encrypted by Splunk upon restart.

The log of the Universal Forwarder shows:

    ERROR AesGcm - Text decryption - error in finalizing: No errors in queue

    ERROR AesGcm - AES-GCM Decryption failed!

    ERROR Crypto - Decryption operation failed: AES-GCM Decryption failed!

    WARN  ConfigEncryptor - Decryption operation failed: AES-GCM Decryption failed!

I have also tried to specify the path to the root CA in the server.conf but this did not help either.

Finally, I have tried to install the Universal Forwarder using the graphical user interface and to specify the certificates in the installation wizard. The strange thing is that the certificate options do not show up in any of the configuration files after the installation is complete and forwarding also does not work.

Has anyone successfully configured forwarding over SSL/TLS from a Windows host or is this only supported on Linux hosts?

[HumanPrinter]

Does anyone have any experience with this?

[youngsuh]

I'd have documentation how to generate the certificate and distribute using DM.  Thus your communication with UF is done with SSL.  Would you be interested in bash script and how to deploy?  Or have you done that already. 

[HumanPrinter · Accepted solution]

We found the solution. Apparently, you can specify a Unix-style path in the configuration. The trick is to use the $SPLUNK_HOME variable to avoid fiddling with Windows drive letters.

We ended up using a path like 'clientCert = $SPLUNK_HOME/etc/apps/ufw_base/local/splunkUfd_chained.pem' in the configuratin and that works like a charm