I'm trying to connect multiple Windows Splunk Universal Forwarders to a TCP receiver. I am running in Splunk Enterprise. I have this working without any issues, making a non-SSL connection, but I want the connection that is made between the Universal Forwarder and the Receiver to be an SSL connection. I do not need the universal forwarder to be authenticated on the Receiver's end (it appears you can require the receiver to verify certs created on the UF to allow the UF as a client).
I have already created self-signed certs on the Receiver (I'm not using Splunk's default certs). By exporting the root public key certificate from the Receiver server, I have been able to successfully get a REST API connection to the receiver server over SSL. My assumption is that the UF should also be able to connect to the Receiver using only the root public key certificate just like a REST API client. However, that post (and everything else I have found on the matter) seems to indicate that the UF needs to provide a "sslCertPath" PEM file that contains the private key. This strikes me as a significant security risk. The private key from my receiver being distributed to every one of my UFs introduces the risk that the key will be used to create certificates that are used maliciously.
I have tried a number of ways to use only the root public key certificate from my Receiver to get a UF connecting via SSL to my Receiver, but none seem to work. Is there a way to do this, or am I running into a quirk in the design of the connections made between a UF and a Receiver? The only possible solution I can think of is I need to create a separate private key that is shared by all the UFs, create a "server" certificate with that private key, distribute that certificate with all the UFs, and use the root certificate (which signs the UF "server" certificate") on the Receiver to verify the UF "server" certificate. But even that creates issues with the common name being different for every UF. It seems so backward to me, like the Receiver is acting as a client instead of a server. Maybe I'm missing something.
The password is initially in clear text but Splunk encrypts it on first use.
I created (but can't share - sorry) a shell script that checks and copies new certs into etc/auth/mydir and has the password in the clear for deployment. It restarts the forwarder so that the password is encrypted and the password can then be removed from the deployment server to minimise the time it's kept anywhere in the clear. Not ideal but possibly as good as Splunk can get it today and still be mass deployed.