How to configure Splunk to use a custom CA SSL certificate for all Splunkd traffic?

[jcspigler2010]

Hello Fellow Splunkers,

I am about to lose my mind! At one time I was able to configure Splunk to use a custom CA cert for all splunkd traffic. Now I cannot, no matter what I do/try/wish/pray for. Let me explain steps I took up to my current status. I have a Microsoft enterprise CA, I issued a new certificate with the EKU for server auth and client auth. I exported it as a pfx file with both the private and public key. I then used open SSL to extract the public key, the private key and converted it to .pem format. I left the passphrase intact on the private key. Next just as stated in the following instructions:

https://docs.splunk.com/Documentation/Splunk/6.5.1/Security/HowtoprepareyoursignedcertificatesforSpl...

Where you cat all these certificates together in this order serverCert.pem > serverKey.pem > IntCACert.pem > RootCACert.pem >> bigCertBundle.pem

I then dropped this in a directory called "C:\Program Files\SplunkUniversalForwarder\etc\auth\myCerts" called "universalForwarderCertBundle.pem"

Next I created a custom app with the following configurations for server.conf

[sslConfig]
serverCert = "C:\Program Files\SplunkUniversalForwarder\etc\auth\myCerts\universalForwarderCertBundle.pem"
sslPassword = Password123

I also commented out the sslPassword attribute located in $SPLUNK_HOME\etc\system\local\server.conf

Permissions are correct on the cert bundle. The Universal Forwarder is being ran by Local System like any other Windows install with a forwarder.

Now, when I start Splunk up it fails with pre-fligh-check errors

ERROR loader - The certificate generation script did not generate the expected certificate file:C:\Program Files\SplunkUniversalForwarder\etc\auth\"C:\Program Files\SplunkUniversalForwarder\etc\auth\myCerts\universalForwarderCertBundle.pem". Splunkd port communication will not work

 -0500 ERROR loader - SSL certificate generation failed.

Heres a btool dump

c:\Program Files\SplunkUniversalForwarder\etc\apps\verbosity_all_forwarderSSL_manage\local\server.conf [sslConfig]
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               allowSslCompression = true
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               allowSslRenegotiation = true
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               caCertFile = $SPLUNK_HOME\etc\auth\cacert.pem
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               caPath = $SPLUNK_HOME\etc\auth
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               certCreateScript = $SPLUNK_HOME\bin\splunk, createssl, server-cert
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               enableSplunkdSSL = true
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               sendStrictTransportSecurityHeader = false
c:\Program Files\SplunkUniversalForwarder\etc\apps\verbosity_all_forwarderSSL_manage\local\server.conf serverCert = "C:\Program Files\SplunkUniversalForwarder\etc\auth\myCerts\universalForwarderCertBundle.pem"
c:\Program Files\SplunkUniversalForwarder\etc\apps\verbosity_all_forwarderSSL_manage\local\server.conf sslPassword = Password123
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               sslVersions = *,-ssl2
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               sslVersionsForClient = *,-ssl2
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               useClientSSLCompression = true
c:\Program Files\SplunkUniversalForwarder\etc\system\default\server.conf                               useSplunkdClientSSLCompression = true

The sslPassword does not hash, Splunk does not start and I'm left scratching my head. Can I not configure the new cert to be used in a new path? Any insight on this would be awesome.

[jwelch_splunk · Accepted solution]

If you take a close look at your Error, you have c:\pro*c:\pro* two times.

In your outputs.conf change your output.conf to this:

serverCert = C:\Program Files\SplunkUniversalForwarder\etc\auth\myCerts\universalForwarderCertBundle.pem
Notice I removed the double quotes, if for whatever reason that does not fix the issue which I suspect it will, then try this version:

serverCert = $SPLUNK_HOME/etc/auth/MyCerts/universalForwarderCertBundle.pem

I know I know it is windows right. Let me know if either one of these fix it for you if not I will set it up and figure it out.

Okie

[jcspigler2010]

Thanks for the reply jwelch. This is actually for management traffic over port 8089 so the change is done in the server.conf. But I finally figured out what my problem was. And you are right in your answer, splunk didn't like using a full path name in Windows speak C:\Program Files\SplunkUniversalForwarder\etc\auth\myCerts\universalForwarderCertBundle.pem. With or with quotes. As soon as I replace this with $SPLUNK_HOME/etc/auth/MyCerts/universalForwarderCertBundle.pem everything started working. Does splunk choked on "\" characters in configuration files?

[gowthammahes]

thankyou. it worked 

[jwelch_splunk]

I think the real issue was the double quotes, normally we (admins) are using those because of the space in Program Files but in our Splunk conf files we don't need or want them.

Sorry I missed the right conf file. As far as the slashes go, AFAIK it sometimes works either way. Best bet is to just look at the /opt/splunk/etc/system/README/server.conf.example or inputs.conf.example or the spec's. I get confused and just look it up from time to time.

Okie