How do I read SSL Certificates from Custom Folder?

[ksextonmacb]

I tried to get my indexer and forwarders communicating using SSL by following this guide. However, I found that I couldn't get it working without just throwing all of the certificates into auth folder. If I do that it works, but if I try to put the certificates outside of that folder it doesn't. The problem is that updating splunk will overwrite that folder, so I'm trying to keep my certificates in a different folder, which should be possible. I've tried setting the variables I can find to point to the new certificate location, but it provides the following error in splunkd.log and doesn't forward data.

12-02-2015 12:01:32.070 -0500 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/certs/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.

Because server.pem works fine inside of the auth folder, I'm guessing that the problem is with supporting files. So far I've tried using outputs.conf sslCertPath and sslRootCAPath, as well as server.conf caPath, sslKeysfile, and caCertFile. Below is the contents of my auth folder.

-rwxr-x--- 1 splunk splunk 3050 Dec 2 12:54 ca.pem
-rwxr-x--- 1 splunk splunk 17 Dec 2 12:54 ca.srl
-rwxr-x--- 1 splunk splunk 1216 Dec 2 12:54 cacert.pem
-rwxr-x--- 1 splunk splunk 1834 Dec 2 12:54 cakey.pem
-rwxr-x--- 1 splunk splunk 1013 Dec 2 12:54 careq.pem
-rw------- 1 splunk splunk 1041 Dec 2 12:54 privKeySecure.pem
-rw------- 1 splunk splunk 566 Dec 2 12:54 req.pem
-rwxr-x--- 1 splunk splunk 4386 Dec 2 12:54 server.pem
-r-------- 1 splunk splunk 255 Dec 2 12:54 splunk.secret
drwx------ 2 splunk splunk 512 Dec 2 12:54 splunkweb

I provided ca.pem, ca.srl, cacert.pem, cakey.pem, careq.pem, server.pem from the certificate generation process; I mirrored all of my certificate names with the default file names. Those files (and only those files) are in the certs folder, which is the folder I made that I want to read certs from. Is there some other Certificate location I'm failing to point to? Or is this a problem somewhere else?

Here's my working outputs.conf:

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 129.52.27.30:9997
compressed = true

[tcpout-server://129.52.27.30:9997]
sslAltNameToCheck = winsplunk
sslCertPath = /opt/splunkforwarder/etc/auth/server.pem
sslCommonNameToCheck = winsplunk
sslPassword = totallyFunctionalHash
sslRootCAPath = /opt/splunkforwarder/etc/auth/cacert.pem
sslVerifyServerCert = true

Here's my working server.conf:

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = AnotherFunctionalHash
serverName = afemssplunk

[sslConfig]
sslKeysfilePassword = SameHashAsOutputs.confSSLPassword,CauseThey'reTheSamePassword
cipherSuite = TLSv1+HIGH:@STRENGTH
sslVersions = tls,-ssl2,-ssl3

Here's my failing outputs.conf:

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 129.52.27.30:9997
compressed = true

[tcpout-server://129.52.27.30:9997]
sslAltNameToCheck = winsplunk
sslCertPath = /opt/splunkforwarder/etc/certs/server.pem
sslCommonNameToCheck = winsplunk
sslPassword = totallyFunctionalHash
sslRootCAPath = /opt/splunkforwarder/etc/certs/cacert.pem
sslVerifyServerCert = true

Here's my failing server.conf:

[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder

[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free

[general]
pass4SymmKey = AnotherFunctionalHash
serverName = afemssplunk

[sslConfig]
sslKeysfilePassword = SameHashAsOutputs.confSSLPassword,CauseThey'reTheSamePassword
cipherSuite = TLSv1+HIGH:@STRENGTH
sslVersions = tls,-ssl2,-ssl3
sslKeysfile = server.pem
caCertFile = cacert.pem
caPath = /opt/splunkforwarder/etc/certs

[ksextonmacb · Accepted solution]

I ended up moving my certificate files into the auth folder to replace the existing files, and then copying the entire auth folder into a second folder I named macbAuth. I did not copy the splunkweb folder from the auth folder; I lacked the necessary permissions. However it seems those files aren't needed. The following configuration worked.

Here's my file structure:

C:\Program Files\SplunkUniversalForwarder\etc\macbAuth>dir
 Volume in drive C is New Volume
 Volume Serial Number is 3A13-D7AF

 Directory of C:\Program Files\SplunkUniversalForwarder\etc\macbAuth

12/07/2015  11:07 AM    <DIR>          .
12/07/2015  11:07 AM    <DIR>          ..
11/12/2015  10:50 AM             1,024 .rnd
10/30/2015  04:08 AM             3,908 appsCA.pem
11/12/2015  10:32 AM             3,050 ca.pem
10/30/2015  04:08 AM             1,860 ca.pem.default
11/12/2015  10:51 AM                17 ca.srl
11/12/2015  10:32 AM             1,216 cacert.pem
10/30/2015  04:08 AM               928 cacert.pem.default
11/12/2015  10:32 AM             1,834 cakey.pem
11/12/2015  10:32 AM             1,013 careq.pem
12/03/2015  02:43 PM             1,041 privKeySecure.pem
12/03/2015  02:43 PM               570 req.pem
11/12/2015  10:51 AM             4,386 server.pem
12/03/2015  02:43 PM               255 splunk.secret
12/03/2015  02:43 PM    <DIR>          splunkweb
              13 File(s)         21,102 bytes
               3 Dir(s)  138,427,678,720 bytes free

Here's my outputs.conf:

 [tcpout]
 defaultGroup = splunkssl

 [tcpout:splunkssl]
 server = 129.52.27.30:9997
 compressed = true

 [tcpout-server://129.52.27.30:9997]
 sslAltNameToCheck = winsplunk
 sslCertPath = /opt/splunkforwarder/etc/macbAuth/server.pem
 sslCommonNameToCheck = winsplunk
 sslPassword = totallyFunctionalHash
 sslRootCAPath = /opt/splunkforwarder/etc/macbAuth/cacert.pem
 sslVerifyServerCert = true

Here's my server.conf:

[lmpool:auto_generated_pool_forwarder]
 description = auto_generated_pool_forwarder
 quota = MAX
 slaves = *
 stack_id = forwarder

 [lmpool:auto_generated_pool_free]
 description = auto_generated_pool_free
 quota = MAX
 slaves = *
 stack_id = free

 [general]
 pass4SymmKey = AnotherFunctionalHash
 serverName = afemssplunk

 [sslConfig]
 sslKeysfilePassword = SameHashAsOutputs.confSSLPassword,CauseThey'reTheSamePassword
 cipherSuite = TLSv1+HIGH:@STRENGTH
 sslVersions = tls,-ssl2,-ssl3
 sslKeysfile = server.pem
 caCertFile = cacert.pem
 caPath = /opt/splunkforwarder/etc/macbAuth

[jplumsdaine22]

That guide is from 5 years ago. Splunk has more than likely changed since then 🙂

Check the documentation here:
http://docs.splunk.com/Documentation/Splunk/6.3.1/Security/Aboutsecuringdatafromforwarders (or whichever versionyou are using - theres a version picker at the top right of the page)

Your configs look different to the current config stanzas

[starcher]

You might go through this as reference.
http://www.georgestarcher.com/wp-content/uploads/2015/09/conf2015_DWaddle_DefensePointSecurity_deplo...