splunk fowarder SSL error ERROR TcpOutputProc

[mcalautti]

on the forwarder I get
ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server

right before that I see
ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/certs/forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.

not sure what that means.
I took the existing certs DIR from an existing splunk build that is working fine with the forwarder

the indexer is version 4.1.5, build 85165
the forwarder is splunkforwarder-4.2.3-105575-Linux-x86_64

output from openssl x509 -inform PEM -in forwarder.pem -text -noout

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
8e:69:04:62:da:36:fa:2b
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=CA, L=SanFrancisco, O=SplunkInc, CN=SplunkCA, O=SplunkUser
Validity
Not Before: Sep 21 18:29:40 2011 GMT
Not After : Sep 20 18:29:40 2014 GMT
Subject: C=US, ST=MA, L=Bedford, O=sb
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:a8:d4:41:84:b2:9f:3c:3f:7f:c3:a2:3d:54:9d:
7f:0b:52:53:73:37:35:85:99:04:ce:09:40:68:38:
5c:ed:0f:52:ff:89:31:e8:b7:c7:f6:82:8d:bc:12:
fa:34:e1:53:65:47:af:4c:5d:03:ab:a0:7f:64:3b:
b2:24:b8:da:db:4e:16:fb:09:3e:11:1f:aa:5e:b3:
b2:20:d6:78:99:3d:ed:c8:74:5b:94:e9:b2:bb:12:
c6:db:85:fa:4c:ec:f3:8b:41:28:6b:03:2e:e4:c6:
11:d4:47:ec:21:c5:8a:70:e0:2f:64:bd:e5:28:f7:
a8:c9:a3:8d:e6:f1:10:b8:59
Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
71:91:88:bd:22:cb:aa:45:3c:9e:ec:43:8d:a3:78:cb:d2:f1:
b6:16:a3:66:80:ba:68:55:c8:18:0d:6f:a9:1a:2a:c2:f3:cf:
a0:c2:b6:0a:f1:8b:f4:9b:e1:e4:70:d3:5c:8c:b1:75:2f:bf:
bd:fc:de:e1:bd:c9:a1:ac:54:fe:99:3e:d1:29:9a:a6:9b:b5:
7c:d3:2f:4e:4e:f2:f6:af:a1:0b:cc:e1:d2:e7:1d:3a:27:0c:
7d:21:4e:78:1e:d1:45:32:da:79:0c:b0:8a:48:3f:3f:fa:23:
3e:71:6b:6f:63:c8:21:ee:c8:a6:86:4e:18:b7:40:52:22:29:
48:38

[nrpeter]

This error occurs when splunk tries to decrypt the RSA private key in server.pem with the wrong password.

I'll show you how to fix this by decrypting the cert by hand as long as you are using default splunk signing CAs.

 1. Delete server.pem from your universal forwarder and restart splunk (this will recreate server.pem with a default password of 'password')
 2. Decrypt the private RSA key 'openssl rsa -in server.pem -out server_unprotected.pem'
 3. Since server_unprotected.pem will only have the decrypted RSA private key in it, copy & paste the other two blocks from server.pem (don't copy over the original encrypted RSA private key)
 4. mv splunk_unprotected.pem server.pem
 5. Restart splunk

[karthi4k]

It works like a charm. Thanks peter. One typo in the point 4 where it should be server.pem and not splunk.pem

[wqiu_splunk]

this should be the answer indeed!!!!

[dhillny]

life saver bro, other than the typo on #4. This is a solid fix.

[moseisleydk]

We just ran into this using conf files working fine on a OpenSolaris 32-bit - not working on Ubuntu 10.04 TLS (64-bit)

The problem was the decryption of the password, pasting the real in as plain text worked, and the resulting encrypted string was different on the Ubuntu.

[andyspusm]

I just saw the same behavior between two Windows 7 Professional x64 machines - pasting the password in plaintext worked. Thanks for the idea.

[mcalautti · Accepted solution]

I just ended up recreating and it worked.. via

http://splunk-base.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certi...

[mikelanghorst]

I'd start with looking at the forwarder.pem file with vi and openssl first.

With vi, it should be readable text with "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" blocks. Next use openssl and inspect the data returned:
openssl x509 -inform PEM -in -text -noout

[mcalautti]

ok.. I see the CERT in the file..

i added the output from openssl to the description.. looks ok to me.. what do you think ?
thanks
mike