topic Re: splunk fowarder SSL error ERROR TcpOutputProc in Getting Data In

splunk fowarder SSL error ERROR TcpOutputProc

mcalautti — Mon, 28 Sep 2020 10:02:39 GMT

on the forwarder I get
ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server

right before that I see
ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/certs/forwarder.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.

not sure what that means.
I took the existing certs DIR from an existing splunk build that is working fine with the forwarder

the indexer is version 4.1.5, build 85165
the forwarder is splunkforwarder-4.2.3-105575-Linux-x86_64

output from openssl x509 -inform PEM -in forwarder.pem -text -noout

Certificate:
Data:
Version: 1 (0x0)
Serial Number:
8e:69:04:62:da:36:fa:2b
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=CA, L=SanFrancisco, O=SplunkInc, CN=SplunkCA, O=SplunkUser
Validity
Not Before: Sep 21 18:29:40 2011 GMT
Not After : Sep 20 18:29:40 2014 GMT
Subject: C=US, ST=MA, L=Bedford, O=sb
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public Key: (1024 bit)
Modulus (1024 bit):
00:a8:d4:41:84:b2:9f:3c:3f:7f:c3:a2:3d:54:9d:
7f:0b:52:53:73:37:35:85:99:04:ce:09:40:68:38:
5c:ed:0f:52:ff:89:31:e8:b7:c7:f6:82:8d:bc:12:
fa:34:e1:53:65:47:af:4c:5d:03:ab:a0:7f:64:3b:
b2:24:b8:da:db:4e:16:fb:09:3e:11:1f:aa:5e:b3:
b2:20:d6:78:99:3d:ed:c8:74:5b:94:e9:b2:bb:12:
c6:db:85:fa:4c:ec:f3:8b:41:28:6b:03:2e:e4:c6:
11:d4:47:ec:21:c5:8a:70:e0:2f:64:bd:e5:28:f7:
a8:c9:a3:8d:e6:f1:10:b8:59
Exponent: 65537 (0x10001)
Signature Algorithm: sha1WithRSAEncryption
71:91:88:bd:22:cb:aa:45:3c:9e:ec:43:8d:a3:78:cb:d2:f1:
b6:16:a3:66:80:ba:68:55:c8:18:0d:6f:a9:1a:2a:c2:f3:cf:
a0:c2:b6:0a:f1:8b:f4:9b:e1:e4:70:d3:5c:8c:b1:75:2f:bf:
bd:fc:de:e1:bd:c9:a1:ac:54:fe:99:3e:d1:29:9a:a6:9b:b5:
7c:d3:2f:4e:4e:f2:f6:af:a1:0b:cc:e1:d2:e7:1d:3a:27:0c:
7d:21:4e:78:1e:d1:45:32:da:79:0c:b0:8a:48:3f:3f:fa:23:
3e:71:6b:6f:63:c8:21:ee:c8:a6:86:4e:18:b7:40:52:22:29:
48:38

Re: splunk fowarder SSL error ERROR TcpOutputProc

mikelanghorst — Mon, 31 Oct 2011 21:19:52 GMT

I'd start with looking at the forwarder.pem file with vi and openssl first.

With vi, it should be readable text with "-----BEGIN CERTIFICATE-----" and "-----END CERTIFICATE-----" blocks. Next use openssl and inspect the data returned:
openssl x509 -inform PEM -in -text -noout

Re: splunk fowarder SSL error ERROR TcpOutputProc

mcalautti — Tue, 01 Nov 2011 02:00:11 GMT

ok.. I see the CERT in the file..

i added the output from openssl to the description.. looks ok to me.. what do you think ?
thanks
mike

Re: splunk fowarder SSL error ERROR TcpOutputProc

mcalautti — Tue, 01 Nov 2011 16:00:14 GMT

I just ended up recreating and it worked.. via

http://splunk-base.splunk.com/answers/7164/how-do-i-set-up-ssl-forwarding-with-new-self-signed-certificates-and-authentication

Re: splunk fowarder SSL error ERROR TcpOutputProc

moseisleydk — Thu, 17 Nov 2011 10:26:21 GMT

We just ran into this using conf files working fine on a OpenSolaris 32-bit - not working on Ubuntu 10.04 TLS (64-bit)

The problem was the decryption of the password, pasting the real in as plain text worked, and the resulting encrypted string was different on the Ubuntu.

Re: splunk fowarder SSL error ERROR TcpOutputProc

andyspusm — Thu, 09 Feb 2012 20:31:08 GMT

I just saw the same behavior between two Windows 7 Professional x64 machines - pasting the password in plaintext worked. Thanks for the idea.

Re: splunk fowarder SSL error ERROR TcpOutputProc

nrpeter — Fri, 12 Jul 2013 21:33:15 GMT

This error occurs when splunk tries to decrypt the RSA private key in server.pem with the wrong password.

I'll show you how to fix this by decrypting the cert by hand as long as you are using default splunk signing CAs.

 1. Delete server.pem from your universal forwarder and restart splunk (this will recreate server.pem with a default password of 'password')
 2. Decrypt the private RSA key 'openssl rsa -in server.pem -out server_unprotected.pem'
 3. Since server_unprotected.pem will only have the decrypted RSA private key in it, copy & paste the other two blocks from server.pem (don't copy over the original encrypted RSA private key)
 4. mv splunk_unprotected.pem server.pem
 5. Restart splunk

Re: splunk fowarder SSL error ERROR TcpOutputProc

karthi4k — Wed, 28 Jan 2015 12:30:58 GMT

It works like a charm. Thanks peter. One typo in the point 4 where it should be server.pem and not splunk.pem

Re: splunk fowarder SSL error ERROR TcpOutputProc

dhillny — Wed, 22 Jun 2016 00:50:06 GMT

life saver bro, other than the typo on #4. This is a solid fix.

Re: splunk fowarder SSL error ERROR TcpOutputProc

wqiu_splunk — Tue, 02 Oct 2018 08:06:21 GMT

this should be the answer indeed!!!!