ERROR TcpInputProc - SSL server certificate not found, or password is wrong - SSL ports will not be opened

[FatDragon1]

Followed the instructions at link text for Configuring Splunk forwarding to use SSL certificates self-signed by a newly generated root certificate

I checked the server cert password with openssl rsa -in /opt/splunk/etc/certs/myServerCertificate.pem and I am able to see RSA Private key

I also have the password type in inputs.conf and it gets hashed

but on the indexer startup log, I get 07-02-2012 23:57:58.047 +0000 ERROR TcpInputProc - SSL server

certificate not found, or password is wrong - SSL ports will not be opened

this is starting to make me mad ..wasting my time for such a small issue or bug!

any idea?

[FatDragon1]

This is crazy..I went to input.conf and retyped the password and restart it and it hashed and then I restarted again few times and everytime is okay now!

I will update you all , if I find ERROR on ssl start for the receiver!!!

[FatDragon1]

maybe this is a bug with password hashing mechanism of inputs.conf password entry

[FatDragon1]

This got to have some dependencies on something. This is crazy!

There are no changes in permission or the files in /opt/splunk/etc/certs/ since
-rw-r--r-- 1 splunk splunk 2909 Jul 2 04:53 myServerCertificate.pem
-rw-r--r-- 1 splunk splunk 973 Jul 2 04:51 myServerPublicCertificate.pem

Same output from earlier time ,shows all is well

07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - Initializing
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - creating tcp pipelineData queue
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - Reconfiguring
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - readConfig - clearing maps
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - global prop enables2sHeartbeat=true
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - global prop s2skeepaliveTimeout=600
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - global prop inputShutdownTimeout=90
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - global prop rdnsMaxDutyCycle=10
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - readConfig - scanning configs
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - SSL serverCert=/opt/splunk/etc/certs/myServerCertificate.pem
07-02-2012 08:42:54.867 +0000 DEBUG TcpInputProc - SSL rootCA=/opt/splunk/etc/certs/myCACertificate.pem
07-02-2012 08:42:54.868 +0000 DEBUG TcpInputProc - Key file password requires decrypting
07-02-2012 08:42:54.868 +0000 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
07-02-2012 08:42:54.868 +0000 INFO TcpInputProc - supporting SSL v2/v3
07-02-2012 08:42:54.868 +0000 DEBUG TcpInputProc - SSL dhfile=
07-02-2012 08:42:54.868 +0000 DEBUG TcpInputProc - SSL requireClientCert=0
07-02-2012 08:42:54.869 +0000 INFO TcpInputProc - IPv4 port 9997 is reserved for splunk 2 splunk
07-02-2012 08:42:54.869 +0000 INFO TcpInputProc - IPv4 port 9997 is not compressed
07-02-2012 08:42:54.869 +0000 INFO TcpInputProc - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
07-02-2012 08:42:54.869 +0000 INFO TcpInputProc - IPv4 port 9997 is compressed
07-02-2012 08:42:54.869 +0000 DEBUG TcpInputProc - readConfig - creating acceptor for IPv4 port 9997
07-02-2012 08:42:54.869 +0000 DEBUG TcpInputProc - Initing Acceptor with SSL
07-02-2012 08:42:54.869 +0000 INFO TcpInputProc - Registering metrics callback for: tcpin_connections

[FatDragon1]

with self-signed certs:

here is the complete output on the server (indexer) and I see the first error sslcommon line below:

07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - Initializing
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - creating tcp pipelineData queue
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - Reconfiguring
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - readConfig - clearing maps
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - global prop enables2sHeartbeat=true
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - global prop s2skeepaliveTimeout=600
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - global prop inputShutdownTimeout=90
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - global prop rdnsMaxDutyCycle=10
07-03-2012 01:20:45.431 +0000 DEBUG TcpInputProc - readConfig - scanning configs
07-03-2012 01:20:45.432 +0000 DEBUG TcpInputProc - SSL serverCert=/opt/splunk/etc/certs/myServerCetificate.pem
07-03-2012 01:20:45.432 +0000 DEBUG TcpInputProc - SSL rootCA=/opt/splunk/etc/certs/myCACertificat.pem
07-03-2012 01:20:45.432 +0000 DEBUG TcpInputProc - Key file password requires decrypting
07-03-2012 01:20:45.432 +0000 INFO TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4RSA:+HIGH:+MEDIUM
07-03-2012 01:20:45.432 +0000 INFO TcpInputProc - supporting SSL v2/v3
07-03-2012 01:20:45.432 +0000 DEBUG TcpInputProc - SSL dhfile=
07-03-2012 01:20:45.432 +0000 DEBUG TcpInputProc - SSL requireClientCert=0
07-03-2012 01:20:45.432 +0000 ERROR SSLCommon - Can't read key file /opt/splunk/etc/certs/myServerertificate.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad derypt.
07-03-2012 01:20:45.432 +0000 ERROR TcpInputProc - SSL server certificate not found, or password i wrong - SSL ports will not be opened
07-03-2012 01:20:45.432 +0000 INFO TcpInputProc - IPv4 port 9997 is reserved for splunk 2 splunk
07-03-2012 01:20:45.433 +0000 INFO TcpInputProc - IPv4 port 9997 is not compressed
07-03-2012 01:20:45.433 +0000 INFO TcpInputProc - IPv4 port 9997 is reserved for splunk 2 splunk SSL)
07-03-2012 01:20:45.433 +0000 INFO TcpInputProc - IPv4 port 9997 is compressed
07-03-2012 01:20:45.433 +0000 DEBUG TcpInputProc - readConfig - creating acceptor for IPv4 port 997
07-03-2012 01:20:45.433 +0000 DEBUG TcpInputProc - Initing Acceptor with Non-SSL
07-03-2012 01:20:45.433 +0000 INFO TcpInputProc - Registering metrics callback for: tcpin_connectons

any ideas?

[FatDragon1]

Same problem if I use
http://wiki.splunk.com/Community:Splunk2Splunk_SSL_DefaultCerts

with the default inputs.conf and password for password and I got all my default files in /opt/splunk/etc/auth as wiki mentions and I get following:

07-03-2012 00:44:02.799 +0000 INFO TcpInputProc - supporting SSL v2/v3
07-03-2012 00:44:02.799 +0000 DEBUG TcpInputProc - SSL dhfile=
07-03-2012 00:44:02.799 +0000 DEBUG TcpInputProc - SSL requireClientCert=0
07-03-2012 00:44:02.800 +0000 ERROR TcpInputProc - SSL server certificate not found, or password is wrong - SSL ports will not be opened
07-03-2012 00:44:02.800 +0000 INFO TcpInputProc - IPv4 port 9997 is reserved for splunk 2 splunk
07-03-2012 00:44:02.800 +0000 INFO TcpInputProc - IPv4 port 9997 is not compressed
07-03-2012 00:44:02.800 +0000 INFO TcpInputProc - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)
07-03-2012 00:44:02.801 +0000 INFO TcpInputProc - IPv4 port 9997 is compressed
07-03-2012 00:44:02.801 +0000 DEBUG TcpInputProc - readConfig - creating acceptor for IPv4 port 9997
07-03-2012 00:44:02.801 +0000 DEBUG TcpInputProc - Initing Acceptor with Non-SSL
07-03-2012 00:44:02.801 +0000 INFO TcpInputProc - Registering metrics callback for: tcpin_connections