Why am I unable to get a Splunk forwarder and indexer to talk over SSL using a non-default CA?

[ryanleerally]

Hey all,

I'm having a really tough time getting my forwarders and indexer to talk over SSL using a non-default CA. I've searched through other answers, but haven't found any resolution. I've been following this guide:
https://wiki.splunk.com/Community:Splunk2Splunk_SSL_SelfSignedCert_NewRootCA

I've also read through http://docs.splunk.com/Documentation/Splunk/6.2.3/Security/ConfigureSplunkforwardingtousesignedcerti....

I generated the CA root cert, generated & signed my CA public cert (myCACertificate.pem), then generated, signed, and my server cert with the public, private, and CA certs (myServerCertificate.pem) with "splunkserver.internal.domain"
I modified my inputs.conf:

# cat /opt/splunk/etc/system/local/inputs.conf
[default]
host = splunkserver

[splunktcp-ssl:9997]
compressed = true
disabled = 0

[SSL]
password = {myServerPrivateKey.key password}
rootCA = $SPLUNK_HOME/etc/certs/myCACertificate.pem
serverCert = $SPLUNK_HOME/etc/certs/myServerCertificate.pem

I restarted Splunk and see this in my splunkd.log, which supposedly means the server has accepted the certificate:

09-10-2015 17:12:30.560 +0000 INFO  loader - Server supporting SSL versions SSL3,TLS1.0,TLS1.1,TLS1.2
09-10-2015 17:12:30.879 +0000 INFO  TcpInputConfig - IPv4 port 9997 is reserved for splunk 2 splunk (SSL)

I copied myCACertificate.pem and myServerCertificate.pem using SCP to the forwarder & modified its outputs.conf:

# cat /opt/splunkforwarder/etc/system/local/outputs.conf
[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
compressed = true
server = {IP_of_server}:9997
sslCertPath = /opt/splunkforwarder/etc/certs/myServerCertificate.pem
sslPassword = {myServerPrivateKey.key password}
sslRootCAPath = /opt/splunkforwarder/etc/certs/myCACertificate.pem
sslVerifyServerCert = true

I restarted the Splunk forwarder and get this output:

Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.

Stopping splunk helpers...

Done.

Splunk> Winning the War on Error

Checking prerequisites...
        Checking mgmt port [8089]: open
        Checking conf files for problems...
Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=587690100 error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error.
Couldn't initialize SSL Context for HTTPClient in ServerConfig
        Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...
Done

I see this in splunkd.log:

09-10-2015 17:32:10.531 +0000 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/auth/server.pem errno=587690100 error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error.
09-10-2015 17:32:10.532 +0000 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong
09-10-2015 17:32:10.532 +0000 ERROR HTTPServer - SSL will not be enabled

I have a few questions:

Why is the Splunk forwarder trying to use /opt/splunkforwarder/etc/auth/server.pem when I specified /opt/splunkforwarder/etc/certs/myServerCertificate.pem? I see that according to the guide, the expected logs from the forwarder use server.pem as well.

Should I be able to run openssl x509 -in /opt/splunkforwarder/etc/certs/myServerCertificate.pem -text -noout and get output without entering a password?
and finally... what the heck am I doing wrong?!

Any help is appreciated! Thanks!

[ryanleerally · Accepted solution]

I figured this out. When I created my certificates, I used the same CN for the CA and server certs. I used a different name for my CA and that worked.

[ryanleerally]

I answered my second question - the correct command is:
openssl rsa -in /opt/splunkforwarder/etc/certs/myServerCertificate.pem -text
I am able to decrypt that key using the same password I entered in the outputs.conf on the forwarder.