Why am I getting errors with my SSL configuration between a universal forwarder and indexers?

[Raghav2384]

Hello Experts,

I have a QA setup with 1 search head, 2 indexers and 1 universal forwarder. I have created the following certificates on splunk search head box as mentioned in the docs:

esCACertificate.csr, esCACertificate.pem, esCAPrivateKey, esForwarderCertificate.csr ,esForwarderCertificate.pem, esForwarderKey.key, esServerCertificate.csr , esServerCertificate.pem, esServerPrivateKey.key, esSplunkWebCert.csr, esSplunkWebCert.pem, esSplunkWebPrivateKey.key

I then copied all these certs to both the indexers and universal forwarder. I was able to setup splunkweb ssl all fine as mentioned here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Security/Self-signcertificatesforSplunkWeb

Now i tried to setup SSL between two indexers and UF as per the docs: Indexer - Forwarder : http://docs.splunk.com/Documentation/Splunk/6.2.1/Security/ConfigureSplunkforwardingtousesignedcerti...

Here's my IDX config

[default]
host = xyz.com
[splunktcp-ssl:9997]
compressed = true
[SSL]
password = server certificate's password
rootCA = $SPLUNK_HOME/etc/auth/escerts/esCACertificate.pem
serverCert = $SPLUNK_HOME/etc/auth/escerts/esServerCertificate.pem

Here's the splunkd.log after i restarted splunk service
******
ERROR TcpInputConfig - SSL server certificate not found, or password is wrong - SSL ports will not be opened
******

My UF outputs.conf:

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
compressed = true
server = x:9997,y:9997
sslCertPath = $SPLUNK_HOME/etc/auth/escerts/esServerCertificate.pem
sslPassword = esServerCertificates password
sslRootCAPath = $SPLUNK_HOME/etc/auth/escerts/esCACertificate.pem
sslVerifyServerCert = true

Restarted UF splunk service. Here's the error from splunkd.log

ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server x:9997  and 
ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server y:9997  

I have verified the passwords and they are correct. Please advise.

Thanks,
Raghav

[aakwah · Accepted solution]

The following configuration is working fine with me:

I generated CA cert and indexer and forwarders certs as per the following:

CA cert:

mkdir -p /opt/splunk/etc/certs
export OPENSSL_CONF=/opt/splunk/openssl/openssl.cnf
/opt/splunk/bin/genRootCA.sh -d /opt/splunk/etc/certs/

indexer cert:
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n indexer -c indexer -p

forwarder cert:
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n forwarder -p

on indexer:

[splunktcp-ssl://9997]
connection_host = none
[SSL]
rootCA = /opt/splunk/etc/system/local/certs/cacert.pem
serverCert = /opt/splunk/etc/system/local/certs/indexer.pem
password = P@ssw0rd
requireClientCert = false

on forwarder:

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = indexer1:9997, indexer2:9997
sslRootCAPath = /opt/splunkforwarder/etc/apps/search/local/certs/cacert.pem
sslCertPath = /opt/splunkforwarder/etc/apps/search/local/certs/forwarder.pem
sslPassword = P@ssw0rd
sslCommonNameToCheck = indexer
sslVerifyServerCert = true

Regards,
Ahmed Elakwah

[Raghav2384]

Hey Thanks for the quick instructions.... What i read is, you have created rootCA, Indexer and forwarder on indexer 1 and copied then over to indexer 2 and forwarder?

I tried the exact same configs....on the forwarder's outputs.config, when you specified sslPassword = P@ssw0rd, is that server certificate's password or forwarder's?

Thanks again for all your help.

Thanks,
Raghav

[guarisma]

The createssl should ask for a password, use that one for the case you're deploying

[aakwah]

Yes I've created rootCA and sign indexer.pem and forwarder.pem with it.

sslPassword is the password you set when you generate forwarder.pem

/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n forwarder -p

Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Make sure that you set the common name to indexer when you generate indexer.pem as it will be checked by forwarder on outputs.conf

sslCommonNameToCheck = indexer

Common Name (e.g. server FQDN or YOUR name) []:indexer

Regards,
Ahmed

[Raghav2384]

Thanks, i promise this would be my last question....how about if i have two indexers? can i just copy the same indexer.pem to the other indexer?

Thanks,
Raghav

[aakwah]

No problems at all .. Yes you can copy indexer.pem to another indexers and it will work fine and also you can create new cert for indexer2 as long as it is signed by the same rootCA but the common namd should be the same as it is checked by forwarder
sslCommonNameToCheck = indexer

Regards,
Ahmed