Getting Data In

Why am I getting errors with my SSL configuration between a universal forwarder and indexers?

Motivator

Hello Experts,

I have a QA setup with 1 search head, 2 indexers and 1 universal forwarder. I have created the following certificates on splunk search head box as mentioned in the docs:

esCACertificate.csr, esCACertificate.pem, esCAPrivateKey, esForwarderCertificate.csr ,esForwarderCertificate.pem, esForwarderKey.key, esServerCertificate.csr , esServerCertificate.pem, esServerPrivateKey.key, esSplunkWebCert.csr, esSplunkWebCert.pem, esSplunkWebPrivateKey.key

I then copied all these certs to both the indexers and universal forwarder. I was able to setup splunkweb ssl all fine as mentioned here: http://docs.splunk.com/Documentation/Splunk/6.2.1/Security/Self-signcertificatesforSplunkWeb

Now i tried to setup SSL between two indexers and UF as per the docs: Indexer - Forwarder : http://docs.splunk.com/Documentation/Splunk/6.2.1/Security/ConfigureSplunkforwardingtousesignedcerti...

Here's my IDX config

[default]
host = xyz.com
[splunktcp-ssl:9997]
compressed = true
[SSL]
password = server certificate's password
rootCA = $SPLUNK_HOME/etc/auth/escerts/esCACertificate.pem
serverCert = $SPLUNK_HOME/etc/auth/escerts/esServerCertificate.pem

Here's the splunkd.log after i restarted splunk service
******
ERROR TcpInputConfig - SSL server certificate not found, or password is wrong - SSL ports will not be opened
******

My UF outputs.conf:

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
compressed = true
server = x:9997,y:9997
sslCertPath = $SPLUNK_HOME/etc/auth/escerts/esServerCertificate.pem
sslPassword = esServerCertificates password
sslRootCAPath = $SPLUNK_HOME/etc/auth/escerts/esCACertificate.pem
sslVerifyServerCert = true

Restarted UF splunk service. Here's the error from splunkd.log

ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server x:9997  and 
ERROR TcpOutputProc - Error initializing SSL context - invalid sslCertPath for server y:9997  

I have verified the passwords and they are correct. Please advise.

Thanks,
Raghav

0 Karma
1 Solution

Builder

The following configuration is working fine with me:

I generated CA cert and indexer and forwarders certs as per the following:

CA cert:

mkdir -p /opt/splunk/etc/certs
export OPENSSL_CONF=/opt/splunk/openssl/openssl.cnf
/opt/splunk/bin/genRootCA.sh -d /opt/splunk/etc/certs/

indexer cert:
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n indexer -c indexer -p

forwarder cert:
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n forwarder -p

on indexer:

[splunktcp-ssl://9997]
connection_host = none
[SSL]
rootCA = /opt/splunk/etc/system/local/certs/cacert.pem
serverCert = /opt/splunk/etc/system/local/certs/indexer.pem
password = P@ssw0rd
requireClientCert = false

on forwarder:

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = indexer1:9997, indexer2:9997
sslRootCAPath = /opt/splunkforwarder/etc/apps/search/local/certs/cacert.pem
sslCertPath = /opt/splunkforwarder/etc/apps/search/local/certs/forwarder.pem
sslPassword = P@ssw0rd
sslCommonNameToCheck = indexer
sslVerifyServerCert = true

Regards,
Ahmed Elakwah

View solution in original post

Builder

The following configuration is working fine with me:

I generated CA cert and indexer and forwarders certs as per the following:

CA cert:

mkdir -p /opt/splunk/etc/certs
export OPENSSL_CONF=/opt/splunk/openssl/openssl.cnf
/opt/splunk/bin/genRootCA.sh -d /opt/splunk/etc/certs/

indexer cert:
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n indexer -c indexer -p

forwarder cert:
/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n forwarder -p

on indexer:

[splunktcp-ssl://9997]
connection_host = none
[SSL]
rootCA = /opt/splunk/etc/system/local/certs/cacert.pem
serverCert = /opt/splunk/etc/system/local/certs/indexer.pem
password = P@ssw0rd
requireClientCert = false

on forwarder:

[tcpout]
defaultGroup = default-autolb-group
[tcpout:default-autolb-group]
server = indexer1:9997, indexer2:9997
sslRootCAPath = /opt/splunkforwarder/etc/apps/search/local/certs/cacert.pem
sslCertPath = /opt/splunkforwarder/etc/apps/search/local/certs/forwarder.pem
sslPassword = P@ssw0rd
sslCommonNameToCheck = indexer
sslVerifyServerCert = true

Regards,
Ahmed Elakwah

View solution in original post

Motivator

Hey Thanks for the quick instructions.... What i read is, you have created rootCA, Indexer and forwarder on indexer 1 and copied then over to indexer 2 and forwarder?

I tried the exact same configs....on the forwarder's outputs.config, when you specified sslPassword = P@ssw0rd, is that server certificate's password or forwarder's?

Thanks again for all your help.

Thanks,
Raghav

0 Karma

Communicator

The createssl should ask for a password, use that one for the case you're deploying

0 Karma

Builder

Yes I've created rootCA and sign indexer.pem and forwarder.pem with it.

sslPassword is the password you set when you generate forwarder.pem

/opt/splunk/bin/splunk createssl server-cert -d /opt/splunk/etc/certs/ -n forwarder -p

Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Make sure that you set the common name to indexer when you generate indexer.pem as it will be checked by forwarder on outputs.conf

sslCommonNameToCheck = indexer

Common Name (e.g. server FQDN or YOUR name) []:indexer

Regards,
Ahmed

0 Karma

Motivator
Thanks, i promise this would be my last question....how about if i have two indexers? can i just copy the same indexer.pem to the other indexer?

Thanks,
Raghav
0 Karma

Builder

No problems at all .. Yes you can copy indexer.pem to another indexers and it will work fine and also you can create new cert for indexer2 as long as it is signed by the same rootCA but the common namd should be the same as it is checked by forwarder
sslCommonNameToCheck = indexer

Regards,
Ahmed

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!