Securing Inter-Splunk-Communication with own certificates fails

[horsefez]

I am trying to secure master <-> indexer communication with server certificates signed by our own company rootCA.
Reason is: Forwarding Master-Data to indexers so master does not index any data itself.

I created certificates for my servers according to the Splunk documentation.
For the sake of this example, I will call the server certificate: servercert.pem
And I will call the rootCA certificate: rootcacert.pem

Lets start at the formatting of the certificates:

The servercert.pem looks like
- servercert in pem format
- privatekey in rsa format (encrypted with secret-key)
- subCAcert in pem format (yes, we have a subCA)
- rootCAcert in pem format

The rootcacert.pem looks like

- rootCAcert in pem format (no subCAcert, only the rootCAcert)

On the master, the outputs.conf looks like this:

[tcpout]
defaultGroup = Splunk_Indexers

[tcpout:Splunk_Indexer]
server = indexer1:9997,indexer2:9997

[tcpout-server://indexer1.ex.amp.le.de:9997]
sslRootCAPath = /opt/splunk/etc/auth/splunkforwarder/rootcacert.pem
sslCertPath = /opt/splunk/etc/auth/splunkforwarder/servercert.pem
sslPassword = <secret-key>
sslVerifyServerCert = true
sslCommonNameToCheck = indexer1.ex.amp.le.de

[tcpout-server://indexer2.ex.amp.le.de:9997]
sslRootCAPath = /opt/splunk/etc/auth/splunkforwarder/rootcacert.pem
sslCertPath = /opt/splunk/etc/auth/splunkforwarder/servercert.pem
sslPassword = <secret-key>
sslVerifyServerCert = true
sslCommonNameToCheck = indexer2.ex.amp.le.de

On the indexers the inputs.conf (distributed to them over the cluster-bundle) looks like this

[SSL]
rootCA = /opt/splunk/etc/auth/receiver/rootcacert.pem
serverCert = /opt/splunk/etc/auth/receiver/servercert.pem
password = <secret-key>

[splunktcp-ssl:9997]
compressed = true

---

Site note: I created a directory called splunkforwarder and receiver for reason of understanding which certificate resides where on the system.

If I open splunkd.log on the master I find the following error:
ERROR TcpOutputFd - Read error. Connection reset by peer

On the indexers splunkd.log looks like this:
ERROR TcpInputProc - Error encountered for connection from src=:38953. error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol

Help would be hugely appreciated, because I've been working on this problem since Monday!

[woodcock · Accepted solution]

You need this:

Who: George Starcher and Duane Waddle, Defense Point Security
What: Avoid the SSLippery SSLope of Default SSL
Recording: https://splunk.webex.com/splunk/lsr.php?RCID=da90ccae281af46da9e4a3b46c076a0b
Slides: Media:SplunkTrustApril-SSLipperySlopeRevisited.pdf

And an older version here::
https://conf.splunk.com/session/2014/conf2014_DuaneWaddleGeorgeStarcher_Self_UsingTrack.pdf

[horsefez]

For the books...
The problem was, that I used, as my companies certificate-specialists called it, "server" certificates. With server certificates it's not possible to actively connect to a host and establish a tcp connection over ssl. What I've found out now, is that you need explicit "server-client" certificates. So splunk can connect to another instance.

This is not documented in the splunk documentation. And I almost spent 1 1/2 weeks on this problem. But it now finally works!

[vtalanki]

Hi @pyro_wood , I know this post is way back but I have a similar usecase now. Basically looking to enable mTLS in splunk Enterprise cluster. Can you please elaborate what did you meant by 'server-client' cert. How can I provide client cert for mutual tls inter-splunk communication with own certificates?

[renjith_nair]

Hi,

Setting up ssl especially with our own certificate is not at all fun. There can be different issues and you might not be getting the exact error message to debug as you might have already encountered.
Just trying to put down few possibilities of your problem and some are silly - please excuse for that

• DefaultGroup name and tcpout: is different. Obviously it's a typo but just in case..
• Try to use FQDN in server=indexer1:9997 also as used in [tcpout-server://indexer1.ex.amp.le.de:9997]
• Make sure you have your sub ca also part of CA cert together with the root CA(the complete chain)
• Remove the private key from the cert file if it's together, even though you have separated the private key and cert file.
• Since you have enabled the compression at receiving side, check the compression is enabled at sending side also(useClientSSLCompression)
• Make sure that your ssl password is the same which you have used to extract pem
• Check if any other forwarder can connect with SSL other than master so that you can concentrate on the forwarding side(master)
• Try openssl s_client from the master to test the connection direct to the indexer
• Make sure that the splunk.secret file is present
• If you haven't tried, just try using auth directory itself insted of separate folder - it was caused an issue for us once.