When updating our certs between universal forwarders and indexers, why am I seeing the following SSL handshake failure?

[pkeller]

I'm attempting to update our certs between our universal forwarders (UF) and indexers in our test environment. I believe I have the certs properly generated and in place. But when the UF attempts to forward, we see this error:

10-19-2018 08:13:14.661 -0600 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.

10-19-2018 14:17:44.863 +0000 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
10-19-2018 14:17:44.863 +0000 ERROR TcpInputProc - Error encountered for connection from src=nn.nn.nn.nn:38438. error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher

This leads me to believe that the cipherSuite needs to be updated ...

indexer server.conf - ( Splunk 7.1.3 ]

[sslConfig]
sslVersions = tls1.2
sslVersionsForClient = tls1.2

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256

( etc/system/local/inputs.conf under [SSL] )
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH


    UF - Splunk 6.6.4 - etc/system/default/server.conf

    [sslConfig]
    cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

    etc/system/default/outputs.conf

    [tcpout]
    sslVersions = tls1.2
    cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256

I've been using this link to generate and set up the new forwarding certs.

https://wiki.splunk.com/images/f/fb/SplunkTrustApril-SSLipperySlopeRevisited.pdf

[lavanyaanne]

From the splunk docs i have observed server.conf ciphersuite is different from inputs.conf and outputs.conf. Check your cipheresuite.
https://docs.splunk.com/Documentation/Splunk/7.1.3/Security/Ciphersuites

[harsmarvania57]

Hi,

While looking at $SPLUNK_HOME/default/etc/system/default/inputs.conf it has below ciphersuite, can you please remove cipherSuite from [SSL] stanza in $SPLUNK_HOME/default/etc/system/local/inputs.conf on Indexer so that it will use default cipherSuite.

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

[cboillot]

I will 2nd this. Is there a reason you aren't using the default CipherSuite?

[pkeller]

Thank you ...

The default didn't work, so I went back and added the content at the end of the list as I'd seen that had solved different SSL issues when I upgraded beyond 6.5 ( guessing on the version )

I've reverted everything back to the default and I'm still getting the same errors.

[sudosplunk]

Couple of things to check, is the sslPassword same on both UFs and Indexer?
And stanza name in outputs.conf is [tcpout] instead of [tcpoutput]
Indexers should be configured to accept encrypted data, meaning, inputs.conf on indexers should have a stanza defined as [splunktcp-ssl:<port>]
* Set to the port on which the forwarder sends the encrypted data

[pkeller]

Thank you for your comments ...

the stanza is definitely [tcpout] ... the error was due to my typing this out in haste. Indexers are definitely listening on the splunktcp-ssl port I configured. I'll edit the post to the correct setting.

[splunk@somewhere ~]$ lsof -Pi :9998
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
splunkd 14568 splunk 47u IPv4 150502287 0t0 TCP *:9998 (LISTEN)

[splunk@somewhere ~]$ /opt/splunk/bin/splunk btool inputs list splunktcp-ssl
[splunktcp-ssl://9998]
_rcvbuf = 1572864
evt_dc_name =
evt_dns_name =

[SSL]
password = +-------redacted encrypted password ----+
rootCA = $SPLUNK_HOME/etc/slave-apps/_cluster/auth/cacert.crt
serverCert = $SPLUNK_HOME/etc/slave-apps/_cluster/auth/secidx.pem