Getting Data In

When updating our certs between universal forwarders and indexers, why am I seeing the following SSL handshake failure?

pkeller
Contributor

I'm attempting to update our certs between our universal forwarders (UF) and indexers in our test environment. I believe I have the certs properly generated and in place. But when the UF attempts to forward, we see this error:

10-19-2018 08:13:14.661 -0600 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.

10-19-2018 14:17:44.863 +0000 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
10-19-2018 14:17:44.863 +0000 ERROR TcpInputProc - Error encountered for connection from src=nn.nn.nn.nn:38438. error:1408A0C1:SSL routines:ssl3_get_client_hello:no shared cipher

This leads me to believe that the cipherSuite needs to be updated ...

indexer server.conf - ( Splunk 7.1.3 ]

[sslConfig]
sslVersions = tls1.2
sslVersionsForClient = tls1.2

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256

( etc/system/local/inputs.conf under [SSL] )
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH


    UF - Splunk 6.6.4 - etc/system/default/server.conf

    [sslConfig]
    cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

    etc/system/default/outputs.conf

    [tcpout]
    sslVersions = tls1.2
    cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-SHA256

I've been using this link to generate and set up the new forwarding certs.

https://wiki.splunk.com/images/f/fb/SplunkTrustApril-SSLipperySlopeRevisited.pdf

0 Karma

lavanyaanne
Path Finder

From the splunk docs i have observed server.conf ciphersuite is different from inputs.conf and outputs.conf. Check your cipheresuite.
https://docs.splunk.com/Documentation/Splunk/7.1.3/Security/Ciphersuites

0 Karma

harsmarvania57
Ultra Champion

Hi,

While looking at $SPLUNK_HOME/default/etc/system/default/inputs.conf it has below ciphersuite, can you please remove cipherSuite from [SSL] stanza in $SPLUNK_HOME/default/etc/system/local/inputs.conf on Indexer so that it will use default cipherSuite.

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
0 Karma

cboillot
Contributor

I will 2nd this. Is there a reason you aren't using the default CipherSuite?

0 Karma

pkeller
Contributor

Thank you ...

The default didn't work, so I went back and added the content at the end of the list as I'd seen that had solved different SSL issues when I upgraded beyond 6.5 ( guessing on the version )

I've reverted everything back to the default and I'm still getting the same errors.

0 Karma

sudosplunk
Motivator

Couple of things to check, is the sslPassword same on both UFs and Indexer?
And stanza name in outputs.conf is [tcpout] instead of [tcpoutput]
Indexers should be configured to accept encrypted data, meaning, inputs.conf on indexers should have a stanza defined as [splunktcp-ssl:<port>]
* Set to the port on which the forwarder sends the encrypted data

0 Karma

pkeller
Contributor

Thank you for your comments ...

the stanza is definitely [tcpout] ... the error was due to my typing this out in haste. Indexers are definitely listening on the splunktcp-ssl port I configured. I'll edit the post to the correct setting.

[splunk@somewhere ~]$ lsof -Pi :9998
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
splunkd 14568 splunk 47u IPv4 150502287 0t0 TCP *:9998 (LISTEN)

[splunk@somewhere ~]$ /opt/splunk/bin/splunk btool inputs list splunktcp-ssl
[splunktcp-ssl://9998]
_rcvbuf = 1572864
evt_dc_name =
evt_dns_name =

[SSL]
password = +-------redacted encrypted password ----+
rootCA = $SPLUNK_HOME/etc/slave-apps/_cluster/auth/cacert.crt
serverCert = $SPLUNK_HOME/etc/slave-apps/_cluster/auth/secidx.pem

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...