Deployment Architecture

After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

a212830
Champion

Hi,

I upgraded a Search Head to 6.6.0, and am getting the following messages continuously...

5-10-2017 13:12:10.558 -0400 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:10.558 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
05-10-2017 13:12:13.181 -0400 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:13.181 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
05-10-2017 13:12:15.624 -0400 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:15.624 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number

abhib89
Explorer

adding cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH parameter under [sslConfig] in server.conf did the trick for us.

We had HTTP event collector servers stopped sending data once upgraded from v6.5 to v7.1.6.

put this in server.conf

[sslConfig]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

andrewjhill
Path Finder

I've battled this issue so many times - nclancy, your comment was very helpful, however - I still had some issues.

At first, I opted to add the following to $SPLUNK_HOME/etc/system/local/inputs.conf:

[applicationsManagement]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

I believe there's a bug, because after a Splunk restart, the btool debug didn't report the change:

$ ./splunk btool inputs list --debug | grep cipher
/opt/splunkforwarder/etc/system/default/inputs.conf                        cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

I ended up editing $SPLUNK_HOME/etc/system/default/inputs.conf and it did the trick. No more SSLv3 errors!

If you're at Splunk and can replicate this issue, I'm happy to provide a diag so we can address this bug.

Thanks!

sloshburch
Splunk Employee
Splunk Employee

I think your specific issue is actually that you should have edited the stanza [SSL], not [applicationsManagement].

Since your changes to default will be reverted upon upgrade, I highly recommend you try adding the stanza in local again but as:

[SSL]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

nclancy_splunk
Splunk Employee
Splunk Employee

Changes to the cipher suites between versions of splunk mean that OOTB the two versions of splunk will not have a common cipher to share the documentation advises providing a common cipher the two versions can agree on.

SSL/TLS are protocols - NOT ciphers. In particular, TLS is an evolution of SSL.

The relevant change is in $SPLUNK_HOM/etc/system/default/server.conf, and is the change to cipherSuite. In 6.4.1 this is set to

TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

and in 6.6.1 this is set to

ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256

TLSv1+HIGH (and corresponding for TLSv1.2) means all ciphers compatible with TLSv1 of HIGH strength. There is some overlap here with the ciphers compatible with SSL3.0. However, none of the SSL3.0 ciphers appear in the 6.6.1 list.

To see this more clearly, take a Linux system with openssl installed (almost any Linux system will do!).

Run:

openssl ciphers SSLv3+HIGH
openssl ciphers TLSv1+HIGH

Note that these give you the same results. However, they all end with SHA. In the explicit list you provide in 6.6.1 they all end with SHA, so it's easy to see that there's no overlap with SSLv3+HIGH and the new list in 6.6.1 - leading to the behaviour observed. Any system (such as Splunk 6.1) which only supports TLS1.0 and below (including SSL3) won't be able to communicate with a Splunk 6.6.1 server with default config only suitable for TLS1.2.

mbrunetto
Path Finder

thank you nclancy, this was a fantastic help.

0 Karma

sloshburch
Splunk Employee
Splunk Employee

@a212830 - Would you accept this answer if it helped?

0 Karma

sloshburch
Splunk Employee
Splunk Employee

Looks like there's some known issues related to SSL and upgrades.

Do any of these items seem like the cause? http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues

sloshburch
Splunk Employee
Splunk Employee

Do you see any communication failures or just these warnings?

Include any SSL related conf from your server.conf or web.conf (probably good to use btool here). Since it was an upgrade, there's always a chance there was a leftover ssl config from a prior release that conflicts with modern security requirements for SSL.

0 Karma

ehorwood
Explorer

I'm also experiencing this after upgrade from 6.5.1 to 6.6.1. Had to rollback to previous version and all worked.

0 Karma

gjanders
SplunkTrust
SplunkTrust

Are you using older universal forwarders pre 6.2.x and sending traffic to a splunk tcp SSL port on the indexer?

In particular the older 6.0/6.1 series releases:
6.0.0 to 6.0.6 forwarders
6.1.0 to 6.1.4 forwarders

If so you can make the changes described in the known issues for 6.6.2 or upgrade your forwarders to a new version.

I suspect that your seeing older forwarders attempting to use an SSL/TLS cipher suite that is no longer supported by a modern version of the Splunk enterprise server.

0 Karma

KPamatian
Engager

any updates on this? Im experiencing the same issue now.

0 Karma

naidusadanala
Communicator

Its a bug ... Roll back to previous version and see

Everything works as normal

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...