topic After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"? in Deployment Architecture

After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

a212830 — Wed, 10 May 2017 17:15:22 GMT

Hi,

I upgraded a Search Head to 6.6.0, and am getting the following messages continuously...

5-10-2017 13:12:10.558 -0400 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:10.558 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
05-10-2017 13:12:13.181 -0400 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:13.181 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number
05-10-2017 13:12:15.624 -0400 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.
05-10-2017 13:12:15.624 -0400 WARN  HttpListener - Socket error from 127.0.0.1 while idling: error:1408A10B:SSL routines:ssl3_get_client_hello:wrong version number

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

naidusadanala — Wed, 10 May 2017 18:37:00 GMT

Its a bug ... Roll back to previous version and see

Everything works as normal

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

sloshburch — Thu, 11 May 2017 12:07:49 GMT

Do you see any communication failures or just these warnings?

Include any SSL related conf from your server.conf or web.conf (probably good to use btool here). Since it was an upgrade, there's always a chance there was a leftover ssl config from a prior release that conflicts with modern security requirements for SSL.

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

sloshburch — Thu, 11 May 2017 12:09:28 GMT

Looks like there's some known issues related to SSL and upgrades.

Do any of these items seem like the cause? http://docs.splunk.com/Documentation/Splunk/latest/ReleaseNotes/Knownissues

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

KPamatian — Thu, 29 Jun 2017 03:23:03 GMT

any updates on this? Im experiencing the same issue now.

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

ehorwood — Mon, 03 Jul 2017 15:17:16 GMT

I'm also experiencing this after upgrade from 6.5.1 to 6.6.1. Had to rollback to previous version and all worked.

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

gjanders — Tue, 04 Jul 2017 08:25:26 GMT

Are you using older universal forwarders pre 6.2.x and sending traffic to a splunk tcp SSL port on the indexer?

In particular the older 6.0/6.1 series releases:
6.0.0 to 6.0.6 forwarders
6.1.0 to 6.1.4 forwarders

If so you can make the changes described in the known issues for 6.6.2 or upgrade your forwarders to a new version.

I suspect that your seeing older forwarders attempting to use an SSL/TLS cipher suite that is no longer supported by a modern version of the Splunk enterprise server.

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

nclancy_splunk — Mon, 24 Jul 2017 06:47:33 GMT

Changes to the cipher suites between versions of splunk mean that OOTB the two versions of splunk will not have a common cipher to share the documentation advises providing a common cipher the two versions can agree on.

SSL/TLS are protocols - NOT ciphers. In particular, TLS is an evolution of SSL.

The relevant change is in $SPLUNK_HOM/etc/system/default/server.conf, and is the change to cipherSuite. In 6.4.1 this is set to

TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

and in 6.6.1 this is set to

ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256

TLSv1+HIGH (and corresponding for TLSv1.2) means all ciphers compatible with TLSv1 of HIGH strength. There is some overlap here with the ciphers compatible with SSL3.0. However, none of the SSL3.0 ciphers appear in the 6.6.1 list.

To see this more clearly, take a Linux system with openssl installed (almost any Linux system will do!).

Run:

openssl ciphers SSLv3+HIGH
openssl ciphers TLSv1+HIGH

Note that these give you the same results. However, they all end with SHA. In the explicit list you provide in 6.6.1 they all end with SHA, so it's easy to see that there's no overlap with SSLv3+HIGH and the new list in 6.6.1 - leading to the behaviour observed. Any system (such as Splunk 6.1) which only supports TLS1.0 and below (including SSL3) won't be able to communicate with a Splunk 6.6.1 server with default config only suitable for TLS1.2.

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

mbrunetto — Fri, 02 Feb 2018 14:29:37 GMT

thank you nclancy, this was a fantastic help.

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

sloshburch — Tue, 06 Feb 2018 13:11:08 GMT

@a212830 - Would you accept this answer if it helped?

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

andrewjhill — Thu, 20 Dec 2018 19:10:04 GMT

I've battled this issue so many times - nclancy, your comment was very helpful, however - I still had some issues.

At first, I opted to add the following to $SPLUNK_HOME/etc/system/local/inputs.conf:

[applicationsManagement]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

I believe there's a bug, because after a Splunk restart, the btool debug didn't report the change:

$ ./splunk btool inputs list --debug | grep cipher
/opt/splunkforwarder/etc/system/default/inputs.conf                        cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

I ended up editing $SPLUNK_HOME/etc/system/default/inputs.conf and it did the trick. No more SSLv3 errors!

If you're at Splunk and can replicate this issue, I'm happy to provide a diag so we can address this bug.

Thanks!

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

sloshburch — Wed, 02 Jan 2019 22:03:58 GMT

I think your specific issue is actually that you should have edited the stanza [SSL], not [applicationsManagement].

Since your changes to default will be reverted upon upgrade, I highly recommend you try adding the stanza in local again but as:

[SSL]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH

Re: After upgrading to Splunk 6.6.0, why am I receiving warning messages such as "WARN SSLCommon - Received fatal SSL3 alert"?

abhib89 — Thu, 30 May 2019 09:01:32 GMT

adding cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH parameter under [sslConfig] in server.conf did the trick for us.

We had HTTP event collector servers stopped sending data once upgraded from v6.5 to v7.1.6.

put this in server.conf

[sslConfig]
cipherSuite = TLSv1+HIGH:TLSv1.2+HIGH:@STRENGTH