Security

How to fix broken Slave to Master SSL Connection after updating to Splunk 6.6.0?

bgstein
Path Finder

We are newly working as a Splunk license Slave of a Master run by another department at our University. Our update of the Slave from 6.5.3 to 6.6.0 broke the SSL communication:

05-05-2017 12:07:38.980 -0700 ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=https://splunk-license.xyz.edu:8089 Error connecting: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure'
05-05-2017 12:08:39.973 -0700 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.

This makes me sad. Any suggestions for how to get our Master to accept us again?

1 Solution

bgstein
Path Finder

Looks like I need to learn to read:

http://docs.splunk.com/Documentation/Splunk/6.6.0/Installation/Aboutupgradingto6.6READTHISFIRST

Connectivity over SSL between version 6.6 and versions less than 5.0 is disabled by default
Because of changes to the security ciphers in version 6.6 of Splunk Enterprise, instances of Splunk software that run on version 5.0 or less cannot connect to instances of version 6.6 or greater by default.

When you upgrade, any instances that run version 5.0 or less will no longer communicate with the upgraded instance over SSL. To work around the problem, edit inputs.conf and outputs.conf on the sending instances to enable ciphers that allow communication between the instances.

View solution in original post

bgstein
Path Finder

Looks like I need to learn to read:

http://docs.splunk.com/Documentation/Splunk/6.6.0/Installation/Aboutupgradingto6.6READTHISFIRST

Connectivity over SSL between version 6.6 and versions less than 5.0 is disabled by default
Because of changes to the security ciphers in version 6.6 of Splunk Enterprise, instances of Splunk software that run on version 5.0 or less cannot connect to instances of version 6.6 or greater by default.

When you upgrade, any instances that run version 5.0 or less will no longer communicate with the upgraded instance over SSL. To work around the problem, edit inputs.conf and outputs.conf on the sending instances to enable ciphers that allow communication between the instances.

aaraneta_splunk
Splunk Employee
Splunk Employee

@bgstein - Glad you were able to find the solution. Please don't forget to click "Accept" to close out your question so others can easily find it. Thank you.

0 Karma

t9445
Path Finder

Interesting - this issue is for compatibility for versions < 5.0 -- we are running thousands of UF instances all v 6.1 or greater (most 6.5+) and still seeing the issue.

0 Karma

t9445
Path Finder

So quick followup @aaraneta (hoping you can help) -- I updated the the certs per the 6.6.0 documentation/read-me-first-documentation (via distribution server) on ALL of our UFs/Heavy-Forwarders in an attempt to resolve this and still getting the errors. - we have upgraded our "Heavy" infrastructure remotely (SHs/Indexers/LicServer/Heavy-forwarders, etc) fully to 6.6.0 except for our distribution server (v6.5.2) -->if we cannot talk to UFs instances < 6.6.0 we'll be in a bad place until resolved -- yes we have also remotely upgraded "most-all" of our UFs to v6.6.0, however due to a lag in the cloud with getting our AMI's updated (our lag, not Splunk's) - cannot rely on new-ami-rollout with splunk-6-6.+ for a bit yet, So appreciate any info or feedback to help resolve (no, we have not as yet opened a ticket with support as yet - process tends to less efficient)

0 Karma

jimt_mt
Loves-to-Learn

Did some quick testing here and it seems that the SSL changes in 6.6.0 broke communications with anything earlier than 6.4.x. I am unable to upgrade all our UF's to the new version, so currently about 1/3 of our environment is "broken". I have a case open with splunk support - waiting for a response.
If I tweak the SSL settings on the DS so it can talk to the forwarders, that breaks SSL with our License Master.

0 Karma

jimt_mt
Loves-to-Learn

Same thing here. Seeing all 6.1.x and 6.2.x forwarders impacted.

0 Karma
Get Updates on the Splunk Community!

Splunk Certification Support Alert | Pearson VUE Outage

Splunk Certification holders and candidates!  Please be advised of an upcoming system maintenance period for ...

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...