How to fix broken Slave to Master SSL Connection after updating to Splunk 6.6.0?

[bgstein]

We are newly working as a Splunk license Slave of a Master run by another department at our University. Our update of the Slave from 6.5.3 to 6.6.0 broke the SSL communication:

05-05-2017 12:07:38.980 -0700 ERROR LMTracker - failed to send rows, reason='Unable to connect to license master=https://splunk-license.xyz.edu:8089 Error connecting: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure'
05-05-2017 12:08:39.973 -0700 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read server hello A', alert_description='handshake failure'.

This makes me sad. Any suggestions for how to get our Master to accept us again?

[bgstein · Accepted solution]

Looks like I need to learn to read:

http://docs.splunk.com/Documentation/Splunk/6.6.0/Installation/Aboutupgradingto6.6READTHISFIRST

Connectivity over SSL between version 6.6 and versions less than 5.0 is disabled by default
Because of changes to the security ciphers in version 6.6 of Splunk Enterprise, instances of Splunk software that run on version 5.0 or less cannot connect to instances of version 6.6 or greater by default.

When you upgrade, any instances that run version 5.0 or less will no longer communicate with the upgraded instance over SSL. To work around the problem, edit inputs.conf and outputs.conf on the sending instances to enable ciphers that allow communication between the instances.

[aaraneta_splunk]

@bgstein - Glad you were able to find the solution. Please don't forget to click "Accept" to close out your question so others can easily find it. Thank you.

[t9445]

Interesting - this issue is for compatibility for versions < 5.0 -- we are running thousands of UF instances all v 6.1 or greater (most 6.5+) and still seeing the issue.

[t9445]

So quick followup @aaraneta (hoping you can help) -- I updated the the certs per the 6.6.0 documentation/read-me-first-documentation (via distribution server) on ALL of our UFs/Heavy-Forwarders in an attempt to resolve this and still getting the errors. - we have upgraded our "Heavy" infrastructure remotely (SHs/Indexers/LicServer/Heavy-forwarders, etc) fully to 6.6.0 except for our distribution server (v6.5.2) -->if we cannot talk to UFs instances < 6.6.0 we'll be in a bad place until resolved -- yes we have also remotely upgraded "most-all" of our UFs to v6.6.0, however due to a lag in the cloud with getting our AMI's updated (our lag, not Splunk's) - cannot rely on new-ami-rollout with splunk-6-6.+ for a bit yet, So appreciate any info or feedback to help resolve (no, we have not as yet opened a ticket with support as yet - process tends to less efficient)

[jimt_mt]

Did some quick testing here and it seems that the SSL changes in 6.6.0 broke communications with anything earlier than 6.4.x. I am unable to upgrade all our UF's to the new version, so currently about 1/3 of our environment is "broken". I have a case open with splunk support - waiting for a response.
If I tweak the SSL settings on the DS so it can talk to the forwarders, that breaks SSL with our License Master.

[jimt_mt]

Same thing here. Seeing all 6.1.x and 6.2.x forwarders impacted.