Why are SSL 3.0 and AWS ELB throwing errors after the upgrade from Splunk 6.5.1 to 7.0.2?

[narenpalepu]

Hi All,

I upgraded search and index clusters to 7.02 from 6.5.1

Seeing the following in splunkd.log

02-11-2018 10:31:34.913 +0000 WARN  SSLCommon - Received fatal SSL3 alert. ssl_state='SSLv3 read client hello C', alert_description='handshake failure'.

and ELB AWS health checks are failing. Tried enabling all the ciphers in AWS did not help.

I am on Ubuntu 12.

Any other changes to be done for SSL or ciphers.

Thanks,

NP

[jkat54]

It’s saying it doesn’t trust the CA.

Check the expiration date of cacert.pem found here: $splunk_home/etc/auth

openssl x509 -in /opt/splunk/etc/auth/cacert.pem -noout -enddate

[narenpalepu]

I am using splunk certs. Any pointer in this direction is appreciated. Not seeing the same issue on indexers.

[narenpalepu]

Thanks for the response.

CONNECTED(00000003)
depth=0 CN = eoe-pdx-splunk-search-0fa4b3c077a58b38b, O = SplunkUser
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = eoe-pdx-splunk-search-0fa4b3c077a58b38b, O = SplunkUser
verify error:num=21:unable to verify the first certificate

verify return:1

Certificate chain
0 s:/CN=eoe--0fa4b3c077a58b38b/O=SplunkUser

i:/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com

Server certificate
-----BEGIN CERTIFICATE-----
MIICPTCCAaYCCQD54aziriM0MjANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJV
UzELMAkGA1UECAwCQ0ExFjAUBgNVBAcMDVNhbiBGcmFuY2lzY28xDzANBgNVBAoM
BlNwbHVuazEXMBUGA1UEAwwOU3BsdW5rQ29tbW9uQ0ExITAfBgkqhkiG9w0BCQEW
EnN1cHBvcnRAc3BsdW5rLmNvbTAeFw0xNzA2MjExODAzMzVaFw0yMDA2MjAxODAz
MzVaMEcxMDAuBgNVBAMMJ2VvZS1wZHgtc3BsdW5rLXNlYXJjaC0wZmE0YjNjMDc3
YTU4YjM4YjETMBEGA1UECgwKU3BsdW5rVXNlcjCBnzANBgkqhkiG9w0BAQEFAAOB
jQAwgYkCgYEApBBvXGG/r72pHNN6q4Mh7Ax5XfYIuOs35uJpzXs6lJ2v0o05Lfqp
J0QfKsSIQAfh3onCwfpsNFWbaKtQuzs4qaQPOX4lQmPKYV0qfAlGJWgjT7FAqKLH
BrtS0RovQ+sp396VPujYyO9hrqZ3d+Qtt+87TXOeteqyyGXu7l+zXMMCAwEAATAN
BgkqhkiG9w0BAQUFAAOBgQCK2vo/ct+EtGF4GpFlB0kp/leoiyNjnpGMrvNxv2z+
DEtavXevgxH0aOIoRgcBI+rPrEYbD9XtqEzFHEBXbLdhRQHsfAf2OQq5Vz4uOu0I
JHfSkgvQDDOifftTUcs3+vphdzexR+anmM5y8ArioEmMRa2UYKhsyGHEVjl7MMHg
RQ==
-----END CERTIFICATE-----
subject=/CN=eoe-pdx-splunk-search-0fa4b3c077a58b38b/O=SplunkUser

issuer=/C=US/ST=CA/L=San Francisco/O=Splunk/CN=SplunkCommonCA/emailAddress=support@splunk.com

No client certificate CA names sent
Peer signing digest: SHA512

Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 1108 bytes and written 431 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 112ADC00DEF5813EA46F7A0CB8F59E88E7B6E119A90417F7C72BA4AAF9FF59A7
Session-ID-ctx:
Master-Key: 1C5B35A20A1247A63A95491FBF6E1FE0C03139433C4262B1CF448C69E56E3E73FB931A8E58620D216DC8E0EB1AB62D29
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 46 c5 6b 10 a3 e0 78 ea-5d f5 c7 17 24 9d 11 cb F.k...x.]...$...
0010 - ea 37 67 49 11 c7 01 9e-93 f7 2c 4e eb 55 52 3a .7gI......,N.UR:
0020 - 72 80 be 81 da 69 26 d2-7d 18 b5 e0 30 b2 b4 c2 r....i&.}...0...
0030 - e9 81 1f 87 9e 5c c3 c2-2f 14 81 6f 47 f7 5a 24 ......./..oG.Z$
0040 - f2 b8 0a dd d7 9f 96 0b-da 8a 0a 6f 06 48 0e cb ...........o.H..
0050 - 2e 01 62 0f 5b c8 1b 5a-0e 7a 96 94 01 c5 b6 da ..b.[..Z.z......
0060 - 6b 26 75 d2 ca 2b fc 0c-55 ad 7f 76 fb e6 c2 d0 k&u..+..U..v....
0070 - 94 9c 6f aa c5 5a dc 8a-6c 43 2d e4 28 e3 14 d1 ..o..Z..lC-.(...
0080 - 79 2c 66 37 0d 6c 64 f0-d6 f2 3a 37 21 0c b5 9f y,f7.ld...:7!...
0090 - b6 e8 1a cf 68 7a 78 78-cc 22 9d 86 0b dc 3d c2 ....hzxx."....=.

Start Time: 1518467892
Timeout   : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)

---

[jkat54]

So the error is right there in the first few lines of your response...

You should have a 0 return code. Instead you have 21.

[jkat54]

Is your load balancer configured for HTTPS or SSLTCP?

Can you connect to the search Heads from the search heads without errors using this command?

openssl s_client -connect localhost:{webport}