Splunk CLI command fails with SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure using high strenghth cipher

[sat94541]

Have issue in 6.2.3 and Search Head Cluster- but I have reproduced it also on out of the box version 6.3.3 Standalone Splunk instance.
Splunk has default out of the Box cipher in server.conf as shown below
Server.con

.../etc/system/local/server.conf [sslConfig]
.../etc/system/default/server.conf allowSslCompression = true
.../etc/system/default/server.conf allowSslRenegotiation = true
.../etc/system/default/server.conf caCertFile = cacert.pem
.../etc/system/default/server.conf caPath = $SPLUNK_HOME/etc/auth
.../etc/system/default/server.conf certCreateScript = $SPLUNK_HOME/bin/splunk, createssl, server-cert
.../etc/system/default/server.conf cipherSuite = TLSv1+HIGH:@STRENGTH
.../etc/system/default/server.conf enableSplunkdSSL = true
.../etc/system/default/server.conf sendStrictTransportSecurityHeader = false
.../etc/system/default/server.conf sslKeysfile = server.pem
.../etc/system/local/server.conf sslKeysfilePassword = $1$LqYAinIu/4eI
.../etc/system/default/server.conf sslVersions = *,-ssl2
.../etc/system/default/server.conf useClientSSLCompression = true
.../etc/system/default/server.conf useSplunkdClientSSLCompression = true

For the web.conf customer is using stronger cipher like
web.conf

…

/etc/system/local/web.conf allowSslRenegotiation = false
…/etc/system/local/web.conf cipherSuite = EECDH:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK
…./etc/system/local/web.conf ecdhCurveName = secp384r1
…/etc/system/local/web.conf enableSplunkWebSSL = True
…/etc/system/local/web.conf sslVersions = tls1.2

*Command below and many other command like cluster status ext fails with error *
$SPLUNK_HOME/bin/splunk help
Couldn't complete HTTP request: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Status command works fine

$SPLUNK_HOME/bin/splunk status
splunkd is running (PID: 23485).
splunk helpers are running (PIDs: 23507 24001 24595 25515).

Due to this we are having an issue that Search Head Cluster members are unable to communicate with other members in group.

[rbal_splunk]

1)In Splunk Version 6.2.3
In the Search Head modified the below mentioned conf files

cat web.conf
[settings]
enableSplunkWebSSL = 1
cipherSuite = EECDH:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK
ecdhCurveName = secp384r1
sslVersions = tls1.2
allowSslRenegotiation = false
cat server.conf
[sslConfig]
sslKeysfilePassword = $1$IzgaP3G/xTrd
cipherSuite = EECDH:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK
ecdhCurveName = secp384r1
....
Please Note: If we don’t specify cipherSuite = EECDH:!SSLv3:!aNULL:!eNULL:!EXPORT:!DES:!DSS:!RC4:!3DES:!MD5:!PSK in server.conf but add only ecdhCurveName = secp384r1 in server.conf then it shows the following:
$SPLUNK_HOME/bin/splunk help

Couldn't complete HTTP request: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

So in order to make everything work successfully, it needs both cipherSuite and ecdhCurveName. This happens only in Splunk 6.2.3 (build 264376)

2) In this distributed Search Environment, we have 1 SH and 2 Search-Peers(Indexers). I have added cipherSuite and ecdhCurveName inside both web.conf and server.conf for one peer [Here:10.222.30.238 but the other peer has No change]

I noticed the peer which has the cipherSuite and ecdhCurveName, is able to send data successfully, that we can search on SH. But, the peer which did not had cipherSuite and ecdhCurveName.

Based on the observation - all the members in the distributed environment [Here 1 SH, 2 peers] needs to have the same configuration changes in both server.conf and web.conf.

3)Next Migrated to Splunk Version 6.3.3 . f we only give ecdhCurveName and miss out cipherSuite , it works successfully. The peer is able to send the data, that can be searched from SH.