I am on Splunk Version : 6.1.3 and trying to use splunk supported cipherSuite from TLSv1.2, but it is causing the CLI command to fail and as a result he is unable to posh the cluster bundle or use the REST end point of the Peer.
Here is Configuration:
../etc/system/local/server.conf [sslConfig]
../etc/system/local/server.conf allowSslCompression = false
../etc/system/default/server.conf allowSslRenegotiation = true
../etc/system/default/server.conf caCertFile = cacert.pem
../etc/system/default/server.conf caPath = ../etc/auth
../etc/system/default/server.conf certCreateScript = ../bin/splunk, createssl, server-cert
../etc/system/local/server.conf cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384
../etc/system/local/server.conf enableSplunkdSSL = true
../etc/system/default/server.conf sendStrictTransportSecurityHeader = false
../etc/system/default/server.conf sslKeysfile = server.pem
../etc/system/local/server.conf sslKeysfilePassword = #####
../etc/system/local/server.conf supportSSLV3Only = True
../etc/system/local/server.conf useClientSSLCompression = false
../etc/system/local/server.conf useSplunkdClientSSLCompression = false
Issue is that CLI command or REST endpoints is failing with error :
./splunk list monitor result in error *
Splunk is not running, and it must be for this operation. To start splunk, run "splunk start". (02)
CLI command to check the bundle status also result in error
splunk show cluster-bundle-status
*Failed to contact the master. ERROR:
Couldn't complete HTTP request: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshak*e failure
The REST endpoint gives error
curl -u admin:changeme -k https://localhost:8089/services/search/jobs -d"search=search *"
curl: (35) SSL connect error
I got this error whilst trying to disable the obsolete (according to Google Chrome) RSA key exchange ciphers by removing the "RSA+AESGCM:RSA+AES" from my cipherSuite.
Upon investigating this issue, it seemed to be caused by the fact that out of the box, splunk cannot use the ECDH ciphers (ecdhCurves = ), so it always does a fallback to the ciphers with RSA as key exchange. But if you remove the RSA ciphers, there are no ciphers left for splunk to use, so it fails.
By specifying "ecdhCurves = prime256v1,secp384r1,secp521r1" the ECDH ciphers can be used, and solves the problem.
side-note;
Using this in combination with the following config enables HSTS headers and Perfect Forward Secrecy:
cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;
ecdhCurves = prime256v1,secp384r1,secp521r1
sendStictTransportSecurityHeader = true
sslVersions = tls1.2
The above configuration is applicable to server.conf under [sslConfig] for the mgmt and kv store port, and in web.conf under [settings] for splunkweb.
(The ciperSuite is taken form the Mozilla SSL Configuration Generator using the modern profile. Older browsers may not support this)
I've been able to recreate it on my side using the following:
Update server.conf with the following:
cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384
Restarted Splunk
Ran the following command:
[root@bporepo01 local]# /opt/splunk/bin/splunk list monitor
Couldn't complete HTTP request: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure
Re-opening the bug.
So how do we do this LD Library Setting?
Thanks
For us it was "export LD_LIBRARY_PATH=/opt/splunk/splunkforwarder/lib"
But it is based off of your install path for the splunk lib directory it would most likely be:
export LD_LIBRARY_PATH=/opt/splunkforwarder/lib
srry for the delay in response
For new install of Splunk 6.2 you need to add following to web.conf
[settings]
supportSSLV3Only = false
cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384
Hope people in need will see this. Had the same issue while trying to enable Search Head Clustering and had to add the above configs to my web.conf as well as server.conf under [sslConfig]
we also needed the settings in both web and server.conf. in addition to the LD_LIBRARY setting.
My issue was a different issue - with HP-UX install, LD_LIBRARY has to be set before attempting any CLI commands.
Rajpal again rocks the house.
Yes that also resolved the issue for us, Rajpal does indeed rock!
This issue has been fixed in Splunk Version 6.2. Please consider upgrade.
um...we just upgraded to splunk 6.2 and we are having the same issue as outlined above. @rbal_splunk are you SURE this was fixed in 6.2 release because it does not seem to be:
$ splunk list forward-server
Couldn't complete HTTP request: Could not find SSL library
$ splunk show cluster-bundle-status
Failed to contact the master. ERROR:
Couldn't complete HTTP request: Could not find SSL library
Failed to contact the peers endpoint. ERROR:
Couldn't complete HTTP request: Could not find SSL library
Hi @mookiie2005
There was some confusion with the postings on this question so I just cleaned everything up. Please refer to @bosburn_splunk's answer and @rbal_splunk's comment below that