topic Re: Splunk using cipherSuite caused CLI command issues in Splunk Search

Splunk using cipherSuite caused CLI command issues

sat94541 — Mon, 28 Sep 2020 18:08:40 GMT

I am on Splunk Version : 6.1.3 and trying to use splunk supported cipherSuite from TLSv1.2, but it is causing the CLI command to fail and as a result he is unable to posh the cluster bundle or use the REST end point of the Peer.

Here is Configuration:

../etc/system/local/server.conf [sslConfig]
../etc/system/local/server.conf allowSslCompression = false
../etc/system/default/server.conf allowSslRenegotiation = true
../etc/system/default/server.conf caCertFile = cacert.pem
../etc/system/default/server.conf caPath = ../etc/auth
../etc/system/default/server.conf certCreateScript = ../bin/splunk, createssl, server-cert
../etc/system/local/server.conf cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384
../etc/system/local/server.conf enableSplunkdSSL = true
../etc/system/default/server.conf sendStrictTransportSecurityHeader = false
../etc/system/default/server.conf sslKeysfile = server.pem
../etc/system/local/server.conf sslKeysfilePassword = #####
../etc/system/local/server.conf supportSSLV3Only = True
../etc/system/local/server.conf useClientSSLCompression = false
../etc/system/local/server.conf useSplunkdClientSSLCompression = false

Issue is that CLI command or REST endpoints is failing with error :

./splunk list monitor result in error *

Splunk is not running, and it must be for this operation. To start splunk, run "splunk start". (02)

CLI command to check the bundle status also result in error
splunk show cluster-bundle-status

*Failed to contact the master. ERROR:
Couldn't complete HTTP request: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshak*e failure

The REST endpoint gives error
curl -u admin:changeme -k https://localhost:8089/services/search/jobs -d"search=search *"

curl: (35) SSL connect error

Re: Splunk using cipherSuite caused CLI command issues

rbal_splunk — Fri, 07 Nov 2014 19:48:51 GMT

This issue has been fixed in Splunk Version 6.2. Please consider upgrade.

Re: Splunk using cipherSuite caused CLI command issues

mookiie2005 — Thu, 13 Nov 2014 14:03:52 GMT

um...we just upgraded to splunk 6.2 and we are having the same issue as outlined above. @rbal_splunk are you SURE this was fixed in 6.2 release because it does not seem to be:

$ splunk list forward-server
Couldn't complete HTTP request: Could not find SSL library

$ splunk show cluster-bundle-status

Failed to contact the master. ERROR:
Couldn't complete HTTP request: Could not find SSL library

Failed to contact the peers endpoint. ERROR:
Couldn't complete HTTP request: Could not find SSL library

Re: Splunk using cipherSuite caused CLI command issues

bosburn_splunk — Mon, 28 Sep 2020 18:12:28 GMT

I've been able to recreate it on my side using the following:

Clean install of Splunk 6.2
Update server.conf with the following:

cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384
Restarted Splunk
Ran the following command:

[root@bporepo01 local]# /opt/splunk/bin/splunk list monitor
Couldn't complete HTTP request: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Re-opening the bug.

Re: Splunk using cipherSuite caused CLI command issues

rbal_splunk — Fri, 14 Nov 2014 23:55:24 GMT

For new install of Splunk 6.2 you need to add following to web.conf

[settings]
supportSSLV3Only = false
cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384

Re: Splunk using cipherSuite caused CLI command issues

ppablo — Mon, 28 Sep 2020 18:11:10 GMT

Hi @mookiie2005

There was some confusion with the postings on this question so I just cleaned everything up. Please refer to @bosburn_splunk's answer and @rbal_splunk's comment below that

Re: Splunk using cipherSuite caused CLI command issues

bosburn_splunk — Wed, 26 Nov 2014 21:02:54 GMT

My issue was a different issue - with HP-UX install, LD_LIBRARY has to be set before attempting any CLI commands.

Rajpal again rocks the house.

Re: Splunk using cipherSuite caused CLI command issues

glancaster — Wed, 26 Nov 2014 21:04:42 GMT

Hope people in need will see this. Had the same issue while trying to enable Search Head Clustering and had to add the above configs to my web.conf as well as server.conf under [sslConfig]

Re: Splunk using cipherSuite caused CLI command issues

mookiie2005 — Wed, 26 Nov 2014 23:22:33 GMT

Yes that also resolved the issue for us, Rajpal does indeed rock!

Re: Splunk using cipherSuite caused CLI command issues

mookiie2005 — Wed, 26 Nov 2014 23:23:30 GMT

we also needed the settings in both web and server.conf. in addition to the LD_LIBRARY setting.

Re: Splunk using cipherSuite caused CLI command issues

jayojo — Thu, 15 Jan 2015 17:31:04 GMT

So how do we do this LD Library Setting?

Thanks

Re: Splunk using cipherSuite caused CLI command issues

mookiie2005 — Mon, 28 Sep 2020 18:50:32 GMT

For us it was "export LD_LIBRARY_PATH=/opt/splunk/splunkforwarder/lib"

But it is based off of your install path for the splunk lib directory it would most likely be:
export LD_LIBRARY_PATH=/opt/splunkforwarder/lib

srry for the delay in response

Re: Splunk using cipherSuite caused CLI command issues

Action01 — Fri, 10 Feb 2017 17:11:00 GMT

I got this error whilst trying to disable the obsolete (according to Google Chrome) RSA key exchange ciphers by removing the "RSA+AESGCM:RSA+AES" from my cipherSuite.

Upon investigating this issue, it seemed to be caused by the fact that out of the box, splunk cannot use the ECDH ciphers (ecdhCurves = ), so it always does a fallback to the ciphers with RSA as key exchange. But if you remove the RSA ciphers, there are no ciphers left for splunk to use, so it fails.

By specifying "ecdhCurves = prime256v1,secp384r1,secp521r1" the ECDH ciphers can be used, and solves the problem.

side-note;
Using this in combination with the following config enables HSTS headers and Perfect Forward Secrecy:

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;
ecdhCurves = prime256v1,secp384r1,secp521r1
sendStictTransportSecurityHeader = true
sslVersions = tls1.2

The above configuration is applicable to server.conf under [sslConfig] for the mgmt and kv store port, and in web.conf under [settings] for splunkweb.
(The ciperSuite is taken form the Mozilla SSL Configuration Generator using the modern profile. Older browsers may not support this)