Splunk Search

Splunk using cipherSuite caused CLI command issues

sat94541
Communicator

I am on Splunk Version : 6.1.3 and trying to use splunk supported cipherSuite from TLSv1.2, but it is causing the CLI command to fail and as a result he is unable to posh the cluster bundle or use the REST end point of the Peer.

Here is Configuration:

../etc/system/local/server.conf [sslConfig]
../etc/system/local/server.conf allowSslCompression = false
../etc/system/default/server.conf allowSslRenegotiation = true
../etc/system/default/server.conf caCertFile = cacert.pem
../etc/system/default/server.conf caPath = ../etc/auth
../etc/system/default/server.conf certCreateScript = ../bin/splunk, createssl, server-cert
../etc/system/local/server.conf cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384
../etc/system/local/server.conf enableSplunkdSSL = true
../etc/system/default/server.conf sendStrictTransportSecurityHeader = false
../etc/system/default/server.conf sslKeysfile = server.pem
../etc/system/local/server.conf sslKeysfilePassword = #####
../etc/system/local/server.conf supportSSLV3Only = True
../etc/system/local/server.conf useClientSSLCompression = false
../etc/system/local/server.conf useSplunkdClientSSLCompression = false

Issue is that CLI command or REST endpoints is failing with error :

./splunk list monitor result in error *

Splunk is not running, and it must be for this operation. To start splunk, run "splunk start". (02)

CLI command to check the bundle status also result in error
splunk show cluster-bundle-status

*Failed to contact the master. ERROR:
Couldn't complete HTTP request: error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshak*e failure

The REST endpoint gives error
curl -u admin:changeme -k https://localhost:8089/services/search/jobs -d"search=search *"

curl: (35) SSL connect error

0 Karma

Action01
Loves-to-Learn

I got this error whilst trying to disable the obsolete (according to Google Chrome) RSA key exchange ciphers by removing the "RSA+AESGCM:RSA+AES" from my cipherSuite.

Upon investigating this issue, it seemed to be caused by the fact that out of the box, splunk cannot use the ECDH ciphers (ecdhCurves = ), so it always does a fallback to the ciphers with RSA as key exchange. But if you remove the RSA ciphers, there are no ciphers left for splunk to use, so it fails.

By specifying "ecdhCurves = prime256v1,secp384r1,secp521r1" the ECDH ciphers can be used, and solves the problem.

side-note;
Using this in combination with the following config enables HSTS headers and Perfect Forward Secrecy:

cipherSuite = ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256;
ecdhCurves = prime256v1,secp384r1,secp521r1
sendStictTransportSecurityHeader = true
sslVersions = tls1.2

The above configuration is applicable to server.conf under [sslConfig] for the mgmt and kv store port, and in web.conf under [settings] for splunkweb.
(The ciperSuite is taken form the Mozilla SSL Configuration Generator using the modern profile. Older browsers may not support this)

0 Karma

bosburn_splunk
Splunk Employee
Splunk Employee

I've been able to recreate it on my side using the following:

  1. Clean install of Splunk 6.2
  2. Update server.conf with the following:

    cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384

  3. Restarted Splunk

  4. Ran the following command:

    [root@bporepo01 local]# /opt/splunk/bin/splunk list monitor
    Couldn't complete HTTP request: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Re-opening the bug.

jayojo
New Member

So how do we do this LD Library Setting?

Thanks

0 Karma

mookiie2005
Communicator

For us it was "export LD_LIBRARY_PATH=/opt/splunk/splunkforwarder/lib"

But it is based off of your install path for the splunk lib directory it would most likely be:
export LD_LIBRARY_PATH=/opt/splunkforwarder/lib

srry for the delay in response

0 Karma

rbal_splunk
Splunk Employee
Splunk Employee

For new install of Splunk 6.2 you need to add following to web.conf

[settings]
supportSSLV3Only = false
cipherSuite = AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDH-RSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-GCM-SHA384

glancaster
Path Finder

Hope people in need will see this. Had the same issue while trying to enable Search Head Clustering and had to add the above configs to my web.conf as well as server.conf under [sslConfig]

mookiie2005
Communicator

we also needed the settings in both web and server.conf. in addition to the LD_LIBRARY setting.

bosburn_splunk
Splunk Employee
Splunk Employee

My issue was a different issue - with HP-UX install, LD_LIBRARY has to be set before attempting any CLI commands.

Rajpal again rocks the house.

0 Karma

mookiie2005
Communicator

Yes that also resolved the issue for us, Rajpal does indeed rock!

rbal_splunk
Splunk Employee
Splunk Employee

This issue has been fixed in Splunk Version 6.2. Please consider upgrade.

mookiie2005
Communicator

um...we just upgraded to splunk 6.2 and we are having the same issue as outlined above. @rbal_splunk are you SURE this was fixed in 6.2 release because it does not seem to be:

$ splunk list forward-server
Couldn't complete HTTP request: Could not find SSL library

$ splunk show cluster-bundle-status

Failed to contact the master. ERROR:
Couldn't complete HTTP request: Could not find SSL library

Failed to contact the peers endpoint. ERROR:
Couldn't complete HTTP request: Could not find SSL library

0 Karma

ppablo
Retired

Hi @mookiie2005

There was some confusion with the postings on this question so I just cleaned everything up. Please refer to @bosburn_splunk's answer and @rbal_splunk's comment below that

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...