topic Re: A writeup: KV store / mongo TLS configuration needs serious work in Security

A writeup: KV store / mongo TLS configuration needs serious work

skalliger — Sun, 22 Oct 2023 00:47:05 GMT

Hello everyone,

so, many hours went by. It all started with the parameters which were introduced in Splunk 9 (docs reference).

Specificially, we should harden the KV store. I've spent several hours in many environments and not a single time I was able to do so. Today, I spent many hours trying to solve it with no success. Here's the problem:

I've configured everything and everything is working fine, except KV store.

[sslConfig] cliVerifyServerName = true sslVerifyServerCert = true sslVerifyServerName = true sslRootCAPath = $SPLUNK_HOME/etc/your/path/your_CA.pem [kvstore] sslVerifyServerCert = true sslVerifyServerName = true serverCert = $SPLUNK_HOME/etc/your/path/your_cert.pem sslPassword = [pythonSslClientConfig] sslVerifyServerCert = true sslVerifyServerName = true [search_state] sslVerifyServerCert = true

(btw, search_state is neither listed in the docs nor does the value display in the UI, however an error is logged if it's not set).

You can put the sslPassword parameter in or not, doesn't matter.

What you'll always end up when enabling sslVerifyServerCert and sslVerifyServerName is in the mongod.log:

2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** WARNING: This server will not perform X.509 hostname validation 2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** This may allow your server to make or accept connections to 2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** untrusted parties 2023-10-22T00:11:28.557Z I CONTROL [initandlisten] 2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** WARNING: No client certificate validation can be performed since no CA file has been provided 2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.

Splunk doesn't seem to be parsing the required parameters to Mongo as it's expecting them, let's dig a bit. This is what you'll find at startups:

2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead. 2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslPEMKeyFile is deprecated. Please use tlsCertificateKeyFile instead. 2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslPEMKeyPassword is deprecated. Please use tlsCertificateKeyFilePassword instead. 2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead. 2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead. 2023-10-21T20:31:54.641Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release. 2023-10-21T20:31:54.644Z W NETWORK [main] Server certificate has no compatible Subject Alternative Name. This may prevent TLS clients from connecting 2023-10-21T20:31:54.645Z W ASIO [main] No TransportLayer configured during NetworkInterface startup

Has anyone ever tested the TLS verification settings? All of the tlsVerify* settings are just very inconsistent in Splunk 9 and I don't see them mentioned often. Also I don't find any bugs or issues listed with KV store encryption.

If you list those parameters on the docs, I expect them to work. A "ps -ef | grep mongo" will list you what options are parsed from Splunk to Mongo, formatted for readability.

mongod --dbpath=/data/splunk/var/lib/splunk/kvstore/mongo --storageEngine=wiredTiger --wiredTigerCacheSizeGB=3.600000 --port=8191 --timeStampFormat=iso8601-utc --oplogSize=200 --keyFile=/data/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --setParameter=oplogFetcherSteadyStateMaxFetcherRestarts=0 --replSet=8B532733-2DEF-42CC-82E5-38E990F3CD04 --bind_ip=0.0.0.0 --sslMode=requireSSL --sslAllowInvalidHostnames --sslPEMKeyFile=/data/splunk/etc/auth/newCerts/machine/deb-spl_full.pem --sslPEMKeyPassword=xxxxxxxx --tlsDisabledProtocols=noTLS1_0,noTLS1_1 --sslCipherConfig=ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-DSS-AES256-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256 --nounixsocket --noscripting

I even tried messing around with old server.conf parameters like

caCertFile or sslKeysPassword, but it seems like the CA is simply never parsed as an argument. Why did no one stumple upon this?

How did I find all of that? I have developed an app which gives an overview of the Splunk environment's mitigation status against current Splunk Vulnerabity Disclosures (SVDs) as well as recommended best practice encryption settings.

If anyone has a working KV store TLS config, I'm eager to see that.

Skalli

Re: A writeup: KV store / mongo TLS configuration needs serious work

arcsight_guru — Tue, 24 Oct 2023 09:06:48 GMT

Hey,

Nice write-up, this is certainly an interesting subject. The whole SSL/TLS implementations seems a bit rushed and indeed not very well documented.

Did you try this on a Search Head Cluster? Because it clearly states "TLS host name validation only works for search head clusters that use App Key Value Store." on the docs page you referred to.

Also in the server.conf.spec for the [kvstore] serverCert setting, it says:

* Only used when Common Criteria is enabled (SPLUNK_COMMON_CRITERIA=1)
  or FIPS is enabled (i.e. SPLUNK_FIPS=1).

My conclusion is that when KV-Store is in stand-alone mode there is no need to verify certificates since there will never be external connections in either direction. When traffic is localhost only I guess Splunk consider it "secure" enough - unless FIPS or CC is enabled.

But I find it very annoying that you get a warning at each start-up that the KV-store is not "secure" even though it is stand-alone.

I haven't had the opportunity to test this out in a clustered environment yet, but I will for sure let you know if I do.

Please let me know if you make any progress in this matter.

Re: A writeup: KV store / mongo TLS configuration needs serious work

jmartens — Mon, 26 Feb 2024 07:15:26 GMT

@skalliger wrote:
How did I find all of that? I have developed an app which gives an overview of the Splunk environment's mitigation status against current Splunk Vulnerabity Disclosures (SVDs) as well as recommended best practice encryption settings.

Nice work @skalliger . Is this app publically available? I would be interested in this functionality.

Re: A writeup: KV store / mongo TLS configuration needs serious work

skalliger — Thu, 14 Nov 2024 20:11:28 GMT

Sorry for the late reply. As I've changed my mail over the years, I don't receive email notifications from replies. Here's the app: https://github.com/skalliger/encryption_and_vulnerability_check

Re: A writeup: KV store / mongo TLS configuration needs serious work

moliminous — Thu, 20 Feb 2025 06:03:48 GMT

You must be running in FIPS mode.

The settings will work fine in non-FIPS, but in FIPS mode if you even add the 1 setting of 'sslRootCAPath' your kvstore will fail.

The only way to get kvstore to run in FIPS mode with your own certs is to rename your certs to be the default filepaths, which is $SPLUNK_HOME/etc/auth/server.pem and cacert.pem

You will also have to cat 'appsCA.pem' to the end of your cacert.pem so that the Manage Apps UI works.

Note that the kvstore is only needed on Search Heads, so you could disable or ignore it on non-SHs.

The other problem you will still have is you cannot set 'requireClientCert=true' in FIPS mode, possibly in non-FIPS I have to confirm that again.

Re: A writeup: KV store / mongo TLS configuration needs serious work

apolo — Sun, 17 Aug 2025 19:13:22 GMT

Hi! I had tried everything to raise the kvstore with mongo embedded version 7, had other errors that were already solved, but still did not take the certificates, I changed the database paths, I was raising mongo manually with certificates created for it and it worked, so I found your answer that matched what I wanted to do ... Putting the default names, so that was the only thing I was missing to finish solving it in a FIPS environment, so your solution with the default names, encouraged me to do it definitely! Splunk does not indicate in its official documentation this ‘casuistry’, so I wanted to put a comment to say that in my case with FIPS it works! Thank you very much!

Re: A writeup: KV store / mongo TLS configuration needs serious work

skalliger — Mon, 18 Aug 2025 07:52:02 GMT

Sorry for the delay, somehow I didn't get notifications for this topic.
My app is here: https://github.com/skalliger/encryption_and_vulnerability_check

It needs some little SPL changes but that's for another topic, not really KV / TLS-related.