A writeup: KV store / mongo TLS configuration needs serious work

[skalliger]

Hello everyone,

so, many hours went by. It all started with the parameters which were introduced in Splunk 9 (docs reference).

Specificially, we should harden the KV store. I've spent several hours in many environments and not a single time I was able to do so. Today, I spent many hours trying to solve it with no success. Here's the problem:

I've configured everything and everything is working fine, except KV store.

[sslConfig]
cliVerifyServerName = true
sslVerifyServerCert = true
sslVerifyServerName = true
sslRootCAPath = $SPLUNK_HOME/etc/your/path/your_CA.pem

[kvstore]
sslVerifyServerCert = true
sslVerifyServerName = true
serverCert = $SPLUNK_HOME/etc/your/path/your_cert.pem
sslPassword =

[pythonSslClientConfig]
sslVerifyServerCert = true
sslVerifyServerName = true

[search_state]
sslVerifyServerCert = true

(btw, search_state is neither listed in the docs nor does the value display in the UI, however an error is logged if it's not set).

You can put the sslPassword parameter in or not, doesn't matter.

What you'll always end up when enabling sslVerifyServerCert and sslVerifyServerName is in the mongod.log:

2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** WARNING: This server will not perform X.509 hostname validation
2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** This may allow your server to make or accept connections to
2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** untrusted parties
2023-10-22T00:11:28.557Z I CONTROL [initandlisten]
2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** WARNING: No client certificate validation can be performed since no CA file has been provided
2023-10-22T00:11:28.557Z I CONTROL [initandlisten] ** Please specify an sslCAFile parameter.

Splunk doesn't seem to be parsing the required parameters to Mongo as it's expecting them, let's dig a bit. This is what you'll find at startups:

2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslMode is deprecated. Please use tlsMode instead.
2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslPEMKeyFile is deprecated. Please use tlsCertificateKeyFile instead.
2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslPEMKeyPassword is deprecated. Please use tlsCertificateKeyFilePassword instead.
2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead.
2023-10-21T22:31:54.640+0200 W CONTROL [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead.
2023-10-21T20:31:54.641Z W CONTROL [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release.
2023-10-21T20:31:54.644Z W NETWORK [main] Server certificate has no compatible Subject Alternative Name. This may prevent TLS clients from connecting
2023-10-21T20:31:54.645Z W ASIO [main] No TransportLayer configured during NetworkInterface startup

Has anyone ever tested the TLS verification settings? All of the tlsVerify* settings are just very inconsistent in Splunk 9 and I don't see them mentioned often. Also I don't find any bugs or issues listed with KV store encryption.

If you list those parameters on the docs, I expect them to work. A "ps -ef | grep mongo" will list you what options are parsed from Splunk to Mongo, formatted for readability.

mongod --dbpath=/data/splunk/var/lib/splunk/kvstore/mongo
--storageEngine=wiredTiger
--wiredTigerCacheSizeGB=3.600000
--port=8191
--timeStampFormat=iso8601-utc
--oplogSize=200
--keyFile=/data/splunk/var/lib/splunk/kvstore/mongo/splunk.key
--setParameter=enableLocalhostAuthBypass=0
--setParameter=oplogFetcherSteadyStateMaxFetcherRestarts=0
--replSet=8B532733-2DEF-42CC-82E5-38E990F3CD04
--bind_ip=0.0.0.0 --sslMode=requireSSL
--sslAllowInvalidHostnames
--sslPEMKeyFile=/data/splunk/etc/auth/newCerts/machine/deb-spl_full.pem
--sslPEMKeyPassword=xxxxxxxx
--tlsDisabledProtocols=noTLS1_0,noTLS1_1
--sslCipherConfig=ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:DHE-DSS-AES256-GCM-SHA384:DHE-DSS-AES256-SHA256:DHE-DSS-AES128-GCM-SHA256:DHE-DSS-AES128-SHA256:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES256-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256
--nounixsocket --noscripting

I even tried messing around with old server.conf parameters like

caCertFile or sslKeysPassword, but it seems like the CA is simply never parsed as an argument. Why did no one stumple upon this?

How did I find all of that? I have developed an app which gives an overview of the Splunk environment's mitigation status against current Splunk Vulnerabity Disclosures (SVDs) as well as recommended best practice encryption settings.

If anyone has a working KV store TLS config, I'm eager to see that.

Skalli

[jmartens]

---

@skalliger wrote:

How did I find all of that? I have developed an app which gives an overview of the Splunk environment's mitigation status against current Splunk Vulnerabity Disclosures (SVDs) as well as recommended best practice encryption settings.

 Nice work @skalliger . Is this app publically available? I would be interested in this functionality.

[arcsight_guru]

Hey,

Nice write-up, this is certainly an interesting subject. The whole SSL/TLS implementations seems a bit rushed and indeed not very well documented.

Did you try this on a Search Head Cluster? Because it clearly states "TLS host name validation only works for search head clusters that use App Key Value Store." on the docs page you referred to.

Also in the server.conf.spec for the [kvstore] serverCert setting, it says:

* Only used when Common Criteria is enabled (SPLUNK_COMMON_CRITERIA=1)
  or FIPS is enabled (i.e. SPLUNK_FIPS=1).

My conclusion is that when KV-Store is in stand-alone mode there is no need to verify certificates since there will never be external connections in either direction. When traffic is localhost only I guess Splunk consider it "secure" enough - unless FIPS or CC is enabled.

But I find it very annoying that you get a warning at each start-up that the KV-store is not "secure" even though it is stand-alone.

I haven't had the opportunity to test this out in a clustered environment yet, but I will for sure let you know if I do.

Please let me know if you make any progress in this matter.