KVstore unable to start after upgrade to Splunk Enterprise 9.4

[gloom]

Hi,

After completing the upgrade from Splunk Enterprise version 9.3.2 to v9.4 the KVstore will no longer start. Splunk has yet to do the KVstore upgrade to v7 as the KVstore cannot start. We were already on 4.2 wiredtiger.

The is no [kvstore] stanza in server.conf so everything should be default.

The relavent lines from splunkd.log are:

INFO  KVStoreConfigurationProvider [9192 MainThread] - Since x509 is not enabled - using a default config from [sslConfig] for Mongod mTLS authentication
WARN  KVStoreConfigurationProvider [9192 MainThread] - Action scheduled, but event loop is not ready yet
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Starting mongod with executable name=mongod-4.2.exe version=kvstore version 4.2
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --dbpath C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo 
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --storageEngine wiredTiger
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using cacheSize=1.65GB
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --port 8191
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --timeStampFormat iso8601-utc
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --oplogSize 200
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --keyFile C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\splunk.key
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --setParameter enableLocalhostAuthBypass=0
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --setParameter oplogFetcherSteadyStateMaxFetcherRestarts=0
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --replSet 4EA2F2AF-2584-4BB0-A2C4-414E7CB68BC2
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --bind_ip=0.0.0.0 (all ipv4 addresses)
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslCAFile C:\Program Files\Splunk\etc\auth\cacert.pem
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --tlsAllowConnectionsWithoutCertificates for version 4.2
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslMode requireSSL
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidHostnames
WARN  KVStoreConfigurationProvider [9192 MainThread] - Action scheduled, but event loop is not ready yet
INFO  KVStoreConfigurationProvider [9192 MainThread] - "SAML cert db" registration with KVStore successful
INFO  KVStoreConfigurationProvider [9192 MainThread] - "Auth cert db" registration with KVStore successful
INFO  KVStoreConfigurationProvider [9192 MainThread] - "JsonWebToken Manager" registration with KVStore successful
INFO  KVStoreBackupRestore [1436 KVStoreBackupThread] - thread started.
INFO  KVStoreConfigurationProvider [9192 MainThread] - "Certificate Manager" registration with KVStore successful
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Found an existing PFX certificate
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslCertificateSelector subject=SplunkServerDefaultCert
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslAllowInvalidCertificates
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --tlsDisabledProtocols noTLS1_0,noTLS1_1
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --sslCipherConfig ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDH-ECDSA-AES256-GCM-SHA384:ECDH-ECDSA-AES128-GCM-SHA256:ECDH-ECDSA-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES128-SHA256
INFO  MongodRunner [7668 KVStoreConfigurationThread] - Using mongod command line --noscripting
WARN  MongoClient [7668 KVStoreConfigurationThread] - Disabling TLS hostname validation for localhost
ERROR MongodRunner [5692 MongodLogThread] - mongod exited abnormally (exit code 14, status: exited with code 14) - look at mongod.log to investigate.
ERROR KVStoreBulletinBoardManager [5692 MongodLogThread] - KV Store process terminated abnormally (exit code 14, status exited with code 14). See mongod.log and splunkd.log for details.
WARN  KVStoreConfigurationProvider [5692 MongodLogThread] - Action scheduled, but event loop is not ready yet
ERROR KVStoreBulletinBoardManager [5692 MongodLogThread] - KV Store changed status to failed. KVStore process terminated..
ERROR KVStoreConfigurationProvider [7668 KVStoreConfigurationThread] - Failed to start mongod on first attempt reason=KVStore service will not start because kvstore process terminated
ERROR KVStoreConfigurationProvider [7668 KVStoreConfigurationThread] - Could not start mongo instance. Initialization failed.
ERROR KVStoreBulletinBoardManager [7668 KVStoreConfigurationThread] - Failed to start KV Store process. See mongod.log and splunkd.log for details.
INFO  KVStoreConfigurationProvider [7668 KVStoreConfigurationThread] - Mongod service shutting down

mogod.log contains the following:

W  CONTROL  [main] Option: sslMode is deprecated. Please use tlsMode instead.
W  CONTROL  [main] Option: sslCAFile is deprecated. Please use tlsCAFile instead.
W  CONTROL  [main] Option: sslCipherConfig is deprecated. Please use tlsCipherConfig instead.
W  CONTROL  [main] Option: sslAllowInvalidHostnames is deprecated. Please use tlsAllowInvalidHostnames instead.
W  CONTROL  [main] Option: sslAllowInvalidCertificates is deprecated. Please use tlsAllowInvalidCertificates instead.
W  CONTROL  [main] Option: sslCertificateSelector is deprecated. Please use tlsCertificateSelector instead.
W  CONTROL  [main] net.tls.tlsCipherConfig is deprecated. It will be removed in a future release.
W  NETWORK  [main] Mixing certs from the system certificate store and PEM files. This may produced unexpected results.
W  NETWORK  [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
W  NETWORK  [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
W  NETWORK  [main] Server certificate has no compatible Subject Alternative Name. This may prevent TLS clients from connecting
W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
W  NETWORK  [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
W  ASIO     [main] No TransportLayer configured during NetworkInterface startup
W  NETWORK  [main] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
I  CONTROL  [initandlisten] MongoDB starting : pid=4640 port=8191 dbpath=C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo 64-bit host=[redacted]
I  CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2
I  CONTROL  [initandlisten] db version v4.2.24
I  CONTROL  [initandlisten] git version: 5e4ec1d24431fcdd28b579a024c5c801b8cde4e2
I  CONTROL  [initandlisten] allocator: tcmalloc
I  CONTROL  [initandlisten] modules: enterprise 
I  CONTROL  [initandlisten] build environment:
I  CONTROL  [initandlisten]     distmod: windows-64
I  CONTROL  [initandlisten]     distarch: x86_64
I  CONTROL  [initandlisten]     target_arch: x86_64
I  CONTROL  [initandlisten] options: { net: { bindIp: "0.0.0.0", port: 8191, tls: { CAFile: "C:\Program Files\Splunk\etc\auth\cacert.pem", allowConnectionsWithoutCertificates: true, allowInvalidCertificates: true, allowInvalidHostnames: true, certificateSelector: "subject=SplunkServerDefaultCert", disabledProtocols: "noTLS1_0,noTLS1_1", mode: "requireTLS", tlsCipherConfig: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RS..." } }, replication: { oplogSizeMB: 200, replSet: "4EA2F2AF-2584-4BB0-A2C4-414E7CB68BC2" }, security: { javascriptEnabled: false, keyFile: "C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo\splunk.key" }, setParameter: { enableLocalhostAuthBypass: "0", oplogFetcherSteadyStateMaxFetcherRestarts: "0" }, storage: { dbPath: "C:\Program Files\Splunk\var\lib\splunk\kvstore\mongo", engine: "wiredTiger", wiredTiger: { engineConfig: { cacheSizeGB: 1.65 } } }, systemLog: { timeStampFormat: "iso8601-utc" } }
W  NETWORK  [initandlisten] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
W  NETWORK  [initandlisten] sslCipherConfig parameter is not supported with Windows SChannel and is ignored.
I  STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1689M,cache_overflow=(file_max=0M),session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000,close_scan_interval=10,close_handle_minimum=250),statistics_log=(wait=0),verbose=[recovery_progress,checkpoint_progress],
W  STORAGE  [initandlisten] Failed to start up WiredTiger under any compatibility version.
F  STORAGE  [initandlisten] Reason: 129: Operation not supported
F  -        [initandlisten] Fatal Assertion 28595 at src\mongo\db\storage\wiredtiger\wiredtiger_kv_engine.cpp 928
F  -        [initandlisten] \n\n***aborting after fassert() failure\n\n

 Does anyone have any idea how to resolve this?

Thanks,

[MaverickT]

Maybe this can be solution to your challenge: https://community.splunk.com/t5/Deployment-Architecture/KVStore-does-not-start-when-running-Splunk-9...

[gloom]

This seems confusing, as Splunk hasn't attempted to do the mongodb upgrade yet, I would expect it to fail after the upgrade if this was the case?

Edit: I ran HWinfo on the box, its showing AVX, AVX2 and AVX-512 supported, so I don't think this is the issue.

[isoutamo]

Hi

Based on log you are running unsupported OS.

 CONTROL  [initandlisten] targetMinOS: Windows 7/Windows Server 2008 R2

On Windows operating systems oldest supported version is Win 2019 or Win 10.

r. Ismo 

[gloom]

Thats incorrect, its a server 2022 box.