Why is there KVStore failure after upgrade to 9.0?

[bigfatyeastroll]

After upgrading to Splunk 9.0 on a single instance, we occasionally get KV Store errors. 

CLI status shows:

This member:
backupRestoreStatus : Ready
disabled : 0
featureCompatibilityVersion : An error occurred during the last operation ('getParameter', domain: '15', code: '13053'): No suitable servers found: `serverSelectionTimeoutMS` expired: [connection closed calling ismaster on '127.0.0.1:8191']
guid : E8254C08-B854-426C-B66D-7072D625D0F6
port : 8191
standalone : 1
status : failed
storageEngine : mmapv1

I've looked on Splunk.com and googled but haven't found anything on single instances beyond reinstalling 9.0 which I've done twice.

[bigfatyeastroll]

Here are the following tails from the log files:

root@splunk:/opt/splunk/var/log/splunk# tail mongod.log
2022-06-28T19:40:57.749Z F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager_openssl.cpp 987
2022-06-28T19:40:57.749Z F - [main] \n\n***aborting after fassert() failure\n\n
2022-06-28T20:26:48.752Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
2022-06-28T20:26:48.763Z F NETWORK [main] The provided SSL certificate is expired or not yet valid.
2022-06-28T20:26:48.763Z F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager_openssl.cpp 987
2022-06-28T20:26:48.763Z F - [main] \n\n***aborting after fassert() failure\n\n
2022-06-29T15:04:51.137Z W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
2022-06-29T15:04:51.149Z F NETWORK [main] The provided SSL certificate is expired or not yet valid.
2022-06-29T15:04:51.149Z F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager_openssl.cpp 987
2022-06-29T15:04:51.149Z F - [main] \n\n***aborting after fassert() failure\n\n
root@splunk:/opt/splunk/var/log/splunk# tail splunkd.log
06-29-2022 11:11:14.993 -0500 WARN AuthorizationManager [44284 SavedSearchFetcher] - Unknown role 'winfra-admin'
06-29-2022 11:11:14.994 -0500 WARN AuthorizationManager [44284 SavedSearchFetcher] - Unknown role 'winfra-admin'
06-29-2022 11:11:14.994 -0500 WARN AuthorizationManager [44284 SavedSearchFetcher] - Unknown role 'winfra-admin'
06-29-2022 11:11:18.206 -0500 WARN HttpListener [44522 webui] - Socket error from 161.31.28.185:52910 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
06-29-2022 11:11:18.214 -0500 WARN HttpListener [44522 webui] - Socket error from 161.31.28.185:52911 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
06-29-2022 11:11:18.219 -0500 WARN HttpListener [44522 webui] - Socket error from 161.31.28.185:52912 while idling: error:140760FC:SSL routines:SSL23_GET_CLIENT_HELLO:unknown protocol
06-29-2022 11:11:24.098 -0500 INFO ExecProcessor [44189 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:228] [get_server_roles] [65977] Fetched server roles, roles=['indexer', 'license_master', 'license_manager']
06-29-2022 11:11:24.107 -0500 INFO ExecProcessor [44189 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:256] [get_cluster_mode] [65977] Fetched cluster mode, mode=disabled
06-29-2022 11:11:24.107 -0500 INFO ExecProcessor [44189 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/instance_id_modular_input.py" [assist::instance_id_modular_input.py:30] [should_run] [65977] should run test, sh=False
06-29-2022 11:11:26.891 -0500 INFO TailReader [44264 tailreader0] - Batch input finished reading file='/opt/splunk/var/spool/splunk/tracker.log'

[pavankumarh]

You logs indicate a old pem cert. have you tried renaming the server.pem file under splunk/etc/auth. then restarting splunk service. most KV store isues are resolved with this action. 

2022-06-29T15:04:51.149Z F NETWORK [main] The provided SSL certificate is expired or not yet valid.

[isoutamo]

Hi

what you found from splunkd.log and mongod.log?
Have you manually migrated mongo version?

r. Ismo