This can happen if the cert used by Splunkd to talk to Mongod has expired. Verify your certs are valid. For example, to validate the expiration date for server.pem you can run:
From $SPLUNK_HOME/etc/auth/
openssl x509 -enddate -noout -in ./server.pem
Results:
notAfter=Dec 10 14:017:25 2015 GMT
In the example above, the cert is expired. If you want to create a new cert you can look at splunk createssl:
$SPLUNK_HOME/bin/splunk help createssl
An example:
$SPLUNK_HOME/bin/splunk createssl server-cert -d $SPLUNK_HOME/etc/auth -n server -c cn.domain.com -l 2048
Simply adjust for your environment requirements/settings. Once the new cert is in place, you can test to confirm it is valid:
From $SPLUNK_HOME/etc/auth/
openssl x509 -enddate -noout -in ./server.pem
Results:
notAfter=Aug 22 15:30:45 2019 GMT
If it is now valid, restart Splunk and validate if KVStore is running:
ps -ax | grep mongod
26108 ? Ssl 62:11 mongod --dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo --port=8191 --timeStampFormat=iso8601-utc --smallfiles --oplogSize=200 --keyFile=/opt/splunk/var/lib/splunk/kvstore/mongo/splunk.key --setParameter=enableLocalhostAuthBypass=0 --replSet=50D25A40-7DD2-4017-A223-732705AD4A96 --sslAllowInvalidHostnames --sslMode=preferSSL --sslPEMKeyFile=/opt/splunk/etc/auth/server.pem --sslPEMKeyPassword=xxxxxxxx --nounixsocket
And also:
$SPLUNK_HOME/bin/splunk _internal call /services/server/info |grep -i kvstore
<s:key name="kvStoreStatus">ready</s:key>
A third way to check is to run the following search from command line on the instance where you have the "Deployment Management Console / Monitor Console" Configured.
$SPLUNK_HOME/bin/splunk search "| rest /services/server/info splunk_server=* | fields splunk_server, kvStoreStatus"
splunk_server kvStoreStatus
------------------------- -------------
indexer01.domain.com ready
indexer02.domain.com ready
indexer03.domain.com ready
indexer04.domain.com ready
indexer05.domain.com ready
Jacob
Sr. Technical Support Engineer
