Hi,
We have a search head cluster of 8 members in which KV store is failing frequently. We used to start services manually.
I'd like to create a report which should contains when exactly kv store got failed & when it got up. I am not sure in which logs we can find this info.
Could anyone help me with the query for same ?
Thanks
I believe you want to search for "MongoDB starting"
index=_internal sourcetype=mongod "MongoDB starting"
In alerts for splunk admins https://splunkbase.splunk.com/app/3796/ I have an alert to detect a lack of logging from mongod.log so combined with richgalloway's answer this might work for you...
Unfortunately, the query doesn't help. I see multiple entries of below message where we didn't do any actual restart.
2021-03-25T08:14:43.060Z I CONTROL [initandlisten] MongoDB starting : pid=18614 port=8191 dbpath=/opt/splunk/var/lib/splunk/kvstore/mongo 64-bit host=xxxxxx