KVstore stopped working, stuck in starting, not connecting to other shcluster members, related to server.pem

[Glasses2]

Hi,
I recently had an issue where my SHCluster was throwing Kvstore errors.
The Kvstore status was abnormal.
The resolution was checking the server.pem expiration date in /opt/splunk/etc/auth
use >>> openssl x509 -in /opt/splunk/etc/auth/server.pem -noout -text
After removing the server.pem and restarting, Kvstore was back up.

Does anyone have a way to monitor the expiration dates for all the server.pem(s) in the deployment?

Thanks

[livehybrid]

Hi @Glasses2 

Im glad you managed to fix your mongo issue - Not the first time SSL expiry has caught people out!

There is an app "SSL Certificate Checker" on Splunkbase at https://splunkbase.splunk.com/app/3172 which looks to solve this. You configure it with the path of the certificates you wish to monitor on a Splunk host and it reports the expiry into Splunk for you to create an alert for. Setting up for /opt/splunk/etc/auth/ should capture most things unless you have other custom certs in use.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing