Looking for a way to measure splunkd service uptime, and alert on when it's down. Not so much worried about system uptime itself, just want to look at measuring splunkd uptime.
You can run this search on your most widely peered Search Head (usually your Monitoring Console) and then check the responding peers against a splunk_infrastructure_hosts.csv lookup and then alert if any are not showing:
If your splunk is running on Unix systems, try installing the unix and linux add-on app then enable the script that captures the uptime of the processes. Then create a gentimes search that will alert you when the process is not running since there will be no logs that will indicate the process is down I believe.
