we have services going down on our splunk server instances , we just want to set up an alert when the service failure occurs and also automatic restart of the services .And for information we dont have clustered Environment . we have 2 forwarders and 5 indexer instances .
Last week there was service down on indexer but we haven't got any alert as out server instances were being monitored by "tivoli" . Kindly help us if we can do this on splunk itself.
If the Splunk service is down, it can't run any alert action by itself to get restarted. You would need to use some external means to monitor and restart Splunk service. A simple crontab job will do the job.