Email Alert Setup if Splunk is down in Production

[nikhilagrawal]

Anybody can suggest how to setup email alerts if Splunk service is down. I am trying to configure alerts in a way so we get email alert if service is down.

Any sort of help will be appreciated.

Thanks
Nik

[yannK]

Who will watch the watchmen.

You may want to use a third party script or tool to check that the splunkd process is up and running

• a simple /opt/splunk/bin/splunk status should do the trick

• or a search on last 5 minutes
  /opt/splunk/bin/splunk search "earliest=-5m index=* | stats count | eval status=if(count>0,'OK','ERROR')" -auth user:password

• or a separate search-head, checking the number of events.

[yannK]

those have to used in a script that will check the result

• this status command will show you if splunk is running
• and the search will show you is events from the last 5 minutes are searchable.

If you want to check if the process is running, then you need a ps script or a monitor. please contact your system administrator, this is his job.

[Dark_Ichigo]

What does this: /opt/splunk/bin/splunk search "earliest=-5m index=* | stats count | eval status=if(count>0,'OK','ERROR')" -auth user:password

Do? and if I turn this into a script that checks for it, will this do the trick?