topic Re: Email Alert Setup if Splunk is down in Production in Alerting

Email Alert Setup if Splunk is down in Production

nikhilagrawal — Fri, 26 Oct 2012 15:09:55 GMT

Anybody can suggest how to setup email alerts if Splunk service is down. I am trying to configure alerts in a way so we get email alert if service is down.

Any sort of help will be appreciated.

Thanks
Nik

Re: Email Alert Setup if Splunk is down in Production

yannK — Fri, 26 Oct 2012 15:41:16 GMT

Who will watch the watchmen.

You may want to use a third party script or tool to check that the splunkd process is up and running

a simple /opt/splunk/bin/splunk status should do the trick
or a search on last 5 minutes
/opt/splunk/bin/splunk search "earliest=-5m index=* | stats count | eval status=if(count>0,'OK','ERROR')" -auth user:password
or a separate search-head, checking the number of events.

Re: Email Alert Setup if Splunk is down in Production

Dark_Ichigo — Tue, 26 Feb 2013 00:09:21 GMT

What does this: /opt/splunk/bin/splunk search "earliest=-5m index=* | stats count | eval status=if(count>0,'OK','ERROR')" -auth user:password

Do? and if I turn this into a script that checks for it, will this do the trick?

Re: Email Alert Setup if Splunk is down in Production

yannK — Tue, 26 Feb 2013 02:32:41 GMT

those have to used in a script that will check the result

this status command will show you if splunk is running
and the search will show you is events from the last 5 minutes are searchable.

If you want to check if the process is running, then you need a ps script or a monitor. please contact your system administrator, this is his job.