topic Re: Send alerts for status recovery in Splunk Enterprise

Send alerts for status recovery

frankroger — Wed, 25 Nov 2020 09:12:03 GMT

We have the below query, which checks whenever a server is down. So we want this query to send an alert when the status changes from Stopped to Running. Now it only sends an alert when the status is Stopped.

index="init_butcher" sourcetype="services_status.out.log" host=* | chart useother=f values(status) as services over host by service limit=0
| eval status=if('abc'="STOPPED", "DOWN", "Critical")
| where 'cfd'="STOPPED" OR 'hij'="STOPPED" OR ''="STOPPED" OR 'jkl'="STOPPED" OR 'mno'="STOPPED" OR 'pqr'="STOPPED" OR 'stu'="STOPPED" OR 'vux'="STOPPED" OR 'yz'="STOPPED"
| fields butcher, host, status
| mvcombine host delim=","
| eval message="Butcher Services are at status: ".status." Host(s):".mvjoin(host,",")

Re: Send alerts for status recovery

thambisetty — Wed, 25 Nov 2020 11:15:21 GMT

@frankroger

you need to identify the value used to find events for running.

index="init_butcher" sourcetype="services_status.out.log" host=* status="running"

Re: Send alerts for status recovery

frankroger — Wed, 25 Nov 2020 11:34:26 GMT

Thanks Thambisetty for your quick respone. But I am new to splunk, I don't understand what is meant by your reply there.

Re: Send alerts for status recovery

thambisetty — Wed, 25 Nov 2020 11:39:59 GMT

@frankroger

can you post sample events and also check is there any field status or abc in your events.

Re: Send alerts for status recovery

frankroger — Wed, 25 Nov 2020 11:46:12 GMT

Here is the results.

Re: Send alerts for status recovery

thambisetty — Wed, 25 Nov 2020 11:57:39 GMT

yes. you have got few results that means you have field status.

The query you have provided in the question is not properly developed. If you just need to be alerted when status is running you can just add below and save as an alert.

| stats count by host,service

Re: Send alerts for status recovery

frankroger — Wed, 25 Nov 2020 12:16:32 GMT

I want the alert to trigger when the services are down and then trigger again once or twice when the server is up.