Send alerts for status recovery

[frankroger]

We have the below query, which checks whenever a server is down. So we want this query to send an alert when the status changes from Stopped to Running. Now it only sends an alert when the status is Stopped.

index="init_butcher" sourcetype="services_status.out.log" host=* | chart useother=f values(status) as services over host by service limit=0
| eval status=if('abc'="STOPPED", "DOWN", "Critical")
| where 'cfd'="STOPPED" OR 'hij'="STOPPED" OR ''="STOPPED" OR 'jkl'="STOPPED" OR 'mno'="STOPPED" OR 'pqr'="STOPPED" OR 'stu'="STOPPED" OR 'vux'="STOPPED" OR 'yz'="STOPPED"
| fields butcher, host, status
| mvcombine host delim=","
| eval message="Butcher Services are at status: ".status." Host(s):".mvjoin(host,",")

[thambisetty]

yes. you have got few results that means you have field status. 

The query you have provided in the question is not properly developed.  If you just need to be alerted when status is running you can just add below and save as an alert.

| stats count by host,service

[frankroger]

I want the alert to trigger when the services are down and then trigger again once or twice when the server is up.  

[thambisetty]

@frankroger 

can you post sample events and also check is there any field status or abc in your events.

[frankroger]

Here is the results.

[frankroger]

Thanks Thambisetty for your quick respone. But I am new to splunk, I don't understand what is meant by your reply there. 

[thambisetty]

@frankroger 

you need to identify the value used to find events for running. 

index="init_butcher" sourcetype="services_status.out.log" host=* status="running"