Stop splunk alerts when the alert results are same as previous result within 30 min window

[ashrafsj]

Hi,

I'm trying to reduce the number of alerts in Splunk, at the moment we receive splunk alert on queue size every 30 min. The problem here is the queue size/depth is the same when it triggers every 30 min, I need a solution so that we could check the previous queue size and the current queue size and stop alerting if they are same.

index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0 
| eval time=strftime(_time,"%d %b %Y %H:%M %p")
| stats latest(QDepth) As QueueSize,max(time) As LastEvent by host, QMan, QName

[FrankVl]

Something like this should work. Run it every 30min over last 60min (instead of your current 30min):

index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0 
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")

Note: I've also moved the timestamp to string conversion to the end, otherwise max() will have incorrect results (the timestamp format you use does not sort chronologically).

[ashrafsj]

Thanks @FrankVl , I tried your query. The original query had returned  52 results and this one has returned 152 results and all the PrevQueueSize values shows 4 for all the queues.  It doesn't seem to pick the previous queue size of each queue. 

[FrankVl · Accepted solution]

@ashrafsj apologies, I indeed forgot to add a by-clause to the eventstats. This should work better:

index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0 
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize by host, QMan, QName
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")

[ashrafsj]

Thanks a lot @FrankVl , that worked for me. Many thanks!