Hi,
I'm trying to reduce the number of alerts in Splunk, at the moment we receive splunk alert on queue size every 30 min. The problem here is the queue size/depth is the same when it triggers every 30 min, I need a solution so that we could check the previous queue size and the current queue size and stop alerting if they are same.
index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0
| eval time=strftime(_time,"%d %b %Y %H:%M %p")
| stats latest(QDepth) As QueueSize,max(time) As LastEvent by host, QMan, QName
@ashrafsj apologies, I indeed forgot to add a by-clause to the eventstats. This should work better:
index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize by host, QMan, QName
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")
Something like this should work. Run it every 30min over last 60min (instead of your current 30min):
index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")
Note: I've also moved the timestamp to string conversion to the end, otherwise max() will have incorrect results (the timestamp format you use does not sort chronologically).
Thanks @FrankVl , I tried your query. The original query had returned 52 results and this one has returned 152 results and all the PrevQueueSize values shows 4 for all the queues. It doesn't seem to pick the previous queue size of each queue.
@ashrafsj apologies, I indeed forgot to add a by-clause to the eventstats. This should work better:
index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize by host, QMan, QName
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")
Thanks a lot @FrankVl , that worked for me. Many thanks!