Alerting

Stop splunk alerts when the alert results are same as previous result within 30 min window

ashrafsj
Path Finder

Hi,

 

I'm trying to reduce the number of alerts in Splunk, at the moment we receive splunk alert on queue size every 30 min. The problem here is the queue size/depth is the same when it triggers every 30 min, I need a solution so that we could check the previous queue size and the current queue size and stop alerting if they are same.

index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0 
| eval time=strftime(_time,"%d %b %Y %H:%M %p")
| stats latest(QDepth) As QueueSize,max(time) As LastEvent by host, QMan, QName


Labels (2)
0 Karma
1 Solution

FrankVl
Ultra Champion

@ashrafsj apologies, I indeed forgot to add a by-clause to the eventstats. This should work better:

index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0 
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize by host, QMan, QName
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")

View solution in original post

FrankVl
Ultra Champion

Something like this should work. Run it every 30min over last 60min (instead of your current 30min):

index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0 
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")

Note: I've also moved the timestamp to string conversion to the end, otherwise max() will have incorrect results (the timestamp format you use does not sort chronologically).

0 Karma

ashrafsj
Path Finder

Thanks @FrankVl , I tried your query. The original query had returned  52 results and this one has returned 152 results and all the PrevQueueSize values shows 4 for all the queues.  It doesn't seem to pick the previous queue size of each queue. 

0 Karma

FrankVl
Ultra Champion

@ashrafsj apologies, I indeed forgot to add a by-clause to the eventstats. This should work better:

index=day sourcetype="mqmon" EventType="QueueDepth" QDepth>0 
| eval time = _time
| bin time span=30min
| stats latest(QDepth) As QueueSize,max(_time) As LastEvent by time,host, QMan, QName
| eval _time=time
| eventstats earliest(QueueSize) as PrevQueueSize by host, QMan, QName
| where QueueSize!=PrevQueueSize
| eval LastEvent=strftime(LastEvent,"%d %b %Y %H:%M %p")

ashrafsj
Path Finder

Thanks a lot @FrankVl , that worked for me. Many thanks!

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...