Alerting

How do I count the number of the occurrences / buckets when the given event happened?

damucka
Builder

Hello,

I am trying to count the time buckets when the specific search returns values and alert on it. My current search looks as follows:

index=mlbso sourcetype=BWP_hanatraces "Out of memory for Pool/JoinEvaluator" | timechart count span=1m as OOM_Pool | eval Occurence = if (OOM_Pool > 0,1,0)

For alerting, I am only interested in the occurrence being 1 or 0, not in the number of events (count) per time bucket. Then, I want to alert when the Occurrences increase with time, which I set in the alert trigger options (> 5 in the last 30 minutes).

The problem is that this is not working and the alert takes not only the Occurrence but also a Count and adds both up. So if I have the 25 events (OOM_Pool) in one minute, then the Occurrence is 1, Count 25 and the alert gets triggered. I tried to overcome this by setting the Custom triggering condition condition:

search Occurence > 5"

but this does not seem to work.

How would I do it properly?

Kind regards,
Kamil

Tags (2)
0 Karma
1 Solution

darrenfuller
Contributor

Hi @damucka

try this:

 index=mlbso sourcetype=BWP_hanatraces "Out of memory for Pool/JoinEvaluator" earliest=-30m
| bin _time span=1m 
| stats count AS OOM_Pool by _time
| eval occurrence=if(OOM_Pool!=0, 1, 0)
| stats count as occurrences_in_last_30_min

Then alert if occurrences_in_last_30_min is >5

View solution in original post

mstjohn_splunk
Splunk Employee
Splunk Employee

hi @damucka

Did the answer below solve your problem? If so, please resolve this post by approving it!
If your problem is still not solved, keep us updated so that someone else can help ya. Thanks for posting!

0 Karma

darrenfuller
Contributor

Hi @damucka

try this:

 index=mlbso sourcetype=BWP_hanatraces "Out of memory for Pool/JoinEvaluator" earliest=-30m
| bin _time span=1m 
| stats count AS OOM_Pool by _time
| eval occurrence=if(OOM_Pool!=0, 1, 0)
| stats count as occurrences_in_last_30_min

Then alert if occurrences_in_last_30_min is >5

*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>