How to trigger an alert if a value dynamically appears morethan threshold.

[mskreddy]

I want to trigger an alert if the same event happened formorethan 10 times in 10 minutes. But the condition for the event is not static.

Example:

index="seach" soursetype="was_debug" site="UK"  "RESP:ABC"

But the ABC can be dynamic like BCD can appear for 10 times and an alert should trigger for BCD.

How to acheive this in Splunk alerting.

Currently i have an alert with Hardcoded ABC but there are a lot of valuesandi need to write a lot of alerts foreach of them and i want to make this in a single alert.

[ITWhisperer]

Hi @mskreddy 

First you need to get ABC into a field. From the limited example you have given perhaps this would work:

index="search" sourcetype="was_debug" site="UK"  "RESP:ABC" | rex field=_raw "RESP:(?<RESP>\w{3})"

The field is more likely to something other than _raw but who knows? (hopefully you do!)

Then as @gcusello said, you can count the occurrences

index="search" sourcetype="was_debug" site="UK"  "RESP:ABC" | rex field=_raw "RESP:(?<RESP>\w{3})" | stats count by RESP | where count > 10

This will return rows for every value of RESP where the count is greater than 10 in your time period (if you only want a specific set of values for RESP to be counted, then a lookup is as good a way as any, but there are other ways)

Now, set the time period of your alert to be the last 10 minutes "-10m@m" and set your trigger to be if any rows are returned, and set the report to run every minute. If you don't need it to be quite so reactive you can run it every 5 minutes but you may miss some instances where 10 occurrences happened, just not in the 10 minute chunks you were looking at

Hope that helps

[gcusello]

Hi @mskreddy,

you could insert the patterns to search in a lookup then run something like this:

your_search [ | inputlookup pattern_lookup.csv | rename pattern AS site | fields dite ]
| stats count BY site
| where count>10

if the lookup is called pattern_lookup.csv and contains a column called "pattern" and the field to search is "site".

Ciao.

Giuseppe