Alerting

how to generate alert based on the count of unique filed value per minute over 5 minutes

Kwip
Contributor

I want to generate an alert when unique field value count is above 10 per minute for 5 minutes.

Example:
my search giving me 3 values in host field.
host name---Count
server 1 ---- 10
server 2 ---- 26
Server 3 ---- 8

I want to generate alert if any of these servers (must be same all over the 5 minutes ) remains more than 10 per 1 minute over continues 5 mins.
Like
Time-----------Host Name-- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server1 ---- 15
03/02 07:24 - server1 ---- 16
03/02 07:25 - server1 ---- 22

I do not want to generate alert if combination of servers remains more than 10 per minute for continues 5 mins.
Like
Time---------- Host Name -- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server2 ---- 15 (During this time server1 count is less than 10. ie, server1 5)
03/02 07:24 - server3 ---- 16 (During this time server1 count is less than 10. ie, server1 9)
03/02 07:25 - server1 ---- 22

1 Solution

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

View solution in original post

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

Kwip
Contributor

@DalJeanis
The above query did the magic!!! Thank you very much!!!

0 Karma

DalJeanis
Legend

Quite welcome. Changed overall time frame to 6m from 7m after verifying that the alert would work correctly with only 6m. I had been wondering why 7m was needed, and found that my test bed had required 7 minutes only because of an internal "| head 1000" pipe on my base search test query that often stopped the search results early... 😉

0 Karma
Get Updates on the Splunk Community!

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...

Splunk Up Your Game: Why It's Time to Embrace Python 3.9+ and OpenSSL 3.0

Did you know that for Splunk Enterprise 9.4, Python 3.9 is the default interpreter? This shift is not just a ...