Alerting

how to generate alert based on the count of unique filed value per minute over 5 minutes

Kwip
Contributor

I want to generate an alert when unique field value count is above 10 per minute for 5 minutes.

Example:
my search giving me 3 values in host field.
host name---Count
server 1 ---- 10
server 2 ---- 26
Server 3 ---- 8

I want to generate alert if any of these servers (must be same all over the 5 minutes ) remains more than 10 per 1 minute over continues 5 mins.
Like
Time-----------Host Name-- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server1 ---- 15
03/02 07:24 - server1 ---- 16
03/02 07:25 - server1 ---- 22

I do not want to generate alert if combination of servers remains more than 10 per minute for continues 5 mins.
Like
Time---------- Host Name -- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server2 ---- 15 (During this time server1 count is less than 10. ie, server1 5)
03/02 07:24 - server3 ---- 16 (During this time server1 count is less than 10. ie, server1 9)
03/02 07:25 - server1 ---- 22

1 Solution

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

View solution in original post

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

Kwip
Contributor

@DalJeanis
The above query did the magic!!! Thank you very much!!!

0 Karma

DalJeanis
Legend

Quite welcome. Changed overall time frame to 6m from 7m after verifying that the alert would work correctly with only 6m. I had been wondering why 7m was needed, and found that my test bed had required 7 minutes only because of an internal "| head 1000" pipe on my base search test query that often stopped the search results early... 😉

0 Karma
Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...