Alerting

how to generate alert based on the count of unique filed value per minute over 5 minutes

Kwip
Contributor

I want to generate an alert when unique field value count is above 10 per minute for 5 minutes.

Example:
my search giving me 3 values in host field.
host name---Count
server 1 ---- 10
server 2 ---- 26
Server 3 ---- 8

I want to generate alert if any of these servers (must be same all over the 5 minutes ) remains more than 10 per 1 minute over continues 5 mins.
Like
Time-----------Host Name-- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server1 ---- 15
03/02 07:24 - server1 ---- 16
03/02 07:25 - server1 ---- 22

I do not want to generate alert if combination of servers remains more than 10 per minute for continues 5 mins.
Like
Time---------- Host Name -- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server2 ---- 15 (During this time server1 count is less than 10. ie, server1 5)
03/02 07:24 - server3 ---- 16 (During this time server1 count is less than 10. ie, server1 9)
03/02 07:25 - server1 ---- 22

1 Solution

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

View solution in original post

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

Kwip
Contributor

@DalJeanis
The above query did the magic!!! Thank you very much!!!

0 Karma

DalJeanis
Legend

Quite welcome. Changed overall time frame to 6m from 7m after verifying that the alert would work correctly with only 6m. I had been wondering why 7m was needed, and found that my test bed had required 7 minutes only because of an internal "| head 1000" pipe on my base search test query that often stopped the search results early... 😉

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

.conf25 Global Broadcast: Don’t Miss a Moment

Hello Splunkers, .conf25 is only a click away.  Not able to make it to .conf25 in person? No worries, you can ...

Observe and Secure All Apps with Splunk

 Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What's New in Splunk Observability - August 2025

What's New We are excited to announce the latest enhancements to Splunk Observability Cloud as well as what is ...