Alerting

how to generate alert based on the count of unique filed value per minute over 5 minutes

Kwip
Contributor

I want to generate an alert when unique field value count is above 10 per minute for 5 minutes.

Example:
my search giving me 3 values in host field.
host name---Count
server 1 ---- 10
server 2 ---- 26
Server 3 ---- 8

I want to generate alert if any of these servers (must be same all over the 5 minutes ) remains more than 10 per 1 minute over continues 5 mins.
Like
Time-----------Host Name-- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server1 ---- 15
03/02 07:24 - server1 ---- 16
03/02 07:25 - server1 ---- 22

I do not want to generate alert if combination of servers remains more than 10 per minute for continues 5 mins.
Like
Time---------- Host Name -- Count
03/02 07:21 - server1 ---- 11
03/02 07:22 - server1 ---- 12
03/02 07:23 - server2 ---- 15 (During this time server1 count is less than 10. ie, server1 5)
03/02 07:24 - server3 ---- 16 (During this time server1 count is less than 10. ie, server1 9)
03/02 07:25 - server1 ---- 22

1 Solution

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

View solution in original post

DalJeanis
Legend
your base search over at least 6 minutes span
| bin _time span=1m
| stats count as eventcount by  _time hostname
| streamstats count(eval(eventcount>=10)) as count10 by hostname time_window=5m
| where count10>4

Notes - This alert requires that the base search code runs over at least a 6 minute period, since the rolling 5-minute timeframe would include a partial minute at the beginning and a partial minute at the end.

Streamstats requires input be sorted by _time, so I corrected the "by" clause of the stats command.

Kwip
Contributor

@DalJeanis
The above query did the magic!!! Thank you very much!!!

0 Karma

DalJeanis
Legend

Quite welcome. Changed overall time frame to 6m from 7m after verifying that the alert would work correctly with only 6m. I had been wondering why 7m was needed, and found that my test bed had required 7 minutes only because of an internal "| head 1000" pipe on my base search test query that often stopped the search results early... 😉

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...