Splunk Search

How to alert when an "error" happens once every 10 min, but not until 3 consecutive hits?

MasterOogway
Communicator

I have an "error-string" and need to alert when I find it not only in the first 10 minute check; not only in the second 10 minute check; but when I run a third consecutive check and find the "error-string" in all three.
Is this possible?

I looked at alert throttling in 4.3.x, but that won't meet my needs. There must be some complex search string that allows for a check of "error-string" every 10 minutes, but don't alert until the third straight positive event.

Thoughts?

0 Karma

yannK
Splunk Employee
Splunk Employee

you can bucket the results per time period, and search over 30 min for :

mysearchwithsearchterms | buckets _time span=10m | stats count by _time | where count > 0

and setup a conditional of alert if number of results > 2

the results will look like :

count _time

1 2012-04-26 10:00:00
32 2012-04-26 10:10:00
4 2012-04-26 10:20:00

or

count _time

2 2012-04-26 11:40:00
6 2012-04-26 11:50:00

Time periods with 0 count will not be present, therefore, is you found less than 3 results, those are not 3 time consecutive.

MickSheppard
Path Finder

Can this string only appear once for each 10 minute period, or could it appear more than once in any 10 minute period?

0 Karma

MasterOogway
Communicator

It can appear any number of times within any given 10 minute window. But it MUST happen in three consecutive 10 minute searches.

0 Karma
Get Updates on the Splunk Community!

Splunk Decoded: Service Maps vs Service Analyzer Tree View vs Flow Maps

It’s Monday morning, and your phone is buzzing with alert escalations – your customer-facing portal is running ...

What’s New in Splunk Observability – September 2025

What's NewWe are excited to announce the latest enhancements to Splunk Observability, designed to help ITOps ...

Fun with Regular Expression - multiples of nine

Fun with Regular Expression - multiples of nineThis challenge was first posted on Slack #regex channel ...