Alerting

subsearch condition which persists for x time

ericrobinson
Path Finder

Is there is a way to have a scheduled search run and only alert if a certain condition is met for a period of time?

We have a search that looks for a delay value that we are extracting with rex. We want to be alerted if this value is > 1 minute for a period of time. (e.g. if the value for this delay filed is greater > 1 minute for 10 minutes, send an alert)

Is this possible with the current monitoring and alerting capabilities? I only see number of events, hosts etc as a conditional alert.

Tags (1)
0 Karma

gkanapathy
Splunk Employee
Splunk Employee

There is a conditional alert, but I suppose you are looking for a value such that the condition is met over a period of time, i.e., you want to refer to a history of running the search on a schedule. The easiest way to do this is to just run a search over a longer period of time and count (e.g., ... | bucket _time span=1min | ... by _time | stats count) but this can be very inefficient. A better way is to use a pair of searches, one to update a lookup table containing the history over time, and the second to report/alert on the lookup table.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

A Four-Part Event Series: Full Stack Observability For the AI Era

As AI reshapes applications, infrastructure, and the way teams operate, the traditional boundaries of ...

SOC4Kafka - New Kafka Connector Powered by OpenTelemetry

The new SOC4Kafka connector, built on OpenTelemetry, enables the collection of Kafka messages and forwards ...

Event Series: Level up your SOC: Advancing with Splunk Enterprise Security

AI has fundamentally raised the stakes for security operations, and this three-part series is your guide to ...