Alerting

scheduling best practices question

rainhailrob
Path Finder

Yes, I read the http://docs.splunk.com/Documentation/Splunk/6.1.1/Alert/Definescheduledalerts document, but I'm still somewhat confused with regard to the following:

Coordinate the alert's search schedule with the search time range. This prevents situations where event data is accidentally evaluated twice by the search (because the search time range exceeds the search schedule, resulting in overlapping event data sets), or not evaluated at all (because the search time range is shorter than the search schedule).

Schedule your alerting searches with at least 60 seconds of delay. This practice is especially important in distributed search Splunk implementations where event data might not reach the indexer precisely at the moment when it is generated. A delay ensures that you are counting all of your events, not just the ones that were quickest to get indexed.

So is this basically telling me I should not do less than 6 minute searches (allowing for 60sec delay)? Essentially. I'd like to know if something occurred in the past 5minutes or would the following work?

earliest: -5m@m
latest: now
cron expression: */5 * * * *

Tags (2)
0 Karma

somesoni2
Revered Legend

It basically says that you should allow sufficient buffer time for events to be written to indexes and while doing the schedule, the search time range should accommodate that buffer time. E.g. You want to search to run every 5 min and process last 5 min data, use something like this

earliest=-7m@m
latest=-2m@m             [2 min buffer time]
cron : 2-59/5 * * * *    [run the search on 3rd minute of the hour, like 2,7,12,17,...
                         minutes. This will allow you to capture data for full 5 min 
                         periods like 0-5 will get captured by search running on minute 7
                         (-7m to -2m) and 6-10 will get captured by search on 12(-7m to -2m).
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...