Alerting

Does the scheduled alerting mechanism allow the search to complete before dispatching an alert

nterry
Path Finder

So I have a search that counts the number of successful dns server health checks over the last 5 minutes for all of our networks. The search runs the individual counts and then pipes to a search command to look for any counts less that a certain value. When I run the search, there is a brief period where a false positive shows up (i.e ~1 second) until the search completes. Is this the reason I am getting false positive alerts? We also suspect that this fires when splunk falls behind on indexing. (We index a LOT of data).

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Methinks there's actually two questions hidden in there.

Does Splunk finish a search before firing an alert?
Certainly. Else anything triggering for "some count is safe over some threshold" would always fire prematurely.

What happens when indexing is delayed?
The search finishes, but doesn't consider events that haven't been indexed yet - here you can get "false" positives.
The key is to not search from e.g. -5m@m to now but rather to allow for indexing delay depending on your environment. If for example you know that you get a minute of delay you could move your search back two minutes to be safe and run it from -7m@m to -2m@m. That way you still have your five-minute windows, but don't get affected by indexing delays.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

To debug time ranges that did trigger a false positive, run the search manually at a later time.
If that search result would not trigger, roll back the reporting and look at the _indextime hidden field for the events used to calculate your trigger value. Check if some events were indexed after the alert was originally run.

martin_mueller
SplunkTrust
SplunkTrust

To feel more comfortable with that you could move the filter-by-count part into the alert's trigger. If that evaluated and triggered prematurely it'd be utterly useless.

nterry
Path Finder

Thats a pretty good idea! Ill give that a go and see if it can't reduce our false positives.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Skimming through the relevant docs page at http://docs.splunk.com/Documentation/Splunk/6.1.3/Alert/Definescheduledalerts I don't see it explicitly say "the alert is triggered after the search has finished", but the example given (alert if count of purchases yesterday is less than 500) would trigger false positives every time if that weren't the case due to the count not yet being 500 in the previewed search results.

0 Karma

nterry
Path Finder

Right. I too skimmed it, but couldn't find anything. The problem is that we pipe the results the search command and do the count filtering there. For just a fraction of a second, we get results that would trigger an alert before the count is met and the result(s) disappear... Im concerned that an alert is triggered for that small window....

0 Karma

nterry
Path Finder

The key is to not search from e.g. -5m@m to now but rather to allow for indexing delay depending on your environment. If for example you know that you get a minute of delay you could move your search back two minutes to be safe and run it from -7m@m to -2m@m.

We already do that 😉

As for logic around Splunk finishing a search, I'm kinda looking for a 'chapter and verse' kind of confirmation. Is there somewhere in the docs that can absolutely confirm this? (I agree that is would be a poor tool if it didn't, but direct confirmation is always desirable)

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.