Alerting

Missing Activity Alert False Positives

mf439
New Member

Hi, 

 

I have the following alert set up: 

query (roughly): 

 

 

index="myindex" "the log message that i am interested in"
| head 1

 

 

Alert type: Scheduled, Run on Cron Schedule */5 * * * * (every 5 minutes) for the last 10 minutes.

Trigger alert when Number of Results is less than 1. I also added throttling and configured an email to be sent when the alert triggers. 

 

It works fine most of the time, but in irregular intervals, I receive alerts. Some of them are valid, because there just were no events. For others, the saved search doesn't show events, but starting the same search again, will show that there actually were events logged, so I'd consider those alerts false positives.

My first idea was that delays in the forwarding and indexing processes lead to this result, but when by checking the _indextime field, this explanation can be ruled out as well. 

Does anyone have an idea why these false positives  appear and how I can get rid of them? 

 

Thanks in advance! 

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...