Why is my scheduled alert not getting triggered?

New Member

My setup is simple. Splunk reads a file /var/log/snmp.log into index "snmp". I created a search: index="snmp" and created an alert for this. It is scheduled to search for results in the last minute every 1 minute and should trigger if the count>1. In the action, I send the alert to alert manager, but it's not working. I could see the search completed on jobs, but no alert showed in triggered alerts nor alert manager.

0 Karma


You might have over 1 minute of indexing / event latency.

Try this in your search instead:

index=snmp _index_earliest=-1m

This will go by the time it was indexed and not by the date timestamp in the events.

It is a best practice to have a "trailing" search for these types of alerts. Something like this would search for the minute of time from 10 minutes ago, and would be more likely to find the events that are latent.

index=snmp earliest=-11m@m latest=-10m@m

Play with _index_earliest, _index_latest, earliest and latest a little and I think you'll solve your problem.

Here's a good write up on the issue I think you're facing:

Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...