Why is my scheduled alert not getting triggered?

New Member

My setup is simple. Splunk reads a file /var/log/snmp.log into index "snmp". I created a search: index="snmp" and created an alert for this. It is scheduled to search for results in the last minute every 1 minute and should trigger if the count>1. In the action, I send the alert to alert manager, but it's not working. I could see the search completed on jobs, but no alert showed in triggered alerts nor alert manager.

0 Karma


You might have over 1 minute of indexing / event latency.

Try this in your search instead:

index=snmp _index_earliest=-1m

This will go by the time it was indexed and not by the date timestamp in the events.

It is a best practice to have a "trailing" search for these types of alerts. Something like this would search for the minute of time from 10 minutes ago, and would be more likely to find the events that are latent.

index=snmp earliest=-11m@m latest=-10m@m

Play with _index_earliest, _index_latest, earliest and latest a little and I think you'll solve your problem.

Here's a good write up on the issue I think you're facing:

Get Updates on the Splunk Community!

Splunk APM & RUM | Upcoming Planned Maintenance

There will be planned maintenance of Splunk APM’s and Splunk RUM’s streaming infrastructure in the coming ...

Part 2: Diving Deeper With AIOps

Getting the Most Out of Event Correlation and Alert Storm Detection in Splunk IT Service Intelligence   Watch ...

User Groups | Upcoming Events!

If by chance you weren't already aware, the Splunk Community is host to numerous User Groups, organized ...