Alerting

Why is my scheduled alert not getting triggered?

dragon0087
New Member

My setup is simple. Splunk reads a file /var/log/snmp.log into index "snmp". I created a search: index="snmp" and created an alert for this. It is scheduled to search for results in the last minute every 1 minute and should trigger if the count>1. In the action, I send the alert to alert manager, but it's not working. I could see the search completed on jobs, but no alert showed in triggered alerts nor alert manager.

0 Karma

jkat54
SplunkTrust
SplunkTrust

You might have over 1 minute of indexing / event latency.

Try this in your search instead:

index=snmp _index_earliest=-1m

This will go by the time it was indexed and not by the date timestamp in the events.

It is a best practice to have a "trailing" search for these types of alerts. Something like this would search for the minute of time from 10 minutes ago, and would be more likely to find the events that are latent.

index=snmp earliest=-11m@m latest=-10m@m

Play with _index_earliest, _index_latest, earliest and latest a little and I think you'll solve your problem.

Here's a good write up on the issue I think you're facing:
https://answers.splunk.com/answers/11870/how-can-i-view-the-indexing-latency-for-incoming-events-in-...

Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...