Alerting

Why is the custom trigger not firing for this alert?

New Member

My search is:

index=soma   source="alarms.txt" StatusID=*
|eval Alarm=if(StatusID=0,"Critical","No")
| table  DateTime EventCode StatusID Message Alarm

The alert runs every minute (cron * * * * *)
and my custom trigger is
search Alarm="No"
There are no fired events for this alert
If I run the search |where Alarm="No" there are 94 events matching that

What am I doing wrong?

0 Karma

Champion

When you say cron *, what is your exact cron expression? I don't believe a single * is a valid expression, so I want to make sure the issue isn't simply that your search isn't running when you think it should be.

0 Karma

New Member

the cron job value is * * * * *
And that's not the issue any longer. Triggers are fired but no emails is generated

0 Karma

What is the time window being monitored for this search that's running every minute? In my experience, if you're running a search every minute (and presumably looking at data logged in the previous minute), the delays of ingestion may prevent you from seeing anything at all.

0 Karma

New Member

Last 7 days. this is a static file i uploaded to the index=soma

0 Karma

Got it. So how about changing the approach. I'd suggest changing your search to this:

index=soma source="alarms.txt" StatusID=* 
| eval Alarm=if(StatusID=0,"Critical","No") 
| table DateTime EventCode StatusID Message Alarm
| search Alarm="No"

And set the trigger to be Number of Results, with is greater than set to 0. Will that achieve what you're trying to do?

0 Karma

New Member

will i get an alert of each instance (that's the desired affect)

0 Karma

New Member

I made the change and I still get There are no fired events for this alert

0 Karma

What type of license do you have? Is there any chance your evaluation has expired? If you're in "Free" status, alerts won't fire.

If that's not the issue, check your Alert configuration. Ensure that under "Add Actions", you've selected "Add to Triggered Alerts".

And if you want one alert for each result, look at the "Trigger" line under "Trigger Conditions". There are two options: "Once" and "For each result". If you want one alert per result, select "For each result".

0 Karma

New Member

I added Added Triggered Alerts to the trigger action and got a trigger but not an email

0 Karma

Then we're on the right track! Under "Trigger Actions", select "Add Actions" and choose "Send email". You'll need to enter your email address in the To field and then save it again.

0 Karma

New Member

I setup the email action but no emails

0 Karma

Have you used this Splunk instance to send any other emails? If not, you'll need to follow the steps on this guide to configure communication between Splunk and your mail server:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Alert/Emailnotification

0 Karma

New Member

Splunk Enterprise Sales Trial 10,240 MB Mar 14, 2018 5:37:43 PM valid

0 Karma