Alerting

Why is the custom trigger not firing for this alert?

ihiesbkalai
New Member

My search is:

index=soma   source="alarms.txt" StatusID=*
|eval Alarm=if(StatusID=0,"Critical","No")
| table  DateTime EventCode StatusID Message Alarm

The alert runs every minute (cron * * * * *)
and my custom trigger is
search Alarm="No"
There are no fired events for this alert
If I run the search |where Alarm="No" there are 94 events matching that

What am I doing wrong?

0 Karma

micahkemp
Champion

When you say cron *, what is your exact cron expression? I don't believe a single * is a valid expression, so I want to make sure the issue isn't simply that your search isn't running when you think it should be.

0 Karma

ihiesbkalai
New Member

the cron job value is * * * * *
And that's not the issue any longer. Triggers are fired but no emails is generated

0 Karma

elliotproebstel
Champion

What is the time window being monitored for this search that's running every minute? In my experience, if you're running a search every minute (and presumably looking at data logged in the previous minute), the delays of ingestion may prevent you from seeing anything at all.

0 Karma

ihiesbkalai
New Member

Last 7 days. this is a static file i uploaded to the index=soma

0 Karma

elliotproebstel
Champion

Got it. So how about changing the approach. I'd suggest changing your search to this:

index=soma source="alarms.txt" StatusID=* 
| eval Alarm=if(StatusID=0,"Critical","No") 
| table DateTime EventCode StatusID Message Alarm
| search Alarm="No"

And set the trigger to be Number of Results, with is greater than set to 0. Will that achieve what you're trying to do?

0 Karma

ihiesbkalai
New Member

will i get an alert of each instance (that's the desired affect)

0 Karma

ihiesbkalai
New Member

I made the change and I still get There are no fired events for this alert

0 Karma

elliotproebstel
Champion

What type of license do you have? Is there any chance your evaluation has expired? If you're in "Free" status, alerts won't fire.

If that's not the issue, check your Alert configuration. Ensure that under "Add Actions", you've selected "Add to Triggered Alerts".

And if you want one alert for each result, look at the "Trigger" line under "Trigger Conditions". There are two options: "Once" and "For each result". If you want one alert per result, select "For each result".

0 Karma

ihiesbkalai
New Member

I added Added Triggered Alerts to the trigger action and got a trigger but not an email

0 Karma

elliotproebstel
Champion

Then we're on the right track! Under "Trigger Actions", select "Add Actions" and choose "Send email". You'll need to enter your email address in the To field and then save it again.

0 Karma

ihiesbkalai
New Member

I setup the email action but no emails

0 Karma

elliotproebstel
Champion

Have you used this Splunk instance to send any other emails? If not, you'll need to follow the steps on this guide to configure communication between Splunk and your mail server:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Alert/Emailnotification

0 Karma

ihiesbkalai
New Member

Splunk Enterprise Sales Trial 10,240 MB Mar 14, 2018 5:37:43 PM valid

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!