Alerting

Why is the custom trigger not firing for this alert?

ihiesbkalai
New Member

My search is:

index=soma   source="alarms.txt" StatusID=*
|eval Alarm=if(StatusID=0,"Critical","No")
| table  DateTime EventCode StatusID Message Alarm

The alert runs every minute (cron * * * * *)
and my custom trigger is
search Alarm="No"
There are no fired events for this alert
If I run the search |where Alarm="No" there are 94 events matching that

What am I doing wrong?

0 Karma

micahkemp
Champion

When you say cron *, what is your exact cron expression? I don't believe a single * is a valid expression, so I want to make sure the issue isn't simply that your search isn't running when you think it should be.

0 Karma

ihiesbkalai
New Member

the cron job value is * * * * *
And that's not the issue any longer. Triggers are fired but no emails is generated

0 Karma

elliotproebstel
Champion

What is the time window being monitored for this search that's running every minute? In my experience, if you're running a search every minute (and presumably looking at data logged in the previous minute), the delays of ingestion may prevent you from seeing anything at all.

0 Karma

ihiesbkalai
New Member

Last 7 days. this is a static file i uploaded to the index=soma

0 Karma

elliotproebstel
Champion

Got it. So how about changing the approach. I'd suggest changing your search to this:

index=soma source="alarms.txt" StatusID=* 
| eval Alarm=if(StatusID=0,"Critical","No") 
| table DateTime EventCode StatusID Message Alarm
| search Alarm="No"

And set the trigger to be Number of Results, with is greater than set to 0. Will that achieve what you're trying to do?

0 Karma

ihiesbkalai
New Member

will i get an alert of each instance (that's the desired affect)

0 Karma

ihiesbkalai
New Member

I made the change and I still get There are no fired events for this alert

0 Karma

elliotproebstel
Champion

What type of license do you have? Is there any chance your evaluation has expired? If you're in "Free" status, alerts won't fire.

If that's not the issue, check your Alert configuration. Ensure that under "Add Actions", you've selected "Add to Triggered Alerts".

And if you want one alert for each result, look at the "Trigger" line under "Trigger Conditions". There are two options: "Once" and "For each result". If you want one alert per result, select "For each result".

0 Karma

ihiesbkalai
New Member

I added Added Triggered Alerts to the trigger action and got a trigger but not an email

0 Karma

elliotproebstel
Champion

Then we're on the right track! Under "Trigger Actions", select "Add Actions" and choose "Send email". You'll need to enter your email address in the To field and then save it again.

0 Karma

ihiesbkalai
New Member

I setup the email action but no emails

0 Karma

elliotproebstel
Champion

Have you used this Splunk instance to send any other emails? If not, you'll need to follow the steps on this guide to configure communication between Splunk and your mail server:
http://docs.splunk.com/Documentation/Splunk/7.0.2/Alert/Emailnotification

0 Karma

ihiesbkalai
New Member

Splunk Enterprise Sales Trial 10,240 MB Mar 14, 2018 5:37:43 PM valid

0 Karma
Get Updates on the Splunk Community!

New Splunk Observability innovations: Deeper visibility and smarter alerting to ...

You asked, we delivered. Splunk Observability Cloud has several new innovations giving you deeper visibility ...

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...