My search returns results, but why are alerts not being triggered?

[jcohen999]

I've set up an alert based on a search that I know returns results. However, the alerts aren't firing.

Here is the search string:

sourcetype=log4j API_Response_Duration>5000 |stats count(API_Response_Duration) as "API Resp > 5 Seconds" by  API_Method

The job runs every hour at 15 minutes past the hour.
I set the alert to fire when Number of Results > 1

[jcohen999 · Accepted solution]

So, I feel a bit silly but it turned out to be that the mail relay was never configured. Our corporate policy is to not allow outbound email directly from the app server. Email worked early on simply because the mail server didn't yet see the mail as spam. I just configured Splunk to use our relay and mail is now flowing as expected. THANKS FOR THE HELP!! Sorry for wasting your time :).

[jwelch_splunk]

Don't sweat it. In a previous life I was a mail guy. So I was suspecting, just wanted you to get there in the end, and you did!

[rmarcum]

I do not think you will get a trigger with "Number of Results > 1". Consider using a "Custom" trigger of "search some_field=some_value

[jcohen999]

turned out to be a configuration issue with my mail server. The Alert did fire with Number or Results > 1.
Thanks

[jwelch_splunk]

Have you looked in the scheduler.log to ensure the search is actually running? It should also tell you if result were found.

index=_internal source=*scheduler.log searchname
or
$SPLUNK_HOME/var/log/splunk/scheduler.log

Then if that is the case, have you looked in python.log to see if the alert is actually being sent, and if so is it possibly stuck on a mail server somewhere and/or in a spam box?

[jcohen999]

entry from today's scheduler.log.. Seems to be running but still email.

07-15-2016 16:16:07.519 +0000 INFO  SavedSplunker - savedsearch_id="nobody;search;API_Response > 5 Seconds", user="admin", app="search", savedsearch_name="API_Response > 5 Seconds", status=success, digest_mode=1, scheduled_time=1468599300, window_time=0, dispatch_time=1468599301, run_time=65.782, result_count=7, alert_actions="email", sid="scheduler__admin__search__RMD58218cf9de9be1944_at_1468599300_39552", suppressed=0, thread_id="AlertNotifierWorker-0"

[jwelch_splunk]

Okay, I will try and repro the "u" part and see if that is normal.

What I want you to do however is assuming this is linux, and your mail is set to go to localhost, this means you are running Postfix or Sendmail as an MTA on your box.

This means that you should have a something like /var/log/maillog.

You should see it accept an email from splunk and then do something with it. Do you see log entries of the sort. Perhaps the mail is queuing on your localbox, or you are getting a relaying denied to upstream host or your DNS is not working so it can't work out who to send it to OR OR OR.

If we know splunk is sending the email to the local instance successfully we can move the troubleshooting to you MTA rather than looking at splunk.

[jcohen999]

Just checked the mail log. The emails are showing up but im seeing DSN=User unknown. Probably a configuration issue with our relay server. I have one of our infra guys looking at it.

[jwelch_splunk]

Good to know. In the OS mail log, do they have the "u" in them or does the email address look normal?

[jcohen999]

Hi, I was able to see the alert in scheduler.log as well as the email log in python.log but still haven't seen any email. Is it normal for each recipient in the list to have u' prepended before each email address? recipients="[u'john.smith@yahoo.com', u'matthew.jones@microsoft.com']?

[somesoni2]

No, the email address should be just be john.smith@yahoo.com,matthew.jones@microsoft.com

[jcohen999]

The email addresses weren't entered that way and they show up correctly in through the front end alert properties. Is this an indication that something is wrong or configured incorrectly?

[jwelch_splunk]

What version are you running? I will create one and see. My question still is, in your conf file what is the mail server set to? Please let me know the answer to both.

[jcohen999]

version is 6.3.2. Mail server is set to localhost.

[jwelch_splunk]

Are you sending email from Splunk to "localhost" mta or are you sending it to a remote MTA. If local, look in the mail logs and see if the email is queued, if remote make sure they accepted it and routed it.

Look at your alert_actions.conf to determine what your MTA is.