Splunk Search

My search returns results, but why are alerts not being triggered?

jcohen999
Explorer

I've set up an alert based on a search that I know returns results. However, the alerts aren't firing.

Here is the search string:

sourcetype=log4j API_Response_Duration>5000 |stats count(API_Response_Duration) as "API Resp > 5 Seconds" by  API_Method

The job runs every hour at 15 minutes past the hour.
I set the alert to fire when Number of Results > 1

0 Karma
1 Solution

jcohen999
Explorer

So, I feel a bit silly but it turned out to be that the mail relay was never configured. Our corporate policy is to not allow outbound email directly from the app server. Email worked early on simply because the mail server didn't yet see the mail as spam. I just configured Splunk to use our relay and mail is now flowing as expected. THANKS FOR THE HELP!! Sorry for wasting your time :).

View solution in original post

0 Karma

jcohen999
Explorer

So, I feel a bit silly but it turned out to be that the mail relay was never configured. Our corporate policy is to not allow outbound email directly from the app server. Email worked early on simply because the mail server didn't yet see the mail as spam. I just configured Splunk to use our relay and mail is now flowing as expected. THANKS FOR THE HELP!! Sorry for wasting your time :).

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Don't sweat it. In a previous life I was a mail guy. So I was suspecting, just wanted you to get there in the end, and you did!

rmarcum
Explorer

I do not think you will get a trigger with "Number of Results > 1". Consider using a "Custom" trigger of "search some_field=some_value

0 Karma

jcohen999
Explorer

turned out to be a configuration issue with my mail server. The Alert did fire with Number or Results > 1.
Thanks

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Have you looked in the scheduler.log to ensure the search is actually running? It should also tell you if result were found.

index=_internal source=*scheduler.log searchname
or
$SPLUNK_HOME/var/log/splunk/scheduler.log

Then if that is the case, have you looked in python.log to see if the alert is actually being sent, and if so is it possibly stuck on a mail server somewhere and/or in a spam box?

0 Karma

jcohen999
Explorer

entry from today's scheduler.log.. Seems to be running but still email.

07-15-2016 16:16:07.519 +0000 INFO  SavedSplunker - savedsearch_id="nobody;search;API_Response > 5 Seconds", user="admin", app="search", savedsearch_name="API_Response > 5 Seconds", status=success, digest_mode=1, scheduled_time=1468599300, window_time=0, dispatch_time=1468599301, run_time=65.782, result_count=7, alert_actions="email", sid="scheduler__admin__search__RMD58218cf9de9be1944_at_1468599300_39552", suppressed=0, thread_id="AlertNotifierWorker-0"
0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Okay, I will try and repro the "u" part and see if that is normal.

What I want you to do however is assuming this is linux, and your mail is set to go to localhost, this means you are running Postfix or Sendmail as an MTA on your box.

This means that you should have a something like /var/log/maillog.

You should see it accept an email from splunk and then do something with it. Do you see log entries of the sort. Perhaps the mail is queuing on your localbox, or you are getting a relaying denied to upstream host or your DNS is not working so it can't work out who to send it to OR OR OR.

If we know splunk is sending the email to the local instance successfully we can move the troubleshooting to you MTA rather than looking at splunk.

0 Karma

jcohen999
Explorer

Just checked the mail log. The emails are showing up but im seeing DSN=User unknown. Probably a configuration issue with our relay server. I have one of our infra guys looking at it.

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Good to know. In the OS mail log, do they have the "u" in them or does the email address look normal?

0 Karma

jcohen999
Explorer

Hi, I was able to see the alert in scheduler.log as well as the email log in python.log but still haven't seen any email. Is it normal for each recipient in the list to have u' prepended before each email address? recipients="[u'john.smith@yahoo.com', u'matthew.jones@microsoft.com']?

0 Karma

somesoni2
SplunkTrust
SplunkTrust

No, the email address should be just be john.smith@yahoo.com,matthew.jones@microsoft.com

0 Karma

jcohen999
Explorer

The email addresses weren't entered that way and they show up correctly in through the front end alert properties. Is this an indication that something is wrong or configured incorrectly?

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

What version are you running? I will create one and see. My question still is, in your conf file what is the mail server set to? Please let me know the answer to both.

0 Karma

jcohen999
Explorer

version is 6.3.2. Mail server is set to localhost.

0 Karma

jwelch_splunk
Splunk Employee
Splunk Employee

Are you sending email from Splunk to "localhost" mta or are you sending it to a remote MTA. If local, look in the mail logs and see if the email is queued, if remote make sure they accepted it and routed it.

Look at your alert_actions.conf to determine what your MTA is.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...