Why are Splunk alerts getting fired but emails not being sent?

[megha0794]

Hi,

The scheduled report was running fine until a few days back. The scheduler.log shows that the search was run successfully.

10-31-2019 03:20:15.655 -0400 INFO SavedSplunker - savedsearch_id="nobody;BMS_WM_INTG_METRICS;Super Critical Interface Alert | Japan Invoice", search_type="scheduled", user="krishv11", app="BMS_WM_INTG_METRICS", savedsearch_name="Super Critical Interface Alert | Japan Invoice", priority=default, status=success, digest_mode=1, scheduled_time=1572506400, window_time=0, dispatch_time=1572506414, run_time=0.129, result_count=1, alert_actions="email", sid="scheduler_krishv11_Qk1TX1dNX0lOVEdfTUVUUklDUw_RMD522d64366074c1a92_at_1572506400_23581", suppressed=0, thread_id="AlertNotifierWorker-0"

When I create an alert with the same search and add to triggered events, it does show that the alert was fired, but emails notifications dont go through.

I dont see any errors related to the same in python.log at this path /opt/splunk/var/log/splunk

Any help is appreciated.

Thanks and Regards,
Megha

[gcusello]

Hi megha0794,
I'm sure that you already configured the email server configurations.
Only one question: what's the dimension of the attachment? if it exceed the limits of your email server surely it will be blocked.

In addition, see in Splunk internal logs if there are message, something like this:

index=_internal sendmail error

Ciao.
Giuseppe

[megha0794]

Hi Giuseppe,

There is no attachment, its an inline alert.
Also I don't see any messages using that query , just events related to the search like:

11-05-2019 03:35:40.651 -0500 INFO StreamedSearch - Streamed search search starting: search_id=remote_ip-172-21-152-54_1572942940.315497, server=ip-172-21-152-54, active_searches=1, search='litsearch (index=_internal error sendmail) | fields keepcolorder=t "*" "_bkt" "_cd" "_si" "host" "index" "linecount" "source" "sourcetype" "splunk_server" | remotetl nb=300 et=1572854400.000000 lt=1572942940.000000 remove=true max_count=1000 max_prefetch=100', remote_ttl=600, apiStartTime='Mon Nov 4 03:00:00 2019', apiEndTime='Tue Nov 5 03:35:40 2019', savedsearch_name=""

Thanks ,
Megha

[gcusello]

Hi Megha,
if you haven't error messages means that the problem is probably outside Splunk, have you other alerts or reports that send emails?
If not create another alert to test the connection and/or use telnet to test the route from Splunk Search Head to email server.
The attachment is usually the main problem that blocks emails; the email message is a little message, not a long message I think, is it correct?

Ciao.
Giuseppe

[megha0794]

Hi Giuseppe,

I tried the below sendemail command from dev, test and Prod:
index=_internal | head 5 | sendemail to="magha.manoj@bms.com" subject="Here is an email from Splunk" message="This is an example message" sendresults=true

I receive mails on dev and test but not Prod. All three are having the same email settings.

Also for the reports run on the server using mailx and mutt, they work fine in Prod as well.
We do not have telnet installed yet.

Thanks,
Megha

[gcusello]

Hi Megha,
Check the firewall routes between Prod and eMail server.
And then re-input eMail configurations on Prod.

Ciao.
Giuseppe

[megha0794]

Thanks Giuseppe,

We found that the splunk hostname had got changed somehow and it was causing issues with sendemail. Would you know how we can troubleshoot why this change happened. Would it be possible to get some logs on this?

Thanks,
Megha