Alerting

Alert suddenly stops sending email as alert action

avoelk
Communicator

Hello there, I hope someone can help!

a report we generated doesn't send emails, or just sporadically. also, the action time for this is very high as you can see here (field "action_time_ms"): 

10-08-2021 14:20:35.156 +0200 INFO SavedSplunker - savedsearch_id="nobody;search;SIGNL4 - High or Critical Notable Events Clone", search_type="scheduled", user="maximilian.wehner", app="search", savedsearch_name="SIGNL4 - High or Critical Notable Events Clone", priority=default, status=success, digest_mode=0, scheduled_time=1633695120, window_time=0, dispatch_time=1633695122, run_time=2.838, result_count=1, alert_actions="email", sid="scheduler_bWF4aW1pbGlhbi53ZWhuZXI__search__RMD55d86aa6233cebf27_at_1633695120_428", suppressed=0, fired=1, skipped=0, action_time_ms=509817, thread_id="AlertNotifierWorker-1", message="", workload_pool=""

 

action_time_ms is a LOT. so something prevents it from being sent or whatever is going on. 

usually I think splunk could send an email without configuring a mailserver but currently we want to use our o365 mailserver for it. this has been tested with another environment and there it definitely works like a charm.  here the config of the alert and the mailserver config: 

--> we've artificially set the maxtime very high to check if splunk is finally sendint the mail after a while. record was over 8 minutes until a mail was sent. 
 
My questions now are how can this happen? is there a way to further investigate and resolve this issue? currently this alert is mandatory for a security view and if this alert only comes every now and then, it's a main issue. 
 
[email]
auth_password = ****
auth_username = user@xyz.de
from = splunk@sxyz.de
mailserver = smtp.office365.com
pdf.header_left = none
pdf.header_right = none
use_tls = 1
reportPaperSize = a4
hostname = somehostname
maxtime = 20m
 

is there something wrong with the config? What can I do to further troubleshoot this issue and hopefully resolve it? I guess this issue has come up in the past

 

thanks a lot for help!

Labels (2)
Tags (1)
0 Karma

Stefanie
Builder

Is that the only alert having this issue? Could there be a setting throttling your results? 
Could you manually recreate that alert?

0 Karma

avoelk
Communicator

hello,

 

yes that's the only alert having that problem. the settings are the same as within another environment and there it doesn't take 30000 ms but 500. the difference between those environments is mainly that the searchheads in which I try to use the alert stand in different networks. so I'm assuming there might be something between the searchhead and the inbox of my email that is throttleing it. could that be? 

I don't really "get" this error, since it's the first time and with an alert that has been used multiple times on different occations.

0 Karma

Stefanie
Builder

Strange, and I'm assuming that the problem alert isn't the only alert on that SH in that environment?

Are you able to send the results of that alert to a different email? That would answer the question if it's something between the SH and your inbox. I doubt it though.

 

avoelk
Communicator

Yes, when sending it to a different, non o365, email address it's going faster. 

But we figured out what might be the cause (still to be confirmed): the firewall resolves the FQDN into several ips so on our firewall, when we allow ooffice365 on a certain port, for example TLS, it won't work that well cause these IPs are bound to change every now and then thanks to microsoft. so I guess in order to fully work around this issue we either need to send the mail to a different inbox and then forwarding it OR allow all IPs from wich the mail might be sended. 

 

I'll update this post once confirmed (or not). fingers crossed

Stefanie
Builder

Good luck! It definitely sounds easier to send it to a different email and have a rule in outlook to forward that email.  Either way glad you found the issue.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...