Alerting

Alert suddenly stops sending email as alert action

avoelk
Communicator

Hello there, I hope someone can help!

a report we generated doesn't send emails, or just sporadically. also, the action time for this is very high as you can see here (field "action_time_ms"): 

10-08-2021 14:20:35.156 +0200 INFO SavedSplunker - savedsearch_id="nobody;search;SIGNL4 - High or Critical Notable Events Clone", search_type="scheduled", user="maximilian.wehner", app="search", savedsearch_name="SIGNL4 - High or Critical Notable Events Clone", priority=default, status=success, digest_mode=0, scheduled_time=1633695120, window_time=0, dispatch_time=1633695122, run_time=2.838, result_count=1, alert_actions="email", sid="scheduler_bWF4aW1pbGlhbi53ZWhuZXI__search__RMD55d86aa6233cebf27_at_1633695120_428", suppressed=0, fired=1, skipped=0, action_time_ms=509817, thread_id="AlertNotifierWorker-1", message="", workload_pool=""

 

action_time_ms is a LOT. so something prevents it from being sent or whatever is going on. 

usually I think splunk could send an email without configuring a mailserver but currently we want to use our o365 mailserver for it. this has been tested with another environment and there it definitely works like a charm.  here the config of the alert and the mailserver config: 

--> we've artificially set the maxtime very high to check if splunk is finally sendint the mail after a while. record was over 8 minutes until a mail was sent. 
 
My questions now are how can this happen? is there a way to further investigate and resolve this issue? currently this alert is mandatory for a security view and if this alert only comes every now and then, it's a main issue. 
 
[email]
auth_password = ****
auth_username = user@xyz.de
from = splunk@sxyz.de
mailserver = smtp.office365.com
pdf.header_left = none
pdf.header_right = none
use_tls = 1
reportPaperSize = a4
hostname = somehostname
maxtime = 20m
 

is there something wrong with the config? What can I do to further troubleshoot this issue and hopefully resolve it? I guess this issue has come up in the past

 

thanks a lot for help!

Labels (2)
Tags (1)
0 Karma

Stefanie
Builder

Is that the only alert having this issue? Could there be a setting throttling your results? 
Could you manually recreate that alert?

0 Karma

avoelk
Communicator

hello,

 

yes that's the only alert having that problem. the settings are the same as within another environment and there it doesn't take 30000 ms but 500. the difference between those environments is mainly that the searchheads in which I try to use the alert stand in different networks. so I'm assuming there might be something between the searchhead and the inbox of my email that is throttleing it. could that be? 

I don't really "get" this error, since it's the first time and with an alert that has been used multiple times on different occations.

0 Karma

Stefanie
Builder

Strange, and I'm assuming that the problem alert isn't the only alert on that SH in that environment?

Are you able to send the results of that alert to a different email? That would answer the question if it's something between the SH and your inbox. I doubt it though.

 

avoelk
Communicator

Yes, when sending it to a different, non o365, email address it's going faster. 

But we figured out what might be the cause (still to be confirmed): the firewall resolves the FQDN into several ips so on our firewall, when we allow ooffice365 on a certain port, for example TLS, it won't work that well cause these IPs are bound to change every now and then thanks to microsoft. so I guess in order to fully work around this issue we either need to send the mail to a different inbox and then forwarding it OR allow all IPs from wich the mail might be sended. 

 

I'll update this post once confirmed (or not). fingers crossed

Stefanie
Builder

Good luck! It definitely sounds easier to send it to a different email and have a rule in outlook to forward that email.  Either way glad you found the issue.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...