Hello there, I hope someone can help!
a report we generated doesn't send emails, or just sporadically. also, the action time for this is very high as you can see here (field "action_time_ms"):
10-08-2021 14:20:35.156 +0200 INFO SavedSplunker - savedsearch_id="nobody;search;SIGNL4 - High or Critical Notable Events Clone", search_type="scheduled", user="maximilian.wehner", app="search", savedsearch_name="SIGNL4 - High or Critical Notable Events Clone", priority=default, status=success, digest_mode=0, scheduled_time=1633695120, window_time=0, dispatch_time=1633695122, run_time=2.838, result_count=1, alert_actions="email", sid="scheduler_bWF4aW1pbGlhbi53ZWhuZXI__search__RMD55d86aa6233cebf27_at_1633695120_428", suppressed=0, fired=1, skipped=0, action_time_ms=509817, thread_id="AlertNotifierWorker-1", message="", workload_pool=""
action_time_ms is a LOT. so something prevents it from being sent or whatever is going on.
usually I think splunk could send an email without configuring a mailserver but currently we want to use our o365 mailserver for it. this has been tested with another environment and there it definitely works like a charm. here the config of the alert and the mailserver config:
is there something wrong with the config? What can I do to further troubleshoot this issue and hopefully resolve it? I guess this issue has come up in the past
thanks a lot for help!
Is that the only alert having this issue? Could there be a setting throttling your results?
Could you manually recreate that alert?
hello,
yes that's the only alert having that problem. the settings are the same as within another environment and there it doesn't take 30000 ms but 500. the difference between those environments is mainly that the searchheads in which I try to use the alert stand in different networks. so I'm assuming there might be something between the searchhead and the inbox of my email that is throttleing it. could that be?
I don't really "get" this error, since it's the first time and with an alert that has been used multiple times on different occations.
Strange, and I'm assuming that the problem alert isn't the only alert on that SH in that environment?
Are you able to send the results of that alert to a different email? That would answer the question if it's something between the SH and your inbox. I doubt it though.
Yes, when sending it to a different, non o365, email address it's going faster.
But we figured out what might be the cause (still to be confirmed): the firewall resolves the FQDN into several ips so on our firewall, when we allow ooffice365 on a certain port, for example TLS, it won't work that well cause these IPs are bound to change every now and then thanks to microsoft. so I guess in order to fully work around this issue we either need to send the mail to a different inbox and then forwarding it OR allow all IPs from wich the mail might be sended.
I'll update this post once confirmed (or not). fingers crossed
Good luck! It definitely sounds easier to send it to a different email and have a rule in outlook to forward that email. Either way glad you found the issue.