topic Re: Scheduled Searches delayed by one hour in Splunk Search

Scheduled Searches delayed by one hour

omuelle1 — Fri, 07 Feb 2020 12:37:05 GMT

Hi,

I have lately seen an issue that some scheduled alerts that contain attachments seem to get emailed to me one hour later than scheduled. I assume this has to do with volume of the alerts and searches scheduled but I haven't been to able to nail it down. When I look in the metadata for information about the delayed scheduled search it does show that it ran at the scheduled time, however the email alert often arrives exactly one hour later in my inbox.

Has anyone experience with this kind of issue?

Oliver

Re: Scheduled Searches delayed by one hour

nickhills — Fri, 07 Feb 2020 12:42:07 GMT

Can you see from the _internal logs when the email was sent. Was it at the anticipated time, or shortly before recieved?

Re: Scheduled Searches delayed by one hour

paulbannister — Fri, 07 Feb 2020 15:01:05 GMT

The exactly one hour part makes me think timezone related for some reason, mainly as we've had to deal with GMT\BST data issues here, possibility at all?

Re: Scheduled Searches delayed by one hour

nickhills — Fri, 07 Feb 2020 15:03:31 GMT

Could well be, reports are scheduled based on the TZ settings of the user who scheduled them.
If the user has the wrong TZ set, they will defer according to that offset.

Re: Scheduled Searches delayed by one hour

paulbannister — Fri, 07 Feb 2020 15:06:17 GMT

^ Exactly this, thanks for expanding

Re: Scheduled Searches delayed by one hour

omuelle1 — Fri, 07 Feb 2020 16:24:14 GMT

I don't think it's the TZ since they are delayed only some of the time.

So the alert fired at 6:00 AM, when it is scheduled to run:

2020-02-07 06:00:27,173 -0500 INFO  sendemail:134 - Sending email. subject="Splunk Alert: 24 Hour Remote Country Login with MFA  Check", results_link="https://XXXX.splunkcloud.com/app/XX_custom_app/@go?sid=scheduler_c2VjX2das9saXZlci5tdWVsXXXbGVyQG9tZy5jcdasdmhvbWcuY29t_T01HX2N1c3RvbV9hcHA__RMD5a1b9cc0db37475e5_at_1581073200_64487", recipients="[u'XXX@XXX.com']", server="XX.XX.XX.1:25"

However the email arrived in my inbox at 7 AM. Yesterday it came at 6 AM as scheduled, so it's not very consistent. But when it is delayed it is delayed but 1 hour.

Re: Scheduled Searches delayed by one hour

nickhills — Fri, 07 Feb 2020 16:27:41 GMT

If Splunk says it sent the alert at 6AM (utc-5h) and it was recieved at 7AM(utc-5) then the delay is external to splunk.

Do you have mailserver logs or a mail provider you can investigate with?

Re: Scheduled Searches delayed by one hour

omuelle1 — Fri, 07 Feb 2020 16:41:26 GMT

Thank you, guys. The mail idea brought me on the right track.

The email apparently sometimes gets quarantined because of its attachment and then released automatically an hour later.