I have lately seen an issue that some scheduled alerts that contain attachments seem to get emailed to me one hour later than scheduled. I assume this has to do with volume of the alerts and searches scheduled but I haven't been to able to nail it down. When I look in the metadata for information about the delayed scheduled search it does show that it ran at the scheduled time, however the email alert often arrives exactly one hour later in my inbox.
Has anyone experience with this kind of issue?
If Splunk says it sent the alert at 6AM (utc-5h) and it was recieved at 7AM(utc-5) then the delay is external to splunk.
Do you have mailserver logs or a mail provider you can investigate with?
I don't think it's the TZ since they are delayed only some of the time.
So the alert fired at 6:00 AM, when it is scheduled to run:
2020-02-07 06:00:27,173 -0500 INFO sendemail:134 - Sending email. subject="Splunk Alert: 24 Hour Remote Country Login with MFA Check", results_link="https://XXXX.splunkcloud.com/app/XX_custom_app/@go?sid=scheduler_c2VjX2das9saXZlci5tdWVsXXXbGVyQG9tZy5jcdasdmhvbWcuY29t_T01HX2N1c3RvbV9hcHA__RMD5a1b9cc0db37475e5_at_1581073200_64487", recipients="[u'XXX@XXX.com']", server="XX.XX.XX.1:25"
However the email arrived in my inbox at 7 AM. Yesterday it came at 6 AM as scheduled, so it's not very consistent. But when it is delayed it is delayed but 1 hour.
Could well be, reports are scheduled based on the TZ settings of the user who scheduled them.
If the user has the wrong TZ set, they will defer according to that offset.