There are a couple of methods to do this, but they generally require a custom scripted input. My favorite is to create an email and send via direct SMTP to the cloud provider, directed at a mailbox inside your organization that drops mail on the floor. Your email should include a well-known subject (e.g. [TIMING TEST] ). Make sure you include the timezone in the date/time string for proper evaluation later on.
In your SMTP logs, you do a search where the recipient is your dropbox and the subject includes [TIMING TEST]. If you are using the Splunk App for Microsoft Exchange, that search is:
eventtype=msexchange-msgtrack source_id=SMTP event_id=RECEIVE recipient="mydropbox@domain.com" subject="*[TIMING TEST]*"
Now, you can do two things here. Firstly, you can calculate the delay - simply extract the date/time that you sent the message from the subject, convert it to a timestamp with eval and strptime(), then subtract the timestamp from _time to get the # seconds between when you sent and when you received it.
Secondly, you can detect when mail is stacking up by checking for the last time you received a timing test. If you send your timing test every 5 minutes and your delivery SLA is 1 minute, you can do an alert if you have not received a message within 6 minutes. I'd add a few seconds to that for delays in logging, so your search looks like:
eventtype=msexchange-msgtrack source_id=SMTP event_id=RECEIVE recipient="mydropbox@domain.com" subject="*[TIMING TEST]*"|eval td=time()-_time|where td > 375|table subject,td
Search over the last 10 minutes and convert to an alert.
I've obviously taken examples from the Splunk App for Microsoft Exchange here, but you could adjust for any SMTP gateway.
... View more
