Heyho!
I am currently working on putting together a summary index and would like to do it via an alert.
The basics are all clear on how to set an alert, scheduling, writing into the index and so on.
However I haven’t been able to find any best practice info here and info on how splunk behaves on delayed alerts (not delayed indexing).
So the Alert I am setting up (and this is already working in a csv) is doing a search for 6h which is at least full 24h ago, and it snaps to the last completed 6h batch of the previous day ( 0,6,12,18). So if you run a search on the 26. of May at 6:01 or at 11:55 it will automatically search for the timeframe of 25. May at 0:00 to 6:00. And that’s all fine in itself, no assistance needed.
What I do not know though is how splunk alerts usually behave.
When I schedule a search in Splunk to run at 0:00 and let’s say select the timeframe in the alert of -6h to now. What happens if due to technical issues the search is delayed and runs at 6 AM? Will it now search from 0:00 to 6:00 instead?
When a search is delayed, will the now() command in the search refer back to the “old” scheduled date or will it accordingly use the date of runtime?
What happens if you schedule a hourly search but due to technical issues no search can run in a 90m time window. Will the delayed search run together with the next search or will it be skipped?
With using these 6h intervals I mentioned before in my plans, do you think that all issues are minimized as far as it gets or do you have any other recommendations?
