I am trying to schedule 60 saved searches with summery indexing. There are for 5 different searches, each with 4 schedules - 10min, 1h, 1d, 1w. The initial 10min search will summarise, from there I want the 1h search to create an SI from the 10min SI, the 1d to create an SI from the 1h SI, etc.
-What would be the best way to do this in terms of setting schedule windows
-How can I catch up for delayed data, populate the latest period and verify the previous one if possible?
To take care of indexing latency, your time range should go little earlier then the current time.
E.g. For 10 min searches, the time range can be
-15m@m to 5m@m, allowing 5 min extra for new events to become searchable. The cron can be this
3-59/10 * * * * (running on 3,13,23.. min of every hour).
For this same 10 min search, the data for last hour will get completed after 3rd min execution of 10 min search, so the time range for 1 hour this can will be
-1h@h to @h with cron as
15 * * * *
The daily search's time range will be
-1d@d to @d, with cron
11 1 * * * (allowing adding 1 hour 11 min for hourly data to be completed)
The weekly search's time range will be
-1w@w to @w, with cron
22 2 * * 1
Thanks for the help! You answered another one of my questions on tstats as well which was very helpful. As mentioned in the above post, I am creating a kinda search inception and am wondering how my 2nd,3rd,4th search string would be written?
EG - (Event count) - The initial 10min search to populate the SI is - | tstats count WHERE index=test* earliest=-10m@m latest=@m by _time span=10m
How would I shape that search for the other 3 SI's populating searches, based on the first SI, so that I can ultimately use anyone of the 4 I choose as a single value with trendline on my dashboard?
After you set your 10 min search using above tstats query, your 10 min SI will have (listing main fields only) fields _time and count, with field count not being an indexed fields (fields gets ingested as kv pair in SI).
So your other SI searches would use regular stats/timechart command, like this
index=your10MinSI source="your10MinSISearchName" earliest=-1h@h latest=@h | timechart span=1h sum(count) as count
index=your1HourSI source="your1HourSISearchName" earliest=-1d@d latest=@d | timechart span=1d sum(count) as count
index=your1DaySI source="your1DaySISearchName" earliest=-1w@w latest=@w | timechart span=1w sum(count) as count
With regards to the above - if I dont want to have buckets that are say 09.48 -0 9.58 can I change the cron jobs to run at 5-59/10 * * * * and so on, so that my time will be on 09.50 - 10.00 etc?