topic Re: Scheduling multiple searches in Splunk Search

Scheduling multiple searches

mansel_scheffel — Tue, 09 Aug 2016 10:05:38 GMT

Hi,

I am trying to schedule 60 saved searches with summery indexing. There are for 5 different searches, each with 4 schedules - 10min, 1h, 1d, 1w. The initial 10min search will summarise, from there I want the 1h search to create an SI from the 10min SI, the 1d to create an SI from the 1h SI, etc.

-What would be the best way to do this in terms of setting schedule windows
-How can I catch up for delayed data, populate the latest period and verify the previous one if possible?

Thanks!

Re: Scheduling multiple searches

somesoni2 — Tue, 09 Aug 2016 14:55:33 GMT

To take care of indexing latency, your time range should go little earlier then the current time.

E.g. For 10 min searches, the time range can be -15m@m to 5m@m, allowing 5 min extra for new events to become searchable. The cron can be this 3-59/10 * * * * (running on 3,13,23.. min of every hour).
For this same 10 min search, the data for last hour will get completed after 3rd min execution of 10 min search, so the time range for 1 hour this can will be -1h@h to @h with cron as 15 * * * *
The daily search's time range will be -1d@d to @d, with cron 11 1 * * * (allowing adding 1 hour 11 min for hourly data to be completed)
The weekly search's time range will be -1w@w to @w, with cron 22 2 * * 1

Re: Scheduling multiple searches

mansel_scheffel — Tue, 09 Aug 2016 15:02:26 GMT

Thanks for the help! You answered another one of my questions on tstats as well which was very helpful. As mentioned in the above post, I am creating a kinda search inception and am wondering how my 2nd,3rd,4th search string would be written?

EG - (Event count) - The initial 10min search to populate the SI is - | tstats count WHERE index=test* earliest=-10m@m latest=@m by _time span=10m

How would I shape that search for the other 3 SI's populating searches, based on the first SI, so that I can ultimately use anyone of the 4 I choose as a single value with trendline on my dashboard?

Re: Scheduling multiple searches

somesoni2 — Tue, 09 Aug 2016 15:20:42 GMT

After you set your 10 min search using above tstats query, your 10 min SI will have (listing main fields only) fields _time and count, with field count not being an indexed fields (fields gets ingested as kv pair in SI).

So your other SI searches would use regular stats/timechart command, like this

Hourly

index=your10MinSI source="your10MinSISearchName" earliest=-1h@h latest=@h | timechart span=1h sum(count) as count

Daily

index=your1HourSI source="your1HourSISearchName" earliest=-1d@d latest=@d | timechart span=1d sum(count) as count

Weekly

index=your1DaySI source="your1DaySISearchName" earliest=-1w@w latest=@w | timechart span=1w sum(count) as count

Re: Scheduling multiple searches

mansel_scheffel — Tue, 09 Aug 2016 15:24:52 GMT

Awesome thanks a lot !!

Re: Scheduling multiple searches

mansel_scheffel — Thu, 11 Aug 2016 09:33:43 GMT

With regards to the above - if I dont want to have buckets that are say 09.48 -0 9.58 can I change the cron jobs to run at 5-59/10 * * * * and so on, so that my time will be on 09.50 - 10.00 etc?