I would like to backfill my index up by 2 months. The query however, is time sensitive and requires the day span to be only between 7am-9pm. Currently, my only method is to manually change the earliest and latest times in both the search and the summary index settings to earliest=-1d@d+7h latest=-1d@d+20h
, and then to earliest=-2d@d+7h latest=-2d@d+20h
, etc. etc.. you can see just how tedious and time-consuming this can become.
Is there any way that I would not have to insert any relative day into my period, to be able to run my overall index search for 30 days with days only involving data between 7am-9pm everyday? if there were an earliest=7h latest=20h kind of deal, that would be great, but I have not found any yet.
Thanks in advance
You can do something like this
1) Create a scheduled saved search for your query. This query should run daily and summarize data for yesterday
2) For the query, add earliest=-1d@d+7h
latest=-1d@d+20h
(within the query)
3) For saved search's Start time use -1d@d
, for Finish time, use @d
(this just to enable scheduling, search'es time range is overridded in the search itself in step 2)
4) For cron use something like this 45 6 * * *
(daily once at 6:45 AM)
5) Use the Splunk backfill utility to backfill any duration at once.
Sample command:
Search Time Range= -1d@d to @d
Since, we summarize yesterday's data, to backfill for Jan-01,2015 to Jan-31-2015, the search should run from Jan-02 (to summarize jan-01) to Feb-01 (to summarize Jan-31)
Backfill job Scheduling time
Earliest Time:01/02/2015 00:00
In Epoch: 1420174800 -- to be used as argument et
Latest Time: 02/01/2015 23:59:59.000000
In Epoch: 1422853199 - to be used as argument lt
Backfill Unix Command (after cd $Splunk_home/bin)
./splunk cmd python fill_summary_index.py -app YourAppName -name "YourSearchName" -et 1420174800 -lt 1422853199 -j 1 -dedup true -auth AdminUserName:AdminPasswordName &
You do know about the backfill
command, right? You should be able use your standard daily SI-populating search as-is as described here:
http://docs.splunk.com/Documentation/Splunk/6.2.5/Knowledge/Managesummaryindexgapsandoverlaps
actually i didn't know about the backfill command. bit of a splunk noob here. thanks!
You can do something like this
1) Create a scheduled saved search for your query. This query should run daily and summarize data for yesterday
2) For the query, add earliest=-1d@d+7h
latest=-1d@d+20h
(within the query)
3) For saved search's Start time use -1d@d
, for Finish time, use @d
(this just to enable scheduling, search'es time range is overridded in the search itself in step 2)
4) For cron use something like this 45 6 * * *
(daily once at 6:45 AM)
5) Use the Splunk backfill utility to backfill any duration at once.
Sample command:
Search Time Range= -1d@d to @d
Since, we summarize yesterday's data, to backfill for Jan-01,2015 to Jan-31-2015, the search should run from Jan-02 (to summarize jan-01) to Feb-01 (to summarize Jan-31)
Backfill job Scheduling time
Earliest Time:01/02/2015 00:00
In Epoch: 1420174800 -- to be used as argument et
Latest Time: 02/01/2015 23:59:59.000000
In Epoch: 1422853199 - to be used as argument lt
Backfill Unix Command (after cd $Splunk_home/bin)
./splunk cmd python fill_summary_index.py -app YourAppName -name "YourSearchName" -et 1420174800 -lt 1422853199 -j 1 -dedup true -auth AdminUserName:AdminPasswordName &