Hi,
I need to automate the backfill script for about 60 searches.. Is there a way to put all 60 searches in a single script and then make them run one by one(wait until previous one is done before continuing to next)
Thanks
Hey @mwdbhyat,
fill_summary_index.py takes an argument called -j <integer>
(max 😎 this will decide the concurrency. If not specified , one search backfill is executed at a time. As far as running automated for 60 searches, are they all in a same app? Can you move them to xyz app (just 60 and nothing more) and use -app xyz -name *
or get creative pragmatically where you have an array and run it through some sort of controlled loop.
What i feel tricky is the time frame you need to select if all these 60 have different windows. I do the backfills quite often and i have wrapped them in shell scripts and tied them up to cron based on their original schedules (offcourse -dedup true)
Hope this helps!
Thanks,
Raghav
Hey @mwdbhyat,
fill_summary_index.py takes an argument called -j <integer>
(max 😎 this will decide the concurrency. If not specified , one search backfill is executed at a time. As far as running automated for 60 searches, are they all in a same app? Can you move them to xyz app (just 60 and nothing more) and use -app xyz -name *
or get creative pragmatically where you have an array and run it through some sort of controlled loop.
What i feel tricky is the time frame you need to select if all these 60 have different windows. I do the backfills quite often and i have wrapped them in shell scripts and tied them up to cron based on their original schedules (offcourse -dedup true)
Hope this helps!
Thanks,
Raghav
Hi Raghav,
Thanks for the reply.. It is for 1 app yes. Basically I have created a txt file with all 60 searches listed with the ./backfill command as above. Initially just to add in previous 12w worth of data I just want it to do the automated backfill once. So if I just run my txt file with 60 searches listed, will it just do them 1 at a time until the whole list is done ?
Correct, keeping the volume of searches aside, that script fill_summary_index.py is designed to execute one search at a time unless asked it to do more.
example:
./splunk cmd python fill_summary_index.py -app xyz -name * -dedup true -showprogress true -et -7d -lt now -j 8 -owner admin
will run / try to run 8 searches at a time (Whole different story if you do not have 8 cores on you search head....rule is 1 search per core).
& if you just not mention -j argument, it will run one search , waits for it to finish and then move on to the next.
Hope this helps!
Thanks,
Raghav
How would I pass the admin username in just once with creds as it is asking me for each search to type in the username and pass?
wrap that in a shell script and pass it one time. That way you do not have to type it every time.
-auth admin:xxxx (Please remove it as soon as your backfill is complete).
Cool, thanks!
Sweet, thanks a lot!
You could have the backfill command run then have it produce an exit code when it's complete which will then trigger the next backfill to begin
Thanks - noob question but could you give me an example of how this would work?
./splunk cmd python fill_summary_index.py -app APPNAME -name SEARCHNAME -et -12w@w -lt now -dedup true
exit script code ?
Assuming that you're working on a Linux machine..
http://bencane.com/2014/09/02/understanding-exit-codes-and-how-to-use-them-in-bash-scripts/
This example will require you to create multiple scripts and one script will trigger the next script after it completes.. Alternatively if you want a single script, you could use a WAIT
for input and have the script wait until the backfill is complete which will start the next backfill..
#!/bin/bash
./splunk cmd python fill_summary_index.py -app APPNAME -name SEARCHNAME -et -12w@w -lt now -dedup true
if [ $? -eq 0 ]
then
echo "Successfully ran backfill"
exit 0
else
echo "Errors running backfill" >&2
exit 1
fi
Here's another method of running which will look at the process ID and execute on a loop until all the backfills run
pid=$(ps -opid= -C your_script_name)
while [ -d /proc/$pid ] ; do
sleep 1
done && ./your_other_script
Awesome, thank you!