I started running the fill_summary_index.py script and my session was interrupted. The summary backfill process never completed and I'm unable to restart it. I've tried restarting Splunk but it still thinks that a fill_summary_index instance is still running:
# ./splunk cmd python fill_summary_index.py Please enter the app that contains the search(es): search An instance of fill_summary_index is already running for app=search
What can I do to cancel it and restart it?
Per Splunk Support, there's a fsidx*.lock file in the app directory that the script was run against. In my case, the search app:
/opt/splunk/etc/apps/search/log]# ls fsidxrlKD2v.lock
After deleting the lock file I was able to relaunch the summary backfill script.