Incremental Saved Search

[arushiseth94]

Hello,

I have a saved search that pushes data to summary index. The summary index has data for last 2 years and data volume is really huge. Suppose I want to add a new field to this data in summary index, I need to re run search for last two years. Since the volume is huge, if I try to run the search for all 2 years data in one time, the search fails or data gets missed. To avoid this, I'll be pushing data in 10 days batch or 30 days batch.

For example - if I have to repopulate my summary index after adding a new field. So, for first batch I'll run for data from 1st Aug 2023 to 10th Aug 2023. Next batch I'll run from 11th Aug to 20th Aug.

Similar thing has to be done for past two years of data to be pushed in summary index.

This task is very cumbersome . Is there a way to automate this task in splunk. Can I schedule my search in such a way that while repushing data , without manual intervention data gets pushed in 10 days batch in summary index?

[arushiseth94]

Hi @richgalloway , Thank you for your response. 

Similar to summary index , I have KV Stores as well , where I am pushing data in similar manner in 10 days batches and appending data in KV Store. Can you please suggest a workaround for KV Stores as well for pushing  2years data in batches without manual intervention.

[richgalloway]

The fill_summary_index.py script referenced in the above link merely runs your saved searches that populate a summary index.  You can use the same script to run other saved searches that populate/update a KVStore.

[richgalloway · Accepted solution]

The backfill script may help here.  See https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Managesummaryindexgapsandoverlaps

You can put a wrapper around this script that runs it multiple times with the appropriate earliest/latest settings.