topic Re: Incremental Saved Search in Splunk Enterprise

Incremental Saved Search

arushiseth94 — Tue, 17 Oct 2023 11:21:04 GMT

Hello,

I have a saved search that pushes data to summary index. The summary index has data for last 2 years and data volume is really huge. Suppose I want to add a new field to this data in summary index, I need to re run search for last two years. Since the volume is huge, if I try to run the search for all 2 years data in one time, the search fails or data gets missed. To avoid this, I'll be pushing data in 10 days batch or 30 days batch.

For example - if I have to repopulate my summary index after adding a new field. So, for first batch I'll run for data from 1st Aug 2023 to 10th Aug 2023. Next batch I'll run from 11th Aug to 20th Aug.

Similar thing has to be done for past two years of data to be pushed in summary index.

This task is very cumbersome . Is there a way to automate this task in splunk. Can I schedule my search in such a way that while repushing data , without manual intervention data gets pushed in 10 days batch in summary index?

Re: Incremental Saved Search

richgalloway — Tue, 17 Oct 2023 13:40:38 GMT

The backfill script may help here. See https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Managesummaryindexgapsandoverlaps

You can put a wrapper around this script that runs it multiple times with the appropriate earliest/latest settings.

Re: Incremental Saved Search

arushiseth94 — Wed, 18 Oct 2023 12:28:16 GMT

Hi @richgalloway , Thank you for your response.

Similar to summary index , I have KV Stores as well , where I am pushing data in similar manner in 10 days batches and appending data in KV Store. Can you please suggest a workaround for KV Stores as well for pushing 2years data in batches without manual intervention.

Re: Incremental Saved Search

richgalloway — Wed, 18 Oct 2023 13:34:03 GMT

The fill_summary_index.py script referenced in the above link merely runs your saved searches that populate a summary index. You can use the same script to run other saved searches that populate/update a KVStore.