Splunk Enterprise

Incremental Saved Search

arushiseth94
Engager

Hello,

I have a saved search that pushes data to summary index. The summary index has data for last 2 years and data volume is really huge. Suppose I want to add a new field to this data in summary index, I need to re run search for last two years. Since the volume is huge, if I try to run the search for all 2 years data in one time, the search fails or data gets missed. To avoid this, I'll be pushing data in 10 days batch or 30 days batch.

For example - if I have to repopulate my summary index after adding a new field. So, for first batch I'll run for data from 1st Aug 2023 to 10th Aug 2023. Next batch I'll run from 11th Aug to 20th Aug.

Similar thing has to be done for past two years of data to be pushed in summary index.

This task is very cumbersome . Is there a way to automate this task in splunk. Can I schedule my search in such a way that while repushing data , without manual intervention data gets pushed in 10 days batch in summary index?

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The backfill script may help here.  See https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Managesummaryindexgapsandoverlaps

You can put a wrapper around this script that runs it multiple times with the appropriate earliest/latest settings.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

arushiseth94
Engager

Hi @richgalloway , Thank you for your response. 

Similar to summary index , I have KV Stores as well , where I am pushing data in similar manner in 10 days batches and appending data in KV Store. Can you please suggest a workaround for KV Stores as well for pushing  2years data in batches without manual intervention.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The fill_summary_index.py script referenced in the above link merely runs your saved searches that populate a summary index.  You can use the same script to run other saved searches that populate/update a KVStore.

---
If this reply helps you, Karma would be appreciated.

richgalloway
SplunkTrust
SplunkTrust

The backfill script may help here.  See https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Managesummaryindexgapsandoverlaps

You can put a wrapper around this script that runs it multiple times with the appropriate earliest/latest settings.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Exporting Splunk Apps

Join us on Monday, October 21 at 11 am PT | 2 pm ET!With the app export functionality, app developers and ...

Cisco Use Cases, ITSI Best Practices, and More New Articles from Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Build Your First SPL2 App!

Watch the recording now!.Do you want to SPL™, too? SPL2, Splunk's next-generation data search and preparation ...