Splunk Enterprise

Incremental Saved Search

arushiseth94
Engager

Hello,

I have a saved search that pushes data to summary index. The summary index has data for last 2 years and data volume is really huge. Suppose I want to add a new field to this data in summary index, I need to re run search for last two years. Since the volume is huge, if I try to run the search for all 2 years data in one time, the search fails or data gets missed. To avoid this, I'll be pushing data in 10 days batch or 30 days batch.

For example - if I have to repopulate my summary index after adding a new field. So, for first batch I'll run for data from 1st Aug 2023 to 10th Aug 2023. Next batch I'll run from 11th Aug to 20th Aug.

Similar thing has to be done for past two years of data to be pushed in summary index.

This task is very cumbersome . Is there a way to automate this task in splunk. Can I schedule my search in such a way that while repushing data , without manual intervention data gets pushed in 10 days batch in summary index?

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The backfill script may help here.  See https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Managesummaryindexgapsandoverlaps

You can put a wrapper around this script that runs it multiple times with the appropriate earliest/latest settings.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

arushiseth94
Engager

Hi @richgalloway , Thank you for your response. 

Similar to summary index , I have KV Stores as well , where I am pushing data in similar manner in 10 days batches and appending data in KV Store. Can you please suggest a workaround for KV Stores as well for pushing  2years data in batches without manual intervention.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

The fill_summary_index.py script referenced in the above link merely runs your saved searches that populate a summary index.  You can use the same script to run other saved searches that populate/update a KVStore.

---
If this reply helps you, Karma would be appreciated.

richgalloway
SplunkTrust
SplunkTrust

The backfill script may help here.  See https://docs.splunk.com/Documentation/SplunkCloud/latest/Knowledge/Managesummaryindexgapsandoverlaps

You can put a wrapper around this script that runs it multiple times with the appropriate earliest/latest settings.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Splunk APM: New Product Features + Community Office Hours Recap!

Howdy Splunk Community! Over the past few months, we’ve had a lot going on in the world of Splunk Application ...

Index This | Forward, I’m heavy; backward, I’m not. What am I?

April 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

A Guide To Cloud Migration Success

As enterprises’ rapid expansion to the cloud continues, IT leaders are continuously looking for ways to focus ...