Knowledge Management

Use of Summary Indexing for Long Term Data from Rolling Index

philh
Explorer

Hi all,

I have the following problem set:

I have an index that rolls out data every 30 days (ie data older than 30 days is removed). There is a subset of data from this index that I would like to query for a longer period of time, say 12 or 24 months. 

I'm fairly new to the idea of summary indexes, but it sounds like the logical solution. However, I'm concerned about losing previous data (that's been removed from the original index) each time the summary index is scheduled to run. Is there a way for a summary index to store the data from old runs so I can build a dataset that encompasses multiple months from the original index? 

 

Thanks in advance!

Labels (1)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

The idea of a summary index is to retain a subset of data from another index.  The summary index should have different retention settings so it holds data longer than the original index.  Summarized data is independent of the data in the original index so the original data can be removed without affecting the summary.

---
If this reply helps you, Karma would be appreciated.

ITWhisperer
SplunkTrust
SplunkTrust

If you still need all the "old" data, extend the retention period for the index. If you only need partial data, extract that to the summary index and keep it for longer.

0 Karma

philh
Explorer

Thanks for the reply. I can't extend the retention period for the index since it is the established company retention length. But you're saying I can extract that partial data and hold it in the summary index without it being overwritten, correct? 

For example, I'd have the summary index run on the original index once every 30 days so it can grab all the partial data before it is removed. So after 2 runs, my summary index would have 60 days of partial data. 

Is this logic correct?

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

So long as the retention period for your summary index is long enough then yes. Summary indexes will still have their own retention periods but are usually longer than the initial raw data indexes. You would have to find out from your administrators how long your summary indexes can be retained for.

Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...