Knowledge Management

How do you determine how many days of data an index has?

R_B
Path Finder

Hi everyone,

What is the best way to determine how many days of data each index is retaining if you only set the retention policy by max size?

In my environment I have a limited amount of storage I can use across all indexes. I have the retention poilcy set for each index to roll to frozen once it hits a max size, so that way we don't go over our allocated storage. However, we want to know how many days each index is actually retaining data.

I know on the indexer, in the "Indexes" page, it shows the earliest event for each index. However, if the earliest event in an index is 2 years ago, that doesn't mean the index is actually retaining 2 years of data. The data could either be parsed/indexed incorrectly or it could have been a newely added archived log from a machine that sent the data to that index.

Has anyone else ran into this scenario? If so, what do you do to determine the data retention for each index?

Tags (1)
0 Karma

valiquet
Contributor

I heard metadata is faster.

| metadata index=risk type=hosts
| rename totalCount as Count firstTime as "First Event" lastTime as "Last Event"
| foreach *Event
[ eval <> = strftime('<>',"%Y-%m-%dT%H:%M:%S.%Q")]

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This may help or it may be the same as what you see on the Indexes page.

| tstats earliest(_time) as earliest latest(_time) as latest where index=* by index 
| eval range=latest-earliest 
| fieldformat earliest=strftime(earliest,"%x %X") 
| fieldformat latest=strftime(latest,"%x %X") 
| fieldformat range=tostring(range, "duration")
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...