Knowledge Management

How do you determine how many days of data an index has?

R_B
Path Finder

Hi everyone,

What is the best way to determine how many days of data each index is retaining if you only set the retention policy by max size?

In my environment I have a limited amount of storage I can use across all indexes. I have the retention poilcy set for each index to roll to frozen once it hits a max size, so that way we don't go over our allocated storage. However, we want to know how many days each index is actually retaining data.

I know on the indexer, in the "Indexes" page, it shows the earliest event for each index. However, if the earliest event in an index is 2 years ago, that doesn't mean the index is actually retaining 2 years of data. The data could either be parsed/indexed incorrectly or it could have been a newely added archived log from a machine that sent the data to that index.

Has anyone else ran into this scenario? If so, what do you do to determine the data retention for each index?

Tags (1)
0 Karma

valiquet
Contributor

I heard metadata is faster.

| metadata index=risk type=hosts
| rename totalCount as Count firstTime as "First Event" lastTime as "Last Event"
| foreach *Event
[ eval <> = strftime('<>',"%Y-%m-%dT%H:%M:%S.%Q")]

0 Karma

richgalloway
SplunkTrust
SplunkTrust

This may help or it may be the same as what you see on the Indexes page.

| tstats earliest(_time) as earliest latest(_time) as latest where index=* by index 
| eval range=latest-earliest 
| fieldformat earliest=strftime(earliest,"%x %X") 
| fieldformat latest=strftime(latest,"%x %X") 
| fieldformat range=tostring(range, "duration")
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...