How do you determine how many days of data an index has?

[R_B]

Hi everyone,

What is the best way to determine how many days of data each index is retaining if you only set the retention policy by max size?

In my environment I have a limited amount of storage I can use across all indexes. I have the retention poilcy set for each index to roll to frozen once it hits a max size, so that way we don't go over our allocated storage. However, we want to know how many days each index is actually retaining data.

I know on the indexer, in the "Indexes" page, it shows the earliest event for each index. However, if the earliest event in an index is 2 years ago, that doesn't mean the index is actually retaining 2 years of data. The data could either be parsed/indexed incorrectly or it could have been a newely added archived log from a machine that sent the data to that index.

Has anyone else ran into this scenario? If so, what do you do to determine the data retention for each index?

[valiquet]

I heard metadata is faster.

| metadata index=risk type=hosts
| rename totalCount as Count firstTime as "First Event" lastTime as "Last Event"
| foreach *Event
[ eval <> = strftime('<>',"%Y-%m-%dT%H:%M:%S.%Q")]

[richgalloway]

This may help or it may be the same as what you see on the Indexes page.

| tstats earliest(_time) as earliest latest(_time) as latest where index=* by index 
| eval range=latest-earliest 
| fieldformat earliest=strftime(earliest,"%x %X") 
| fieldformat latest=strftime(latest,"%x %X") 
| fieldformat range=tostring(range, "duration")