Splunk Search

How to find the retention period of an index

kteng2024
Path Finder

Hi here,

Query to find the retention period of an particular index in days and all the configurations associated with that index .

0 Karma
1 Solution

mattymo
Splunk Employee
Splunk Employee

The Monitoring Console provides easy access to index details. You can find it under Settings > Monitoring Console > Indexing > Indexes and Volumes > Index Detail

alt text

I point you here because each of the panels provide you access to an underlying search that relies on a rest command that will bring you back all the details you could want, and gives you great examples of the type of splunk searches you could build for this info. Press the "open in search" button to reveal what is essentially this search (obviously replace with your servername):

| rest splunk_server=n00bserver /services/data/indexes
  | join type=outer title [
    | rest splunk_server=n00bserver /services/data/indexes-extended
  ]

With a ton of great eval tricks to present the data. Slap an eval that divides the frozenTimePeriodInSecs by 86400 and you have your retention in days...

A quick search I slapped together for an index called n00blab

| rest splunk_server=n00bserver /services/data/indexes
   | join type=outer title [
     | rest splunk_server=n00bserver /services/data/indexes-extended
   ]
| search title=n00blab
|  eval retentionInDays=frozenTimePeriodInSecs/86400
| table title retentionInDays

You can also retrieve this information from the cli using the btool command ./splunk btool indexes list <nameOfYourIndex> --debug

- MattyMo

View solution in original post

woodcock
Esteemed Legend

Index data retention is enforced the bucket level (not as individual events), based on the earliest timestamp value stored within bucket. So, to keep the data for a 6 months/180 days, you should ensure that bucket roll from hot to warm every day, regardless of how slow the flow is. Something like this should work in indexes.conf:

[YourIndexName]
homePath   = $SPLUNK_DB/YourIndexName/db
coldPath   = $SPLUNK_DB/YourIndexName/colddb
thawedPath = $SPLUNK_DB/YourIndexName/thaweddb
#Rollover data bucket everyday
maxHotIdleSecs = 86400
#NEVER keep more than 180 days of data
frozenTimePeriodInSecs = 15552000
#NEVER let the index get bigger than 5G
homePath.maxDataSizeMB=5000

Even so, I almost never use frozenTimePeriodInSecs because NOT doing so allows me to see the gradual impact of the increase in daily volume on my index. If I do NOT use that setting then my index is ONLY constrained by space (not by date) and I can actually watch as older events are dumped off the back end to make room for new data on the front end (you need to watch both ends). If the data for your index grows so large that 60 days no longer fits, splunk will roll buckets sooner than your frozenTimePeriodInSecs value specifies and you need to know this (preferably before it gets to that point). Whenever a bucket is rolled, there is a log that indicates this. The other answers make sure that you expire data when you should but you ALSO need to have a handle on the opposite to make sure that you are NOT expiring data when you should NOT be. To do this you need to track that the size of 6-months worth of data always ("still") fits within the filespace that you have allotted for this index. To do this, you need to setup an alert or periodic report based on this search:

index=_internal sourcetype=splunkd bucketmover 
| rex "[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db[\/\\\]db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+" 
| rex "db_(?<newestTime>\d+)_(?<oldestTime>\d+)_\d+.*?[\/\\\](?<indexname>[^\/\\\]*)[\/\\\][^\/\\\]*db" 
| sort 0 - oldestTime
| dedup indexname
| eval retentionDays=(_time-oldestTime)/(60*60*24)
| table _time indexname retentionDays

mattymo
Splunk Employee
Splunk Employee

The Monitoring Console provides easy access to index details. You can find it under Settings > Monitoring Console > Indexing > Indexes and Volumes > Index Detail

alt text

I point you here because each of the panels provide you access to an underlying search that relies on a rest command that will bring you back all the details you could want, and gives you great examples of the type of splunk searches you could build for this info. Press the "open in search" button to reveal what is essentially this search (obviously replace with your servername):

| rest splunk_server=n00bserver /services/data/indexes
  | join type=outer title [
    | rest splunk_server=n00bserver /services/data/indexes-extended
  ]

With a ton of great eval tricks to present the data. Slap an eval that divides the frozenTimePeriodInSecs by 86400 and you have your retention in days...

A quick search I slapped together for an index called n00blab

| rest splunk_server=n00bserver /services/data/indexes
   | join type=outer title [
     | rest splunk_server=n00bserver /services/data/indexes-extended
   ]
| search title=n00blab
|  eval retentionInDays=frozenTimePeriodInSecs/86400
| table title retentionInDays

You can also retrieve this information from the cli using the btool command ./splunk btool indexes list <nameOfYourIndex> --debug

- MattyMo
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...