Deployment Architecture
Highlighted

Indexes having more data than retention period defined

Engager

Hi Everyone,

We have one index defined in indexes.conf with frozenTimePeriodInSecs as 365 days (31536000 seconds), but there are 3 years of data stored in index.It seems not working if we just define retention time period in frozenTimePeriodInSecs. Can someone help ?

$ view ./apps/launcher/local/indexes.conf
[Indexname]
coldPath = $SPLUNKDB/Indexname/colddb
homePath = $SPLUNK
DB/Indexname/db
thawedPath = $SPLUNK_DB/Indexname/thaweddb

Maximum index total size in MB

maxTotalDataSizeMB = 35000

frozenTimePeriodInSecs = 31536000

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

SplunkTrust
SplunkTrust

If you define the changes via the config, you will need to restart the splunk, to make effect.

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

SplunkTrust
SplunkTrust

Also, the data (bucket) is frozen only when the most recent event in the bucket is older then the retention period. Sometimes, a bucket can have data for varying/larger dates-range (e.g. a bucket has data for whole 1 year) and doesn't roll until the event with newest time is older than retention period. See this for more information on the same.

https://docs.splunk.com/Documentation/Splunk/7.2.3/Indexer/Setaretirementandarchivingpolicy#Freeze_d...

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

Engager

Do i need to check configuration file precedence if the settings mentioned by me are fine.

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

SplunkTrust
SplunkTrust

Sure. Run btool command on your indexers to see what frozenTimePeriodInSecs is effective.

Go to $SPLUNK_HOME/bin and then run this:

./splunk btool indexes list IndexName --debug

This should show you what the effective configuration is and from what location.

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

Engager

Yes it is showing 365 days only

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

Engager

Yes Splunk has been restarted and it has most recent data in Indexes.

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

SplunkTrust
SplunkTrust

ok. As a good practice, pls put all your config in a custom 'app' or under 'search' app if its temporary. If the issue is resolved, pls accept the answer to close the thread.

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

SplunkTrust
SplunkTrust

To clarify, you pulled this from your indexer and not the search head right?

0 Karma
Highlighted

Re: Indexes having more data than retention period defined

Engager

I pulled data from Search head and found that it has 3 years of data but when i logged in to check configuration files it shows me 1 year retention settings

0 Karma