topic Re: Index Retention Time in Getting Data In

How to set up Index Retention Time?

mc210274 — Thu, 27 Apr 2023 13:25:29 GMT

Hello,

I did some reading up on the hot, warm and cold buckets and data retention of indexes but I am not sure I 100% get it.

What I am simply trying to do is to set my indexes to keep data for 180 days and then whatever data is older should be deleted.
There seems to be this frozen data timer but I am not able to find any settings based on time. every setting I see seemed to be based on how much storage the index\bucket uses.

What am I missing here?

Thank you
Marcus

Re: Index Retention Time

pgoyal_splunk — Sat, 30 Nov 2019 08:44:37 GMT

You can set indexes to keep your data for 180 days,
just need to configure 'frozenTimePeriodInSecs' setting in indexes.conf.

frozenTimePeriodInSecs =
The number of seconds after which indexed data rolls to frozen. meaning: if "frozenTimePeriodInSecs" seconds have passed, data could prematurely roll to frozen

Default: 188697600 (6 years)

In your case: It is like-

[]
frozenTimePeriodInSecs = 15552000 (180 Days)

Re: Index Retention Time

gcusello — Sat, 30 Nov 2019 15:48:01 GMT

Hi @mc210274,
I think that you should read at https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Setaretirementandarchivingpolicy
In this pare is described that your data pass through three states:

Hot: data is just indexed and stored in a bucket that is modyfied time by time with new data;
Warm: data are indexed from not much time and they are frequently used, buckets aren't modified by new data;
Cold: data are indexed fron much time and they aren't frequently used, buckets aren't modified by new data.

Dimension of buckets in each state is configurable.

After it's possible to discard data or store offline using a script.

Anyway the data discard from cold is configurable in two ways:

by retention period using (in your case) frozenTimePeriodInSecs = 15552000;
by index dimension using maxTotalDataSizeMB = .

You can find Infos about this at https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf .

If you use a retention policy, remember that discard is related to buckets, this means that a bucket is discarded when the newest event in the bucket exceeds the retention time, in othe words, do not be surprised if you find in an index events that exceed the retention period: this happens because they are in a bucket where there are also events that have not yet passed the retention period.

Ciao.
Giuseppe

Re: Index Retention Time

saramamurthy_sp — Sat, 30 Nov 2019 18:39:50 GMT

This question is about buckets, and I would advise you to reffer the below document which will help you to understand what is the buckets and what is the time range and what is the rolling of buckets.

https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/HowSplunkstoresindexes

Coming to your question you need to either make it, on the time period or on the size of the bucket. Since you require 180 days of the data then you need to make changes in the indexes.conf

frozenTimePeriodInSecs = 15552000 (180 Days)

This is the time you are setting to make the data into frozen, you can read more details on this in the below document.

https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf

Re: Index Retention Time

mc210274 — Thu, 05 Dec 2019 03:26:17 GMT

Thanks everybody - these answers are very helpful.

Re: Index Retention Time

gcusello — Thu, 27 Apr 2023 06:52:47 GMT

Hi @mc210274 ,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the contributors 😉