Getting Data In

Index Retention Time

mc210274
New Member

Hello,

I did some reading up on the hot, warm and cold buckets and data retention of indexes but I am not sure I 100% get it.

What I am simply trying to do is to set my indexes to keep data for 180 days and then whatever data is older should be deleted.
There seems to be this frozen data timer but I am not able to find any settings based on time. every setting I see seemed to be based on how much storage the index\bucket uses.

What am I missing here?

Thank you
Marcus

0 Karma
1 Solution

gcusello
Legend

Hi @mc210274,
I think that you should read at https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Setaretirementandarchivingpolicy
In this pare is described that your data pass through three states:

  • Hot: data is just indexed and stored in a bucket that is modyfied time by time with new data;
  • Warm: data are indexed from not much time and they are frequently used, buckets aren't modified by new data;
  • Cold: data are indexed fron much time and they aren't frequently used, buckets aren't modified by new data.

Dimension of buckets in each state is configurable.

After it's possible to discard data or store offline using a script.

Anyway the data discard from cold is configurable in two ways:

  • by retention period using (in your case) frozenTimePeriodInSecs = 15552000;
  • by index dimension using maxTotalDataSizeMB = .

You can find Infos about this at https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf .

If you use a retention policy, remember that discard is related to buckets, this means that a bucket is discarded when the newest event in the bucket exceeds the retention time, in othe words, do not be surprised if you find in an index events that exceed the retention period: this happens because they are in a bucket where there are also events that have not yet passed the retention period.

Ciao.
Giuseppe

View solution in original post

0 Karma

mc210274
New Member

Thanks everybody - these answers are very helpful.

0 Karma

saramamurthy_sp
Splunk Employee
Splunk Employee

Hi

This question is about buckets, and I would advise you to reffer the below document which will help you to understand what is the buckets and what is the time range and what is the rolling of buckets.

https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/HowSplunkstoresindexes

Coming to your question you need to either make it, on the time period or on the size of the bucket. Since you require 180 days of the data then you need to make changes in the indexes.conf

frozenTimePeriodInSecs = 15552000 (180 Days)

This is the time you are setting to make the data into frozen, you can read more details on this in the below document.

https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf

0 Karma

gcusello
Legend

Hi @mc210274,
I think that you should read at https://docs.splunk.com/Documentation/Splunk/8.0.0/Indexer/Setaretirementandarchivingpolicy
In this pare is described that your data pass through three states:

  • Hot: data is just indexed and stored in a bucket that is modyfied time by time with new data;
  • Warm: data are indexed from not much time and they are frequently used, buckets aren't modified by new data;
  • Cold: data are indexed fron much time and they aren't frequently used, buckets aren't modified by new data.

Dimension of buckets in each state is configurable.

After it's possible to discard data or store offline using a script.

Anyway the data discard from cold is configurable in two ways:

  • by retention period using (in your case) frozenTimePeriodInSecs = 15552000;
  • by index dimension using maxTotalDataSizeMB = .

You can find Infos about this at https://docs.splunk.com/Documentation/Splunk/8.0.0/Admin/Indexesconf .

If you use a retention policy, remember that discard is related to buckets, this means that a bucket is discarded when the newest event in the bucket exceeds the retention time, in othe words, do not be surprised if you find in an index events that exceed the retention period: this happens because they are in a bucket where there are also events that have not yet passed the retention period.

Ciao.
Giuseppe

0 Karma

pgoyal_splunk
Splunk Employee
Splunk Employee

You can set indexes to keep your data for 180 days,
just need to configure 'frozenTimePeriodInSecs' setting in indexes.conf.

frozenTimePeriodInSecs =
The number of seconds after which indexed data rolls to frozen. meaning: if "frozenTimePeriodInSecs" seconds have passed, data could prematurely roll to frozen

Default: 188697600 (6 years)

In your case: It is like-

[]
frozenTimePeriodInSecs = 15552000 (180 Days)

0 Karma
Get Updates on the Splunk Community!

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...

Ready, Set, SOAR: How Utility Apps Can Up Level Your Playbooks!

 WATCH NOW Powering your capabilities has never been so easy with ready-made Splunk® SOAR Utility Apps. Parse ...