Behavior of frozenTimePeriodSecs

[justinjohn83]

In my indexes.conf I've set "frozenTimePeriodSecs" to "3888000" => 45 days. I've specified no coldToFrozenScript so I am assuming that any data older than 45 days should be discarded. The trouble is I am still seeing data with timestamps older than 45 days in the search results? Am I misunderstanding how this parameter is supposed to work. I am running splunk 4.1.6.

Thanks,

Justin

[David · Accepted solution]

What is your data volume? With small volumes, what will often happen is that the data will never leave the hot buckets, and then the warm buckets will never turn to frozen (e.g., be deleted).

Alternatively, it could be that the particular bucket may have just not rolled over yet. If you have a large volume, are you seeing data way older than 45 days? Part of this is that buckets roll over; events don't. The buckets contain the events, so it is almost the same thing, but any given bucket is going to contain a range of events (dependent on the bucket size).

You might find value looking at these two Answers:

• http://answers.splunk.com/questions/11234/managing-bucket-sizes
• http://answers.splunk.com/questions/1696/index-settings-using-auto-what-is-my-bucket-size

Hopefully that's of some value, and not way too basic.

[gkanapathy · Accepted solution]

Data will eventually leave hot buckets, as long as it keeps coming in till one is full. A bucket can get up to 10 GB in size (by default) but could be smaller.

Data will only be deleted when all data in a bucket is older than frozenTimePeriodInSecs. So if you have older data that is sharing a bucket with more recent data (up to 10 GB [compressed] of more recent data) then the older data may not be deleted until that has all aged off.

[yannK]

hot and thawed buckets will not be frozen, and buckets will only be frozen because of frozenTimePeriodSecs if ALL events in it are older than the retention.

[David · Accepted solution]

What is your data volume? With small volumes, what will often happen is that the data will never leave the hot buckets, and then the warm buckets will never turn to frozen (e.g., be deleted).

Alternatively, it could be that the particular bucket may have just not rolled over yet. If you have a large volume, are you seeing data way older than 45 days? Part of this is that buckets roll over; events don't. The buckets contain the events, so it is almost the same thing, but any given bucket is going to contain a range of events (dependent on the bucket size).

You might find value looking at these two Answers:

• http://answers.splunk.com/questions/11234/managing-bucket-sizes
• http://answers.splunk.com/questions/1696/index-settings-using-auto-what-is-my-bucket-size

Hopefully that's of some value, and not way too basic.