Deployment Architecture

Behavior of frozenTimePeriodSecs

justinjohn83
Explorer

In my indexes.conf I've set "frozenTimePeriodSecs" to "3888000" => 45 days. I've specified no coldToFrozenScript so I am assuming that any data older than 45 days should be discarded. The trouble is I am still seeing data with timestamps older than 45 days in the search results? Am I misunderstanding how this parameter is supposed to work. I am running splunk 4.1.6.

Thanks,

Justin

0 Karma
2 Solutions

David
Splunk Employee
Splunk Employee

What is your data volume? With small volumes, what will often happen is that the data will never leave the hot buckets, and then the warm buckets will never turn to frozen (e.g., be deleted).

Alternatively, it could be that the particular bucket may have just not rolled over yet. If you have a large volume, are you seeing data way older than 45 days? Part of this is that buckets roll over; events don't. The buckets contain the events, so it is almost the same thing, but any given bucket is going to contain a range of events (dependent on the bucket size).

You might find value looking at these two Answers:

Hopefully that's of some value, and not way too basic.

View solution in original post

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

Data will eventually leave hot buckets, as long as it keeps coming in till one is full. A bucket can get up to 10 GB in size (by default) but could be smaller.

Data will only be deleted when all data in a bucket is older than frozenTimePeriodInSecs. So if you have older data that is sharing a bucket with more recent data (up to 10 GB [compressed] of more recent data) then the older data may not be deleted until that has all aged off.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Data will eventually leave hot buckets, as long as it keeps coming in till one is full. A bucket can get up to 10 GB in size (by default) but could be smaller.

Data will only be deleted when all data in a bucket is older than frozenTimePeriodInSecs. So if you have older data that is sharing a bucket with more recent data (up to 10 GB [compressed] of more recent data) then the older data may not be deleted until that has all aged off.

yannK
Splunk Employee
Splunk Employee

hot and thawed buckets will not be frozen, and buckets will only be frozen because of frozenTimePeriodSecs if ALL events in it are older than the retention.

0 Karma

David
Splunk Employee
Splunk Employee

What is your data volume? With small volumes, what will often happen is that the data will never leave the hot buckets, and then the warm buckets will never turn to frozen (e.g., be deleted).

Alternatively, it could be that the particular bucket may have just not rolled over yet. If you have a large volume, are you seeing data way older than 45 days? Part of this is that buckets roll over; events don't. The buckets contain the events, so it is almost the same thing, but any given bucket is going to contain a range of events (dependent on the bucket size).

You might find value looking at these two Answers:

Hopefully that's of some value, and not way too basic.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...