hi, when we set frozenTimePeriodInSecs=30 days for an index. i have read through the documentation and forums that the bucket moves to frozen state when all the data in the given bucket is more than 30 days old. but i need some more clarity. basically the condition i think is one of the below for the latest event in the bucket.
_time > frozenTimePeriodInSecs
Please let me know which time controls the retention.
Thanks Luke.. The explanation in wiki link was a bit complex and took time to understand but it certainly supports your answer.
I was running two parallel daily reports - one based on _time and the other based on _indextime to verify if my retention values were working as expected. But both confusing me.
Now i will concentrate on _time based one for my further analysis.
It is complex, but necessarily so because while you may want old data, you probably most certainly want current data.
What may not be evident in the wiki post is the indexing very old logs and logs 'in the future'.
Let's say for example you have logs that are older than the MAXDAYSAGO parameter from props.conf (default 2000 days) - all events older than the MAXDAYSAGO will have the time value of the last acceptable timestamp in that log file, and if all events are older than MAXDAYS_AGO, then all events will have the current (index time) timestamp for _time.
So, in addition to knowing that Freezing data is based on time, you must also understand that it is based on the most recent time in the buckets (restart Splunkd = new buckets), and that indexing very old logs may give you event times (time) that are not what you expected (MAXDAYSAGO defaults to 2000 days).