bucket retention and frozenTimePeriodInSecs

mataharry — Thu, 16 Jan 2014 01:05:41 GMT

My index has a retention of 6 months with frozenTimePeriodInSecs=15552000.
But I still see some events that are older than the retention.

By example events that are from 6 month and 2 weeks.

Any thoughts ?

Re: bucket retention and frozenTimePeriodInSecs

yannK — Thu, 16 Jan 2014 01:10:55 GMT

A bucket that contains events overlapping the time retention will not be frozen until all the events are older than the retention.

By default indexes.conf has buckets with up to 3 months of span. So It's possible that you have buckets still overlapping
[main] maxHotSpanSecs = 7776000 frozenTimePeriodInSecs = 188697600
A workaround may be to reduce the maxHotSpanSecs to a week, to force the buckets to be smaller, and rotate more often.

To verify the status of your buckets, and estimate if they meet the condition to be frozen you can use those dbinspect searches on the indexer.

for Splunk 6, for all indexes with autodetection from frozenTimePeriodInSecs

for splunk 5 and 4, you have to manually add the values and do one index at a time

|dbinspect index=main | eval frozenTimePeriodInSecs= 2592000 | convert timeformat="%m/%d/%Y:%H:%M:%S" mktime(earliestTime) AS endEpoch | eval toNow=now()-endEpoch | convert num(toNow) | convert num(frozenTimePeriodInSecs) | eval shouldBeFrozen=if( ( state!="hot" AND state!="thawed" ) AND toNow>frozenTimePeriodInSecs,"yes","no") | table path id state earliestTime latestTime endEvent shouldBeFrozen toNow frozenTimePeriodInSecs

Re: bucket retention and frozenTimePeriodInSecs

hajducko — Thu, 10 Apr 2014 23:45:38 GMT

I think the Splunk 5/4 search is wrong. Wouldn't you want to make endEpoch as the latestTime as every event in the bucket needs to be older than the frozenTimePeriod. If I run that search, I get 'yes' for buckets that have an earliestTime that is older than the frozenTimePeriod, but with a latestTime that is newer. Those buckets wouldn't be deleted.

topic Re: bucket retention and frozenTimePeriodInSecs in Getting Data In

bucket retention and frozenTimePeriodInSecs

Re: bucket retention and frozenTimePeriodInSecs

Re: bucket retention and frozenTimePeriodInSecs