I have my frozen time set like this
frozenTimePeriodInSecs = 47304000 (1.5 years)
yet when I do this search
| metadata index=foo type=hosts | stats max(lastTime) as lastTime, min(firstTime) as firstTime | convert ctime(*Time)
my "firstTime" is more than two years from my "lastTime"
lastTime firstTime 02/06/2017 07:10:40 01/16/2015 09:18:53
this is more than 2 years of data.
How can I force retention time pruning or find out why pruning it not running correctly?
So, it's interesting to find out what the value of
maxHotSpanSecs is now and what it was 1.5 years ago, because its default value of 1 day in seconds, enforces rotation of the bucket on a daily basis. Keep in mind that this value for a slow growing index can produce lots of small buckets...
events deletion is related to buckets time period: a bucket is deleted when the latest bucket's event is out of the retention period, it isn't possible to force the bucket's deletion.
Giuseppe, @cusello Thanks for the response. I have the retention set to 1.5 years and there is more than 2 years of data. This is a high volume index "Firewall" logs. Since I have 2 yeas worth of data, bucket data would have to span more than 6 moths in order not to be deleted in the 1.5 year time frame. this seems unlikely to me.
I wish there was a way of looking into the buckets to see which ones should be removed and are not being removed.
Thanks for your response.
Have you checked the old events? Do their timestamps match their index time? In other words, are they reporting as 2 years old for LOG TIME, but they were actually indexed more recently?