topic Re: How to force retention time pruning in Getting Data In

How to force retention time pruning

hartfoml — Mon, 06 Feb 2017 13:24:57 GMT

I have my frozen time set like this frozenTimePeriodInSecs = 47304000 (1.5 years)
yet when I do this search

| metadata index=foo type=hosts | stats max(lastTime) as lastTime, min(firstTime) as firstTime | convert ctime(*Time)

my "firstTime" is more than two years from my "lastTime"

lastTime               firstTime
02/06/2017 07:10:40     01/16/2015 09:18:53

this is more than 2 years of data.
How can I force retention time pruning or find out why pruning it not running correctly?

Re: How to force retention time pruning

gcusello — Mon, 06 Feb 2017 13:53:50 GMT

Hi hartfoml,
events deletion is related to buckets time period: a bucket is deleted when the latest bucket's event is out of the retention period, it isn't possible to force the bucket's deletion.

Bye.
Giuseppe

Re: How to force retention time pruning

ddrillic — Mon, 06 Feb 2017 14:38:28 GMT

So, it's interesting to find out what the value of maxHotSpanSecs is now and what it was 1.5 years ago, because its default value of 1 day in seconds, enforces rotation of the bucket on a daily basis. Keep in mind that this value for a slow growing index can produce lots of small buckets...

Re: How to force retention time pruning

twinspop — Mon, 06 Feb 2017 14:47:19 GMT

I'm not sure about that default. On all my 6.5.1 systems, the default is 7776000, which is 90 days.

Re: How to force retention time pruning

ddrillic — Mon, 06 Feb 2017 14:49:30 GMT

Really interesting, because I'm pretty sure on older versions it was a day...

Re: How to force retention time pruning

hartfoml — Mon, 06 Feb 2017 14:57:37 GMT

Giuseppe, @cusello Thanks for the response. I have the retention set to 1.5 years and there is more than 2 years of data. This is a high volume index "Firewall" logs. Since I have 2 yeas worth of data, bucket data would have to span more than 6 moths in order not to be deleted in the 1.5 year time frame. this seems unlikely to me.

I wish there was a way of looking into the buckets to see which ones should be removed and are not being removed.

Thanks for your response.

Re: How to force retention time pruning

twinspop — Mon, 06 Feb 2017 15:00:18 GMT

Have you checked the old events? Do their timestamps match their index time? In other words, are they reporting as 2 years old for LOG TIME, but they were actually indexed more recently?

Re: How to force retention time pruning

ddrillic — Mon, 06 Feb 2017 15:00:36 GMT

You see, each bucket has two epoch time stamps, in their file name, which define their time interval. So, you can check them on the file system.