Getting Data In

How to force retention time pruning

Motivator

I have my frozen time set like this frozenTimePeriodInSecs = 47304000 (1.5 years)
yet when I do this search

| metadata index=foo type=hosts | stats max(lastTime) as lastTime, min(firstTime) as firstTime | convert ctime(*Time)

my "firstTime" is more than two years from my "lastTime"

lastTime               firstTime
02/06/2017 07:10:40     01/16/2015 09:18:53

this is more than 2 years of data.
How can I force retention time pruning or find out why pruning it not running correctly?

0 Karma

Ultra Champion

So, it's interesting to find out what the value of maxHotSpanSecs is now and what it was 1.5 years ago, because its default value of 1 day in seconds, enforces rotation of the bucket on a daily basis. Keep in mind that this value for a slow growing index can produce lots of small buckets...

0 Karma

Influencer

I'm not sure about that default. On all my 6.5.1 systems, the default is 7776000, which is 90 days.

0 Karma

Ultra Champion

Really interesting, because I'm pretty sure on older versions it was a day...

0 Karma

SplunkTrust
SplunkTrust

Hi hartfoml,
events deletion is related to buckets time period: a bucket is deleted when the latest bucket's event is out of the retention period, it isn't possible to force the bucket's deletion.

Bye.
Giuseppe

0 Karma

Motivator

Giuseppe, @cusello Thanks for the response. I have the retention set to 1.5 years and there is more than 2 years of data. This is a high volume index "Firewall" logs. Since I have 2 yeas worth of data, bucket data would have to span more than 6 moths in order not to be deleted in the 1.5 year time frame. this seems unlikely to me.

I wish there was a way of looking into the buckets to see which ones should be removed and are not being removed.

Thanks for your response.

0 Karma

Ultra Champion

You see, each bucket has two epoch time stamps, in their file name, which define their time interval. So, you can check them on the file system.

0 Karma

Influencer

Have you checked the old events? Do their timestamps match their index time? In other words, are they reporting as 2 years old for LOG TIME, but they were actually indexed more recently?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!