topic Re: how to configure retirement policy? in Getting Data In

how to configure retirement policy?

giovere — Fri, 09 Sep 2011 15:45:48 GMT

I'm trying to make indexes retire after 60 seconds, here is how my indexes.conf looks like:

[default]

frozenTimePeriodInSecs = 60

rotatePeriodInSecs = 10

If I restart Splunk, it indeed removes all events older than 60 seconds, but I could not achieve the same without restarting Spunk. Am I missing any additional rotate parameter?

If the problem is related to maximum buket sizes (defualt 300) is there a way to make them age dependent?

Re: how to configure retirement policy?

yannK — Fri, 09 Sep 2011 16:23:28 GMT

Why would you do such a thing ?
what is the purpose of keeping 60 seconds of data in splunk ?
Also You shouldn't setup default rules, they will apply to ALL your indexes, including the internal splunk ones, and all your summary indexes.

the reasons why your rule doesn't remove all the old events instantly are :

the events are stored in buckets per time period, and a bucket is been rotated only if all the events of a buckets are older than the retention period.
also a hot bucket is not immediately frozen, it has to be at least rotated to warm.

The reason why it worked after a restart :

When you restart splunk (since 4.2), all the hot buckets are rotated, therefore your settings apply to them.

The only clean way to achieve your goal is to change the maximum size and span of a bucket to keep them small.
see maxDataSize and maxHotSpanSecs.

see explanations
http://www.splunk.com/wiki/Deploy:BucketRotationAndRetention

Re: how to configure retirement policy?

gkanapathy — Fri, 09 Sep 2011 16:51:58 GMT

I think that you're perhaps fundamentally misunderstanding the the retirement/freezing settings. I don't think that you should even attempt to achieve the goal of deleting data once it is more than 60 seconds old, and I doubt that you will be able to guarantee that no matter what you set maxDataSize or maxHotSpanSecs to.

Re: how to configure retirement policy?

giovere — Mon, 12 Sep 2011 08:20:04 GMT

thanks for comments, of course 60 secs is not real life problem, I just wanted test it on lower time limit to make sure it is working properly.

In other words my goal is: make indexes age independently of their size or position in a bucket, avoiding rotation between hot,warm,cold (just delete when it is old).

My current version looks like this:

[main]

frozenTimePeriodInSecs = 60

rotatePeriodInSecs = 10

maxHotSpanSecs = 10

Current index contains only date > date.log which is old enough to be removed, but is not. Any suggestions will help. Thanks ...

Re: how to configure retirement policy?

giovere — Mon, 12 Sep 2011 11:49:24 GMT

maxHotIdleSecs did the trick, so finally it looks like this:

[main]

frozenTimePeriodInSecs = 10

rotatePeriodInSecs = 10

maxHotIdleSecs = 120

indexes spend 120 secs in hot buckets (starting from the last write action to a bucket), then rolled to warm spend 10 seconds there and finally they are deleted. If you sense any pitfall here, please let me know I'm quite new to all Splunk things.