topic Re: Remove data when it ages in Getting Data In

Remove data when it ages

shangshin — Mon, 25 Jun 2012 20:33:26 GMT

Hi, I saw the doc on how to remove data when it ages.
Most of my log data goes to the default index db which is "main"

So assuming I want to index data for ONLY the latest one hour, I added these 2 lines in /etc/system/default/indexes.conf

[main]
frozenTimePeriodInSecs = 3600

On top of that I also created a new file /etc/system/local/indexes.conf with the same 2 lines.

[main]
frozenTimePeriodInSecs = 3600

Then I restarted splunk server but I still saw data older than 1 hour ago.

Can anyone shed some light on it? Thank you!!!

Re: Remove data when it ages

yannK — Mon, 25 Jun 2012 21:24:09 GMT

Hi Shangshin

Keeping only one hour of data is pretty short, what is your use case ?

A short frozenTimePeriodInSecs will not be enough to ensure that the data is frozen quickly.
This is because the events are written in buckets, a bucket have a certain span of time, or a maximum size. The bucket being written are in hot state, and roll to warm (or cold) state when they became read only. But only warm or cold state buckets can actually be frozen, and only if all the events their contains are older than the frozenTimePeriodInSecs or the maximum size of the index is reached. Then the ones containing the oldest events will be frozen.

The maxHotSpanSecs parameter can be used to limit the span times for your buckets, and force them to roll from hot to warm more often. Then the frozenTimePeriodInSecs will kick in.

see http://wiki.splunk.com/Deploy:BucketRotationAndRetention
http://docs.splunk.com/Documentation/Splunk/5.0/Admin/Indexesconf

Re: Remove data when it ages

shangshin — Tue, 26 Jun 2012 18:53:52 GMT

Thanks for detail explanation. It's very helpful.