topic Re: Is there a method for rolling data completely out of an idex based on its age? in Splunk Search

Is there a method for rolling data completely out of an idex based on its age?

cdavidy — Thu, 04 Nov 2010 22:55:57 GMT

I have a need to automatically roll data completely out of an index so that it's no longer searchable after a number of days. Is this possible? how do I go about it?

Re: Is there a method for rolling data completely out of an idex based on its age?

Simeon — Thu, 04 Nov 2010 23:11:21 GMT

You can set the retention period of an index via the indexes.conf file:

http://www.splunk.com/base/Documentation/latest/admin/Indexesconf

frozenTimePeriodInSecs = <integer>
* Number of seconds after which indexed data rolls to frozen.
* If you do not specify a coldToFrozenScript, this data is erased.
* IMPORTANT: Every event in the DB must be older than frozenTimePeriodInSecs before it will roll.
* frozenTimePeriodInSecs will be frozen the next time splunkd checks.
* Defaults to 188697600.

You should apply the above value within your specific index stanza. By default, frozen implies that data is deleted.

Re: Is there a method for rolling data completely out of an idex based on its age?

BunnyHop — Fri, 05 Nov 2010 20:19:39 GMT

This is the answer to do the rolling automatically.

http://answers.splunk.com/questions/664/how-can-i-trigger-migration-of-buckets-from-warm-to-cold

Keep in mind that both hot and warm are still searchable. If you want to search the cold bucket, you can specify that.

Re: Is there a method for rolling data completely out of an idex based on its age?

gkanapathy — Fri, 05 Nov 2010 20:24:05 GMT

In 4.x, there is no functional distinction in search between data in hot or cold buckets, i.e., searches will look in cold automatically as appropriate.

Re: Is there a method for rolling data completely out of an idex based on its age?

gkanapathy — Fri, 05 Nov 2010 20:50:48 GMT

You can set frozenTimePeriodInSecs, but it does not necessarily guarantee that data will be removed when it reaches that age. What it does specify is that data may be rolled out when it hits that age, provided that everything in its data bucket is also aged enough to be rolled out.

You can control to some degree by setting maxHotSpanSecs, but this setting can have significant impact on search performance and changing it probably requires changing other index configurations, and generally should not be done without official Splunk recommendations.