Adding data retention to the main index

[watsm10]

I've tried to add a 6 month retention policy to the main index. As the main index is already defined in the default indexes.conf, I only need to specify the following in the local indexes.conf:

[main]
frozenTimePeriodInSecs=15552000

After I've restarted my indexers for the configuration to take affect, the data stops being indexed into main.

Anyone got any ideas as to where I'm going wrong?

Cheers.

[watsm10 · Accepted solution]

Hi Dimitri,
Thanks for your reply. I have since found that the issue is with the high CPU usage. There are a lot of buckets over 6 months old, so Splunk takes time and CPU to process these and the indexing queue backs up and fills in no time, so the indexer blocks all incoming data on port 9997 until the buckets have been frozen.

[Dimitri_McKay]

So, I'm not sure if you copied and pasted directly from your indexes.conf, but you're missing a space on either side of the equal sign, it looks like.

For everyone else:

You can use the age of data to determine when a bucket gets rolled to frozen (aka deleted). When the most recent data in a particular bucket reaches the configured age, the entire bucket is rolled.

To specify the age at which data should freeze, edit the frozenTimePeriodInSecs attribute in indexes.conf. This attribute specifies the number of seconds to elapse before data gets frozen. The default value is 188697600 seconds, or approximately 6 years. This example configures Splunk to cull old events from its index when they become more than 180 days (15552000 seconds) old:

[main]
frozenTimePeriodInSecs = 15552000

Restart Splunk for the new setting to take effect. Depending on how much data there is to process, it can take some time for Splunk to begin to move buckets out of the index to conform to the new policy. You might see high CPU usage during this time.