How can I rotate all Hot and Warm Buckets older than 30 days into Cold storage?

[Steven_McGrath]

How can I rotate all Hot and Warm Buckets older than 30 days into Cold storage? I would like to do this as I have limited local disk and would like to roll data older than 30 days onto the San. My current index.conf file is below:

[default]

[_audit]
homePath = /var/local/splunk/warm/_audit
coldPath = /var/lun/splunk/cold/_audit
thawedPath = /var/local/splunk/thawed/_audit
frozenTimePeriodInSecs = 31536000

[_blocksignature]
homePath = /var/local/splunk/warm/_blocksignature
coldPath = /var/lun/splunk/cold/_blocksignature
thawedPath = /var/local/splunk/thawed/_blocksignature
frozenTimePeriodInSecs = 31536000

[_internal]
homePath = /var/local/splunk/warm/_internal
coldPath = /var/lun/splunk/cold/_internal
thawedPath = /var/local/splunk/thawed/_internal
frozenTimePeriodInSecs = 31536000

[_thefishbucket]
homePath = /var/local/splunk/warm/_thefishbucket
coldPath = /var/lun/splunk/cold/_thefishbucket
thawedPath = /var/local/splunk/thawed/_thefishbucket
frozenTimePeriodInSecs = 31536000

[history]
homePath = /var/local/splunk/warm/history
coldPath = /var/lun/splunk/cold/history
thawedPath = /var/local/splunk/thawed/history
frozenTimePeriodInSecs = 31536000

[main]
homePath = /var/local/splunk/warm/main
coldPath = /var/lun/splunk/cold/main
thawedPath = /var/local/splunk/thawed/main
frozenTimePeriodInSecs = 31536000

[os]
homePath = /var/local/splunk/warm/os
coldPath = /var/lun/splunk/cold/os
thawedPath = /var/local/splunk/thawed/os
frozenTimePeriodInSecs = 31536000

[sample]
homePath = /var/local/splunk/warm/sample
coldPath = /var/lun/splunk/cold/sample
thawedPath = /var/local/splunk/thawed/sample
frozenTimePeriodInSecs = 31536000

[splunklogger]
homePath = /var/local/splunk/warm/splunklogger
coldPath = /var/lun/splunk/cold/splunklogger
thawedPath = /var/local/splunk/thawed/splunklogger
frozenTimePeriodInSecs = 31536000

[summary]
homePath = /var/local/splunk/warm/summary
coldPath = /var/lun/splunk/cold/summary
thawedPath = /var/local/splunk/thawed/summary
frozenTimePeriodInSecs = 31536000

[gkanapathy]

There is not a way to roll from warm to cold by time. However, that really isn't your requirement. Your actual requirement is to limit the space used by warm. For that purpose, you can use the setting maxWarmDBCount which limits the number of warm buckets in an index. Note that the size of a warm bucket is limited in turn by maxDataSize.

[sowings]

Note also that maxDataSize is just the upper limit for a bucket's size. Other factors (such as restarting Splunk, time-based rotation, etc) may mean that a bucket is actually smaller than this size. The product of maxWarmDBCount and maxDataSize then represents an upper bound on the warm space in use.

[dave_duvall · Accepted solution]

The way we have always managed the warm to cold transition it to monitor the index to determine how long it takes to fill a bucket. If for example on the main index I have the following configured:

maxDataSize = 5000 
maxWarmDBCount=40
frozenTimePeriodinSecs = 34186669

This way if I know I fill a 5000 MB bucket every day, then I know I should roll to cold every 40 days or so. Another way to think of this is that I know that my warm buckets for the main index will take up no more that 200 GB of space with this configuration which is much more predictable than basing things on keeping 30 days in warm storage which could vary greatly in size.