Deployment Architecture

How can I rotate all Hot and Warm Buckets older than 30 days into Cold storage?

Steven_McGrath
Engager

How can I rotate all Hot and Warm Buckets older than 30 days into Cold storage? I would like to do this as I have limited local disk and would like to roll data older than 30 days onto the San. My current index.conf file is below:

[default]

[_audit]
homePath = /var/local/splunk/warm/_audit
coldPath = /var/lun/splunk/cold/_audit
thawedPath = /var/local/splunk/thawed/_audit
frozenTimePeriodInSecs = 31536000

[_blocksignature]
homePath = /var/local/splunk/warm/_blocksignature
coldPath = /var/lun/splunk/cold/_blocksignature
thawedPath = /var/local/splunk/thawed/_blocksignature
frozenTimePeriodInSecs = 31536000

[_internal]
homePath = /var/local/splunk/warm/_internal
coldPath = /var/lun/splunk/cold/_internal
thawedPath = /var/local/splunk/thawed/_internal
frozenTimePeriodInSecs = 31536000

[_thefishbucket]
homePath = /var/local/splunk/warm/_thefishbucket
coldPath = /var/lun/splunk/cold/_thefishbucket
thawedPath = /var/local/splunk/thawed/_thefishbucket
frozenTimePeriodInSecs = 31536000

[history]
homePath = /var/local/splunk/warm/history
coldPath = /var/lun/splunk/cold/history
thawedPath = /var/local/splunk/thawed/history
frozenTimePeriodInSecs = 31536000

[main]
homePath = /var/local/splunk/warm/main
coldPath = /var/lun/splunk/cold/main
thawedPath = /var/local/splunk/thawed/main
frozenTimePeriodInSecs = 31536000

[os]
homePath = /var/local/splunk/warm/os
coldPath = /var/lun/splunk/cold/os
thawedPath = /var/local/splunk/thawed/os
frozenTimePeriodInSecs = 31536000

[sample]
homePath = /var/local/splunk/warm/sample
coldPath = /var/lun/splunk/cold/sample
thawedPath = /var/local/splunk/thawed/sample
frozenTimePeriodInSecs = 31536000

[splunklogger]
homePath = /var/local/splunk/warm/splunklogger
coldPath = /var/lun/splunk/cold/splunklogger
thawedPath = /var/local/splunk/thawed/splunklogger
frozenTimePeriodInSecs = 31536000

[summary]
homePath = /var/local/splunk/warm/summary
coldPath = /var/lun/splunk/cold/summary
thawedPath = /var/local/splunk/thawed/summary
frozenTimePeriodInSecs = 31536000
Tags (1)
1 Solution

dave_duvall
Explorer

The way we have always managed the warm to cold transition it to monitor the index to determine how long it takes to fill a bucket. If for example on the main index I have the following configured:

maxDataSize = 5000 
maxWarmDBCount=40
frozenTimePeriodinSecs = 34186669

This way if I know I fill a 5000 MB bucket every day, then I know I should roll to cold every 40 days or so. Another way to think of this is that I know that my warm buckets for the main index will take up no more that 200 GB of space with this configuration which is much more predictable than basing things on keeping 30 days in warm storage which could vary greatly in size.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

There is not a way to roll from warm to cold by time. However, that really isn't your requirement. Your actual requirement is to limit the space used by warm. For that purpose, you can use the setting maxWarmDBCount which limits the number of warm buckets in an index. Note that the size of a warm bucket is limited in turn by maxDataSize.

sowings
Splunk Employee
Splunk Employee

Note also that maxDataSize is just the upper limit for a bucket's size. Other factors (such as restarting Splunk, time-based rotation, etc) may mean that a bucket is actually smaller than this size. The product of maxWarmDBCount and maxDataSize then represents an upper bound on the warm space in use.

0 Karma

dave_duvall
Explorer

The way we have always managed the warm to cold transition it to monitor the index to determine how long it takes to fill a bucket. If for example on the main index I have the following configured:

maxDataSize = 5000 
maxWarmDBCount=40
frozenTimePeriodinSecs = 34186669

This way if I know I fill a 5000 MB bucket every day, then I know I should roll to cold every 40 days or so. Another way to think of this is that I know that my warm buckets for the main index will take up no more that 200 GB of space with this configuration which is much more predictable than basing things on keeping 30 days in warm storage which could vary greatly in size.

Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...