How do we change indexes.conf's cold path in a clustered Splunk environment?

[jaracan]

Hi Team,

Here is our scenario:

Our current directory in our coldPath parameter in master-apps/org_all_indexes/local/indexes.conf is almost full in disk space. We are planning to change the coldPath and point it to a new directory with more disk space.

Since we have a clustered environment, it is safe to just update the coldPath parameter in master-apps/org_all_indexes/local/indexes.conf? Else, what are the factors needed to consider first to avoid unnecessary repercussions and what are the best practices to migrate cold buckets into a new directory?

[bcyates]

Cold path or cold volume? Is your hot/warm storage and cold storage on the same partition or do they each have their own?

Judging by that app naming convention, it sounds like you had Professional Services help at some point. They should have set a parameter called maxVolumeDataSizeMB for that partition in indexes.conf. The max size should be set close to the total amount of storage available on that partition with a little bit of buffer.

Essentially, when your cold storage reaches that point, it will begin rolling the oldest data to frozen. By default, rolling to Frozen just deletes the data. If you did not specify a coldToFrozenScript or coldToFrozenDir for Frozen data, then that is what would happen.

I would not change your current cold directory. I would add a partition for Frozen data if you do not want data to be deleted, then just set the coldToFrozenDir. Or if you have no reason to retain the data, then just let Splunk roll the oldest stuff to frozen.

[ansif]

https://answers.splunk.com/answers/478697/migrating-hotwarm-and-cold-buckets-to-separate-dri.html

The above link will be helpful for you to start.

Please let me know if you find any difficulties.