How will Splunk respond if a cold database path is not present when data is going to be rolled from warm to cold?

[koshyk]

hi folks,

We have an issue with our cold database filesystem and the estimate to bring it back is around 10 days.

So my question is:
What happens if a cold database path is not present and there is data to be rolled over from warm to cold?

Will warm buckets be kept till it get's hold of the cold database path? or will it be deleted? or will Splunk stop abruptly?

[JMichaelis · Accepted solution]

From the wiki:

Bad settings for the max number of hot and warm buckets or bad bucket size, (too many hot+warm buckets for your partition)
may cause your buckets to never go to the cold location, and to fill up your hot+warm location, and stop Splunk.

https://wiki.splunk.com/Deploy:BucketRotationAndRetention

Your case should be similar to a wrong config: No possibility to switch buckets to cold -> filled hot and warm buckets -> splunk stops.

[muebel]

Hi koshyk, if Splunk even starts without having access to the defined Cold Volume, my expectation is that it will fail when it initiates the warm to cold bucket rollover process, or when someone searches within a timeframe that includes a cold bucket (Splunk goes to open read on cold bucket, can't find it, fails).

As a workaround, you might be able to temporarily define a folder on the Warm volume to use for cold, and then once you have your cold volume move those buckets there and reconfigure everything back.

Please let me know if this answers your question! 😄

[koshyk]

@muebel thanks for the answer. So if the rollover fails, we are Ok as long as it won't bring down Splunk installation itself.(We are not really concerned about the data loss to cold bucket as of now).