What happens if cold bucket storage is unavaliable when rolling buckets from warm to cold?

[mihelic]

Our Splunk setup holds its hot and warm buckets on local disks, cold buckets are on FC storage.
What would happen if the FC storage becomes unavailable, e.g. crashed, unmounted, unresponsive?

Would Splunk try to roll the buckets to cold and become unresponsive itself?

Would Splunk hold off the rolling to cold and fill local warm buckets until FC storage becomes available?

Regards,
Mitja

[MuS]

Hi mihelic,

there was an excellent answer to this, but to make it short: if your colddb is offline and bucket are rolled from warm to cold they are gone. Splunk does not check if the colddb is available or not and it will not stop rolling buckets to cold if the colddb is not available.

I'll keep looking for the other answer and link it here if I find it.

Hope this helps ...

cheers, MuS

[kvm]

Can't we put the indexer/ cluster master in maintenance mode, so that the bucket rolling will stop temporarily?

[PickleRick]

Firstly, let me point out that you're digging up a very old thread. You'd get better chance of reasonable (i.e. not mine ;)) answer if you created a new question and optionally provided a link to the old one for reference.

But to the point.

I suppose there are several scenarios that can happen during an attempt to roll and accounting for all of them and handling them could be:

1) hard to do

2) not necessarily desired in some situations.

Imagine a cluster with RF=1. If you stop rolling buckets you're on the best path to filling up the storage and rendering the indexer inoperable.

But if you can find a good use case for some particular behaviour that could be implemented in the maintenance thread, feel free to open an idea with https://ideas.splunk.com/

[mihelic]

Yes, I meant storage connected via Fibre Channel.

[mendesjo]

Curious if you received an answer? I have cold storage on NAS and they need to do maintenance.. trying to figure out what will happen or if need to take splunk down.

[yannK]

candid question by FC storage, do you mean storage connected with a Fibre Channel ?