Deployment Architecture

What happens if cold bucket storage is unavaliable when rolling buckets from warm to cold?

mihelic
Path Finder

Our Splunk setup holds its hot and warm buckets on local disks, cold buckets are on FC storage.
What would happen if the FC storage becomes unavailable, e.g. crashed, unmounted, unresponsive?

Would Splunk try to roll the buckets to cold and become unresponsive itself?

Would Splunk hold off the rolling to cold and fill local warm buckets until FC storage becomes available?

Regards,
Mitja

Tags (3)

MuS
Legend

Hi mihelic,

there was an excellent answer to this, but to make it short: if your colddb is offline and bucket are rolled from warm to cold they are gone. Splunk does not check if the colddb is available or not and it will not stop rolling buckets to cold if the colddb is not available.

I'll keep looking for the other answer and link it here if I find it.

Hope this helps ...

cheers, MuS

0 Karma

kvm
Explorer

Can't we put the indexer/ cluster master in maintenance mode, so that the bucket rolling will stop temporarily?

Tags (2)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

Firstly, let me point out that you're digging up a very old thread. You'd get better chance of reasonable (i.e. not mine ;)) answer if you created a new question and optionally provided a link to the old one for reference.

But to the point.

I suppose there are several scenarios that can happen during an attempt to roll and accounting for all of them and handling them could be:

1) hard to do

2) not necessarily desired in some situations.

Imagine a cluster with RF=1. If you stop rolling buckets you're on the best path to filling up the storage and rendering the indexer inoperable.

But if you can find a good use case for some particular behaviour that could be implemented in the maintenance thread, feel free to open an idea with https://ideas.splunk.com/

0 Karma

mihelic
Path Finder

Yes, I meant storage connected via Fibre Channel.

0 Karma

mendesjo
Path Finder

Curious if you received an answer? I have cold storage on NAS and they need to do maintenance.. trying to figure out what will happen or if need to take splunk down.

0 Karma

yannK
Splunk Employee
Splunk Employee

candid question by FC storage, do you mean storage connected with a Fibre Channel ?

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...