What happens if the cold mount becomes unavailable?

[maciep]

We've been informed that the cold mount on half of our indexers needs to be moved to a different file system. And for that to happen there will be a downtime of about an hour when it needs to be unmounted from our indexers.

I'm assuming that Splunk won't like that. But does anyone know what exactly would happen if that cold path becomes unavailable? Does Splunk just queue up those buckets that need to be rolled from warm to cold? Or does it handle less gracefully? Does replication start breaking too maybe?

Our initial thought was just to put the cluster in maintenance mode and take those indexers offline during the downtime. But we'll ingest a decent amount of data in an hour, so if we could keep the indexers up to receive new data, that would be ideal.

Anyone insight would be helpful. Thanks!

[MuS · Accepted solution]

Hi maciep,

find the answer regarding the move to cold here http://answers.splunk.com/answers/287056/if-my-coldtofrozendir-is-full-or-unavailable-do-i.html
Regarding the replication; this should not be effected at all (anyone correct me if I'm wrong!)

cheers, MuS

[maciep]

Thanks! I think we have plenty of disk space on our hot/warm mount to allow Splunk to keep running. If we decide to go that way, then I'll definitely follow up here with how it went.

[maciep]

The change was made last night. All that I did was put the cluster in maintenance mode. I think if the master would have seen all cold data disappear from half or our indexers, it would have tried to clean things up. I'm not sure if that was necessary, but wasn't going to take a chance otherwise.

Looking in the logs on the servers that lost their cold path, I see a lot of these messages. They start at the time of the change and they stop after the share was remounted. That was it.

10-14-2015 19:34:05.437 -0400 WARN  DatabaseDirectoryManager - Directory='/[path to cold]/[some bucket]' does not exist.  The directory representing this bucket might have just rolled.