In our clustered indexers, the cold storage is "network attached". There is a software upgrade for the storage platform which requires all of the cold storage locations to be intermittently unavailable for some duration. So this impacts all of the indexer peers
have you guys got experience on how
- Splunk indexer peer can handle detachment of network attached cold storage location? (Will it just show errors or shutdown?)
- Is there any command to stop hot to cold data movement for a definite period?
I have worked on environments with this same situation and we simply coordinated with the storage teams, and relied on the redundancy of multisite to allow us to completely shutdown the indexers during the work. Our UF were configured to send data to both sites to ensure that we could take down an indexing site when need be.
There is no command to stop rolling of buckets to cold.
In my experience - and this may depend on your environment/configuration - Splunk simply began writing locally. I don't think this is a Splunk designed behavior, was probably provided by the underlying os ! Your results may vary! PLEASE TEST! I would think it is completely within the realm of possibility that you could lose the indexer completely.
Do you have a multi-site cluster?
Do you have a lab environment where you can run this scenario to understand your configuration and what you can expect in your environment?
Have you ever lost your NAS in your environment?
My personal opinion is that your environment should be built to withstand failure of your storage, but also to accommodate the inevitable outages that maintenance work brings. If this is a production environment, these failure scenarios should be tested!
Hope that helps and that you have a smooth storage upgrade!
Thanks mate. we do have multi-site, but wanted to check if we really need to whole site down or Splunk can handle this gracefully. My main query is What will happen when the cold storage is removed all-of-a-sudden from a working indexer? Will it continue indexing or abruptly shutdown?
np! yeah, I was surprised to see the loss of the NAS mount point cause splunk to simply write to the same location locally. I am not confident that will be true in all cases, thus the encouragement to test, but I can definitely advise that is one experience I have had with losing NAS while splunk is still running.
I guess the main idea is that it is hard to know what to expect in that scenario. I have totally seen indexers die when losing attached storage, so it wouldn't surprise me to see it fall over....anyway you can test in the lab? would love to know what you end up experiencing.