Repairing buckets on index cluster

[snyderm_dos]

Is bucket repair on an index cluster any different from non-clustered indexers? 

• Should splunkd be running on the cluster master? Should it be in maintenance mode?
• When using network storage, should it be mounted to all of the indexers or only one?
• Is the fsck command run from the cluster master or from one of the indexers?

[richgalloway]

Yes, the CM can remain running.  Maintenance mode is a good idea.

Each indexer should have its own storage.  They can all use the same device as long as it looks private to the indexer.  That is, no two indexers should write to the same space.

Run fsck on the indexer.

[snyderm_dos]

---

@richgalloway wrote:

Each indexer should have its own storage.  They can all use the same device as long as it looks private to the indexer.  That is, no two indexers should write to the same space.

---

Does this also apply to cold storage? Our hot/warm are on peered NFS shares, but the cold looks like it was created as one single share.

[richgalloway]

NFS often is referred to as "Not For Splunk".  It can work if you're careful, but can be painful otherwise.  See https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Considerations_r...

One issue with shared storage is I/O rate.  If you access cold buckets infrequently, then shared storage may work for you.