topic Re: Repairing buckets on index cluster in Getting Data In

Repairing buckets on index cluster

snyderm_dos — Thu, 18 Nov 2021 15:19:12 GMT

Is bucket repair on an index cluster any different from non-clustered indexers?

Should splunkd be running on the cluster master? Should it be in maintenance mode?
When using network storage, should it be mounted to all of the indexers or only one?
Is the fsck command run from the cluster master or from one of the indexers?

Re: Repairing buckets on index cluster

richgalloway — Thu, 18 Nov 2021 19:18:06 GMT

Yes, the CM can remain running. Maintenance mode is a good idea.

Each indexer should have its own storage. They can all use the same device as long as it looks private to the indexer. That is, no two indexers should write to the same space.

Run fsck on the indexer.

Re: Repairing buckets on index cluster

snyderm_dos — Thu, 18 Nov 2021 20:55:06 GMT

@richgalloway wrote:
Each indexer should have its own storage. They can all use the same device as long as it looks private to the indexer. That is, no two indexers should write to the same space.

Does this also apply to cold storage? Our hot/warm are on peered NFS shares, but the cold looks like it was created as one single share.

Re: Repairing buckets on index cluster

richgalloway — Fri, 19 Nov 2021 13:56:46 GMT

NFS often is referred to as "Not For Splunk". It can work if you're careful, but can be painful otherwise. See https://docs.splunk.com/Documentation/Splunk/latest/Installation/Systemrequirements#Considerations_regarding_Network_File_System_.28NFS.29

One issue with shared storage is I/O rate. If you access cold buckets infrequently, then shared storage may work for you.