Try running Splunk's fsck command. See https://docs.splunk.com/Documentation/Splunk/Latest/Troubleshooting/CommandlinetoolsforusewithSuppor... and https://docs.splunk.com/Documentation/Splunk/8.0.6/Indexer/Bucketissues
I have a clustered indexers in a multi site environment. DO I need to shutdown the full cluster to run the
fsck command?
You only need to stop the indexer on which the corrupt bucket resides.
If I shutdown one node alone can this case search factor and replication factor met failure? Will the data generated during the shutdown period will be replicated to this nodes after we bring it up?
Shutting down an indexer may affect the ability for Splunk for maintain the desired search and replication factors. It depends on your architecture and the specific RF/SF settings. It will only be temporary, though. Once the indexer comes back up the RF and SF will resolve themselves.
Data onboarded while the indexer is down will be stored on the other indexer(s) and will be replicated to the restarted node, if necessary.