Is bucket repair on an index cluster any different from non-clustered indexers?  Should splunkd be running on the cluster master? Should it be in maintenance mode? When using network storage, should it be mounted to all of the indexers or only one? Is the fsck command run from the cluster master or from one of the indexers? ... View more

I recently performed a data migration to correct some mistakes made by the person who built our environment. Afterward, I found I had to run `splunk fsck repair` due to errors that are preventing splunk from starting. After running the command with "--all-buckets-all-indexes" or "--all-buckets-one-index --index-name=linux" it stops without seeming to do anything. After it stops I get, as an example Process delayed by 56.174 seconds, perhaps system was suspended? Stopping WatchdogThread. We have four indexers in a cluster. I've put the cluster master in maintenance mode and stopped splunk on all of the indexers. I'm running the command on a single indexer since the data is shared via NFS. One thing I haven't done is unmount the share on all of the other indexers. What is the cause of this error and what do I need to do to move past it? ... View more

I recently had to realign our storage. Specifically, write cold data to one NFS share and hot/warm to another. Prior to this, all data was being written to the same storage which was not per our design. I placed our cluster master in maintenance-mode, stopped splunk on all indexers, then used rsync to copy data to the proper shares. After moving data around and ensuring that NFS shares were then mounted in the proper locations, I attempted to bring everything back online. The cluster master starts fine. The indexers, though, do not. I have only been able to start one indexer out of four. It seems to not be one specific indexer, though. I had splunk running on indexer1, but indexer2, indexer3, and indexer4 then failed. Later, I was able to start splunk on indexer2, but indexer1, indexer3, and indexer4 failed. Examples of the errors I'm seeing are ERROR STMgr - dir='/splunk/audit/db/hot_v1_64' st_open failure: opts=1 tsidxWritingLevel=1 (No such file or directory) ERROR StreamGroup - Failed to open THING for dir=/splunk/audit/db/hot_v1_64 exists=false isDir=false isRW=false errno='No such file or directory' Your .tsidx files will be incomplete for this bucket, and you may have to rebuild it. ERROR StreamGroup - failed to add corrupt marker to dir=/splunk/audit/db/hot_v1_64 errno=No such file or directory and ERROR HotDBManager - Could not service the bucket: path=/splunk/_introspection/db/hot_v1_388/rawdata not found. Remove it from host bucket list. WARN  TimeInvertedIndex - Directory /splunk/_introspection/db/hot_v1_388 appears to have been deleted FATAL MetaData - Unable to open tempfile=/splunk/_introspection/db/hot_v1_388/Strings.data.temp for reason="No such file or directory";  this=MetaData: {file=/splunk/_introspection/db/hot_v1_388/Strings.data description=Strings totalCount=761 secsSinceFullService=0 global=WordPositionData: { count=0 ET=n/a LT=n/a mostRecent=n/a } and FATAL HotDBManager - Hot bucket with id=389 already exists. idx=_introspection dir=/splunk/_introspection/db/hot_v1_389 I've run 'splunk fsck repair --all-buckets-all-indexes' more than once, but these issues persist. Can the underlying issues be corrected or should we cut our losses and start our collections fresh? Fortunately, this is an option we can use as a last resort. ... View more