Indexer fails on startup

[a238574]

When I try and restart one of my indexers after an OS upgrade I am seeing the following messages. My 2 other indexers are up and running. How do I fix this. I found one articale where they talk about fixing the offending buckets but don't say how and I am not positive this is the same issue

09-06-2018 06:37:17.576 -0400 ERROR DatabaseDirectoryManager - idx=main bid=main~392~F18EA0F4-48F1-4D8C-8209-5B40F
0B66E1E bucket=392_F18EA0F4-48F1-4D8C-8209-5B40F0B66E1E Detected directory manually copied into its database, caus
ing id conflicts [path1='/opt/splunk/var/lib/splunk/defaultdb/db/rb_1535649215_1535592644_392_F18EA0F4-48F1-4D8C-8
209-5B40F0B66E1E' path2='/opt/splunk/var/lib/splunk/defaultdb/db/392_F18EA0F4-48F1-4D8C-8209-5B40F0B66E1E'].
09-06-2018 06:37:17.579 -0400 ERROR IndexerService - Error intializing IndexerService: idx=main bid=main~392~F18EA
0F4-48F1-4D8C-8209-5B40F0B66E1E bucket=392_F18EA0F4-48F1-4D8C-8209-5B40F0B66E1E Detected directory manually copied
into its database, causing id conflicts [path1='/opt/splunk/var/lib/splunk/defaultdb/db/rb_1535649215_1535592644_
392_F18EA0F4-48F1-4D8C-8209-5B40F0B66E1E' path2='/opt/splunk/var/lib/splunk/defaultdb/db/392_F18EA0F4-48F1-4D8C-82
09-5B40F0B66E1E'].

09-06-2018 06:37:17.584 -0400 FATAL IndexerService - One or more indexes could not be initialized. Cannot disable
indexes on a clustering slave.

[harsmarvania57]

Hi @a238574,

Based on logs there is Bucket ID conflict in your main index.

You have Bucket ID 392 at /opt/splunk/var/lib/splunk/defaultdb/db/rb_1535649215_1535592644_392_F18EA0F4-48F1-4D8C-8209-5B40F0B66E1E and /opt/splunk/var/lib/splunk/defaultdb/db/392_F18EA0F4-48F1-4D8C-8209-5B40F0B66E1E in same index which is causing issue here. The thing is bucket starting with rb_ is a replicated bucket and should replicated from different indexer but here it looks like you have same GUID - F18EA0F4-48F1-4D8C-8209-5B40F0B66E1E across different indexers. Each Indexer should have different GUID so first check across your indexer with duplicated GUID.