Why does one index get marked as "disabled" in my indexer cluster with two indexers?

[sat94541]

I have Cluster Master with 2 Indexers. These are on version 6.3.5
We noticed that on one of the indexers and for an index “winevent”, we hit the following errors:

1-11-2017 09:31:54.947 -0500 ERROR TcpInputProc - event=replicationData status=failed err="Could not open file for bid= winevent ~127~051261C5-3A44-451D-B5EA-48DEC971466A err="Cannot find config for idx=ads_main" (The operation completed successfully.)"
1-11-2017 08:25:25.289 -0500 WARN IndexerService - Received event for unconfigured/disabled/deleted index= winevent with source="source::WinEventLog:Security" host="host::SDRT0086" sourcetype="sourcetype::WinEventLog:Security". So far received events from 1 missing index(es).
01-11-2017 08:22:20.879 -0500 ERROR DatabaseDirectoryManager - idx= winevent bucket=db_1483621671_1483545612_420_051261C5-3A44-451D-B5EA-48DEC971466A Detected directory manually copied into its database, causing id conflicts [path1='\\gofisilon01.nac.ppg.com\Archive\Splunk\Idx2\ winevent \colddb\rb_1483621671_1483545612_420_051261C5-3A44-451D-B5EA-48DEC971466A' path2='\\gofisilon01.nac.ppg.com\Archive\Splunk\Idx2\ winevent \colddb\db_1483621671_1483545612_420_051261C5-3A44-451D-B5EA-48DEC971466A'].

Due to these duplicate buckets the index “winevent” got marked as “Disabled? What can be done for this issue?

[rbal_splunk · Accepted solution]

A bucket id is made of three parts, index, local sequence, and GUID. "index" is the name of the data index and "GUID" is the global unique ID of the indexer (or Splunk server). For a given index, both "index" and "GUID" are invariant when splunkd generates a new bucket id. The only variant part is the bucket local sequence that is saved in the metadata file "var/lib/splunk/.dat". So, the root cause of bucket id conflicts is that two buckets have the same local sequence number accidentally when splunkd adds a new hot bucket or replicates a clustered bucket. Moreover, restoring the buckets from backup archives likely causes the bucket id conflicts.
Below is a list of possible scenarios with bucket id conflicts.
• Adding a hot bucket conflicts with a local warm or cold bucket
• Adding a hot bucket conflicts with a replicated hot, warm or cold bucket
• Adding a hot bucket conflicts with a cached warm or cold bucket
• Replicating a hot, warm or cold bucket conflicts with a replicated hot, warm or cold bucket
• Replicating a hot bucket conflicts with a cached warm or cold bucket
• Restoring a hot, warm or cold bucket conflicts with a local hot, warm or cold bucket
• Restoring a hot, warm or cold bucket conflicts with a replicated hot, warm or cold bucket
• Restoring a hot, warm or cold bucket conflicts with a cached warm or cold bucket
Currently when the duplicate buckets are created, the only workaround is to delete one of the duplicate buckets. In future , Splunk is working to improve the code so that splunk auto detect and address such conflicts.