Bucket ID conflict issue while thawing the data

[khusain_splunk]

The problem we are having is that of bucket ID conflicts. We end up having same bucket IDs from the same indexer that have different earliest and latest event timestamps. Even though that makes the bucket directory itself different, the ID conflict still causes ERROR with the DatabaseDirectoryManager component. See below example of ERROR log from splunkd.log

05-21-2019 06:40:04.620 -0700 ERROR DatabaseDirectoryManager - idx=XXXXXXX_XXXXXXXXX bid=XXXXXX_XXXXXXX~1199~1BEAF658-2C75-41B7-ADF2-C4F077C2A57C bucket=db_1543859820_1539917640_1199_1BEAF658-2C75-41B7-ADF2-C4F077C2A57C Detected directory manually copied into its database, causing id conflicts [path1='/opt/splunk/var/lib/splunkthawed/xxxxxxxxxxxxx/xxxxxxxxxxxxxxxx/db_1542977580_1535519520_1199_xxxxxxxxxxxxxxxxxxxxxxxxxxxx' path2='/opt/splunk/var/lib/splunkthawed/xxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxx/db_1543859820_xxxxxxxxxxxxxxxxxxxxxxxxxxxx'].

[khusain_splunk]

Hi,

A bucket id is made of three parts, index, local sequence, and GUID. "index" is the name of the data index and "GUID" is the global unique ID of the indexer (or Splunk server). For a given index, both "index" and "GUID" are invariant when splunkd generates a new bucket id. The only variant part is the bucket local sequence that is saved in the metadata file "var/lib/splunk/.dat". So, the root cause of bucket id conflicts is that two buckets have the same local sequence number accidentally when splunkd adds a new hot bucket or replicates a clustered bucket. Moreover, restoring the buckets from backup archives likely causes the bucket id conflicts.

Write a script to either disable the duplicate bucket or rename it before rebuild it.

Thanks!