Getting Data In

Bucket ID conflict issue while thawing the data

khusain_splunk
Splunk Employee
Splunk Employee

The problem we are having is that of bucket ID conflicts. We end up having same bucket IDs from the same indexer that have different earliest and latest event timestamps. Even though that makes the bucket directory itself different, the ID conflict still causes ERROR with the DatabaseDirectoryManager component. See below example of ERROR log from splunkd.log

05-21-2019 06:40:04.620 -0700 ERROR DatabaseDirectoryManager - idx=XXXXXXX_XXXXXXXXX bid=XXXXXX_XXXXXXX~1199~1BEAF658-2C75-41B7-ADF2-C4F077C2A57C bucket=db_1543859820_1539917640_1199_1BEAF658-2C75-41B7-ADF2-C4F077C2A57C Detected directory manually copied into its database, causing id conflicts [path1='/opt/splunk/var/lib/splunkthawed/xxxxxxxxxxxxx/xxxxxxxxxxxxxxxx/db_1542977580_1535519520_1199_xxxxxxxxxxxxxxxxxxxxxxxxxxxx' path2='/opt/splunk/var/lib/splunkthawed/xxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxx/db_1543859820_xxxxxxxxxxxxxxxxxxxxxxxxxxxx'].

Tags (1)
0 Karma

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

A bucket id is made of three parts, index, local sequence, and GUID. "index" is the name of the data index and "GUID" is the global unique ID of the indexer (or Splunk server). For a given index, both "index" and "GUID" are invariant when splunkd generates a new bucket id. The only variant part is the bucket local sequence that is saved in the metadata file "var/lib/splunk/.dat". So, the root cause of bucket id conflicts is that two buckets have the same local sequence number accidentally when splunkd adds a new hot bucket or replicates a clustered bucket. Moreover, restoring the buckets from backup archives likely causes the bucket id conflicts.

Write a script to either disable the duplicate bucket or rename it before rebuild it.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...

Want a chance to win $500 to the Splunk shop? Take our IT Incident Management Survey!

  Top Trends & Best Practices in Incident ManagementSplunk is partnering up with Constellation Research to ...