Getting Data In

Bucket ID conflict issue while thawing the data

khusain_splunk
Splunk Employee
Splunk Employee

The problem we are having is that of bucket ID conflicts. We end up having same bucket IDs from the same indexer that have different earliest and latest event timestamps. Even though that makes the bucket directory itself different, the ID conflict still causes ERROR with the DatabaseDirectoryManager component. See below example of ERROR log from splunkd.log

05-21-2019 06:40:04.620 -0700 ERROR DatabaseDirectoryManager - idx=XXXXXXX_XXXXXXXXX bid=XXXXXX_XXXXXXX~1199~1BEAF658-2C75-41B7-ADF2-C4F077C2A57C bucket=db_1543859820_1539917640_1199_1BEAF658-2C75-41B7-ADF2-C4F077C2A57C Detected directory manually copied into its database, causing id conflicts [path1='/opt/splunk/var/lib/splunkthawed/xxxxxxxxxxxxx/xxxxxxxxxxxxxxxx/db_1542977580_1535519520_1199_xxxxxxxxxxxxxxxxxxxxxxxxxxxx' path2='/opt/splunk/var/lib/splunkthawed/xxxxxxxxxxxxxxxxx/xxxxxxxxxxxxxxxxxx/db_1543859820_xxxxxxxxxxxxxxxxxxxxxxxxxxxx'].

Tags (1)
0 Karma

khusain_splunk
Splunk Employee
Splunk Employee

Hi,

A bucket id is made of three parts, index, local sequence, and GUID. "index" is the name of the data index and "GUID" is the global unique ID of the indexer (or Splunk server). For a given index, both "index" and "GUID" are invariant when splunkd generates a new bucket id. The only variant part is the bucket local sequence that is saved in the metadata file "var/lib/splunk/.dat". So, the root cause of bucket id conflicts is that two buckets have the same local sequence number accidentally when splunkd adds a new hot bucket or replicates a clustered bucket. Moreover, restoring the buckets from backup archives likely causes the bucket id conflicts.

Write a script to either disable the duplicate bucket or rename it before rebuild it.

Thanks!

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...