Why is Cluster Peer unable to start due to bucket conflict after upgrade to Splunk 6.2? How to resolve this?

[sat94541]

When trying to upgrade to Splunk 6.2 after successfully upgrading and starting the instance, splunkd immediately fails. This is a cluster. 3 of the other peer nodes upgraded OK and one is hard down. Attached is a diag.

The error I'm seeing is:

2014-11-22 07:31:20.962 +0000 splunkd started (build 237341)
Bypassing local license checks since this instance is configured with a remote license master.
2014-11-22 07:31:23.402 +0000 splunkd started (build 237341)
terminate called after throwing an instance of 'DatabaseDirectoryManagerException'
what(): idx=summary bucket=db_1384819200_1384819200_31_2FB35CAB-A5AE-47A8-AB5D-C2775C0B3368 Detected directory manually copied into its database, causing id conflicts [path1='/opt/splunk/data/hotdb/summary/31_2FB35CAB-A5AE-47A8-AB5D-C2775C0B3368' path2='/opt/splunk/data/hotdb/summary/db_1384819200_1384819200_31_2FB35CAB-A5AE-47A8-AB5D-C2775C0B3368'].

Looking at the directory above I don't see any foreign directories, everything looks legit both from a naming and permissions perspective.

[rbal_splunk · Accepted solution]

Based on the error messages two buckets have the same bucket as shown below

/opt/splunk/data/hotdb/summary/db_1384819200_1384819200_31_2FB35CAB-A5AE-47A8-AB5D-C2775C0B3368 >>>>Warm bucket

/opt/splunk/data/hotdb/summary/31_2FB35CAB-A5AE-47A8-AB5D-C2775C0B3368 >>>>>Bucket created due to summarization

Splunk doesn’t like two buckets in same location generated from same indexer to have same Bucket_id, this situation is treated as conflict. Pre 6.2 version index with conflicting bucket_id use to get disabled. In Splunk Version 6.2 these conflicting buckets are causing indexers to crash. This behavior is being treated as BUG- SPL-93102

So, here issue is that [summary] index has two buckets starting with (bucket_id)number like “31_2FB35CAB-A5AE-47A8-AB5D-C2775C0B3368”.The duplicates are not actually buckets but accelerated data (summary acceleration) from other indexes, that happens to be created in your summary homepath, because of the way the folders are created.

[volume:hot0]
path = /opt/splunk/data/hotdb

[main]
repFactor = auto
homePath = volume:hot0/main
coldPath = volume:cold0/main
summaryHomePath = volume:summary0/main
tstatsHomePath = volume:datamodelsummary0/main
thawedPath = /opt/splunk/data/thaweddb/main

the unspecified summaryHomePath in splunk will be created using this method summaryHomePath = homePath cd .. mkdir summary -> /opt/splunk/data/summary and be on your actual summary index homepath, and for all your indexes.

There are two solutions :

---Solution 1-----
- enforce a different homepath for your each index to add a extra level
example :
homePath = volume:hot0/main/db

will cause the automatic summaryHomePath to be /opt/splunk/main/summary
It will require a downtime, and to create the folder and move the buckets to the new location, then restart.

----Solution 2-----

specify the summaryHomePath for each indexes to a specific location
example

[volume:summary0]
path = /opt/splunk/data/summarydb

[main]
repFactor = auto
homePath = volume:hot0/main
coldPath = volume:cold0/main
summaryHomePath = volume:summary0/main
tstatsHomePath = volume:datamodelsummary0/main
thawedPath = /opt/splunk/data/thaweddb/main
frozenTimePeriodInSecs = 34186700

etc ...

And in all cases :
- You need to just clear summary buckets from summary index (the one starting with *, not the db* or the hot_v1_* ) .

• check your accelerated report on the search-head, they will rebuild themselves

For details about the summary accelerated settings :
spec : http://docs.splunk.com/Documentation/Splunk/6.1.2/admin/Indexesconf

summaryHomePath =

* An absolute path where transparent summarization results for data in this index
should be stored. Must be different for each index and may be on any disk drive.
* May contain a volume reference (see volume section below).
* Volume reference must be used if data retention based on data size is desired.
* If not specified it defaults to a directory 'summary' in the same location as homePath
* For example, if homePath is "/opt/splunk/var/lib/splunk/index1/db",
then summaryHomePath would be "/opt/splunk/var/lib/splunk/index1/summary".
* CAUTION: Path must be writable.
* Must restart splunkd after changing this parameter; index reload will not suffice.